DATAcnv1.doc - Convert EBCDIC to ASCII - Contents

Documentation in 9 parts

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

DATAcnv1.doc - Convert EBCDIC to ASCII - Contents (continued)

Copyright 1993-2009, UV Software Inc.

Owen Townsend, UV Software, 4667 Hoskins Rd., North Vancouver BC, V7K2R3

 Tel: 604-980-5434 Fax: 604-980-5404

mailto:owen@uvsoftware.ca

http://www.uvsoftware.ca

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_0 mainframe DATA conversion Plans

DATA Conversion Alternatives

UV Software offers 2 strategies for converting mainframe data files for use on Unix/Linux systems. We are concerned here with the Sequential & Indexed files used on the mainframe, not with existing mainframe DBMS's. Software for converting mainframe DBMS tables is usually supplied by the vendor.

UV Software supplies powerful utilities to convert the Sequential & Indexed files, automatically from the COBOL copybooks, allowing for complex files, with multiple record types, occurs, etc.

Plan 'A' is to retain existing record layouts, which allows you to convert quickly, since COBOL program logic needs no changes.

Plan 'B' is to convert data files to pipe delimited text files for loading RDBMS tables. Any packed/binary fields are unpacked & edited with signs & decimal points as required for loading RDBMS tables. We also automatically generate the SQL Loader control files from the COBOL copybooks.

Plan A (layouts unchanged)

1. Existing Record layouts are retained, allowing you to convert quickly, since COBOL program logic requires no changes.
   
   
2. Character fields are translated from EBCDIC to ASCII, but packed/binary fields are preserved unchanged, as required for Micro Focus COBOL on Unix/Linux.
   
   
3. UV Software provides utilities to generate the 'data conversion jobs' automatically from the COBOL copybook record layouts.
   
   
4. The auto generated conversion jobs usually require no manual changes before execution, but changes are required for files with multiple record types.
   
   
5. UV Software's conversion techniques have been proven to handle complex data files with multiple record types based on several different fields that could not be handled by other vendors.
   
   
6. The data conversion jobs for hundreds of data files can all be executed with 1 command, which is a big advantage, since the data conversion will usually be repeated several times during a 6 month or year long conversion. Of course the important repeat is on the 'go live' weekend.
   

See sample conversions for plan A (layouts unchanged, preserve packed fields) at http://www.uvsoftware.ca/datacnv1.htm#Part_3

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part 0 DATA Conversion Alternatives

plan B - for RDBMS's

1. The 2nd plan is to convert the data files to pipe delimited text files for loading RDBMS tables (for DB2, Oracle, SQL server, etc).
   
   
2. Any packed/binary fields are unpacked & edited with signs & decimal points as required for loading RDBMS tables.
   
   
3. UV Software provides utilities for generating the 'pipe delimit jobs' automatically from the COBOL copybook record layouts.
   
   
4. We also automatically generate the 'SQL Loader control tables' from the COBOL copybooks.
   
   
5. Note that mainframe files with multiple record types must be split into separate tables, since RDBMS tables do not allow multiple record types. UV Software's techniques can easily handle this situation.
   
   
6. RDBMS tables do not allow 'occurs'. UV Software's techniques make it easy to append a sequence number on such fields, or split the occurs data out to a separate table, inserting record ID fields as required.
   

See sample conversions for plan B (unpack to delimited files to load RDBMSs) at http://www.uvsoftware.ca/datacnv1.htm#Part_4

plan C = Plan A + Plan B on your timetable

We recommend plan A to get off the mainframe as quickly as possible. Then you can convert files to RDBMS tables depending on your priorities & timetable.

related documentation

 SQLconvert.htm - convert flat files to pipe delimited for Microsoft SQL server
                  - driven by COBOL copybooks
                  - also generate scripts to create & load tables
                  - generates jobs for all copybooks & all data files
                    (vs this DATAcnv1.doc for 1 copybook/file at a time)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_1 Data Conversion Preparation

Part1 - Contents

 1D1. Using 'uvhd' to investigate EBCDIC files that you wish to convert to ASCII.
     - determine/verify record size & look for packed fields & zoned signs
     - look for complex files (multi record types, variable length records, etc)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1A1. Data Conversion Preparation

DATAcnv1.doc - OVERVIEW

The power & flexibility of these procedures are necessarily accompanied by some underlying complexity, but we want you to know at the outset, that there are master scripts to simplify these procedures.

We recommend you follow the step by step procedures to convert the test/demo file & perhaps your first data file. This will help you understand how these conversions work & allow you to modify the procedures for complex files (multi record types or variable length records).

For most files, you can use the 'gencnvD1' or 'gencnvD2' master scripts to generate all conversion jobs with 1 command. Converting any 1 data file is then simplified to the following (using 'citytax1' demo file for illustration):

 #1. Copy the data file to the expected input subdir
     (supplied demo file dat1/citytax1 may be copied to cnvdata/d1ebc).

 #2. gencnvD1 citytax1 r128d4 <-- generate All conversion jobs for demo file
     ========================

 #3. uvcopy pfx3/citytax1     <-- execute EBCDIC to ASCII conversion
     ====================       - from d1ebc/citytax1 to d2asc/citytax1
                                  (preserves packed, fixes zoned signs)

 #4. uvcopy pfp3/citytax1     <-- execute conversion to pipe delimited text
     ====================       - from d2asc/citytax1 to d4pipe/citytax1

 #5. Converted outputs are left in following subdirs:
 #5a. d2asc/citytax1      - ASCII file (same layout as input)
 #5b. d4pipe/citytax1.dat - all text pipe delimited file (for loading RDB's)
 #5c. sqls/citytax1       - SQL Loader Control file to load relational DB

If you already have the COBOL copybook (datafile record description), you would use master script 'gencnvD2' which saves the 1st 2 of the 6 steps in gencnvD1 (scanning the data file & generating the copybook).

These jobs are simplified by setting up several subdirs within a superdir, which allows us to maintain the same file name as we generate jobs & convert data thru the various steps.

You don't have to modify the generated jobs if you copy the data file to the expected input subdir (d1ebc), and rename same as copybook. After conversion you can copy the converted from the output subdir (d2asc), and rename back to your desired datafilename.

But is is a simple matter to modify the generated conversion jobs to specify your I/O data pathnames. You might do this for large files, especially if you need several re-conversions during your conversion project.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1A2. Data Conversion Preparation

DATAcnv1.doc - OVERVIEW

1. DATAcnv1 (this document) is intended to help you convert EBCDIC data files whether or not the COBOL copybook is available. Other conversion documents in the Vancouver Utilities presume the COBOL copybooks are available.
   
   
2. When the COBOL copybook is not available, we have procedures to scan the EBCDIC data to determine the record layout. 'signscanA' can determine the locations of packed fields & zoned signs which would be destroyed if you translate the entire record to ASCII.
   
   
3. When the COBOL copybook is available, you can use it with this document, by skipping the signscanA & sign2cpyA procedures. All the data conversion procedures are driven by the copybook whether supplied or created by scanning the data.
   
   
4. DATAcnv1 is intended to convert 1 file at a time. Other conversion documents (see below) are intended to convert entire mainframe sites with hundreds of data files. DATAcnv1 is applicable to sites that receive data files from outside sources which supply the data in varying formats with or without copybooks.
   
   
5. The procedures in DATAcnv1 are much simpler & easier to execute than those in the other documents. However you may still need to reference the other documents for more complex situations (multiple record types, variable length records, magnetic tape input, etc).
   
   

   Other Data Conversion Documentation

There are several other Vancouver Utility documents that describe the EBCDIC to ASCII data conversion in much more detail than this DATAcnv1.doc.

1. VSEDATA.doc - Advanced VSE Mainframe DATA conversion - for more complex files & higher volumes - assumes that COBOL copybooks are available. - convert EBCDIC data files to '|' pipe delimited ASCII files suitable for loading Relational data bases. - reformat data files based on old & new versions of copybooks
   
   
2. MVSDATA.doc - same as above for MVS (vs VSE)
   
   

   UV Software Training & Support

UV Software recommends a minimum 1 week training course with the initial installation of the Vancouver Utilities. With this training, you should be able to convert most EBCDIC files. UV Software provides customers with a generous amount of free support, so please call if you can use some help with a complex file. UV Software can also be contracted to convert the unusually complex files if the problem is not easily resolved via telephone and emails.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1A3. Data Conversion Preparation

DATAcnv1 - Utilities Summarized

1. 'uvhd' is the first tool you will use to investigate EBCDIC files that you wish to convert to ASCII. Use uvhd to determine or verify the correct the record size & to check for packed fields, zoned signs, and complex files (multiple record types, variable length records, etc). Note that you cannot use 'vi' or most other unix tools on EBCDIC files, or ASCII files with packed fields, or any file without linefeeds (most mainframe files do not have linefeeds so the entire file appears in vi as 1 long line).
   
   
2. 'signscanA' will help you determine the record layout of EBCDIC data files when no COBOL copybook is available. Script signscanA runs uvcopy job 'signscan1' which scans for packed fields, zoned signs, date fields, and right blank filled fields (such as Name & Address). It is most to determine packed fields & zoned sign positions for the subsequent script 'gencnvA' which performs the EBCDIC to ASCII conversion.
   
   
3. 'sign2cpyA' converts the signscanA record layout to a COBOL copybook (an industry standard method of describing record field layouts). The copybook is used by the following scripts to perform various conversions. Note that if a copybook is supplied with the EBCDIC data file, you can use it with the following scripts.
   
   
4. 'cobmapA' converts COBOL copybooks to 'cobmaps' (record layouts), adding field start,end,length,& type on the right hand side of the copybook. List the cobmap & use the 'uvhd' utility to verify and make corrections & enhancements. This is most important if you have just created the copybook using signscanA & sign2cpyA.
   
   
5. 'uvhdcob' displays the data beside the COBOL copybook fieldnames making it easy to see if the copybook is consistent with the data. I recommend you list out the 1st few records with both uvhd & uvhdcob to verify the layout & resolve any problems if file is supplied by outside sources.
   
   
6. 'gencnvA' generates a uvcopy job to translate EBCDIC files to ASCII, preserving packed fields (would be destroyed by translation), and correcting any zoned decimal signs from the EBCDIC conventions to ASCII conventions.
   
   
7. 'genpipeA' generates a uvcopy job to convert data to '|' pipe delimited ASCII text files. Packed & numeric fields will be converted to zero suppressed numeric fields with leading sign +/-. The pipe delimited format is often used for loading Relational data bases.
   
   
8. 'listpipe1' lists a '|' delimited output file to verify conversion. The copybook fieldnames are listed beside the data field contents.
   
   
9. 'sqlcreate1' generates batch-file & control-file to create & load table
   
   
10. 'genrfmA' generates a uvcopy job to unpack packed fields increasing recsize. Use genrfmA when the application for which the data file is intended does not allow packed fields.
    

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1B1. Data Conversion Preparation

Initial Setup for DataFile Conversion

I recommend you setup a separate login user account to perform these data conversions. I suggest userid 'cnvdata' which will be used in this document. The home directory will then be /home/cnvdata.

If you cannot setup a separate userid, you could use a subdirectory within your home directory (but this may not be advisable if there is not enough space for your projected requirements).

Setting up a separate userid might be better if you do not want to modify your current .profile. Note that you must modify the .profile to setup PATH, etc to run the Vancouver Utility scripts & uvcopy jobs. Please see install.htm for Unix/Linux, WindowsSFU.htm for Windows with SFU, or WindowsDOS.htm for native Windows.

Sub-directories are made for the different types of files, so we can keep the same name as the original datafile & copybook, as we convert copybooks to record layouts, to uvcopy jobs. For example our demo file is named 'citytax1' & the same name is used as we convert thru several subdirs:

 d1ebc ---------> signs ---------> cpys ---------> maps ----------> pfx1
       signscan1        sign2cpy1        cobmap1         uvdata51

Keeping the same name thruout the various subdirs greatly simplifies our operating instructions. This also means you must drop any extensions on your datafile or copybook names.

making subdirs for data conversions

Preferably setup a separate userid 'cnvdata' to perform these conversions, But if not possible make directory 'cnvdata' within your home dir or elsewhere. For Windows, just make directory C:\cnvdata.

 #1. login cnvdata ---> /home/cnvdata
     =============

 #2. cnvdatadirs       <-- script to create sub-dirs (listed on next page)
     ===========

       - - - OR - - -

 #1. login youruserid ---> yourhomedir (/home/yourid)

 #2. mkdir cnvdata     <-- make conversion superdir within your homedir

 #3. cd cnvdata        <-- change into conversion superdir

 #4. cnvdatadirs       <-- script to create sub-dirs (listed on next page)
     ===========

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1B2. DATAcnv1.doc - Convert EBCDIC to ASCII - INITIAL SETUP

cnvdata subdirs (dtree illustration)

 /home/mvstest        <-- setup 'mvstest' user ?
 /home/userxx         <-- OR setup cnvdata withing your $HOME dir
 :-----cnvdata        <-- cnvdata superdir for DATA conversion
 :     :-----cpys       - copybooks
 :     :-----ctl        - control files
 :     :-----d1ebc      - EBCDIC data files
 :     :-----d2asc      - ASCII data files
 :     :-----d3unpk     - unpacked data files (seldom used)
 :     :-----d4pipe     - pipe delimited all text for RDBMSs
 :     :-----maps       - copybooks with record layouts on right side
 :     :-----pf         - misc uvcopy jobs
 :     :-----pfp1       - uvcopy jobs generated for pipe delimited data
 :     :-----pfp2       - reserved in case uvdata54 to insert datafilenames
 :     :-----pfp3       - copied here for modify/execute
 :     :-----pfx1       - uvcopy jobs generated to convert EBCDIC to ASCII
 :     :-----pfx2       - with data filenames inserted (vs copybook names)
 :     :-----pfx3       - copied here for modify/execute
 :     :-----sf         - misc scripts
 :     :-----signs      - record layouts created by signscan1
 :     :-----sqls       - SQL loader control files generated from copybooks
 :     :-----tmp        - temp files (keep working directory clean)
 :     :
 :-----testdata       <-- testdata superdir for later testing
 :     :-----joblog     - see MVSJCL.doc
 :     :-----jobtmp
 :     :-----mstr
 :     :-----rpts
 :     :-----tmp
 :     :-----wrk
 :     :
 :-----testlibs       <-- testlibs superdir for COBOL & JCL conversion
 :     :-----cbls       - see MVSCOBOL.doc & MVSJCL.doc
 :     :-----cblx
 :     :-----cpys
 :     :-----ctl
 :     :-----jcls
 :     :-----maps
 :     :-----parms
 :     :-----pf
 :     :-----procs
 :     :-----sf
 :     :-----sfun
 :     :-----tmp
 :     :-----xref

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1B3. DATAcnv1.doc - Convert EBCDIC to ASCII - INITIAL SETUP

subdirs required for conversion scripts

 cnvdatadirs    <-- script to create the sub-dirs listed below
 ===========

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1B4. DATAcnv1.doc - Convert EBCDIC to ASCII - INITIAL SETUP

subdirs required for conversion scripts (continued)

additional subdir required for signscan

You will not need subdir 'signs' if you have the COBOL copybooks & do not need to scan the data (signscan1) to automatically create the copybooks. Note that signscan1 is not foolproof & may require correction.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1B4. DATAcnv1.doc - Convert EBCDIC to ASCII - INITIAL SETUP

additional subdirs to create unpacked data records

You will need the following subdirs if you wish to create unpacked data records. The previous page assumed you wished only to convert the EBCDIC data to pipe delimited ASCII text records for loading into SQL databases.

stay in the working-dir (not the subdirs)

As you run the various scripts & uvcopy jobs, you should stay in the working directory (/home/cnvdata/ or whatever you call it). Do NOT change into the subdirs, since you must be back at the working directory to run the next step.

Address the files thru 1 level of subdir for various view/print commands. For example, to display the EBCDIC to ASCII conversion job:

 more pfx1/citytax1      <-- address files thru 1 level of subdir
 ==================

 cd pfx1                 <-- do not change into & out of subdirs
 more citytax1             - would waste 3 commands vs 1
 cd ..                     - since next step requires working directory level

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1C1. DATAcnv1.doc - Convert EBCDIC to ASCII - INITIAL SETUP

Load Test Data & CopyBooks - for Unix/Linux

Before you attempt to use these procedures to convert your own EBCDIC data files, I suggest you execute the various procedures using the 'citytax1' demo file supplied. You can get familiar with the various operating procedures and verify that you get the same results as presented with the operating instructions.

 #1a. cp /home/uvadm/dat1/citytax1 d1ebc
      ==================================
 #1b. cp /home/uvadm/dat1/citytax2 d1ebc
      ==================================

 #2a. cp /home/uvadm/tf/citytax1.cpy cpys/citytax1
      ============================================
 #2b. cp /home/uvadm/tf/citytax2.cpy cpys/citytax2
      ============================================

Loading the copybooks as well as the data files presumes that you are skipping the signscan jobs (because you have COBOL copybooks & don't need to generate them). Or you could re-transfer the copybooks after you test/demo signscan so you can test the rest of the system using the copybooks with meaningful field names (vs the numbered fields resulting from signscan).

If you had many data files & copybooks with extensions, you could copy them all using wildcards (*) & then remove extensions with rename-X & convert filenames to lower case if desired.

 #3a. rename-X d1ebc .dat      - remove any .dat extensions from data files
      ===================
 #3b. renameL d1ebc            - convert all data file names to lower case
      =============

 #4a. rename-X cpys .cpy       - remove any .cpy extensions from copybooks
      ==================
 #4b. renameL cpys             - convert all copybook names to lower case
      ============

Load Test Data & CopyBooks - for Windows/Dos

 #1a. copy C:\uvwin\dat1\citytax1 d1ebc
      =================================
 #1b. copy C:\uvwin\dat1\citytax2 d1ebc
      =================================

 #2a. copy C:\uvwin\tf\citytax1.cpy cpys\citytax1
      ===========================================
 #2b. copy C:\uvwin\tf\citytax2.cpy cpys\citytax2
      ===========================================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1C2. DATAcnv1.doc - Convert EBCDIC to ASCII - INITIAL SETUP

Initial Data File Investigation (with uvhd)

You can demonstrate the data conversion scripts, by setting up the directories suggested on the previous pages,& loading the supplied test files.

 cd cnvdata    - change to superdir setup on prior pages
 ==========      (with subdirs: d1ebc,d2asc,d4pipe,cpys,maps,etc)

Inspect the test/demo file - with 'uvhd'

 uvhd d1ebc/citytax1 r128as7   - inspect citytax1 demo file
 ===========================   - option 'r' (r128) specifies record size
                               - option 'a' translates characters to Ascii
                               - option 's7' spaces between scale & data below

                       1         2         3         4         5         6
 r#        1 0123456789012345678901234567890123456789012345678901234567890123

           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
             1013014000168508559800000000000000018150266550961400000000005153

          64 OUVER           BCV9S1H1.bi...Wb....g...qq.00014906A970530191809
             DEECD44444444444CCEFEFCF0888005880008000770FFFFFFFFCFFFFFFFFFFFF
             645590000000000023592181029C0072C0047C0111C000149061970530191809

--> i3 <-- immediate print 1st 3 records == --> q <-- quit uvhd

When you have a new EBCDIC file to be converted, the first thing you need to do is investigate the file (using 'uvhd') to determine or verify the correct record size & to check for packed fields, zoned signs, and complex files (multiple record types, variable length records, etc).

Your first objective is to determine the 'CORRECT RECORD SIZE'. In this case I already knew it & specified it on the command line (see 'r128' above). If you don't specify any option 'r', the record size defaults to 256.

'uvhd' makes it easy to determine the correct record size. When displaying record #1, uvhd will suggest the next lower & next higher record-size. If you have it wrong you will see the data shifting left or right as you browse thru the file (just hit enter to advance to the next record).

Note that you cannot use 'vi' or most other unix tools on EBCDIC files, or ASCII files with packed fields, or any file without linefeeds (most mainframe files do not have linefeeds so the entire file appears in vi as 1 long line).

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1D1. DATAcnv1.doc - Convert EBCDIC to ASCII with or without COBOL copybook

Investigating EBCDIC files - with 'uvhd'

When you have a new EBCDIC file to be converted, you should investigate the file using 'uvhd' since you cannot use 'vi' with EBCDIC files. Your 1st concern is to determine the correct record size or verify the size given to you by the source or copybook. Uvhd defaults record size to 256 & it is usually easy to determine record size by looking for data repetition.

'uvhd' displays data files in vertical hex. Data is displayed in 64 byte segments on 3 lines (characters, hex zones,& hex digits). Use option 'a' with EBCDIC files to translate the character line to ASCII while still showing the EBCDIC data as hex zones & hex digits.

For the citytax1 demo file we know the record size is 128 which is specified via option 'r' (r128). We will also specify option 'a' to translate the character line to ASCII. If we also specify 's2' to space between the 3 line groups of characters, zones,& digits the option string will be 'r128as2'.

 uvhd d1ebc/citytax1 r128as2
 ===========================

                       1         2         3         4         5         6
 r#        1 0123456789012345678901234567890123456789012345678901234567890123

           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
             1013014000168508559800000000000000018150266550961400000000005153

          64 OUVER           BCV9S1H1.bi...Wb....g...qq.00014906A970530191809
             DEECD44444444444CCEFEFCF0888005880008000770FFFFFFFFCFFFFFFFFFFFF
             645590000000000023592181029C0072C0047C0111C000149061970530191809

 rec#=1 recs=10 rsiz=128 fsiz=1280 d1ebc/citytax1
 enter command--> i3  <-- immediate print 1st 3 records

'uvhd' displays the 1st record in the file & waits for you to enter a command. Please see uvhd.htm for the complete command repertoire or enter '?' for the Help screens. In this case we entered 'i3' to immediately print the 1st 3 records.

Note the packed fields in bytes 88-105 & the zoned sign in byte 115. You can calculate byte positions by adding the scale position (from top line) to the offset of the current row of data. For example packed fields begin at scale position 24 + offset 64 = byte 88 (zero relative) = column 89 (1 relative).

If you have long records & have a large screen (WinDos allows 60 lines/screen) use the 'm' option of uvhd to records up to 1000 bytes on 1 screen, for example:

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1D2. DATAcnv1 - Convert EBCDIC to ASCII with or without COBOL copybook

Printing data files with uvhd

The 'immediate printing' (i3 above) feature of uvhd is very handy, but if it does not work on your computer, you can use the 'print' command (p3 vs i3). The 'print' command stores the formatted file in the 'tmp' subdir, named the same as the datafile with a date/time suffix. You could print separately:

 uvlp12 tmp/citytax1.yymmddhhmmssP
 =================================

For the 'i' command, uvhd calls the 'uvlp12' script (or 'uvlpr12' if WinDos). The printer for 'uvlp12' is defined by env-var 'UVLPDEST'. The WinDos script 'uvlpr12.bat' redirects output to 'LPT1'. See more on page C1 of uvhd.doc.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1D3. DATAcnv1.doc - Converting EBCDIC to ASCII

Investigating EBCDIC files - with 'uvhd' (continued)

Before we leave the discussion of using 'uvhd' with EBCDIC files, we should show you what happens if you don't supply any options to uvhd:

 uvhd d1ebc/citytax1            <-- uvhd with No Options
 ===================

                 1         2         3         4         5         6
     1 0123456789012345678901234567890123456789012345678901234567890123 <--scale
     0 ........@@....@.....@@@@@@@@@@@@@@@....@.....@....@@@@@@@@@@.... <--chars
       FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC <--zones
       1013014000168508559800000000000000018150266550961400000000005153 <--digits
    64 .....@@@@@@@@@@@..............W.........qq......................
       DEECD44444444444CCEFEFCF0888005880008000770FFFFFFFFCFFFFFFFFFFFF
       645590000000000023592181029C0072C0047C0111C000149061970530191809
   128 ........@@......@.....@@@@@@@@@@@@@....@....@......@......@@....
       FFFFFFFF44CCDDCC4CDDED4444444444444FFFF4CCEE4DCDCCD4EEDCCE44ECDC
       1013992300756975029665000000000000012500512307554590239553005153
   192 .....@@@@@@@@@@@..............W.........xRL.....................
       DEECD44444444444CCEFDFEF0888005880008800754FFFFFFFFDFFFFFFFFFFFF
       645590000000000023553161029C0072C0047C0182C000146261980601190156

1. When no options are specified, uvhd defaults to single spacing. On the previous page we used option 's2' to double space between the 3 line groups of character,zones,& digits for each 64 byte segment.
   
   
2. uvhd defaults to expecting ASCII, so for EBCDIC files we will get garbage. Note that this applies only to the character line, the zones & digits will be the same since they represent the data in hexadecimal. The 'a' option tells uvhd to translate the character line to ASCII. With option 'a' the 1st 64 byte portion from above would be:
   

 uvhd d1ebc/citytax1 a          <-- uvhd with option 'a'
 =====================

                  10        20        30        40        50        60
     1 0123456789012345678901234567890123456789012345678901234567890123
     0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC <--chars
       FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC <--zones
       1013014000168508559800000000000000018150266550961400000000005153 <--digits

3. If you are working with EBCDIC files, you should have a good understanding of the EBCDIC & ASCII character set representations in hexadecimal. For example a '1' is x'F1' in EBCDIC & x'31' in ASCII, an 'A' is x'C1' in EBCDIC & x'41' in ASCII (the IBM code card shows you all this).
   
   
4. You can identify the packed fields by looking for bytes with '0' zones, & x'_C' or x'_D' in the digit of the last byte of a field which is the sign '+' or '-'.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1D4. DATAcnv1.doc - Converting EBCDIC to ASCII

Investigating citytax2 with uvhd

'citytax2' is a multi-record-type file with 3 record types 'H','T',or 'P' in byte 8 (Header records, Tax Records,& Payment records).

The record size is 128 (same as citytax1), but we will use '512' in the example below to illustrate some points.

 uvhd d1ebc/citytax2 r128as3  <-- display citytax2 (see options below)
 ===========================

                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123

           0         H December 31, 1998
             44444444C4C88898894FF64FFFF4444444444444444444444444444444444444
             000000008045354259031B019980000000000000000000000000000000000000

          64
             4444444444444444444444444444444444444444444444444444444444444444
             0000000000000000000000000000000000000000000000000000000000000000

                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123

         128 10130140T JOHN HENRY               1815 BOWEN ROAD          VANC
             FFFFFFFFE4DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
             1013014030168508559800000000000000018150266550961400000000005153

          64 OUVER           BCV9S1H1.bi...Wb....g...qq.00014906A970530191809
             DEECD44444444444CCEFEFCF0888005880008000770FFFFFFFFCFFFFFFFFFFFF
             645590000000000023592181029C0072C0047C0111C000149061970530191809

                      10        20        30        40        50        60
 r#        3 0123456789012345678901234567890123456789012345678901234567890123

         256 10130140P       1999...A....A....A....A....A....A....A....A....A
             FFFFFFFFD4444444FFFF00042000420004200042000420004200042000420004
             101301407000000019990001C0001C0001C0001C0001C0001C0001C0001C0001

          64 ....A....A....A.
             2000420004200042444444444444444444444444444444444444444444444444
             C0001C0001C0001C000000000000000000000000000000000000000000000000

 uvhd d1ebc/citytax2 r128as3   <-- note options explained below
 ===========================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1E1. DATAcnv1.doc - Converting EBCDIC to ASCII

Recommendations to Convert Your Own Files

1. First perform the 'citytax1' demo conversions illustrated in this document.
   
   
2. You can use the demo set of subdirs for your own file conversions, or setup a similar set (superdir cnvdata with subdirs d1ebc,d2asc, d3unpk,d4pipe,cpys,maps,pfx1,pfx2,pfx3,pfp1,pfp2,pfp3,sqls,tmp,etc).
   
   
3. If possible name your input data file & copybook the same & keep it short to save keystrokes & minimize confusion. The uvcopy conversion jobs are generated assuming datafile name same as copybook name. Multiple separate subdirs allow you to keep the same name for the various datafiles & uvcopy jobs throughout the system.
   
   
4. Whenever you wish to view or print your copybooks - DON'T, always view or print the corresponding 'cobmap' since it is much more meaningful with the start/end/length bytes calculated on the right side. The cobmaps are generated in the 'maps' subdir by many of the scripts, but if you have just updated with the editor, you can run 'cobmapA' to regenerate the cobmap.
   
   
5. Whenever you wish to print your cobmaps or uvcopy jobs, use the 'uvlp12' script (or uvlpr12 for WinDos). These scripts add headings with filenames, dates, page#s, etc. See uvlist.doc for the environmental variable (UVHDPRINT) used by uvhd for its desination printer. For files with long lines such as the pipe delimited files, you can use uvlp14L, uvlp18L, etc to print Landscape at 14 cpi or more to get up to 200 chars per line. For Windows/DOS the script names are: uvlpr12, uvlpr12L, uvlpr20L, etc.
   
   
6. When naming your files, it is usually a good idea to append a numeric digit to your filenames, since you will often find the necessity to create alternate versions. For example the test files here are named citytax1, citytax2, etc.
   
   
7. Most of these procedures apply to 'fixed length records' that may have packed decimal fields. For variable length files see 'Part_7'.
   
   
8. Our primary test/demo file has only 1 record type, but procedures for multi-record type files are illustrated later in sections M,N,O,P.
   
   
9. For simple files (only 1 record type), you can use the master script 'gencnvD2' (see page '4H1'), which generates all conversion jobs with just 1 command. But you should do the step by step procedures for your 1st file or 2 until you thoroughly understand these procedures.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_2 DATAcnv1 - convert EBCDIC to ASCII without COBOL copybook

Part 2 - Scanning Data to determine Record Layout - Contents

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2A1. DATAcnv1 - Convert EBCDIC to ASCII without COBOL copybook

signscan1/signscanA - Op-Instrns

Please see page '1B2' for our operating environment, changed into cnvdata superdir with various subdirs (d1ebc,d2asc,...,signs,...,etc). Note that most of these procedures are uvcopy jobs which have long commands, but we have provided scripts to allow shorter commands. By convention the uvcopy jobs have a digit suffix (1) & scripts have an UPPER case alpha (A).

 #1a. uvcopy signscan1,fili1=d1ebc/citytax1,filo1=signs/citytax1,uop=r128
      ===================================================================

 #1b. signscanA citytax1 r128  <-- use script equivalent for shorter command
      =======================    - ksh for unix/linux or .bat file for Win/Dos

signscan1 option selection screen

 uop=b2c1d0e99f80g33r0000s0t200y4z15 - option defaults
     b2                       - 2 blanks on right identify N&A type field
     b0                       - inhibit scan for blank right filled fields
       c0                     - drop no overlapped entries
       c1                     - drop type x overlapped entries (default)
       c8                     - drop all overlapped entries
         d1                   - scan for dates assuming format mmddyy
         d2                   - scan for dates assuming format mmddyyyy
         d4                   - scan for dates assuming format yymmdd
         d8                   - scan for dates assuming format yyyymmdd
         d0                   - inhibit scan for date fields (default)
           e99                - end year for date scans (e02 for 2002)
                                (use e99 if possible since more unique)
              f80             - from year for date recognition
                 g33          - percentage to qualify date scans
                    r????     - record size must agree with data file
                    r8192     - record size max (increase via was= below)
                s000          - skip count before testing starts
            t200              - terminate count (end testing)
          y4                  - disregard zoned signed fields shorter than
       z15                    - max length for zoned signed fields
 User OPtion (uop) defaults  = q1b2c1d0e99f80g33r0s0t200y4z15
  null to accept or re-specify (1 or more) --> r128   <-- ENTER REC SIZE

You must enter at least the record size option 'r128' in this case. Note the other option defaults & enter any you wish to change. For example if your date format was yymmdd & you wanted to test the 1st 500 records you would specify options as --> r128d4t500

The options are shown on the 3rd line of the output report (see next page) and in this example they would appear as:

 # default options=q1b2c1d0e99f80g33r0s0t200y4z15r128d4t500
 ================================================**********

The default options on the left are overridden by any options you enter which appear here on the right (underlined by '*'s vs '='s).

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2A2. DATAcnv1 - Convert EBCDIC to ASCII without COBOL copybook

signscan1/signscanA - Op-Instrns (repeated from previous page)

 #1a. uvcopy signscan1,fili1=d1ebc/citytax1,filo1=signs/citytax1,uop=r128
      ===================================================================

 #1b. signscanA citytax1 r128  <-- use script equivalent for shorter command
      =======================    - allows options for record-size, etc

 uvhd d1ebc/citytax1 r128a  <-- uvhd option 'a' (translate chars to ASCII)
 =========================

sample input file (uvhd display)

                    1         2         3         4         5         6
 r#     1 0123456789012345678901234567890123456789012345678901234567890123
        0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
          FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
          1013014000168508559800000000000000018150266550961400000000005153
       64 OUVER           BCV9S1H1.bi...Wb....g...qq.00014906A970530191809
          DEECD44444444444CCEFEFCF0888005880008000770FFFFFFFFCFFFFFFFFFFFF
          645590000000000023592181029C0072C0047C0111C000149061970530191809

sample output report

 #2a. vi signs/citytax1         <-- inspect output with vi
      =================

 #2b. uvlp12 signs/citytax1     <-- list output with uvlp12
      =====================         (uvlpr12 for Windows/DOS batch script)

 # signscan1 - report signed fields in EBCDIC data files 2004/02/21_13:31:35
 # recsize=00128,datafile=d1ebc/citytax1
 # options=q1b2d0e99f80g33r0s0t200y4z15r128d4
 #start(length)   end   count (p=packed,z=+0-0 signs,?=unreliable signs A-I,J-R)
 .00000(00010x) 00009 0000010
 .00010(00025x) 00034 0000010
 .00035(00025x) 00059 0000010
 .00060(00020x) 00079 0000010
 .00080(00008x) 00087
 .00088(00004p) 00091 0000007
 .00092(00005p) 00096 0000008
 .00097(00005p) 00101 0000009
 .00102(00005p) 00106 0000006
 .00107(00009z)?00115 0000008
 .00116(00006d) 00121 0000010 yymmdd
 .00122(00006x) 00127

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2A3. DATAcnv1 - Convert EBCDIC to ASCII without COBOL copybook

                   ** signscan1 report field-layout

           1         2         3         4
 01234567890123456789012345678901234567890

 # signscan1 - report signed fields in EBCDIC data files 2004/02/21_13:31:35
 # recsize=00128,datafile=d1ebc/citytax1
 # options=q1b2d0e99f80g33r0s0t200y4z15r128d4
 #start(length)   end   count (p=packed,z=+0-0 signs,?=unreliable signs A-I,J-R)
 .00000(00010x) 00009 0000010
 .00010(00025x) 00034 0000010
 .00035(00025x) 00059 0000010
 .00060(00020x) 00079 0000010
 .00080(00008x) 00087
 .00088(00004p) 00091 0000007
 .00092(00005p) 00096 0000008
 .00097(00005p) 00101 0000009
 .00102(00005p) 00106 0000006
 .00107(00009z)?00115 0000008
 .00116(00006d) 00121 0000010 yymmdd
 .00122(00006x) 00127

 00(1) - '.' identifies an active entry (vs '#' comment)
 01(5)  - end position of signed field
 07(5)  - start position of signed field
 13(5)  - length of signed field
 18(1)  - field type (p,z,x,d) see descriptions below:
 21(7)  - count of records with this signed field detected

signscan1 field types detected

 'p'   - packed fields detected by scan for x'0C' or x'0D'
       - reliable since thee codes would not otherwise occur in EBCDIC files

 'z'   - zoned signs detected by scan for x'C0' or x'D0' ('{' or '}')
       - reliable since '{' & '}' unlikely characters in EBCDIC files

 'z)?' - zoned signs detected by scan for single alpha among numeric digits
       - scans for pattern '##@##' where '#' any numeric & '@' any numeric
       - this could be zoned sign if alpha is 'A'-'I' or 'J'-'R'
         which represent signs for +1 thru +9 or -1 thru -9
       - unreliable since there could be legitimate single alphas in numerics
       - you need to investigate the file with uvhd to verify these fields
         (if invalid remove from sign report file before generating copybook)

 'x'   - character field detected by scanning for blank fill on right side
       - detects fields such as name & address
       - field start assumed to be end of prior field which might not be
         correct if no packed, zoned, date, or other blank filled on left

 'd'   - date field detected by scanning for date field patterns
       - specify option d1=mmddyy, d2=mmddyyyy, d4=yymmdd, d8=yyyymmdd
       - use uvhd to investigate file & determine which date pattern applies
       - you can combine options if more than 1 format present in 1 file
         for example d12 includes d4+d8 & the 4 digit year is scanned 1st

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2B1. DATAcnv1 - Convert EBCDIC to ASCII without COBOL copybook

scanning for Packed & Signed fields

We are primarily concerned with accurate EBCDIC to ASCII file conversion preserving any packed fields & correcting any EBCDIC zoned signs '}JKLMNOPQR' to the ASCII equivalents 'pqrstuvwxy'.

We only need to care about the accurate detection of packed & signed fields (type 'p' & 'z'). We don't need to worry about accurate boundaries for type 'x' fields since the entire record will be translated from EBCDIC to ASCII & then the packed & signed fields will be overlaid.

But if you also want to generate an accurate copybook (that reflects the application field boundaries), then you can modify the signscan results before conversion to copybook (or modify the copybook fields after). Using the previous page example, consider the 1st 2 fields detected by signscan:

 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
 10139923  GEORGE BROWN             1250 EAST PENDER STREET  VANCOUVER

 # end  start(length)
 .00009 00000(00010x)
 .00034 00010(00025x)

From an application viewpoint the 1st 10 byte field should probably be 2 fields (1 8 byte field & 1 2 byte field), so if desired you could change to:

 # end  start(length)
 .00007 00000(00008x)    f00000 --> rollno
 .00009 00008(00002x)    f00008 --> flags
 .00034 00010(00025x)    f00010 --> owner-name

After conversion to copybook (by sign2cpy1), you can assign meaningful names if desired. The default names will be f00000 (8 bytes), f00008 (2 bytes), & f00010 (25 bytes). Using your application knowledge, you might rename as 'rollno', 'flags',& 'owner-name'.

But remember if all you want is an accurate conversion, you don't need to do any alphanumeric field splitting & renaming. You only need to care about the accurate detection of packed & signed fields (types 'p' & 'z').

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2B2. DATAcnv1 - Convert EBCDIC to ASCII without COBOL copybook

signscan1 watch-out-for situations

'signscan1' will detect all packed & zoned signed fields that are logically possible to detect. However there are some data patterns where it is impossible to detect (for certain), the boundaries of packed & zoned signed fields. You should understand & watch for the following possible situations.

#1 - unpacked signed fields without +0 or -0

Unpacked signed fields are detected by the sign codes in the units position.

--> Signed +0 thru +9 are x'C0' thru x'C9' displayed as '{ABCDEFGHI'. --> Signed -0 thru -9 are x'D0' thru x'D9' displayed as '}JKLMNOPQR'

See the EBCDIC file display on page '2A2', using uvhd with option 'a' to translate the character line to ASCII, but still displaying the zones & digits in hexadecimal.

+0 x'C0' '{' and -0 x'D0' '}' are unique but A-I & J-R are not unique. Therefore if we find only A-I or J-R as single alphas among numerics, we will flag these entries with '?', as shown by our sample citytax1 file:

 .00102(00005p) 00106 0000006
 .00107(00009z)?00115 0000008         <-- '?' on zoned fields w/o '{' '}'
 .00116(00006d) 00121 0000010 yymmdd

#2 - unsigned fields prior to signed fields

The left hand boundary of unpacked signed fields is determined by scanning left until we find a non-numeric byte. If an unsigned field immediately precedes a signed field, we will not know where the signed field ends. For example consider the following file:

 product#  price   qty    amount
        unsigned signed    signed
 hammer  0000495000000A000000049E   <-- 4.95 * 1 =  4.95
 wrench  0000995000000C000000298E   <-- 9.95 * 3 = 29.85

The amount field is easily determined to be 9 digits long since the preceding field is also a signed field with unique codes in the units position. However we don't know for certain how long the 'qty' field is, since the preceding field is unsigned & has no unique codes in the zone of the units.

The good news here is that it does not matter, since the translations of unsigned & signed fields are the same except for the units position of the signed fields. In this example, signscan1 would report the price & qty as 1 signed field 14 digits long.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2B3. DATAcnv1 - Convert EBCDIC to ASCII without COBOL copybook

#3 - packed field Right boundary

Packed fields are identified by the sign codes in the right hand byte:

--> Negative fields will have x'0D' thru x'9D' in the right hand byte. --> Positive fields will have x'0C' thru x'9C' or x'0F' thru x'9F'.

x'_C' & x'_D' are unique, but some of the x'_F' bytes are not unique. x'4F' is ']', x'5F' is '^', x'6F' is '?', x'7F' is '"'.

If a packed field only had 1 of these last 4 codes in the sign byte (in all records tested), we could not uniquely identify it as a packed sign byte. But this is highly unlikely. We might increase our terminate count to test more records in the file.

#4 - packed field left boundary

The left boundary of a packed field is determined by scanning left until we reach a byte whose zone or digit is not 0 thru 9. This could be the sign byte of another packed field or any unpacked numeric, alpha, or special character.

If we encounter a x'40', we would not know for sure whether it is 2 digits of the packed field 4 & 0, or it could be an EBCDIC character blank. But if we encounter 2 blanks (x'4040'), we will assume this is the end of the packed field (a small possibility of ending too soon).

If a packed field always had 1 x'40' on its left side, we might report the length as 1 longer than it should be.

#5. - option to drop overlapped entries

signscan1 builds a table in memory & dumps to the report file at EOF. Before dumping, the table is sorted on startbyte(length)endbyte & any overlapped type 'x' entries are dropped (default option 'c1'). Type x (alphanumeric) fields are superfluous to type 'p' & 'z' which are vital for accurate file conversions of packed & signed fields.

If you are curious, you could try option 'c0' to inhibit these drops, but you should rerun with option 'c1' before you continue to 'sign2cpy1', which would have problems with the overlapped fields. Overlapped fields are probably caused by 2 blanks in name & address fields or by 2 blanks x'4040' in packed fields (which must be ignored).

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2B4. DATAcnv1 - Convert EBCDIC to ASCII without COBOL copybook

Tips to Improve Scan Results

1. Adjust the scan terminate count at the prompt by signscan1. Option 't' defaults to t200. You might increase if you suspect that the record data is not uniform thruout the file. For example, the negative unpacked signed fields may be further down the file. This would only be significant if the +0's were just '0' x'F0' vs '{' x'C0'.
   
   
2. You might set the max length for unpacked signed fields using option 'z' (default 'z15'). If you knew the actual max signed field length, you could prevent unsigned fields being considered part of max length signed fields.
   
   
3. Probably the best way to verify the scan results is to continue these procedures until you get to 'uvhdcob'.
   
   
   

   2C1.	sign2cpy1 - convert sign report to copybook
   2D1.	cobmap1 - convert copybook to cobmap
   3D1.	uvhdcob - display data field contents beside copybook field names

See the uvhdcob example on page '3D1'. The original auto assigned fieldnames (f00000, f00008, f00010, etc) have already been replaced with meaningful names, but that is not necessary to check field boundaries.

Using uvhdcob, you can quickly spot any packed or signed fields with incorrect boundaries.

Tips to Improve DATE Scan Results

1. Use uvhd to investigate the data, identify the date formats present so you can specify the correct date scan format options. Most data files have only 1 date format, but it is possible to combine options (for example d12 = d8 + d4). Note that the scan sequence is #1=yyyymmdd(d8), #2=mmddyyyy(d2), #3=yymmdd(d4), #4=mmddyy(d1) which ensures we scan for 4 digit years before 2 digit years. Four digit years (beginning with '19' or '20) are much more unique.
   
   
2. If possible specify end year (option 'e') as e99 (from year default f80). Restricting year scan from '80' to '99' will enhance date recognition by avoiding '00' which occurs very frequently in data records
   
   
3. You might have to modify option 'g' percentage for date qualification which defaults to g33 (33 % of records have date at this byte#).
   
   
4. You can run the signscan1 job several times modifying these options until you get optimum results
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2C1. DATAcnv1 - Convert EBCDIC to ASCII without COBOL copybook

sign2cpy1 - convert signscan1 report to COBOL copybook

Pre-requisite is uvcopy job 'signscan1' which created sign report file by scanning an EBCDIC data file for packed & zoned signs. This job will convert the sign report to a COBOL copybook which may be used as input for several Vancouver Utility jobs (see further below).

sign2cpy1 - Operating Instructions

 #1a. uvcopy sign2cpy1,fili1=signs/citytax1,filo1=cpys/citytax1
      =========================================================

 #1b. sign2cpyA citytax1      <-- use script equivalent for shorter command
      ==================        - ksh for unix/linux or .bat file for Win/Dos

sample input (from signscan1)

 # signscan1 - report signed fields in EBCDIC data files 2004/02/21_13:31:35
 # recsize=00128,datafile=d1ebc/citytax1
 # options=q1b2d0e99f80g33r0s0t200y4z15r128d4
 #start(length)   end   count (p=packed,z=+0-0 signs,?=unreliable signs A-I,J-R)
 .00000(00010x) 00009 0000010
 .00010(00025x) 00034 0000010
 .00035(00025x) 00059 0000010
 .00060(00020x) 00079 0000010
 .00080(00008x) 00087
 .00088(00004p) 00091 0000007
 .00092(00005p) 00096 0000008
 .00097(00005p) 00101 0000009
 .00102(00005p) 00106 0000006
 .00107(00009z)?00115 0000008
 .00116(00006d) 00121 0000010 yymmdd
 .00122(00006x) 00127

sample output (copybook)

 ******* copybook for: citytax1 generated from signscan1 report
        01  citytax1.
            10 f00000                pic  x(00010).
            10 f00010                pic  x(00025).
            10 f00035                pic  x(00025).
            10 f00060                pic  x(00020).
            10 f00080                pic  x(00008).
            10 f00088                pic s9(00007) comp-3.
            10 f00092                pic s9(00009) comp-3.
            10 f00097                pic s9(00009) comp-3.
            10 f00102                pic s9(00009) comp-3.
            10 f00107                pic s9(00009).
            10 f00116yymmdd          pic  x(00006).
            10 f00122                pic  x(00006).

Before you attempt to make corrections & enhancements to the copybook, you should run cobmap1/cobmapA which converts the copybook to a 'cobmap' showing field start/end/length/type on the right side. Please see next page --->

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2D1. DATAcnv1 - Convert EBCDIC to ASCII with or without COBOL copybook

cobmap1/cobmapA - convert copybook to record-layout

 #1a. uvcopy cobmap1,fili1=cpys/citytax1,filo1=maps/citytax1
      ======================================================

 #1b. cobmapA citytax1       <-- use script equivalent for shorter command
      ================         - ksh for unix/linux or .bat file for Win/Dos

 #2a. more maps/citytax1       - display copybook map for review & enhancement
      ==================

 #2b. uvlp12 maps/citytax1     - or list copybook map with UV script for lasers
      ====================       (uvlpr12 for Windows/DOS batch script)

sample cobmap output report - from signscan1

 cobmap1  start-end bytes for cobol record fields    200212051728  pg# 0001
 cpys/citytax1                   citytax1            RCSZ=00128  bgn  end  lth typ
 * copybook for: citytax1
 * generated from signscan1 report $datetime
  01  citytax1.
      10 f00000                pic  x(00010).                 0000 0009  010
      10 f00010                pic  x(00025).                 0010 0034  025
      10 f00035                pic  x(00025).                 0035 0059  025
      10 f00060                pic  x(00020).                 0060 0079  020
      10 f00080                pic  x(00008).                 0080 0087  008
      10 f00088                pic s9(00007) comp-3.          0088 0091  004pns 7
      10 f00092                pic s9(00009) comp-3.          0092 0096  005pns 9
      10 f00097                pic s9(00009) comp-3.          0097 0101  005pns 9
      10 f00102                pic s9(00009) comp-3.          0102 0106  005pns 9
      10 f00107                pic s9(00009).                 0107 0115  009 ns 9
      10 f00116yymmdd          pic  x(00006).                 0116 0121  006
      10 f00122                pic  x(00006).                 0122 0127  006
 *RCSZ=00128                                                       0128

 #3. vi cpys/citytax1      - modify the generated copybook (see next page)
     ================

Since the following conversions are based upon this automatically scan generated copybook, you should inspect it & if possible modify it to improve the results of the following data file conversions.

Please see the example on the next page --->

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2D2. DATAcnv1 - Convert EBCDIC to ASCII with or without COBOL copybook

sample signscan1 cobmap report (BEFORE enhancement)

  01  citytax1.
      10 f00000                pic  x(00010).                 0000 0009  010
      10 f00010                pic  x(00025).                 0010 0034  025
      10 f00035                pic  x(00025).                 0035 0059  025
      10 f00060                pic  x(00020).                 0060 0079  020
      10 f00080                pic  x(00008).                 0080 0087  008
      10 f00088                pic s9(00007) comp-3.          0088 0091  004pns 7
      10 f00092                pic s9(00009) comp-3.          0092 0096  005pns 9
      10 f00097                pic s9(00009) comp-3.          0097 0101  005pns 9
      10 f00102                pic s9(00009) comp-3.          0102 0106  005pns 9
      10 f00107                pic s9(00009).                 0107 0115  009 ns 9
      10 f00116yymmdd          pic  x(00006).                 0116 0121  006
      10 f00122                pic  x(00006).                 0122 0127  006
 *RCSZ=00128                                                       0128

Enhancing the scan-generated copybook

 #3. vi cpys/citytax1      - modify the copybook (not the cobmap)
     ================

1. Split type 'x' areas to known components
2. Assign meaningful names
3. Correct unreliable zoned signs (flagged with '?' on the signscan1 report) These were detected by A-I & J-R in numeric fields (vs signs { & } +0 & -0 which are more reliable).
   

   AFTER editing fieldnames & rerun cobmapA

 cobmap1  start-end bytes for cobol record fields    200306092029  pg# 0001
 cpys/citytax1                  citytax1           RCSZ=00128  bgn  end  lth typ
 * citytax1 copybook, from signscan1 + edit meaningful names
  01  citytax1.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-3.          0088 0091  004pns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  x(00006).                 0116 0121  006
      10 filler001             pic  x(00006).                 0122 0127  006
 *RCSZ=00128                                                       0128

Note that you should plan your changes using the 'cobmap' listing, but be sure to modify the copybook (not the cobmap). Then rerun the cobmapA script to regenerate & re-list the cobmap for the new copybook to verify that your changes result in the same record size.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_3 Generating uvcopy jobs to convert data files

Part 3 - Generating uvcopy jobs to convert data files - Contents

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3A1. Converting Mainframe Data Files from EBCDIC to ASCII

Generating uvcopy jobs to convert data files - Introduction

The Vancouver Utilities have jobs to convert mainframe data files for use on Unix, Linux, or Windows.

A 'uvcopy job' is generated (for each data file) automatically from the COBOL copybook. The uvcopy job has 1 instruction to translate the entire record from EBCDIC to ASCII. Then we generate instructions to preserve any packed/binary fields which are the same on unix/linux/windows.

We assume here that the data files are intended for use by Micro Focus COBOL which provides for packed/binary fields exactly the same as on the mainframe.

generate all jobs to convert EBCDIC datafiles to ASCII

'scripts' to generate jobs to convert EBCDIC to ASCII

super scripts to generate ALL jobs

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3A2. translating data files EBCDIC to ASCII, preserving packed

Directories Required

 /home/cnvdata
 :-----cpys       <-- copybooks with no extensions
 :-----d1ebc      <-- EBCDIC data files (renamed same as copybooks)
 :-----d2asc      <-- output data files converted to ASCII
 :-----maps       <-- record layouts (cobmaps) created from copybooks
 :-----pfx1       <-- uvcopy conversion jobs generated from cobmaps by uvdata51
 :-----pfx2       <-- reserved in case you use uvdata52 to insert datafilenames
 :-----pfx3       <-- uvcopy jobs copied here before execution
                    - modified if required (if multiple record types)

Pre-Requisites

1. data file stored in d1ebc/...
2. data file renamed same as copybook (with any extension removed)
3. copybook stored in cpys/... (with any extension removed)
   

   Illustrating EBCDIC to ASCII conversion

We will illustrate the conversion using our demo data file 'd1ebc/citytax1' and the corresponding COBOL copybook 'cpys/citytax1'. We need to use the 'uvhd' utility to display the data file since it is EBCDIC with packed fields.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3A3. translating data files EBCDIC to ASCII, preserving packed

sample input file (uvhd display)

 uvhd d1ebc/citytax1 r128a  <-- uvhd option 'a' (translate chars to ASCII)
 =========================

                    1         2         3         4         5         6
 r#     1 0123456789012345678901234567890123456789012345678901234567890123
        0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
          FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
          1013014000168508559800000000000000018150266550961400000000005153
       64 OUVER           BCV9S1H1.bi...Wb....g...qq.00014906A970530191809
          DEECD44444444444CCEFEFCF0888005880008000770FFFFFFFFCFFFFFFFFFFFF
          645590000000000023592181029C0072C0047C0111C000149061970530191809

corresponding COBOL copybook (with layout on right)

 cobmap1  start-end bytes for cobol record fields    200306092029  pg# 0001
 cpys/citytax1                  citytax1           RCSZ=00128  bgn  end  lth typ
 * citytax1 copybook, from signscan1 + edit meaningful names
  01  citytax1.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-3.          0088 0091  004pns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  x(00006).                 0116 0121  006
      10 filler001             pic  x(00006).                 0122 0127  006
 *RCSZ=00128                                                       0128

uvcopy data conversion job - generated from copybook

 # citytax1 - uvcopy job, EBCDIC to ASCII, preserve packed, fix signs
 opr='citytax1 - uvcopy code generated from copybook: citytax1 '
 was=a33000b33000
 fili1=?d1ebc/citytax1,rcs=00128,typ=RSF
 filo1=?d2asc/citytax1,rcs=00128,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
        mvc    b88(19),a88             pns post-date:face-value
        trt    b107(9),$trtsea          ns maint-tax
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3B1. translating data files EBCDIC to ASCII, preserving packed

Generating uvcopy job to translate EBCDIC to ASCII

          uvcopy cobmap1         uvcopy uvdata51
 copybook --------------> cobmap ---------------> uvcopy job (EBCDIC to ASCII)
 cpys/citytax1         maps/citytax1            pfx1/citytax1

 We will present 2 ways to generate the EBCDIC to ASCII conversion job
 - run the 2 uvcopy jobs separately (cobmap1 & uvdata51), OR run the script
 (gencnvA) which runs the 2 jobs within the script.

The only input required by genpipeA is the COBOL copybook for the data file.

1st - convert the copybook to a cobmap (record layout)

 #1a. uvcopy cobmap1,fili1=cpys/citytax1,filo1=maps/citytax1
      ======================================================
      - convert any 1 copybook to a 'cobmap' (record layout)

         - - - OR - - -

 #1b. cobmapA citytax1        <-- script to convert any 1 copybook to cobmap
      ================          - same as above, but shorter command

copybook converted to cobmap (layout on right)

 cobmap1  start-end bytes for cobol record fields    200306092029  pg# 0001
 cpys/citytax1                  citytax1           RCSZ=00128  bgn  end  lth typ
 * citytax1 copybook, from signscan1 + edit meaningful names
  01  citytax1.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-3.          0088 0091  004pns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  x(00006).                 0116 0121  006
      10 filler001             pic  x(00006).                 0122 0127  006
 *RCSZ=00128                                                       0128

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3B2. Converting Mainframe Data Files from EBCDIC to ASCII

Generating uvcopy jobs to convert data files - Op. Instrns

 #2a. uvcopy uvdata51,fili1=maps/citytax1,filo1=pfx1/citytax1  <-- uvcopy command
      =======================================================

 #2b. gencnvA citytax1     <-- script equivalent of above (shorter command)
      ================       - ksh for unix/linux or .bat file for Windows/Dos
                             - see script listed on page '9D1'

uvdata51/gencnvA output - uvcopy job to convert EBCDIC to ASCII

 # citytax1 - uvcopy job, EBCDIC to ASCII, preserve packed, fix signs
 opr='citytax1 - uvcopy code generated from copybook: citytax1 '
 uop=q0
 was=a33000b33000
 fili1=?d1ebc/citytax1,rcs=00128,typ=RSF
 filo1=?d2asc/citytax1,rcs=00128,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
        mvc    b88(19),a88             pns post-date:face-value
        trt    b107(9),$trtsea          ns maint-tax
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

Notes re uvcopy job to convert EBCDIC to ASCII

The uvcopy jobs are automatically generated by gencnvA/uvdata51 based on the copybook record definitions which define the locations of packed & zoned signed fields.

These jobs are generated from a template into which we insert filenames, record sizes,& the instructions to preserve the packed fields & correct the zoned signs.

In the 'citytax1' example there were only 2 instructions inserted to preserve packed & correct signs, because the 4 packed fields were contiguous.

Note that for some real life data files, there could be hundreds of packed & zoned signed fields. The number of mvc/trt instructions inserted depends on how contiguous the packed/zoned fields are.

See detailed explanations on the next page --->

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3B3. Generating uvcopy jobs to convert DATA files

generated uvcopy job to convert EBCDIC to ASCII

 # citytax1 - uvcopy job, EBCDIC to ASCII, preserve packed, fix signs   #01
 opr='citytax1 - uvcopy code generated from copybook: citytax1 '        #02
 uop=q0                                                                 #03
 was=a33000b33000                                                       #04
 fili1=?d1ebc/citytax1,rcs=00128,typ=RSF                                #05
 filo1=?d2asc/citytax1,rcs=00128,typ=RSF                                #06
 @run                                                                   #07
        opn    all                                                      #08
 loop   get    fili1,a0                                                 #09
        skp>   eof                                                      #10
        mvc    b0(00128),a0       move inrec to outarea before prcsng   #11
        tra    b0(00128)          translate entire outarea to ASCII     #12
 #      ---               <-- insert R/T tests here for redefined recs  #13
        mvc    b88(19),a88        pns post-date:face-value              #14
        trt    b107(9),$trtsea     ns maint-tax                         #15
 put1   put    filo1,b0                                                 #16
        skp    loop                                                     #17
 eof    cls    all                                                      #18
        eoj                                                             #19

The uvcopy job generated by 'gencnvA/uvdata51' is re-listed above with line#s on the right so we can give a brief explanation of the more relevant instructions. For complete explanations, please refer to uvcopy3.htm.

Lines 5&6 declare the I/O (fili1/filo1) filenames. The generation has inserted the correct filenames & record sizes.

Line 8 (opn) opens all files.

Line 9 (get) is the beginning of the record processing loop. 'get' reads the next record from fili1 into area 'a' starting at byte '0' (a0).

Line 10 (skp>) will skip to eof at End Of File, because the 'get' will set the condition code '>' when there are no more records.

Line 11 (mvc) moves the record to output area 'b' starting at byte '0' for a length of 128 bytes(recsize), from input area 'a' starting at byte '0'.

Line 12 (tra) translates to ASCII, entire record in area 'b' from byte '0'.

Line 13 is a #comment indicating where you can add code to test Record Types.

Line 14 (mvc) moves the packed fields from the input area 'a' beginning at byte 88 to the output area 'b' beginning at byte 88 for a length of (19), which is the total of 4 packed fields (4+5+5+5).

Line 15 (trt) translates the zoned signed field b107(9) using translate table 'trtsea', which translates Ebcdic zoned signs to Ascii zoned signs.

Line 16 (put) writes to the output file from area 'b' starting from byte '0' for the record length declared on the filo1= (line 6).

Line 17 closes the files & line 18 ends the job.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3C1. uvcopy jobs to convert EBCDIC to ASCII

allowing for modifications before Execution

 #3. cp pfx1/citytax1 pfx3    <-- copy auto-generated job to pfx3
     =====================      - before any manual changes & execution

 #4. vi pfx3/citytax1         <-- make any manual changes required
     ================             (if multiple record types)

Executing generated job to convert EBCDIC to ASCII

 #1a. uvcopy pfx3/citytax1,fili1=d1ebc/citytax1,filo1=d2asc/citytax1
      ==============================================================

 #1b. uvcopy pfx3/citytax1   <-- easier (filenames default as shown above)
      ====================

sample input EBCDIC

 #2. uvhd d1ebc/citytax1 r128a  <-- display EBCDIC input file
     =========================

                       1         2         3         4         5         6
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
             1013014000168508559800000000000000018150266550961400000000005153
          64 OUVER           BCV9S1H1.bi...Wb....g...qq.00014906A970530191809
             DEECD44444444444CCEFEFCF0888005880008000770FFFFFFFFCFFFFFFFFFFFF
             645590000000000023592181029C0072C0047C0111C000149061970530191809

sample output ASCII

 #3. uvhd d2asc/citytax1 r128   <-- display ASCII output file
     ========================

                       1         2         3         4         5         6
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             3333333322444424445522222222222222233332445442544422222222225444
             1013014000AF8E085E29000000000000000181502F75E02F14000000000061E3
          64 OUVER           BCV9S1H1......W.........qq.000149061970530191809
             4554522222222222445353430888005880008000770333333333333333333333
             F56520000000000023693181029C0072C0047C0111C000149061970530191809

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3C2. Converting Mainframe Data Files from EBCDIC to ASCII

Notes re EBCDIC to ASCII conversion

1. Note that the character fields have been translated from EBCDIC to ASCII For example the 1st record byte '1' is x'F1' in the EBCDIC record, but it is x'31' in the ASCII record.
   
   
2. The reason we can read the character lines in the uvhd EBCDIC display is because we specified option 'a' (r128a) to translate the character line of the EBCDIC file display (vs the uvhd ASCII display which omitted option 'a').
   
   
3. Note that the packed fields are preserved in the ASCII file, because the conversion job (see prior page) moved them (unchanged) from input to output after the translation of the entire record (which would have destroyed them).
   
   
4. Note the zoned sign at byte 115 (51+64) has been corrected from 'A' (x'C1') in EBCDIC to '1' (x'31') in ASCII. See more explanation on the next page -->
   
   

   converting Zoned signs

Zoned (unpacked) signs must be converted from the mainframe codes to the codes accepted by Micro Focus COBOL on Unix/Linux/Windows systems.

• mainframe EBCDIC zoned signs for +0:+9 are: {ABCDEFGHI (x'C0' - x'C9')
• mainframe EBCDIC zoned signs for -0:-9 are: }JKLMNOPQR (x'D0' - x'D9')
• unix COBOL ASCII zoned signs for -0:-9 are: pqrstuvwxy (x'70' - x'79')
• unix COBOL ASCII zoned signs for +0:+9 are: 0123456789 (x'30' - x'39')

Notes

1. Without identifying & correcting signed fields, you would simply get the EBCDIC alpha codes translated to their ASCII alpha codes, which would not be meaningful to Micro Focus COBOL on Unix/Linux/Windows.
   
   
2. The 'trt' instruction with translate table '$trtsea' is used to translate EBCDIC zoned signs to ASCII zoned signs, for example:
   

    trt   b107(9),$trtsea
    =====================

3. The above instruction translates the entire field, but it is only the units position (where the sign is) that needs the translation. (IE - the translate does not change any digits, only the alpha sign codes)
   
   
4. The fact that ASCII signs appear as lower case alphas 'pqrstuvwxy' is simply a byproduct of them having x'7_' zones.
   
   
5. There is no separate set of codes for +digits on unix, x'3_' zones are the only + zones required.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3C3. Converting Mainframe Data Files from EBCDIC to ASCII

Here is a sample sign conversion based on record #8 of the demo file citytax1. I am showing only bytes 64-128 of these 128 byte records & have replaced some bytes surrounding the signed field in bytes 107-115 with '...'. Note that 'R' EBCDIC -9 x'D9' is converted to 'y' ASCII -9 x'79'.

          64 OUVER           ALT6C4J2.Ri....Y@..........00000909R............
             DEECD44444444444CDEFCFDF0589000570000000090FFFFFFFFDFFFFFFFFFFFF
             645590000000000013363412029C0039C0000C0099C000009099980601190157

          64 OUVER           ALT6C4J2.R.....Y|..........00000909y............
             4554522222222222445343430589000570000000090333333337333333333333
             F5652000000000001C4634A2029C0039C0000C0099C000009099980601190157

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3C4. uvcopy jobs to convert EBCDIC to ASCII

Manual Changes Required ??

Manual changes may be required to the auto-generated uvcopy job, before you get the optimum results. The most common reason is to allow for multiple record types (with different layouts).

I recommend you do NOT modify the auto-generated jobs in subdir 'pfx1', but first copy them to subdir 'pfx3' before making any manual changes. This also protects your manual changes in pfx3 in case of regenerating (after changes to the copybook.).

Regardless of whether changes are required or not, I suggest you always copy the jobs from pfx1 to pfx3, before executing (from pfx3). Often you wont realize you need modifications until after you inspect the output data.

 #3.  cp pfx1/citytax1 pfx3   - copy auto-generated job to pfx3
      =====================     before any manual changes &/or execution

 #4.  vi pfx3/citytax1        - make any manual changes required
      ================          (record type tests, see example page '5B2')

sample manual change - Record-Type test

The most common change required is to insert code to test record types for files with redefined records (different layouts in the same file). Our 'citytax1' test file does not have multiple record types and the generated job requires no changes before execution.

But for illustration purposes, let us suppose it had a header record with no packed or zoned fields. It is a simple matter to insert 2 instructions to test for this & bypass the packed/zoned instructions which would destroy parts of our all character header record. We will assume that the header record is identified with an 'H' in byte 8 (zero relative).

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3C5. uvcopy jobs to convert EBCDIC to ASCII

demo job after inserting Record-Type test code

 # citytax1 - uvcopy job, EBCDIC to ASCII, preserve packed, fix signs
 opr='citytax1 - uvcopy code generated from copybook: citytax1 '
 uop=q0
 was=a33000b33000
 fili1=?d1ebc/citytax1,rcs=00128,typ=RSF
 filo1=?d2asc/citytax1,rcs=00128,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #------------------------------------------------------------------------
 #      ---                <-- insert R/T tests here for redefined records
        cmc    b8(1),'H'          Header record ? (no packed/zoned fields)
        skp=   put1               ifso - bypass packed/zoned mvc/trt
 #------------------------------------------------------------------------
        mvc    b88(19),a88             pns post-date:face-value
        trt    b107(9),$trtsea          ns maint-tax
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

 We have inserted instructions to compare (cmc) byte 8 to 'H' & if =, to skip
 to the 'put1' label, bypassing the mvc/trt which would destroy part of our
 header record. I have also added the '#------' lines to emphasize our inserted
 instructions, but you would not need to bother with these.

See the Operating Instructions to execute the uvcopy job on the next page --->

Note that we did not need to modify the 'citytax1' job before executing since the citytax1 datafile does not in fact have any header record.

For an example of a multi record type file (with 3 record types) & the coding changes required, please see the 'citytax2' demo beginning on page '5A1'.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3D1. Converting Mainframe Data Files from EBCDIC to ASCII

uvhdcob - list data fields beside copybook field names

'uvhdcob' is similar to 'uvhd', but also lists the copybook fieldnames beside the data field contents. You must have run cobmap1/cobmapA to convert the copybook to the 'cobmap' required by uvhdcob. To display our citytax1 demo file, we will specify option 'r128' for recsize + option 'a' to translate the characters line to ASCII.

     uvhdcob datafile cobmap [options]       <-- command format
     =================================

 #1. uvhdcob d1ebc/citytax1 maps/citytax1 a  <-- display citytax1 EBCDIC file
     ======================================      'a' option translate to ASCII

 d1ebc/citytax1 rec#=1 recs=10 rsz=128 fsiz=1280 fptr=0
 cobmap=maps/citytax1 today=0306092049 lastmod=02120515
 rec#       1 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 folio                        0   9    10130140
 002 name                        10  34    JOHN HENRY
 003 address                     35  59    1815 BOWEN ROAD
 004 city                        60  79    VANCOUVER
 005 province                    80  81    BC
 006 zip                         82  87    V9S1H1
 007 post-date                   88  91pns 0082898C
 008 land-value                  92  96pns 000057828C
 009 improv-value                97 101pns 000004870C
 010 face-value                 102 106pns 000171710C
 011 maint-tax                  107 115 ns 00014906A
 012 purchase-date              116 121    970530
 013 filler001                  122 127    191809
 rec#=1 of 10,null=next,r#=rec,s=search,u=update,p=print,i=iprint,w=write
 ,t=tally,c=chkseq,e=exit to uvhd,q=quit,?=help -->

 enter command --> p1    <-- print 1st record
 enter command --> q     <-- quit

Note you can use uvhdcob on either the EBCDIC file or the ASCII file. For the EBCDIC file, specify option 'a' to translate to ASCII (as above). For the ASCII file, omit the 'a' option. You might want to specify option 'm' to increase the 'maximum' lines per screen if you have records > 320 bytes. If you are on a Windows terminal you can specify m60, for example:

 #1a. uvhdcob d2asc/citytax1 maps/citytax1 m60  <-- ASCII data + fieldnames
      ========================================    - 60 lines/scrn or recsize

 #1b. --> i3f1   <-- immediate print 1st 3 records, formfeed each page
          ====

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_4 Generating uvcopy jobs to convert data files

converting to pipe delimited text files to load RDBMS - Contents

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4A1. Converting Data Files to ASCII Delimited for loading SQL DBs

uvcopy jobs to generate pipe delimit & SQL table create/load

scripts to generate pipe delimit & SQL table create/load

super script to generate ALL jobs

 gencnvD2 citytax1  <-- script to generate all jobs + batch-file + loader-file
 =================    - from the demo copybook 'cpys/citytax1'

related documentation

 SQLconvert.htm - convert flat files to pipe delimited for Microsoft SQL server
                  - driven by COBOL copybooks
                  - also generate scripts to create & load tables
                  - generates jobs for all copybooks & all data files
                    (vs this DATAcnv1.doc for 1 copybook/file at a time)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4A2. Converting Data Files to ASCII Delimited for loading SQL DBs

Directories Required

 /home/cnvdata
 :-----cpys       <-- copybooks with no extensions
 :-----d1ebc      <-- EBCDIC data files (renamed same as copybooks)
 :-----d2asc      <-- data files converted to ASCII (by uvdata51/gencnvA Part_3)
 :-----d4pipe     <-- output files in pipe delimited format
 :-----maps       <-- record layouts (cobmaps) created from copybooks
 :-----pfp1       <-- uvcopy pipe delimit jobs generated from cobmaps by genpipe1
 :-----pfp2       <-- reserved in case you use uvdata54 to insert datafilenames
 :-----pfp3       <-- uvcopy jobs copied here before execution
 :-----             - modified if required (if multiple record types)
 :-----sqls       <-- SQL batch-files & loader control-files

Pre-Requisites

1. data file already converted to ASCII & stored in d2asc/...
   
   
2. data file renamed same as copybook (with any extension removed)
   
   
3. copybook stored in cpys/... (with any extension removed)
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4A3. Converting Data Files to ASCII Delimited for loading SQL DBs

sample data file with packed data

 uvhd d2asc/citytax1 r128    <-- display sample input data with 'uvhd'
 ========================      - note packed fields in bytes 88-106
                               - will show only 1st 2 of 10 records in the file

                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             3333333322444424445522222222222222233332445442544422222222225444
             1013014000AF8E085E29000000000000000181502F75E02F14000000000061E3
          64 OUVER           BCV9S1H1......W.........qq.000149061970530191809
             4554522222222222445353430888005880008000770333333333333333333333
             F56520000000000023693181029C0072C0047C0111C000149061970530191809

                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 10139923  GEORGE BROWN             1250 EAST PENDER STREET  VANC
             3333333322444544245454222222222222233332445525444452555445225444
             101399230075F275022F7E0000000000000125005134005E45203425540061E3
          64 OUVER           BCV5L1W1......W.........xRL00014626q980601190156
             4554522222222222445343530888005880008800754333333337333333333333
             F5652000000000002365C171029C0072C0047C0182C000146261980601190156

sample output - pipe delimited text

 10130140|JOHN HENRY|1815 BOWEN ROAD|VANCOUVER|BC|V9S1H1|+82898|+57828|+4870|+171710|+1490.61|1997/05/30|191809|
 10139923|GEORGE BROWN|1250 EAST PENDER STREET|VANCOUVER|BC|V5L1W1|+82898|+57828|+4878|+178524|-1462.61|1998/06/01|190156|
 10147615|BONNIE SMITH|44430 YALE ROAD WEST|VANCOUVER|BC|V2P6J1|+121395|+39623|+0|+51914|+376.92|1995/06/01|201031|
 10149304|TONY TATTERSAL|3350 S.W. MARINE DRIVE|VANCOUVER|BC|V6N3Y9|+50594|+76800|+0|+208391|+1925.03|1993/06/01|201512|
 10154688|MEL RAHAL|1110 - 625 HOWE STREET|VANCOUVER|BC|V6C2T6|+82898|+44880|+0|+125038|+1154.79|1996/11/26|082351|
 10173320|SAM SMITH|2381 ROGERS AVE|VANCOUVER|BC|V3K5Y2|+50498|+60600|+0|+169086|+1562.00|1997/05/30|191810|
 10204700|JOHN JENSEN|12611-100 ST|VANCOUVER|AL|T8V4H2|+122396|+15720|+0|+43744|-404.40|1996/11/26|081702|
 10223240|BOB ANDERSON|7215 ARGYLL ROAD|VANCOUVER|AL|T6C4J2|+52899|+3597|+0|+9990|-90.99|1998/06/01|190157|
 10308685|TOM PERRY|3932 - 3A ST. N.W.|VANCOUVER|AL|T2E6R4|+70192|+68155|+0|+125773|+1065.10|1992/06/01|221650|
 10318833|HARRY SMIRNOFF|BOX 308, STN J|VANCOUVER|AL|T2A4X6|+11399|+54541|+42046|+93502|+761.46|1997/05/30|191811|

The output file will be an all text ASCII file and can be easily listed with the 'uvlp' scripts or other unix utilities. We will later illustrate another utility 'sqlcreate1' to generate batch-file & control-file to create & load table from the pipe delimted file into a Relational Data Base (see page '4F1').

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4B1. Converting Data Files to ASCII Delimited for loading SQL DBs

Generating uvcopy job to convert data to pipe delimited text

          uvcopy cobmap1         uvcopy genpipe1
 copybook --------------> cobmap ---------------> uvcopy job (to pipe delimit)
 cpys/citytax1         maps/citytax1            pfp1/citytax1

We will present 2 ways to generate the pipe delimit job - run the 2 uvcopy jobs separately (cobmap1 & genpipe1), OR run the script genpipeA which runs the 2 jobs within the script.

The only input required by genpipeA is the COBOL copybook for the data file.

#1 - convert the copybook to a cobmap (record layout)

 #1a. uvcopy cobmap1,fili1=cpys/citytax1,filo1=maps/citytax1
      ======================================================
      - convert any 1 copybook to a 'cobmap' (record layout)

         - - - OR - - -

 #1b. cobmapA citytax1        <-- script to convert any 1 copybook to cobmap
      ================          - same as above, but shorter command

input - copybook (cobmap listed below)

 cobmap1  start-end bytes for cobol record fields    200306092029  pg# 0001
 cpys/citytax1                  citytax1           RCSZ=00128  bgn  end  lth typ
 * citytax1 copybook, from signscan1 + edit meaningful names
  01  citytax1.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-3.          0088 0091  004pns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  x(00006).                 0116 0121  006
      10 filler001             pic  x(00006).                 0122 0127  006
 *RCSZ=00128                                                       0128

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4B2. Converting Data Files to ASCII Delimited for loading SQL DBs

#2 - convert the cobmap to a uvcopy pipe-delimit job

 #2a. uvcopy genpipe1,fili1=maps/citytax1,filo1=pfp1/citytax1\
      ========================================================
             ,arg1=delimiter,arg2=enclose-character,uop=translate-option
             ===========================================================
             ,arg1=7C,arg2=20,uop=a0   <-- default arguments & options
             =======================
                       - arg1 & arg2 must be specified in hex (if specified)
             ,arg1=7C  - delimiter '|' x'7C' (the default, no need to specify)
             ,arg1=2C  - alternate delimiter ',' x'2C' - see example page '4D1'
             ,arg1=FC  - example for umlat x'FC' (u with 2 dots above)
              arg2     - enclose char for A/N fields, numeric fields not enclosed
             ,arg2=20  - field enclose char (blank x'20' default for none)
             ,arg2=22  - alternate enclose-char " x'22' - see example page '4D1'
             ,uop=a0   - do NOT translate input data (a0/off is the default)
             ,uop=a1   - translate input from EBCDIC to ASCII (see page '4D2')

 #2b. uvcopy genpipe1,fili1=maps/citytax1,filo1=pfp1/citytax1
      =======================================================
      - only need to specify I/O files if default delimiter & no translate OK

 #2c. genpipeA citytax1     <-- easy script equivalent of above (shorter command)
      =================       - use this if defaults OK for arg1, arg2, uop
                              - combines cobmap1 & genpipe1

console log & options for genpipe1

 User OPtion (uop) defaults  = q1a0d1s1
  null to accept or re-specify (1 or more) -->      <-- null to accept defaults

 genpipe1:050418:114651: EOF fili01 19 rds, 0 wrts, 1315 size; maps/citytax1
 genpipe1:050418:114651: EOF filo01 0 rds, 46 wrts, 2125 size; pfp1/citytax1

EOJ, Output File written to: pfp1/citytax1 enter command: vi,cat,more,lp,uvlp12,null --> more <-- to display output code

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4B3. Converting Data Files to ASCII Delimited for loading SQL DBs

genpipeA sample output - '|' pipe delimited ASCII text

 # citytax1 - uvcopy job to convert EBCDIC/ASCII data to delimited text file
 opr='citytax1 - uvcopy code generated from copybook: citytax1 '
 rop=j200000r1  #Run OPtions: increase instrn storage & prompt display output
 was=a33000b33000c999000d33000e33000
 fili1=?d2asc/citytax1,rcs=00128,typ=RSTm10000
 filo1=?d4pipe/citytax1.dat,rcs=9000,typ=LSTtd3  #<-- DOS option d3 for CR/LF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
 # area a input, see 'get' instrn
 # area b translated to ASCII, in case mainframe EBCDIC file (with packed?)
 # area c data fields fixed 100 bytes apart in prep for var | delimit instrn
 # area d output, see 'put' instrn at end
        mvc    b0(00128),a0       move input area a to area b
 ###    tra    b0(00128)          uncomment for EBCDIC input (or option a1)
 #      ---                   <-- & insert instrns from uvdata51 if packed fields
 #      ---                   <-- insert R/T tests if redef records
        mvc    c0(10),b0(10)                        #1 folio
        mvc    c100(25),b10(25)                     #2 name
        mvc    c200(25),b35(25)                     #3 address
        mvc    c300(20),b60(20)                     #4 city
        mvc    c400(2),b80(2)                       #5 province
        mvc    c500(6),b82(6)                       #6 zip
        edta1  c600(9),b88(4p),'+zzzzzzzzz9'        #7 post-date
        edta1  c700(11),b92(5p),'+zzzzzzzzz9'       #8 land-value
        edta1  c800(11),b97(5p),'+zzzzzzzzz9'       #9 improv-value
        edta1  c900(11),b102(5p),'+zzzzzzzzz9'      #10 face-value
        edta1  c1000(11),b107(9),'+zzzzzzz.99'      #11 maint-tax
        mvc    c1100(6),b116(6)                     #12 purchase-date
        bal    dat6n,'1100'
        mvc    c1200(6),b122(6)                     #13 filler001
        var    d0(32000),c0(100),0013,'|'
        trt    d0(32000),$trtchr
        clr    c0(001300),' '
 put1   put    filo1,d0
        skp    loop
 #
 eof    cls    all
        eoj
 @pf2=genpipe1.sub

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4C1. Converting Data Files to ASCII Delimited for loading SQL DBs

Executing pipe delimit jobs - Allowing for manual change

In case manual changes are later required, I recommend you copy the jobs from subdir pfp1/... (generation) to subdir pfp2/... (for execution). This will protect any manual changes you might make in pfp2/... in case you later run mass re-generations to pfp1/...

Manual changes will be required for files with multiple record types/layouts in the same file. 'citytax1' has only 1 record type, so we will execute it now.

 #1.  cp pfp1/citytax1 pfp2   - copy auto-generated job to pfp2, before
      =====================     optional insert datafilenames vs copybooknames

 #2.  cp pfp2/citytax1 pfp3   - copy to pfp3
      =====================     before any manual changes &/or execution

 #2.  vi pfp3/citytax1        - make any manual changes required
      ================          (record type tests, see example page '5B3')

Executing jobs to pipe delimit data file

 #3a. uvcopy pfp3/citytax1,fili1=d2asc/citytax1,filo1=d4pipe/citytax1.dat
      ===================================================================

 #3b. uvcopy pfp3/citytax1   <-- easier (filenames default as shown above)
      ====================

sample input file ASCII (or EBCDIC) with packed

 #4. uvhd d2asc/citytax1 r128
     ========================

                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             3333333322444424445522222222222222233332445442544422222222225444
             1013014000AF8E085E29000000000000000181502F75E02F14000000000061E3
          64 OUVER           BCV9S1H1......W.........qq.000149061970530191809
             4554522222222222445353430888005880008000770333333333333333333333
             F56520000000000023693181029C0072C0047C0111C000149061970530191809

                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 10139923  GEORGE BROWN             1250 EAST PENDER STREET  VANC
             3333333322444544245454222222222222233332445525444452555445225444
             101399230075F275022F7E0000000000000125005134005E45203425540061E3
          64 OUVER           BCV5L1W1......W.........xRL00014626q980601190156
             4554522222222222445343530888005880008800754333333337333333333333
             F5652000000000002365C171029C0072C0047C0182C000146261980601190156

Only 1st 2 records of input file are shown above since we must use uvhd (vs vi/lp/uvlp12/etc) because the data contains packed fields & no linefeeds.

We will show all records of the output file on the next page, since the output data is now a text file with linefeeds & easily printed or displayed.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4C2. Converting Data Files to ASCII Delimited for loading SQL DBs

sample output - pipe delimited text

 #5. more d4pipe/citytax1.dat  <-- examine output (or vi/cat/lp/uvlp14/etc)
     ========================

 10130140|JOHN HENRY|1815 BOWEN ROAD|VANCOUVER|BC|V9S1H1|+82898|+57828|+4870|+171710|+1490.61|1997/05/30|191809|
 10139923|GEORGE BROWN|1250 EAST PENDER STREET|VANCOUVER|BC|V5L1W1|+82898|+57828|+4878|+178524|-1462.61|1998/06/01|190156|
 10147615|BONNIE SMITH|44430 YALE ROAD WEST|VANCOUVER|BC|V2P6J1|+121395|+39623|+0|+51914|+376.92|1995/06/01|201031|
 10149304|TONY TATTERSAL|3350 S.W. MARINE DRIVE|VANCOUVER|BC|V6N3Y9|+50594|+76800|+0|+208391|+1925.03|1993/06/01|201512|
 10154688|MEL RAHAL|1110 - 625 HOWE STREET|VANCOUVER|BC|V6C2T6|+82898|+44880|+0|+125038|+1154.79|1996/11/26|082351|
 10173320|SAM SMITH|2381 ROGERS AVE|VANCOUVER|BC|V3K5Y2|+50498|+60600|+0|+169086|+1562.00|1997/05/30|191810|
 10204700|JOHN JENSEN|12611-100 ST|VANCOUVER|AL|T8V4H2|+122396|+15720|+0|+43744|-404.40|1996/11/26|081702|
 10223240|BOB ANDERSON|7215 ARGYLL ROAD|VANCOUVER|AL|T6C4J2|+52899|+3597|+0|+9990|-90.99|1998/06/01|190157|
 10308685|TOM PERRY|3932 - 3A ST. N.W.|VANCOUVER|AL|T2E6R4|+70192|+68155|+0|+125773|+1065.10|1992/06/01|221650|
 10318833|HARRY SMIRNOFF|BOX 308, STN J|VANCOUVER|AL|T2A4X6|+11399|+54541|+42046|+93502|+761.46|1997/05/30|191811|

The output file is now an all text ASCII file and can be easily listed with the 'uvlp' scripts or other unix utilities. The input file requires a utility such as uvhd to display the packed fields.

 #6. uvlp14L d4pipe/citytax1.dat e1    <-- list 1st page (e1=end after 1 page)
     ==============================

 #6a. uvlp20L d4pipe/citytax1.dat e1   <-- use uvlp20L for long records
      ==============================     - prints 200 chars per line Landscape

      - - - OR for WindowsDOS, use 'uvlpr...' scripts - - -

 #6. uvlpr14L d4pipe/citytax1.dat e1   <-- list 1st page (e1=end after 1 page)
     ===============================

 #6a. uvlpr20L d4pipe/citytax1.dat e1  <-- use uvlp20L for long records
      ===============================    - prints 200 chars per line Landscape

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4D1. Converting Data Files to ASCII Delimited for loading SQL DBs

Alternate Delimiter &/or Enclose Character

arg1 may specify an alternate field separator (vs '|' delimiter). If used, arg1 must be specified as 2 digit hexadecimal value. For example: arg1=2C defines a comma ',' & arg1=FC defines an umlat (u with 2 dots above).

arg2 may specify a character to enclose alpha-numeric data fields. If used, arg2 must be specified as 2 digit hexadecimal value. For example: arg2=22 for '"' double quote, arg2=5E for '^' caret.

Here is the command to regenerate the delimit job specifying the separator as a comma x'2C' & the data enclose character as a double quote x'22'. Followed by the command to execute the job, followed by a display of 1st few records.

 #1. uvcopy genpipe1,fili1=maps/citytax1,filo1=pfp1/citytax1a,arg1=2C,arg2=22
     ========================================================================

 #2. cp pfp1/citytax1a pfp2      - copy to standard execution pfp2 subdir
     ======================

 #4. uvcopy pfp2/citytax1a,fili1=d2asc/citytax1,filo1=d4pipe/citytax1.dat
     ====================================================================

1st few records from alternate job

 "10130140  ","JOHN HENRY               ","1815 BOWEN ROAD          ","VANCOUVER           ","BC"
 ,"V9S1H1",+82898,+57828,+4870,+171710,+1490.61,0000/00/00,"191809",
 "10139923  ","GEORGE BROWN             ","1250 EAST PENDER STREET  ","VANCOUVER           ","BC"
 ,"V5L1W1",+82898,+57828,+4878,+178524,-1462.61,0000/00/00,"190156",
 "10147615  ","BONNIE SMITH             ","44430 YALE ROAD WEST     ","VANCOUVER           ","BC"
 ,"V2P6J1",+121395,+39623,+0,+51914,+376.92,0000/00/00,"201031",
 "10149304  ","TONY TATTERSAL           ","3350 S.W. MARINE DRIVE   ","VANCOUVER           ","BC"
 ,"V6N3Y9",+50594,+76800,+0,+208391,+1925.03,0000/00/00,"201512",
 "10154688  ","MEL RAHAL                ","1110 - 625 HOWE STREET   ","VANCOUVER           ","BC"
 ,"V6C2T6",+82898,+44880,+0,+125038,+1154.79,0000/00/00,"082351",
 "10173320  ","SAM SMITH                ","2381 ROGERS AVE          ","VANCOUVER           ","BC"
 ,"V3K5Y2",+50498,+60600,+0,+169086,+1562.00,0000/00/00,"191810",

I have split the records (after "BC") so you can see the numeric fields (at the end of the record) are not enclosed in the specified enclose character. COBOL type 'x' & unpacked unsigned type '9' are enclosed, while type '9' signed or packed or binary fields are not enclosed.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4D2. Converting Data Files to ASCII Delimited for loading SQL DBs

EBCDIC/ASCII convert is Pre-Requisite of Pipe Delimit

Conversion from EBCDIC to ASCII is normally a pre-requisite to pipe-delimiting. For large files, it may be worth combining the EBCDIC to ASCII instructions into the pipe delimit job.

genpipe1 option to use EBCDIC file as input

You might do this to save time & space for large files where you only needed the pipe delimited output. The genpipeA script generates a uvcopy job to read the ASCII file by default. If you want the generated job to read the original EBCDIC file you could run the genpipe1 job with the 'a1' option as shown below.

 #1. uvcopy genpipe1,fili1=maps/citytax1,filo1=pfp1/citytax1,uop=a1  <-- generate
     ==============================================================

Or if already generated, you can simply uncomment the following line:

 ###    tra    b0(00128)          uncomment for EBCDIC input (or option a1)

You also need to transfer the instructions (that preserve packed fields & correct zoned signs) from the EBCDIC to ASCII conversion job to the pipe delimit job at the point marked by:

 #    ---             <-- insert instrns from uvdata51 if packed fields

From our EBCDIC to ASCII job pfx2/citytax1 listed on page '3B2', these instructions are:

      mvc    b88(19),a88             pns taxduedate:landbldgtax
      trt    b107(9),$trtsea          ns f00107

 Insert these instructions into the pipe delimit job (listed on page '4B3')
 after the '#     ----' indicator line.

The execute command would then specify the EBCDIC file as input (vs the ASCII file on the execute command on the previous page).

 #2a. uvcopy pfp2/citytax1,fili1=d1ebc/citytax1,filo1=d4pipe/citytax1.dat
      ===========================*****===================================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4D3. Converting Data Files to ASCII Delimited for loading SQL DBs

option s0/s1 for machine architecture

Mainframe binary fields are BIG-end whereas INTEL binary fields are little-end.

The default option is 's1' which assumes you are generating & executing the conversion jobs on an INTEL machine which does little-end arithmetic.

If you are generating & executing on a unix/linux RISC machine (HP-UX, AIX, DEC Alpha, SUN sparc), you must specify option 's0'.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4E1. Converting Data Files to ASCII Delimited for loading SQL DBs

listpipe1 - list pipe delimited data with copybook field names

 #1a. uvcopy listpipe1,fili1=d4pipe/citytax1.dat,fili2=maps/citytax1
                      ,filo1=tmp/citytax1
      ==============================================================

 #1b. listpipeA citytax1    <-- script equivalent of above (shorter command)
      ==================      - ksh for unix/linux or .bat file for Win/Dos

listpipe1 OPTIONS screen

 uop=q1f1l1s3   - default options
       f1       - FormFeed after each record, f2 every 2nd rec, etc
         l1     - list data recs horizontally before vertically
           s3   - stop count default 1st 3 records in file
 User OPtion (uop) defaults  = q1f1l1s3
  null to accept or re-specify (1 or more) -->    <-- null reply lists 3 records
 report=tmp/citytax1, enter: vi/more/lp/etc
 --> more       <-- display output (shown below)

sample listpipe1 report

 listpipe1 - list '|' delimited files with copybook fieldnames
           - to verify files created by genpipe1 (see DATAcnv1.htm#Part_4)
 datafile=d4pipe/filename?  copybookmap=maps/filename?  datetime=2003/06/09_20:26:34
 SEQ# FIELDNAME              COBOL PICTURE      DATA
 datafile = d4pipe/citytax1.dat, record# = 1, field count = 13

 10130140|JOHN HENRY|1815 BOWEN ROAD|VANCOUVER|BC|V9S1H1|+82898|+57828|+4870|+171
 710|+1490.61|1997/05/30|191809|

 001 folio                   x(00010).          10130140
 002 name                    x(00025).          JOHN HENRY
 003 address                 x(00025).          1815 BOWEN ROAD
 004 city                    x(00020).          VANCOUVER
 005 province                x(00002).          BC
 006 zip                     x(00006).          V9S1H1
 007 post-date               s9(00007) comp-3.  +82898
 008 land-value              s9(00009) comp-3.  +57828
 009 improv-value            s9(00009) comp-3.  +4870
 010 face-value              s9(00009) comp-3.  +171710
 011 maint-tax               s9(00007)v99.      +1490.61
 012 purchase-date           x(00006).          1997/05/30
 013 filler001               x(00006).          191809

1. Only 1 record is shown here, the default is to list the 1st 3 records, which is controlled by the stop option 's3' above. (reply 's5' for 1st 5, etc).
   
   
2. Note that the pipe delimited text record is first listed 'as is' & then listed field by field with COBOL copybook names.
   
   
3. The pipe delimited text records could be very long if large records with many fields.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4F1. Converting Data Files to ASCII Delimited for loading SQL DBs

input - copybook/cobmap

 cobmap1  start-end bytes for cobol record fields    200306092029  pg# 0001
 cpys/citytax1                  citytax1           RCSZ=00128  bgn  end  lth typ
 * citytax1 copybook, from signscan1 + edit meaningful names
  01  citytax1.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-3.          0088 0091  004pns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  x(00006).                 0116 0121  006
      10 filler001             pic  x(00006).                 0122 0127  006
 *RCSZ=00128                                                       0128

generate batch-file & control-file to create & load table

 #1. uvcopy cobmap1,fili1=cpys/citytax1,filo1=maps/citytax1
     ======================================================
     - convert copybook to 'cobmap' (record layout)

 #2. uvcopy sqlcreate1,fili1=maps/citytax1,filo1=sqls/create_citytax1.sql
     ====================================================================
                                          ,filo2=sqls/load_citytax1.ctl
     ====================================================================
     - convert cobmap to batch-file & control-file to create & load table

      - - - OR use script 'sqlcreateA' (combines above into 1 short command)

 #12. sqlcreateA citytax1    <-- script equivalent of above (shorter command)
      ===================      - ksh for unix/linux or .bat file for Win/Dos

 #3a. more sqls/create_citytax1.sql   <-- display batch-file output
      =============================

 #3b. more sqls/load_citytax1.ctl     <-- display control-file output
      ===========================

See the output files listed on the next page --->

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4F2. Oracle Database SQL test/demo

Oracle SQL demo - CREATE & LOAD table

 #1. Login demo1 --> /home/demo1
     ===========

 #2. sqlplus demo1/demo1pw     <-- start SQL*Plus
     =====================

 #3. SQL> create table citytax1 (folio char (10), name char (25),
                 address char (25), city char (20), province char (02),
                 zip char (06), post_date date, land_value number (09),
                 improv_value number (09), face_value number (09),
                 maint_tax number (09,2), purchase_date date);

 #4. SQL> describe citytax1
          =================

  Name					   Null?    Type
  ----------------------------------------- -------- ----------------------------
  FOLIO						    CHAR(10)
  NAME						    CHAR(25)
  ADDRESS					    CHAR(25)
  CITY						    CHAR(20)
  PROVINCE					    CHAR(2)
  ZIP						    CHAR(6)
  POST_DATE					    DATE
  LAND_VALUE					    NUMBER(9)
  IMPROV_VALUE					    NUMBER(9)
  FACE_VALUE					    NUMBER(9)
  MAINT_TAX					    NUMBER(9,2)
  PURCHASE_DATE					    DATE

 #5. SQL> exit    <-- exit to unix shell to run SQLLDR
          ====

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4F3. Oracle Database SQL test/demo

Oracle 'batch-file' to create table

Creating tables manually (as shown above) would be very laborious for files with many fields. UV Software provides uvcopy job 'sqlcreate1' to automatically generate an Oracle batch-file to create the table. The job also generates the SQL*LOADER control file to load the table. See 'sqlcreate1' fully documented in DATAcnv1.htm#Part_4.

Oracle batch-file to create table

 /* create_citytax1.sql - SQL batch file to create a table */
 /*                     - generated from COBOL copybook 20081223:125830*/
 /* see www.uvsoftware.ca/datacnv1.htm#Part_4 & sqldemo.htm#Part_4    */
 /*                                                                   */
 /* sqlplus user/pass @sqls/create_citytax1.sql                       */
 /* ===========================================                       */
 /*                                                                   */
 /*Note - changed purchase_date length from 6 to 10             */
 /*     - pipe delimit job inserted century & edited ccyy/mm/dd */
 /*                                                             */
 DROP TABLE citytax1;
 CREATE TABLE citytax1 (
 folio                         char     (0010) , --#001 x(00010).
 name                          char     (0025) , --#002 x(00025).
 address                       char     (0025) , --#003 x(00025).
 city                          char     (0020) , --#004 x(00020).
 province                      char     (0002) , --#005 x(00002).
 zip                           char     (0006) , --#006 x(00006).
 post_date                     number   (07)   , --#007 s9(00007) comp-3.
 land_value                    number   (09)   , --#008 s9(00009) comp-3.
 improv_value                  number   (09)   , --#009 s9(00009) comp-3.
 face_value                    number   (09)   , --#010 s9(00009) comp-3.
 maint_tax                     number   (09,2) , --#011 s9(00007)v99.
 purchase_date                 char     (0010)   --#012 x(00006).
 );
 exit;

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4F4. Oracle Database SQL test/demo

Oracle SQL control file to LOAD table

 -- load_citytax1.ctl - SQL*LOADER control file to load Oracle table
 --                   - generated from COBOL copybook 20081221:080752
 -- see www.uvsoftware.ca/datacnv1.htm#Part_4 & sqldemo.htm#Part_4
 --
 -- sqlldr user/pass control=sqls/load_citytax1.ctl
 -- ===============================================
 --
 load data infile 'd4pipe/citytax1.dat' into table citytax1
 fields terminated by '|' optionally enclosed by '"'
 (
 folio                         , --char     (0010) , --#001 x(00010).
 name                          , --char     (0025) , --#002 x(00025).
 address                       , --char     (0025) , --#003 x(00025).
 city                          , --char     (0020) , --#004 x(00020).
 province                      , --char     (0002) , --#005 x(00002).
 zip                           , --char     (0006) , --#006 x(00006).
 post_date                     , --datetime        , --#007 s9(00007) comp-3.
 land_value                    , --number   (09)   , --#008 s9(00009) comp-3.
 improv_value                  , --number   (09)   , --#009 s9(00009) comp-3.
 face_value                    , --number   (09)   , --#010 s9(00009) comp-3.
 maint_tax                     , --number   (09,2) , --#011 s9(00007)v99.
 purchase_date                   --datetime          --#012 x(00006).
 )

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4F5. Oracle Database SQL test/demo

create & load table using batch-file & control-file

 #1. Login demo1 --> /home/demo1
     ===========

 #2. sqlplus user/pass @sqls/create_citytax1.sql
     ===========================================
     - run SQL 'batch-file' to create the table

 #3. sqlldr user/pass control=sqls/load_citytax1.ctl
     ===============================================
     - run SQL*LOADER control file to load the table

use SQL*Plus to verify table loaded OK

 #4. sqlplus demo1/demo1pw       <-- start SQL*Plus
     =====================

 #5. SQL> select * from citytax1 <-- SELECT all rows/fields from table
          ======================   - headings & 1st 2 rows shown below:

 FOLIO	   NAME 		     ADDRESS
 CITY		     PR ZIP	POST_DATE LAND_VALUE IMPROV_VALUE FACE_VALUE
  MAINT_TAX PURCHASE_D
 ============================================================================

 10130140   JOHN HENRY		     1815 BOWEN ROAD
 VANCOUVER	     BC V9S1H1	    82898      57828	     4870     171710
    1490.61 1997/05/30

 10139923   GEORGE BROWN 	     1250 EAST PENDER STREET
 VANCOUVER	     BC V5L1W1	    82898      57828	     4878     178524
   -1462.61 1998/06/01

batch-file to list table rows

 /* select_all_citytax1.sql - batch file to select all rows from citytax1  */
 /*                      - batch files useful for complex repeated selects */
 select * from citytax1;
 exit;

You could use the batch-file listed above as an alternative to logging in to SQL*PLUS to list the table rows:

 #6. SQL> exit      <-- exit from SQL*PLUS (back to Unix/Linux prompt)

 #7. sqlplus user/pass @sqls/select_all_citytax1.sql
     ===============================================
     - run SQL 'batch-file' to create the table

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4G1. Converting Mainframe Data Files from EBCDIC to ASCII

gencnvD1/gencnvD2 - Generate All Data Conversion Jobs

 'gencnvD1' - script to run all conversion GENERATION jobs.
            - signscan1, sign2cpy1, cobmap1, uvdata51, genpipe1, sqlcreate1

 'gencnvD2' - script to create GENERATION jobs required when copybooks present.
            - signscan1, sign2cpy1, cobmap1, uvdata51, genpipe1, sqlcreate1

I suggest you convert your 1st file using the individual scripts or uvcopy jobs (as described earlier in this documentation), to gain a good understanding of these procedures. Running them 1 by 1 allows you to examine the various outputs & check for any errors.

Use 'gencnvD1' when no COBOL copybook is available & you need to scan the datafile to create the copybook. Use alternate script 'gencnvD2' when the COBOL copybook is available (skips signscan1 & sign2cpy1 jobs).

The input EBCDIC datafile must be stored in the d1ebc subdir & the outputs will be named the same as the input datafile & will be stored in separate subdirs (signs, cpys, maps, pfx1,pfx2,pfx3 pfp1,pfp2,pfp3,& sqls).

related documentation

 SQLconvert.htm - convert flat files to pipe delimited for Microsoft SQL server
                  - driven by COBOL copybooks
                  - also generate scripts to create & load tables
                  - generates jobs for all copybooks & all data files
                    (vs this DATAcnv1.doc for 1 copybook/file at a time)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4H1. Converting Mainframe Data Files from EBCDIC to ASCII

gencnvD2 - Generate conversion jobs from CopyBooks

Script 'gencnvD2' runs conversion GENERATION jobs based on CopyBooks. (cobmap1, uvdata51, genpipe1, sqlcreate1). Use script 'gencnvD2' when the COBOL copybook is available (skips signscan1 & sign2cpy1 jobs).

gencnvD2 - Operating Instructions

 #1. gencnvD2 file-name               <-- command format
     ==================

 #1a. gencnvD2 citytax1                <-- example for test/demo file
      =================

copy to alt subdir before changes/execution

Regardless of whether changes are required or not, I suggest you always copy the jobs from pfx1 to pfx3 & from pfp1 to pfp3, before executing. Often you wont realize you need modifications until after you inspect the output data.

 #2a. cp pfx1/citytax1 pfx3   - copy auto-generated EBCDIC to ASCII job
      =====================     before any manual changes &/or execution
 #2b. cp pfp1/citytax1 pfp3   - copy auto-generated pipe delimit job
      =====================     before any manual changes &/or execution

 #3a. vi pfx3/citytax1        - make any manual changes required
      ================          (for record type tests, see page '5D2')
 #3b. vi pfp3/citytax1        - make any manual changes required
      ================

executing jobs generated by gencnvD2

 #4. uvcopy pfx3/citytax1 <-- converts d1ebc/citytax1 to d2asc/citytax1
     ====================     (EBCDIC to ASCII, preserving packed, fixing signs)

 #5. uvcopy pfp3/citytax1 <-- converts d2asc/citytax1 to d4pipe/citytax1.dat
     ====================     (converts ASCII file to delimited text file)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4H2. Converting Mainframe Data Files from EBCDIC to ASCII

checking gencnvD2 outputs (Unix/Linux)

 #1. vi cpys/citytax1    - copybook generated by sign2cpy1
                         - better to check maps with calculated
                           field start/end/length on right side

 #2. vi maps/citytax1    - cobmap generated by cobmap1

 #3. vi pfx1/citytax1    - EBCDIC to ASCII conversion job

 #4. vi pfp1/citytax1    - job to convert ASCII file to text delimited

 #5. vi sqls/create_citytax1.sql - batch file to create table

 #6. vi sqls/load_citytax1.ctl   - control file to load table

checking gencnvD2 outputs (Windows)

 #1. edit cpys\citytax1    - copybook generated by sign2cpy1
                           - better to check maps with calculated
                             field start/end/length on right side

 #2. edit maps\citytax1    - cobmap generated by cobmap1

 #3. edit pfx1\citytax1    - EBCDIC to ASCII conversion job

 #4. edit sqls\create_citytax1.sql - batch file to create table

 #5. edit sqls\load_citytax1.ctl   - control file to load table

checking data files

 #1. uvhd d1ebc/citytax r128a  - original EBCDIC input

 #2. uvhd d2asc/citytax r128   - ASCII output from pfx3/citytax1

 #3a. vi d4pipe/citytax1.dat   - delimited output from pfp3/citytax1.dat

 #3b. edit d4pipe\citytax1.dat - delimited output (for Windows vs Unix)

 #3c. listpipeA citytax1       - list 1st 3 records of delimited output with
                                 copybook fieldnames beside data contents

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4H3. Converting Mainframe Data Files from EBCDIC to ASCII

modifying script gencnvD2

Please see the gencnvD2 script listed on page '9E1' for the Unix/Linux Korn shell & page '9E2' for the Windows batch script. You can modify the various options as appropriate for your site. For example if you wanted to use a semicolon as the delimiter, you would add ',arg1=\;' to line 38 as below:

 uvcopy genpipe1,fili1=maps/$f,filo1=pfp1/$f,uop=q0,rop=r0,arg1=\;
 =================================================================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4I1. Converting Mainframe Data Files from EBCDIC to ASCII

genrfmA Operating Instructions - unpack packed fields

'genrfmA' reformats data records based on old & new versions of the copybook. We already have the copybook for the original data file with packed fields. We can easily create the new record layout simply by copying the original and removing all the 'comp-3's. Our copybook filename (citytax1) can remain the same, since we will copy to a different subdir (from cpys to cpyu).

 #1. cp cpys/citytax1 cpyu/           <-- copy original copybook for unpacking
     ======================

 #2. vi cpyu/citytax1                 <-- use 'vi' to remove all 'comp-3's
     ================

 #3. uvcopy cobmap1,fili1=cpyu/citytax1,filo1=mapu/citytax1 <-- generate cobmap
     ======================================================

 #4. uvlp12 mapu/citytax1            <-- list cobmap to verify results desired
     ====================                (uvlpr12 for Windows/DOS batch script)

cobmap with all 'comp-3's removed

 cobmap1  start-end bytes for cobol record fields    200306092029  pg# 0001
 cpyu/citytax1                  citytax1           RCSZ=00143  bgn  end  lth typ
 * citytax1 copybook, from signscan1 + edit meaningful names
  01  citytax1.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) .                0088 0094  007 ns 7
      10 land-value            pic s9(00009) .                0095 0103  009 ns 9
      10 improv-value          pic s9(00009) .                0104 0112  009 ns 9
      10 face-value            pic s9(00009) .                0113 0121  009 ns 9
      10 maint-tax             pic s9(00007)v99.              0122 0130  009 ns 9
      10 purchase-date         pic  x(00006).                 0131 0136  006
      10 filler001             pic  x(00006).                 0137 0142  006
 *RCSZ=00143                                                       0143

Note that the record size has increased from 128 to 143, due to unpacking. Since there was one 4 byte field (which is now 7) & three 5 byte fields (which become 9), the output is (1*3 + 3*4) = 3+12 = 15 bytes larger. The new record is 128 + 15 = 143 bytes, confirmed by the cobmap above. The conversion job option t1 (default) will add 1 more for a terminating LF.

 #5. genrfmA citytax1       <-- generate uvcopy job to unpack packed fields
     ================         - ksh for unix/linux or .bat file for Win/Dos

Please see the genrfmA output listed on the next page --->

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4I2. Converting Mainframe Data Files from EBCDIC to ASCII

 #5. genrfmA citytax1        <-- generate uvcopy job to unpack packed fields
     ================          - ksh for unix/linux or .bat file for Win/Dos

genrfmA output - uvcopy job to unpack packed fields

 opr='JOBNAME citytax1 - generated by cobmap1,reform2,uvdata52'
 uop=q1d2m2t1z0g7n4p0s0w50x1y0q1t1    <-- options from reform2 gen
 was=a9000b9000
 fili1=?d2asc/FILEIN,rcs=00128,typ=RSF
 filo1=?d3unpk/FILEOUT,rcs=00144,typ=RST
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        clr    b0(00145),' '             init output area to all blanks
 #      ---                          <-- insert R/T tests if redef records
        mvc    b0(88),a0(88)                 folio : zip
        bal    rfmpn,'a00088b07c07d00088e04f07','post-date'
        bal    rfmpn,'a00095b09c09d00092e05f09','land-value'
        bal    rfmpn,'a00104b09c09d00097e05f09','improv-value'
        bal    rfmpn,'a00113b09c09d00102e05f09','face-value'
        mvc    b122(21),a107(21)             maint-tax : filler001
 #
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj
 @pf2=reform2.sub

Since this DATAcnv1.doc is intended to be the easiest to use document for file conversions, I wont go into detail here about the generated code. If you wish more explanation & discussion about possible modifications for complex files, please see REFORMjobs.htm.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4I3. Converting Mainframe Data Files from EBCDIC to ASCII

Manual Changes Required ? (to reformat jobs)

Manual changes may be required to the auto-generated uvcopy job (usually to allow for multiple record types/layouts in the same file).

Regardless of whether changes are required or not, I suggest you always copy the jobs from pfr1 to pfr2, before executing (from pfr2). Often you wont realize you need modifications until after you inspect the output data.

 #6.  cp pfr1/citytax1 pfr2   - copy auto-generated job to pfr2
      =====================     before any manual changes &/or execution

 #7.  vi pfr2/citytax1        - make any manual changes required
      ================          (for record type tests, see page '5D2')

execute generated job to unpack packed fields

 #8a. uvcopy pfr2/citytax1,fili1=d2asc/citytax1,filo1=d3unpk/citytax1
      ===============================================================

 #8b. uvcopy pfr2/citytax1  <-- easier (filenames default as shown above)
      ====================

Please see next page for sample data I/O listed with uvhd --->

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4I4. Converting Mainframe Data Files from EBCDIC to ASCII

sample input file ASCII (or EBCDIC)

 #9. uvhd d2asc/citytax1 r128s2
     ==========================

                       1         2         3         4         5         6
 r#        1 0123456789012345678901234567890123456789012345678901234567890123

           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             3333333322444424445522222222222222233332445442544422222222225444
             1013014000AF8E085E29000000000000000181502F75E02F14000000000061E3

          64 OUVER           BCV9S1H1......W.........qq.000149061970530191809
             4554522222222222445353430888005880008000770333333333333333333333
             F56520000000000023693181029C0072C0047C0111C000149061970530191809

sample output - unpacked

 #10. uvhd d3unpk/citytax1 r144s2
      ===========================

                       1         2         3         4         5         6
 r#        1 0123456789012345678901234567890123456789012345678901234567890123

           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             3333333322444424445522222222222222233332445442544422222222225444
             1013014000AF8E085E29000000000000000181502F75E02F14000000000061E3

          64 OUVER           BCV9S1H10082898000057828000004870000171710000149
             4554522222222222445353433333333333333333333333333333333333333333
             F565200000000000236931810082898000057828000004870000171710000149

         128 061970530191809..
             3333333333333330
             061970530191809A

Note that the record size has increased from 128 to 143, due to unpacking. Since there was one 4 byte field (which is now 7) & three 5 byte fields (which become 9), the output is (1*3 + 3*4) = 3+12 = 15 bytes larger. The conversion job option t2 (default) added 2 more for terminating CR+LF. Option t1 would add just 1 more for LF alone (unix vs dos/windows). Option t0 would mean no record termination.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_5 Special Situations Requiring Manual Changes

Part5 Special Situations Requiring Manual Changes

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5A1. DATAcnv1 - Converting Multiple Record Type Files

citytax2 - copybook with redefined records

 cobmap1  start-end bytes for cobol record fields    200403021116  pg# 0001
 cpys/citytax2                  citytax2           RCSZ=00128  bgn  end  lth typ
 * citytax2 copybook for redefined multi record type file
 * tax record (type 'T')
  01  citytax2.
      10 folio                 pic  x(00008).                 0000 0007  008
      10 rectype               pic  x.                        0008 0008  001
      10 filler001             pic  x.                        0009 0009  001
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-3.          0088 0091  004pns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  x(00006).                 0116 0121  006
      10 filler002             pic  x(00006).                 0122 0127  006
 *
 * monthly payment record (type 'P')
 *RCSZ=00128                                                       0128
  01 mthpayrec redefines citytax2.
      10 folio                 pic  x(00008).                 0000 0007  008
      10 rectype               pic  x.                        0008 0008  001
      10 filler003             pic  x(7).                     0009 0015  007
      10 tax-year              pic  9(4).                     0016 0019  004 n  4
      10 mthpayments           pic  s9(9) comp-3 occurs 12.   0020 0024  005pns 9
      10 filler004             pic  x(48).                    0080 0127  048
 *
 * file header record (type 'H')
 *RCSZ=00128                                                       0128
  01 headerrec redefines citytax2.
      10 folio                 pic  x(00008).                 0000 0007  008
      10 rectype               pic  x.                        0008 0008  001
      10 filler005             pic  x.                        0009 0009  001
      10 year-end              pic  x(30).                    0010 0039  030
      10 filler006             pic  x(88).                    0040 0127  088
 ************** end of citytax2 copybook ********************
 *RCSZ=00128                                                       0128

 uvcopy cobmap1,fili1=cpys/citytax2,filo1=maps/citytax2
 ======================================================

 cobmapA citytax2   <-- script to simplify running cobmap1 (above)
 ================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5A2. DATAcnv1 - Converting Multiple Record Type Files

test data file to demo Multi-Record-Type files

 uvhd d1ebc/citytax2 r128a    <-- 'a' option to translate char lines to ASCII
 =========================

                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123

           0         H December 31, 1998
             44444444C4C88898894FF64FFFF4444444444444444444444444444444444444
             000000008045354259031B019980000000000000000000000000000000000000

          64
             4444444444444444444444444444444444444444444444444444444444444444
             0000000000000000000000000000000000000000000000000000000000000000

                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123

         128 10130140T JOHN HENRY               1815 BOWEN ROAD          VANC
             FFFFFFFFE4DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
             1013014030168508559800000000000000018150266550961400000000005153

          64 OUVER           BCV9S1H1.bi...Wb....g...qq.00014906A970530191809
             DEECD44444444444CCEFEFCF0888005880008000770FFFFFFFFCFFFFFFFFFFFF
             645590000000000023592181029C0072C0047C0111C000149061970530191809

                      10        20        30        40        50        60
 r#        3 0123456789012345678901234567890123456789012345678901234567890123

         256 10130140P       1999...A....A....A....A....A....A....A....A....A
             FFFFFFFFD4444444FFFF00042000420004200042000420004200042000420004
             101301407000000019990001C0001C0001C0001C0001C0001C0001C0001C0001

          64 ....A....A....A.
             2000420004200042444444444444444444444444444444444444444444444444
             C0001C0001C0001C000000000000000000000000000000000000000000000000

Please relate the 3 record types here (H,T,P in byte 8 above) to the redefined records in the copybook listed on the previous page.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5B1. DATAcnv1 - Converting Multiple Record Type Files

Generate uvcopy job to convert MRT files

 #1a. uvcopy uvdata51,fili1=maps/citytax2,filo1=pfx1/citytax2  <-- uvcopy command
      ======================================================

 #1b. gencnvA citytax2     <-- script equivalent of above (shorter command)
      ================       - ksh for unix/linux or .bat file for Win/Dos

pfx1/citytax2 - uvcopy job for multi record types

 opr='JOBNAME citytax2 - genby: cobmap1,uvdata51,uvdata52'
 uop=q0
 was=a33000b33000
 fili1=?d1ebc/citytax2,rcs=00128,typ=RSF
 filo1=?d2asc/citytax2,rcs=00128,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
        mvc    b88(19),a88             pns post-date:face-value
        trt    b107(9),$trtsea          ns maint-tax
 #      ---                                              redef, R/T test?
        mvc    b20(60),a20             pns mthpayments
 #      ---                                              redef, R/T test?
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

uvcopy jobs generated from copybooks with redefined records require manual coding to test the record type & skip to the proper set of conversion instructions.

Please see this job re-listed on the next page, AFTER record type test coding has been inserted --->

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5B2. DATAcnv1 - Converting Multiple Record Type Files

Inserting code to test Record-Types

Before modifying, copy the auto generated job to an alternate subdir, for protection in case the generation job gets rerun in future.

 cp pfx1/citytax2 pfx3    <-- copy auto-generated job to alternate subdir
 =====================

 vi pfx3/citytax2         <-- modify the job as shown below:
 ================

uvcopy job AFTER manual changes

 opr='JOBNAME citytax2 - genby: cobmap1,uvdata51,uvdata52'
 uop=q0
 was=a33000b33000
 fili1=?d1ebc/citytax2,rcs=00128,typ=RSF
 filo1=?d2asc/citytax2,rcs=00128,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #------------------------------------------------------------------------
 # test Record Type & skip to appropriate instrns to preserve packed & signed
        cmc    b8(1),'H'           Header rec (no packed or signed)
        skp=   put1
        cmc    b8(1),'T'           Tax rec ?
        skp=   tax1
        cmc    b8(1),'P'           Payment rec ?
        skp=   pay1
        msg    b0(80)              show record with invalid type
        msgw   'Error Rec Type invalid - enter to bypass'
        skp    loop
 #-------------------------------------------------------------------------
 tax1   mvc    b88(19),a88             pns post-date:face-value
        trt    b107(9),$trtsea          ns maint-tax
        skp    put1                 skip to output
 #
 pay1   mvc    b20(60),a20             pns mthpayments
 #      ---
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

Please compare this to the previous page (BEFORE changes for multi R/T's).

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5B3. DATAcnv1 - Converting Multiple Record Type Files

execute uvcopy job to convert Multi-Record-Type file

 #2a. uvcopy pfx3/citytax2,fili1=d1ebc/citytax2,filo1=d2asc/citytax2
      ==============================================================

 #2b. uvcopy pfx3/citytax2   <-- easier (filenames default as shown above)
      ====================

sample output data file (ASCII)

 uvhd d2asc/citytax2 r128
 ========================

                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123

           0         H December 31, 1998
             2222222242466666672332233332222222222222222222222222222222222222
             00000000804535D252031C019980000000000000000000000000000000000000

          64
             2222222222222222222222222222222222222222222222222222222222222222
             0000000000000000000000000000000000000000000000000000000000000000

                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123

         128 10130140T JOHN HENRY               1815 BOWEN ROAD          VANC
             3333333352444424445522222222222222233332445442544422222222225444
             1013014040AF8E085E29000000000000000181502F75E02F14000000000061E3

          64 OUVER           BCV9S1H1......W.........qq.000149061970530191809
             4554522222222222445353430888005880008000770333333333333333333333
             F56520000000000023693181029C0072C0047C0111C000149061970530191809

                      10        20        30        40        50        60
 r#        3 0123456789012345678901234567890123456789012345678901234567890123

         256 10130140P       1999...A,...A,...A,...A,...A,...A,...A,...A,...A
             3333333352222222333300042000420004200042000420004200042000420004
             101301400000000019990001C0001C0001C0001C0001C0001C0001C0001C0001

          64 ,...A,...A,...A,
             2000420004200042222222222222222222222222222222222222222222222222
             C0001C0001C0001C000000000000000000000000000000000000000000000000

Please relate these 3 ASCII data records to the original EBCDIC records shown on page '5A2'. Note that option 'a' was used on page '5A2' to translate the character line to ASCII, but not used here since this is ASCII. You should also see that the zones & digits are EBCDIC on page '5A2', but are ASCII here. For example EBCDIC blanks are x'40', but ASCII blanks are x'20'.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5C1. DATAcnv1 - Converting Multiple Record Type Files

Converting MRT files & Splitting to Separate files

Splitting different record types to separate files is a minor change to the uvcopy conversion job listed on page '5B2', since that already includes the code to test the record types. First we will copy & rename in order not to destroy our job that converts record types into 1 output file. We will append an 's' to the original jobname to signify 'split'.

 cp pfx3/citytax2 pfx3/citytax2s   <-- copy/rename previous job
 ===============================
 vi pfx3/citytax2s                 <-- modify the job as shown below:
 =================

uvcopy job AFTER changes to write separate files

 opr='citytax2s - genby: cobmap1,uvdata51 convert EBCDIC to ASCII'
 opr=' - preserving packed fields & correcting signs of zoned fields'
 opr=' - record type tests added for redefined records'
 opr=' - alternate version to split different rec types to different files'
 uop=q0
 was=a33000b33000
 fili1=?d1ebc/citytax2,rcs=00128,typ=RSF    #input EBCDIC 3 rec types
 filo1=?d2asc/citytax2a,rcs=00128,typ=RSF   #output #1 tax recs
 filo2=?d2asc/citytax2b,rcs=00128,typ=RSF   #output #2 payment recs       <--
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #------------------------------------------------------------------------
 # test Record Type & skip to appropriate instrns to preserve packed&signed
        cmc    b8(1),'H'           Header rec (no packed or signed)
        skp=   loop                drop Header rec                          <--
        cmc    b8(1),'T'           Tax rec ?
        skp=   tax1
        cmc    b8(1),'P'           Payment rec ?
        skp=   pay1
        msg    b0(80)              show record with invalid type
        msgw   'Error Rec Type invalid - enter to bypass'
        skp    loop
 #-----------------------------------------------------------------
 tax1   mvc    b88(19),a88             pns post-date:face-value
        trt    b107(9),$trtsea          ns maint-tax
        put    filo1,b0                write tax rec to file#1
        skp    loop                    return to get next
 pay1   mvc    b20(60),a20             pns mthpayments
        put    filo2,b0                write payment rec to file#2          <--
        skp    loop                    return to get next
 eof    cls    all
        eoj

Please compare this to page '5B2' (before changes to write separate files). We have marked the few changes required with '<--' on the right side above.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5C2. DATAcnv1 - Converting Multiple Record Type Files

Executing file split job

 uvcopy pfx3/citytax2s   <-- convert & split MRTs to separate files
 =====================

Tax records separated to d2asc/citytax2a

 uvhd d2asc/citytax2a r128    <-- display Tax records (1st 2 of 10)
 =========================

                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140T JOHN HENRY               1815 BOWEN ROAD          VANC
             3333333352444424445522222222222222233332445442544422222222225444
             1013014040AF8E085E29000000000000000181502F75E02F14000000000061E3
          64 OUVER           BCV9S1H1......W.........qq.000149061970530191809
             4554522222222222445353430888005880008000770333333333333333333333
             F56520000000000023693181029C0072C0047C0111C000149061970530191809

                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 10139923T GEORGE BROWN             1250 EAST PENDER STREET  VANC
             3333333352444544245454222222222222233332445525444452555445225444
             101399234075F275022F7E0000000000000125005134005E45203425540061E3
          64 OUVER           BCV5L1W1......W.........xRL00014626q980601190156
             4554522222222222445343530888005880008800754333333337333333333333
             F5652000000000002365C171029C0072C0047C0182C000146261980601190156

Payment records separated to d2asc/citytax2a

 uvhd d2asc/citytax2b r128    <-- display Payment records (1st 2 of 10)
 =========================

                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140P       1999...A,...A,...A,...A,...A,...A,...A,...A,...A
             3333333352222222333300042000420004200042000420004200042000420004
             101301400000000019990001C0001C0001C0001C0001C0001C0001C0001C0001
          64 ,...A,...A,...A,
             2000420004200042222222222222222222222222222222222222222222222222
             C0001C0001C0001C000000000000000000000000000000000000000000000000

                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 10139923P       1999...B....B....B....B....B....B....B....B....B
             3333333352222222333300048000480004800048000480004800048000480004
             101399230000000019990002C0002C0002C0002C0002C0002C0002C0002C0002
          64 ....B....B....B.
             8000480004800048222222222222222222222222222222222222222222222222
             C0002C0002C0002C000000000000000000000000000000000000000000000000

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5D1. DATAcnv1 - Converting Multiple Record Type Files

Converting Separated Record-Types to Pipe Delimited for SQL DBs

In the section above, we have split the data records to separate files, and we will now split the copybook redefined records to separate copybooks, so we can use them to process the separate data files.

 vi cpys/citytax2     <-- split redefined records to separate copybooks
 ================

 #1. write Tax record as cpys/citytax2a
 #2. write Payment record as cpys/citytax2b
 #3. ignore the Header/Date record (not required)

We will not illustrate the Tax record cpys/citytax2a, since it is the same as cpys/citytax1, already illustrated previously.

We will illustrate the Payment record, which requires special consideration since it contains recurring data defined by OCCURS (pic s9(9) occurs 12). SQL databases do not allow for 'occurs', so we must define these fields individually. The easiest way to do this is to modify the copybook & then generate the pipe delimit job.

maps/citytax2b AFTER replacing OCCURS with separate fields

 cobmap1  start-end bytes for cobol record fields    200306090839  pg# 0001
 cpys/citytax2b                 mthpayrec          RCSZ=00128  bgn  end  lth typ
 * citytax2b copybook
 * monthly payment record (type 'P')
 * - isolated out of citytax2 (multi rec type)
 *   to convert mthpay recs to pipe delimited
 * - occurs 12 manually split to 12 separate fields
 *   since occurs not allowed in sql data bases
  01 mthpayrec.
      10 folio                 pic  x(00008).                 0000 0007  008
      10 rectype               pic  x.                        0008 0008  001
      10 filler001             pic  x(7).                     0009 0015  007
      10 tax-year              pic  9(4).                     0016 0019  004 n  4
 *    10 mthpayments           pic  s9(9) comp-3 occurs 12.
 * original split to sep fields sql does not allow occurs
      10 mthpayments-01        pic  s9(9) comp-3.             0020 0024  005pns 9
      10 mthpayments-02        pic  s9(9) comp-3.             0025 0029  005pns 9
      10 mthpayments-03        pic  s9(9) comp-3.             0030 0034  005pns 9
      10 mthpayments-04        pic  s9(9) comp-3.             0035 0039  005pns 9
      10 mthpayments-05        pic  s9(9) comp-3.             0040 0044  005pns 9
      10 mthpayments-06        pic  s9(9) comp-3.             0045 0049  005pns 9
      10 mthpayments-07        pic  s9(9) comp-3.             0050 0054  005pns 9
      10 mthpayments-08        pic  s9(9) comp-3.             0055 0059  005pns 9
      10 mthpayments-09        pic  s9(9) comp-3.             0060 0064  005pns 9
      10 mthpayments-10        pic  s9(9) comp-3.             0065 0069  005pns 9
      10 mthpayments-11        pic  s9(9) comp-3.             0070 0074  005pns 9
      10 mthpayments-12        pic  s9(9) comp-3.             0075 0079  005pns 9
      10 filler002             pic  x(48).                    0080 0127  048
 ************** end of citytax2b copybook *******************
 *RCSZ=00128                                                       0128

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5D2. DATAcnv1 - Converting Multiple Record Type Files

Generate Pipe Delimit job for Payment file

 #1a. uvcopy genpipe1,fili1=maps/citytax2b,filo1=pfp1/citytax2b,uop=a0 <-hard way
      ================================================================

 #1b. genpipeA citytax2b   <-- easy script equivalent of above (shorter command)
      ==================     - no-ext(ksh) for unix/linux or .bat for Win/Dos

pfp1/citytax2b - job to pipe delimit payment file

 opr='JOBNAME citytax2b - generated by cobmap1,genpipe1,uvdata52'
 # - uvcopy job to convert & fix data fields 100 bytes apart
 #   for compress & '|' delimit (see 'var' instrn below)
 rop=j200000r1  #Run OPtions: increase instrn storage & prompt display output
 was=a33000b33000c999000d33000e33000
 fili1=?d2asc/citytax2b,rcs=00128,typ=RSFm10000
 filo1=?d4pipe/citytax2b,rcs=9000,typ=LSTtd3  #<-- DOS option d3 for CR/LF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
 # area a input, see 'get' instrn
 # area b translated to ASCII, in case mainframe EBCDIC file (with packed?)
 # area c data fields fixed 100 bytes apart in prep for var | delimit instrn
 # area d output, see 'put' instrn at end
        mvc    b0(00128),a0       move input area a to area b
 ###    tra    b0(00128)          uncomment for EBCDIC input (or option a1)
 #      ---                   <-- & insert instrns from uvdata51 if packed fields
 #      ---                   <-- insert R/T tests if redef records
        mvc    c0(8),b0(8)                          #1 folio
        mvc    c100(1),b8(1)                        #2 rectype
        mvc    c200(7),b9(7)                        #3 filler001
        mvc    c300(4),b16(4)                       #4 tax-year
        edta1  c400(11),b20(5p),'+zzzzzzzzz9'       #5 mthpayments-01
        edta1  c500(11),b25(5p),'+zzzzzzzzz9'       #6 mthpayments-02
        edta1  c600(11),b30(5p),'+zzzzzzzzz9'       #7 mthpayments-03
        edta1  c700(11),b35(5p),'+zzzzzzzzz9'       #8 mthpayments-04
        edta1  c800(11),b40(5p),'+zzzzzzzzz9'       #9 mthpayments-05
        edta1  c900(11),b45(5p),'+zzzzzzzzz9'       #10 mthpayments-06
        edta1  c1000(11),b50(5p),'+zzzzzzzzz9'      #11 mthpayments-07
        edta1  c1100(11),b55(5p),'+zzzzzzzzz9'      #12 mthpayments-08
        edta1  c1200(11),b60(5p),'+zzzzzzzzz9'      #13 mthpayments-09
        edta1  c1300(11),b65(5p),'+zzzzzzzzz9'      #14 mthpayments-10
        edta1  c1400(11),b70(5p),'+zzzzzzzzz9'      #15 mthpayments-11
        edta1  c1500(11),b75(5p),'+zzzzzzzzz9'      #16 mthpayments-12
        mvc    c1600(48),b80(48)                    #17 filler002
        var    d0(32000),c0(100),0017,'|'
        trt    d0(32000),$trtchr
        clr    c0(001700),' '
 put1   put    filo1,d0
        skp    loop
 eof    cls    all
        eoj
 @pf2=genpipe1.sub

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5D3. DATAcnv1 - Converting Multiple Record Type Files

execute generated job to create '|' pipe delimited data file

 #2a. uvcopy pfp1/citytax2b,fili1=d2asc/citytax2b,filo1=d4pipe/citytax2b
      ==================================================================
 #2b. uvcopy pfp1/citytax2b   <-- easier (filenames default as shown above)
      =====================

sample output - pipe delimited text

 #3a. more d4pipe/citytax2b       <-- display output
      =====================

 #3a. uvlp12 d4pipe/citytax2b     <-- list the output file
      =======================         (uvlpr12 for Windows/DOS batch script)

 10130140|P||1999|+412|+412|+412|+412|+412|+412|+412|+412|+412|+412|+412|+412||
 10139923|P||1999|+428|+428|+428|+428|+428|+428|+428|+428|+428|+428|+428|+428||
 10147615|P||1999|+124|+124|+124|+124|+124|+124|+124|+124|+124|+124|+124|+124||
 10149304|P||1999|+500|+500|+500|+500|+500|+500|+500|+500|+500|+500|+500|+500||
 10154688|P||1999|+300|+300|+300|+300|+300|+300|+300|+300|+300|+300|+300|+300||
 10173320|P||1999|+405|+405|+405|+405|+405|+405|+405|+405|+405|+405|+405|+405||
 10204700|P||1999|+104|+104|+104|+104|+104|+104|+104|+104|+104|+104|+104|+104||
 10223240|P||1999|+23|+23|+23|+23|+23|+23|+23|+23|+23|+23|+23|+23||
 10308685|P||1999|+301|+301|+301|+301|+301|+301|+301|+301|+301|+301|+301|+301||
 10318833|P||1999|+224|+224|+224|+224|+224|+224|+224|+224|+224|+224|+224|+224||

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5D4. DATAcnv1 - Converting Multiple Record Type Files

list pipe delimited data with copybook field names

 #1a. uvcopy listpipe1,fili1=d4pipe/citytax2b,fili2=maps/citytax2b,filo1=tmp/citytax2b
      ================================================================================

 #1b. listpipeA citytax2b    <-- script equivalent of above (shorter command)
      ===================      - ksh for unix/linux or .bat file for Win/Dos

sample listpipe1 report for Payments

 listpipe1 - list '|' delimited files with copybook fieldnames
           - to verify files created by genpipe1 (see DATAcnv1.htm#Part_4)
 datafile=d4pipe/filename?  copybookmap=maps/filename?  datetime=2003/06/09_13:05:26
 SEQ# FIELDNAME              COBOL PICTURE      DATA

datafile = d4pipe/citytax2b, record# = 1, field count = 17

 10130140|P||1999|+412|+412|+412|+412|+412|+412|+412|+412|+412|+412|+412|+412||

 001 folio                   x(00008).          10130140
 002 rectype                 x.                 P
 003 filler001               x(7).
 004 tax-year                9(4).              1999
 005 mthpayments-01          s9(9) comp-3.      +412
 006 mthpayments-02          s9(9) comp-3.      +412
 007 mthpayments-03          s9(9) comp-3.      +412
 008 mthpayments-04          s9(9) comp-3.      +412
 009 mthpayments-05          s9(9) comp-3.      +412
 010 mthpayments-06          s9(9) comp-3.      +412
 011 mthpayments-07          s9(9) comp-3.      +412
 012 mthpayments-08          s9(9) comp-3.      +412
 013 mthpayments-09          s9(9) comp-3.      +412
 014 mthpayments-10          s9(9) comp-3.      +412
 015 mthpayments-11          s9(9) comp-3.      +412
 016 mthpayments-12          s9(9) comp-3.      +412
 017 filler002               x(48).

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5E1. DATAcnv1 - Converting Multiple Record Type Files

generate batch-file & control-file to create & load table

 #1. uvcopy cobmap1,fili1=cpys/citytax2b,filo1=maps/citytax2b
     ========================================================
     - convert copybook to 'cobmap' (record layout)

 #2. uvcopy sqlcreate1,fili1=maps/citytax2b,filo1=sqls/create_citytax2b.sql
     ======================================================================
                                          ,filo2=sqls/load_citytax2b.ctl
     ===================================================================
     - convert cobmap to batch-file & control-file to create & load table

      - - - OR use script 'sqlcreateA' (combines above into 1 short command)

 #12. sqlcreateA citytax2b    <-- script equivalent of above (shorter command)
      ====================      - ksh for unix/linux or .bat file for Win/Dos

 #3a. more sqls/create_citytax2b.sql   <-- display batch-file output
      ==============================

 #3b. more sqls/load_citytax2b.ctl     <-- display control-file output
      ============================

maps/citytax2b AFTER replacing OCCURS with separate fields

 cobmap1  start-end bytes for cobol record fields    200306090839  pg# 0001
 cpys/citytax2b                 mthpayrec          RCSZ=00128  bgn  end  lth typ
 * citytax2b copybook monthly payment record (type 'P')
 * - occurs 12 manually split to 12 separate fields
  01 mthpayrec.
      10 folio                 pic  x(00008).                 0000 0007  008
      10 rectype               pic  x.                        0008 0008  001
      10 filler001             pic  x(7).                     0009 0015  007
      10 tax-year              pic  9(4).                     0016 0019  004 n  4
 *    10 mthpayments           pic  s9(9) comp-3 occurs 12.
 * original split to sep fields sql does not allow occurs
      10 mthpayments-01        pic  s9(9) comp-3.             0020 0024  005pns 9
      10 mthpayments-02        pic  s9(9) comp-3.             0025 0029  005pns 9
      10 mthpayments-03        pic  s9(9) comp-3.             0030 0034  005pns 9
      10 mthpayments-04        pic  s9(9) comp-3.             0035 0039  005pns 9
      10 mthpayments-05        pic  s9(9) comp-3.             0040 0044  005pns 9
      10 mthpayments-06        pic  s9(9) comp-3.             0045 0049  005pns 9
      10 mthpayments-07        pic  s9(9) comp-3.             0050 0054  005pns 9
      10 mthpayments-08        pic  s9(9) comp-3.             0055 0059  005pns 9
      10 mthpayments-09        pic  s9(9) comp-3.             0060 0064  005pns 9
      10 mthpayments-10        pic  s9(9) comp-3.             0065 0069  005pns 9
      10 mthpayments-11        pic  s9(9) comp-3.             0070 0074  005pns 9
      10 mthpayments-12        pic  s9(9) comp-3.             0075 0079  005pns 9
      10 filler002             pic  x(48).                    0080 0127  048
 ************** end of citytax2b copybook *******************
 *RCSZ=00128                                                       0128

See the output files listed on the next page --->

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5E2. Oracle Database SQL test/demo

Oracle 'batch-file' to create table

 /* create_citytax2b.sql - SQL batch file to create a table */
 /*                     - generated from COBOL copybook 20090220:094034 */
 /* see www.uvsoftware.ca/datacnv1.htm#Part_4 & sqldemo.htm#Part_4      */
 /*                                                                     */
 /* sqlplus user/pass @sqls/create_citytax2b.sql                        */
 /* ===========================================                         */
 DROP TABLE citytax2b;
 CREATE TABLE citytax2b (
 folio                         char     (0008) , --#001 x(00008).
 rectype                       char     (0001) , --#002 x.
 tax_year                      number   (04)   , --#003 9(4).
 mthpayments_01                number   (09)   , --#004 s9(9) comp-3.
 mthpayments_02                number   (09)   , --#005 s9(9) comp-3.
 mthpayments_03                number   (09)   , --#006 s9(9) comp-3.
 mthpayments_04                number   (09)   , --#007 s9(9) comp-3.
 mthpayments_05                number   (09)   , --#008 s9(9) comp-3.
 mthpayments_06                number   (09)   , --#009 s9(9) comp-3.
 mthpayments_07                number   (09)   , --#010 s9(9) comp-3.
 mthpayments_08                number   (09)   , --#011 s9(9) comp-3.
 mthpayments_09                number   (09)   , --#012 s9(9) comp-3.
 mthpayments_10                number   (09)   , --#013 s9(9) comp-3.
 mthpayments_11                number   (09)   , --#014 s9(9) comp-3.
 mthpayments_12                number   (09)     --#015 s9(9) comp-3.
 );
 exit;

Oracle SQL control file to LOAD table

 -- load_citytax2b.ctl - SQL*LOADER control file to load Oracle table
 --                   - generated from COBOL copybook 20081221:080752
 -- see www.uvsoftware.ca/datacnv1.htm#Part_4 & sqldemo.htm#Part_4
 --
 -- sqlldr user/pass control=sqls/load_citytax2b.ctl
 -- ===============================================
 load data infile 'd4pipe/citytax2b.dat' into table citytax2b
 fields terminated by '|' optionally enclosed by '"'
 (
 folio                         , --char     (0008) , --#001 x(00008).
 rectype                       , --char     (0001) , --#002 x.
 tax_year                      , --number   (04)   , --#003 9(4).
 mthpayments_01                , --number   (09)   , --#004 s9(9) comp-3.
 mthpayments_02                , --number   (09)   , --#005 s9(9) comp-3.
 mthpayments_03                , --number   (09)   , --#006 s9(9) comp-3.
 mthpayments_04                , --number   (09)   , --#007 s9(9) comp-3.
 mthpayments_05                , --number   (09)   , --#008 s9(9) comp-3.
 mthpayments_06                , --number   (09)   , --#009 s9(9) comp-3.
 mthpayments_07                , --number   (09)   , --#010 s9(9) comp-3.
 mthpayments_08                , --number   (09)   , --#011 s9(9) comp-3.
 mthpayments_09                , --number   (09)   , --#012 s9(9) comp-3.
 mthpayments_10                , --number   (09)   , --#013 s9(9) comp-3.
 mthpayments_11                , --number   (09)   , --#014 s9(9) comp-3.
 mthpayments_12                  --number   (09)     --#015 s9(9) comp-3.
 )

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5E3. Oracle Database SQL test/demo

create & load table using batch-file & control-file

 #1. Login demo1 --> /home/demo1
     ===========

 #2. sqlplus user/pass @sqls/create_citytax2b.sql
     ============================================
     - run SQL 'batch-file' to create the table

 #3. sqlldr user/pass control=sqls/load_citytax2b.ctl
     ================================================
     - run SQL*LOADER control file to load the table

use SQL*Plus to verify table loaded OK

 #4. sqlplus demo1/demo1pw       <-- start SQL*Plus
     =====================

 #5. SQL> select * from citytax2b <-- SELECT all rows/fields from table
          ======================   - headings & 1st 2 rows shown below:

>>>>> to be supplied

batch-file to list table rows

 /* select_all_citytax2b.sql - batch file to select all rows from citytax2b  */
 /*                      - batch files useful for complex repeated selects */
 select * from citytax2b;
 exit;

You could use the batch-file listed above as an alternative to logging in to SQL*PLUS to list the table rows:

 #6. SQL> exit      <-- exit from SQL*PLUS (back to Unix/Linux prompt)

 #7. sqlplus user/pass @sqls/select_all_citytax2b.sql
     ================================================
     - run SQL 'batch-file' to create the table

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_6 DATAcnv1.doc - Converting EBCDIC to ASCII - Contents

Generating & Executing ALL jobs to convert ALL data files

related documentation

 SQLconvert.htm - convert flat files to pipe delimited for Microsoft SQL server
                  - driven by COBOL copybooks
                  - also generate scripts to create & load tables
                  - generates jobs for all copybooks & all data files
                    (vs this DATAcnv1.doc for 1 copybook/file at a time)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6A1. DATAcnv1.doc - Converting EBCDIC to ASCII

Generating & Executing ALL jobs - Overview

This document (DATAcnv1.doc) is intended to convert 1 file at a time. We make it easy to use by renaming the datafiles the same as the copybook names. This is possible because we have different subdirs for different types of files (cpys/..., maps/..., pfx1/..., pfx2/..., pfx3/... d1ebc/..., d2asc/...).

If you have dozens or hundreds of datafiles to be converted, you should follow the procedures documented in VSEDATA.htm or MVSDATA.htm.

Those procedures use a control file to relate the data file names to the copybook names.

Pages '6D1' + will apply the control file method in a limited manner to the procedures documented here in DATAcnv1.

The control file method allows for the distinct possibility of having multiple datafiles using the same copybook.

But without using the control file method you could still copy the copybook named job in pfx1 to pfx3, renaming for the datafilename. You would also change the names on the fili1=... & filo1=... from the generated copybook names to the actual datafilenames.

But first (in case you are OK with naming datafiles same as copybooks), we will show you how you can generate ALL jobs to convert ALL files with 2 commands & then how to execute all data conversion jobs with 1 command.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6B1. Converting Mainframe DATA to unix/linux/windows

Generating ALL jobs to convert ALL datafiles EBCDIC to ASCII

 cpys-------->maps---------->pfx1-------------->pfx2----------->pfx3--------->
     cobmap1       uvdata51         uvdata52          copy/edit       execute
             convert copybooks  change copybooknames
               to uvcopy jobs     to datafilenames

 #0. Login as cnvdata --> /home/cnvdata

 #1. uvcopyx cobmap1 cpys maps uop=q0i7p0
     ====================================
     - convert all copybooks to record layouts (cobmaps)

 #2. uvcopyx uvdata51 maps pfx1 uop=q0i7
     ===================================
     - generate uvcopy jobs from cobmaps
     - uvcopy jobs have the same filenames as copybooks
     - copybooknames used for datafilenames on fili1=... & filo1=...

 #3a. cp pfx1/* pfx2/
      ===============
 #3b. cp pfx2/* pfx3/
      ===============
      - copy all auto-generated jobs from pfx1/... to pfx2/... & pfx3/*
      - do this copy all/* 1st time only, else 1 at a time
      - will make any manual changes in pfx3 (for multi record type files)
      - protects our changes in case mass generation is repeated

           - - - - OR - - - -

 #4. See optional '6D1' to replace the copybook names (used as I/O filenames)
     with the actual data filenames (as we copy from pfx2 to pfx3).

 #4a. uvcopy uvdata52,fili1=ctl/ctlfile2,fild2=pfx1,fild3=pfx2,uop=r1s1t0
      ===================================================================
      - insert data filenames into EBCDIC to ASCII conversion jobs
      - see options explained on page '6D4'

 #5. cp pfx2/* pfx3/
     ===============
     - copy all jobs to pfx3/... for execution
     - do this copy all/* 1st time only, else 1 at a time
     - will make any manual changes in pfx3 (for multi record type files)
     - protects our changes in case mass generation is repeated

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6C1. Converting Mainframe DATA to unix/linux/windows

Executing jobs to convert ALL data files

 d1ebc------------------->d2asc------------------>$TESTDATA/mstr
        EBCDIC->ASCII          copy files to test dirs
       uvcopyxx 'pfx3/*'       cp d2asc/* $TESTDATA/mstr

 #0. Login as cnvdata --> /home/cnvdata

 #1. uvcopyxx 'pfx3/*'
     =================
     - execute ALL uvcopy jobs in pfx3/... to convert all data files
     - copies all data files from d1ebc/... to d2asc/...
     - translating char fields from EBCDIC to ASCII
     - preserving any packed/binary fields (same as on mainframe)
     - converting any zoned signs from mainframe to Micro Focus conventions
       (pos {ABCDEFGHI ---> 0123456789 & neg }JKLMNOPQR ---> pqrstuvwxy)

 #2. uvcopyxx 'pfp3/*'
     =================
     - execute ALL uvcopy jobs in pfp3/... to pipe delimit all data files
       as we copy from d2asc/... to d4pipe/...

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6D1. DATAcnv1 - Convert EBCDIC to ASCII with COBOL copybooks

Inserting actual datafile names into conversion jobs

Up to this point, we have been using the same name for the data files (EBCDIC,ASCII,& Delimited) as for the copybook & uvcopy conversion jobs. This simplified the operating instructions and was made possible by having different subdirs for each type of file.

This also required loading the original EBCDIC data file into the expected input subdir & copying the converted data file from the designated output subdir out to wherever we might want it.

For large conversions that may have to be repeated (for testing, retesting,& final conversion), it might be better to insert the actual data file pathnames into the uvcopy conversion jobs.

Inserting actual datafile names - Advantages

1. We would not have to change the name of the datafile to match the copybook or vice-versa.
   
   
2. This will also solve the possible problem of having 2 data files using the same copybook.
   
   
3. The control file can also supply Indexed file types & key locations to be coded on the filo1=... definition
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6D2. DATAcnv1 - Convert EBCDIC to ASCII with COBOL copybooks

Inserting actual datafile names - the Plan

1. We will generate the uvcopy conversion jobs in the same manner as previously documented, outputting the EBCDIC to ASCII conversion job into subdir pfx1 & the pipe delimit job into subdir pfp1, and naming the conversion jobs the same as the copybook.
   
   
2. We could copy pfx1/... to pfx2/... & pfx3/... and make any manual changes in pfx3/... (allowing for multi record type files).
   
   
3. OR we can create a 'control file' to relate the actual data file name to the copybook name.
   
   
4. We will run an additional utility job 'uvdata52' that will read the control file & copy conversion jobs from pfx1 to a new subdir pfx2 where they are named for the actual datafile & have the datafile names inserted on the fili1=... & filo1=... declarations.
   
   
5. We can use a similar job 'uvdata54' to create the pipe delimit jobs in pfp2 with actual data filenames.
   
   
6. The directories for fili1=... & filo1=... will be coded as ${CNVDATA}/d1ebc & ${CNVDATA}/d2asc. This allows flexibility in data file locations by exporting CNVDATA other than /home/cnvdata, but the subdirs are hardcoded in the uvcopy jobs as d1ebc/d2asc & d2asc/d4pipe.
   
   
7. We can use the 'uvcopyxx' Unix/Linux script to execute ALL conversion jobs in subdirs pfx3 (or pfp3) with 1 command.
   
   
8. Whether you use the control file method or not, I recommend always storing the final version of the uvcopy jobs in pfx3/... or pfp3/... This gives you some flexibility if you later decide the control file is worth the effort.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6D3. DATAcnv1 - Convert EBCDIC to ASCII with COBOL copybooks

Inserting actual datafile names - Test/Demo

We will illustrate inserting actual data file names into our conversion jobs (& executing them), using the same test data files as previously, but renaming them (different from the copybook names), as shown below.

 #1a. Login cnvdata ---> /home/cnvdata  (OR cd $CNVDATA)

 #1b. cd $CNVDATA    - change to our superdir with same subdirs as previous
      =============    (d1ebc,d2asc,d4pipe,cpys,maps,pfx1/2/3,pfp1/2/3,etc)

 #2a. cp /home/uvadm/dat1/citytax1 d1ebc/citytax1data
      ===============================================
 #2b. cp /home/uvadm/dat1/citytax2 d1ebc/citytax2data
      ===============================================
         - copy & change names of demo datafiles (different from copybooks)

 #3a. mkdir pfx2 pfx3 pfp2 pfp3  - make subdirs if not already present
      =========================

 #3b. mkdir ctl        - make subdir for the control file
      =========

 #4a. cp /home/uvadm/ctl/ctlfile2 ctl  - copy supplied demo control file
      ===============================

 #4b. vi ctl/ctlfile2                  - OR create the control file with vi
      ===============

sample control file

 citytax1data                  cpy=citytax1 rcs=00128
 citytax2data                  cpy=citytax2 rcs=00128
 citytax3data                  cpy=citytax3 rcs=00128 keys=(0,10n,10,25d)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6D4. DATAcnv1 - Inserting actual DataFileNames vs CopyBookNames

insert datafilenames (vs copybooknames)

 #5a. uvcopy uvdata52,fili1=ctl/ctlfile2,fild2=pfx1,fild3=pfx2,uop=r1s1t0
      ===================================================================
      - insert data filenames into EBCDIC to ASCII conversion jobs

prompt for options

 uop=r1s1t0 - use these for uvdata51 jobs (EBCDIC to ASCII & typ=RSF)
 uop=r1s2t0 - use these for genverify1 jobs (pfv1-->pfv2 jobs)
 uop=r1s2t0 - also use these for genacum2 jobs (pfa1-->pfa2 jobs)
     r1     - override rcs=... on fili1 (from cobmap) using ctlfile rcs=...
              if typ=RSF (N/A to typ=RDW)
       s0   - leave input typ as coded from prior job (usually typ=RSF)
       s1   - set input typ from ctlfile typ=...
       s2   - force input file typ=RDWz2
         t0 - leave output typ as coded from prior job
              (uvdata51 typ=RSF, genpipe1,genverify1,genacum2 typ=LSTt)
         t2 - set output typ=RDWz2 (unconditionally if no t1 bit)
         t1 - (t1+t2=t3) set output typ=RDWz2 ONLY IF input typ=RDW__
 User OPtion (uop) defaults  = q1r1s1t2
 -->null to accept or enter/override -->

 #6a. cp pfx2/* pfx3    <-- copy EBCDIC to ASCII jobs to pfx3 before execution
      ==============
 #6b. cp pfp2/* pfp3    <-- copy pipe delimit jobs to pfp3 before execution
      ==============

 #5b. uvcopy uvdata54,fili1=ctl/ctlfile2,fild2=pfp1,fild3=pfp2
      ========================================================
      - insert data filenames into pipe delimit jobs
      - alternative to #5a above for jobs from pipedelim1 (vs uvdata51)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6D5. DATAcnv1 - Inserting actual DataFileNames vs CopyBookNames

sample conversion job - BEFORE inserting data filenames

 # citytax1 - uvcopy job, EBCDIC to ASCII, preserve packed, fix signs
 opr='citytax1 - uvcopy code generated from copybook: citytax1 '
 uop=q0
 was=a33000b33000
 fili1=?d1ebc/citytax1,rcs=00128,typ=RSF   # 'citytax1' is copybook name
 filo1=?d2asc/citytax1,rcs=00128,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
        mvc    b88(19),a88             pns post-date:face-value
        trt    b107(9),$trtsea          ns maint-tax
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

sample conversion job - AFTER inserting data filenames

 # citytax1data - uvcopy job, EBCDIC to ASCII, preserve packed, fix signs
 opr='citytax1data - uvcopy code generated from copybook: citytax1 '
 uop=q0
 was=a33000b33000
 fili1=${CNVDATA}/citytax1data,rcs=00128,typ=RSF  #'citytax1data' is datafilename
 filo1=${CNVDATA}/citytax1data,rcs=00128,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
        mvc    b88(19),a88             pns post-date:face-value
        trt    b107(9),$trtsea          ns maint-tax
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

1. Note the changes on lines 5&6 (fili1=... & filo1=...) of the Before & After listings. The copybook filenames have been replaced with the data filenames from the control file.
   
   
2. The path to data filenames is coded as an environmental variables ${CNVDATA}. This allows flexibility in data file locations by exporting CNVDATA other than /home/cnvdata, but the subdirs are hardcoded in the uvcopy jobs as d1ebc/d2asc or d2asc/d4pipe.
   
   
3. The conversion jobs in the output subdirs (pfx3 & pfp3) have been renamed to the data filenames (vs the copybook filenames).
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6D6. DATAcnv1 - Inserting actual DataFileNames vs CopyBookNames

Executing Data Conversion uvcopy jobs

 #0a. Login cnvdata ---> /home/cnvdata  (OR cd $CNVDATA)

 #0b. export CNVDATA=/home/cnvdata
      ============================
      - this export should already be in the profile

 #1. cd $CNVDATA    - change to our superdir with same subdirs as previous
     =============    (d1ebc,d2asc,d4pipe,cpys,maps,pfx1/2/3,pfp1/2/3,etc)

 #2. uvcopyxx 'pfx3/*'
     =================
     - execute all EBCDIC to ASCII conversion jobs
     - copies & converts all data files from d1ebc to d2asc

 #3. uvcopyxx 'pfp3/*'
     =================
     - execute all pipe delimit conversion jobs
     - copies & pipe-delimits all datafiles from d2asc to d4pipe

Notes

1. We have exported CNVDATA as our conversion superdir /home/cnvdata The conversion jobs define I/O as $CNVDATA/d1ebc/... & $CNVDATA/d2asc/... The pipe delimit jobs define I/O as $CNVDATA/d2asc/... & $CNVDATA/d4pipe/...
   
   
2. The '*' means ALL jobs in the subdir will be executed, but we could modify this to execute only those jobnames matching a pattern. For example 'gl*' would execute all jobs whose name begins with 'gl'.
   

     uvcopyxx '/home/cnvdata/pfx3/gl*'
     =================================

3. You must enclose the pathname in single quotes, since the '*' expansion is to occur within the script 'uvcopyxx' (not when it is called by the shell).
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_7 DATAcnv1 - Converting Mainframe Data to Unix/Linux

Part 7 - Variable Length Files - Contents

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7A1. Variable Length RDW (Record Descriptor Word) files

Introduction to variable length RDW files

RDW (Record Descriptor Word) variable length files are often used to FTP variable length EBCDIC files from the mainframe to unix/linux/windows systems. uvcp provides for 2 types of RDW files - standard & non-standard (option z2).

standard RDW files

• 'typ=RDWz4' is standard ('typ=RDW' defaults to 'typ=RDWz4').
• 4 byte record prefix with 2 byte binary record size & 2 unused nulls
• record size in prefix is the total record size including the prefix
• 2 byte record size is 'BIG-END' binary (like mainframe or SUN,AIX,DEC)
• 'little-end' machines (Intel,AMD) need to switch ends to perform math

Non-standard RDW files

• 2 byte record prefix (BIG-END binary, same as standard RDW)
• record size in prefix is the data record size excluding the prefix
• uvcp provides option 'z2' (typ=RDWz2) to process non-standard RDW files

test/demo files supplied in /home/uvadm/dat1

We will illustrate how to convert RDW EBCDIC files to text or fixed length files using: uvhd, uvcp,& uvcopy.

1. use 'uvhd' when there you only have a few files to do,& when the interactive utility is preferred.
   
   
2. use 'uvcp' when you want to write a script to convert several files and for repeatability.
   
   
3. use 'uvcopy' when you need to do some data manipulation beyond just the simple conversion from 1 file format to another. You could write custom uvcopy jobs to convert files with packed/binary, but DATAcnv1.htm documents pre-programmed jobs to generate uvcopy jobs from COBOL copybooks.
   

We will 1st investigate the testRDWe file using 'uvhd'. We can not use unix tools (vi, lp, etc) since the file is EBCDIC & contains binary record sizes.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7B1. Variable Length RDW Files

testRDWe - test file to demo RDW files

 uvhd dat1/testRDWe as3 <-- examine the RDW test file
 ======================   - option 'a' translates character line to ASCII
                          - option 's3' to space between scale & 3 line groups

                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123

           0 ....DELL00 - Dell Inc.  ....HP0000 - Hewlett Packard....IBM000 -
             0100CCDDFF464C8994C984440100CDFFFF464C8A98AA4D8898980200CCDFFF46
             080045330000045330953B000C008700000008563533071321940C0092400000

          64  International Business Machines....MFC000 - Micro Focus COBOL
             4C9A8998A899894CAA898AA4D888898A0200DCCFFF464D88994C98AA4CDCDD44
             0953595139651302429552204138955200004630000004939606634203626300

         128 ....MS0000 - Microsoft Corp.....REDHAT - Red Hat Linux  ....SUN0
             0100DEFFFF464D8899A98A4C99940100DCCCCE464D884C8A4D89AA440200EEDF
             0C0042000000049396266303697B0C0095481300095408130395470004002450

         192 00 - Sun Microsystems Ltd   ....UVSI00 - UV Software Inc.
             FF464EA94D8899AAAA89A4DA84440200EEECFF464EE4E98AA8984C984444
             00000245049396282354203340000000452900000450266361950953B000

The entire file is only 252 bytes & contains 8 short variable length records:

• 1st (DELL) starts at byte 00 - length x'0018' = 1*16+8*1 = 24 bytes
• 2nd (HP) starts at byte 24 - length x'001C' = 1*16+12*1 = 28 bytes
• 3rd (IBM) starts at byte 52 - length x'002C' = 2*16+12*1 = 44 bytes

You cannot display RDW files with 'vi', because vi cannot handle binary, there are no LineFeeds to separate the records, so the entire file appears as 1 long line (unless there just happened to be a x'0A' in the length field, which would be interpreted as a LineFeed by vi).

uvhd option 'z' to recognize RDW files

OPtion 'z' would tell uvhd to look for the 'RDW' record prefixes & show 1 record at a time, but we did not specify option z above, and the default is to show any file in 256 byte blocks (4 groups of 3 64 byte lines for characters, zones,& digits).

See the next page where we will specify option 'z' to show RDW files 1 record at a time. You can press enter to browse forward until EOF reached. Then you could enter '1' to return to the begining of the file.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7C1. Variable Length RDW Files

Investigating RDWz4 files with 'uvhd'

 uvhd dat1/testRDWe z4a   <-- display RDWz4 file using option 'z'
 ======================     - option 'a' to display character line in ASCII
                            - null entries browse forward until EOF reached

                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ....DELL10 - Dell Inc.
             0100CCDDFF464C8994C98444
             080045331000045330953B00
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
          24 ....HP0010 - Hewlett Packard
             0100CDFFFF464C8A98AA4D889898
             0C00870010000856353307132194
                      10        20        30        40        50        60
 r#        3 0123456789012345678901234567890123456789012345678901234567890123
          52 ....IBM010 - International Business Machines
             0200CCDFFF464C9A8998A899894CAA898AA4D888898A
             0C009240100009535951396513024295522041389552
                      10        20        30        40        50        60
 r#        4 0123456789012345678901234567890123456789012345678901234567890123
          96 ....MFC010 - Micro Focus COBOL
             0200DCCFFF464D88994C98AA4CDCDD44
             00004630100004939606634203626300
                      10        20        30        40        50        60
 r#        5 0123456789012345678901234567890123456789012345678901234567890123
         128 ....MS0010 - Microsoft Corp.
             0100DEFFFF464D8899A98A4C9994
             0C0042001000049396266303697B
                      10        20        30        40        50        60
 r#        6 0123456789012345678901234567890123456789012345678901234567890123
         156 ....REDH10 - Red Hat Linux
             0100DCCCFF464D884C8A4D89AA44
             0C00954810000954081303954700
                      10        20        30        40        50        60
 r#        7 0123456789012345678901234567890123456789012345678901234567890123
         184 ....SUN010 - Sun Microsystems Ltd
             0200EEDFFF464EA94D8899AAAA89A4DA8444
             040024501000024504939628235420334000
                      10        20        30        40        50        60
 r#        8 0123456789012345678901234567890123456789012345678901234567890123
         220 ....UVSI10 - UV Software Inc.
             0200EEECFF464EE4E98AA8984C984444
             0000452910000450266361950953B000

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7D1. Variable Length RDW Files

converting EBCDIC RDWz4 files to ASCII - with 'uvhd'

On this page we will show you how to convert EBCDIC RDWz4 files to ASCII text using 'uvhd'. On following pages we will do the same with 'uvcp' & 'uvcopy'.

We will specify uvhd options 'za3p4y7' which mean:

 uvhd dat1/testRDWe z4a3p4y7   <-- display 1st record & wait for command
 ===========================

                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ....DELL10 - Dell Inc.
             0100CCDDFF464C8994C98444
             080045331000045330953B00

 ---> w99999  <-- write all records (8) to tmp/testRDWe_yymmdd_hhmmssW
                - output filename will be date/time stamped
                - On Dec 18, 20007 at 12:15 tmp/testRDWe_071218_121500W

 ---> q       <--- quit uvhd

 vi tmp/testRDWe_071218_121500W
 ==============================
 vi tmp/*00W       <-- shortcut to display desired file
 ===========

text records extracted from RDWz4 file

     DELL00 - Dell Inc.
     HP0000 - Hewlett Packard
     IBM000 - International Business Machines
     MFC000 - Micro Focus COBOL
     MS0000 - Microsoft Corp.
     REDHAT - Red Hat Linux
     SUN000 - Sun Microsystems Ltd
     UVSI00 - UV Software Inc.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7E1. Variable Length RDW Files

converting EBCDIC RDWz4 files to ASCII - with 'uvcp'

We will now use 'uvcp' to perform the conversion from EBCDIC RDWz4 to ASCII.

Instead of variable length text, we will create fixed length output records that are required for most COBOL input files. For this demo, we will convert to fixed 64 byte records & insert a LineFeed in the last byte so we can have it both ways (fixed length for COBOL, but LFs allow investigation with vi).

 uvcp "fili1=dat1/testRDWe,rcs=64,typ=RDWz4,filo1=tmp/testz4,typ=RST,tra=0(64)"
 ==============================================================================

confirm conversion OK with uvhd

 uvhd tmp/testz4 r64h2     <-- display output file with uvhd
 =====================       - option r64 for record size
                             - option 'h2' for hex display to see LineFeed
                             - only 1st record shown below

                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 DELL10 - Dell Inc.                                             .
             4444332224666246622222222222222222222222222222222222222222222220
             45CC100D045CC09E3E000000000000000000000000000000000000000000000A

Confirmations shown by uvhd

1. Binary record prefixes have been dropped off, because output typ=RST extracts the data record only & omits the typ=RDW record prefix.
   
   
2. LineFeeds in last byte of each record (x'0A's).
   
   
3. Output is ASCII 1st char 'D' is x'44' here vs x'C4' on page 'K2'&'K3' Translation performed by instruction 'tra=0(64)'.
   
   
4. See pages uvcp.htm#M0 thru M17 for other instructions you can use to reformat data (mvc, clr, rep, tre, tru, trl, sel, del, etc...).
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7F1. Variable Length RDW Files

converting EBCDIC RDW files to ASCII - with 'uvcopy varfix11'

'varfix11' is a uvcopy utility job to convert an RDW variable length file to a specified fixed length (large enough to hold largest variable record).

On your mainframe migration, you may have EBCDIC RDW & BDW/RDW files to be transferred to & converted on your unix/linux system.

For a mainframe migration you would not want to use the interactive 'uvhd' for hundreds of files. Here we will illustrate a batch utility 'varfix11' that you could use to convert varlth RDW EBCDIC files to ASCII fixed records.

Note that this job can not be used for EBCDIC files with packed/binary fields. For those you need the procedures documented in 'Part_3' of this DATAcnv1.doc or in MVSDATA.htm#Part_4.

 uvcopy varfix11,fili1=dat1/testRDWe,filo1=tmp/testRDWaf,uop=a1b1h1r64t1
 =======================================================================

 uop=a0b0h1r2048t0 - option defaults
     a0            - no translate (input EBCDIC with packed/binary fields)
     a1            - translate from EBCDIC to ASCII
       b0          - do NOT convert nulls to blanks
       b1          - DO convert nulls to ASCII blanks
       b2          - DO convert nulls to EBCDIC blanks
         h0        - drop the 4 byte binary record-size headers
         h1        - replace 4 byte binary recsize hdr with numeric chars
           r2048   - output fixed records 2048 bytes
           r8192   - max output fixed size 8192 bytes
                t0 - do NOT insert LineFeed in last byte of record
                t1 - DO insert LineFeed in last byte of record
  null to accept or re-specify (1 or more) -->
 071128:172717:varfix11: EOF fili01 rds=16 size=252: dat1/testRDWe
 071128:172717:varfix11: EOF filo01 wrts=8 size=512: tmp/testRDWaf

 cat tmp/testRDWaf       <-- display output file
 =================

 0024DELL10 - Dell Inc.
 0028HP0010 - Hewlett Packard
 0044IBM010 - International Business Machines
 0032MFC010 - Micro Focus COBOL
 0028MS0010 - Microsoft Corp.
 0028REDH10 - Red Hat Linux
 0036SUN010 - Sun Microsystems Ltd
 0032UVSI10 - UV Software Inc.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7G1. Variable Length RDW Files

creating table summary of record sizes in RDW files

'varstat1' is a uvcopy job that you can run to create a summary table of record-sizes found in all records of a variable file typ=RDWz4 or typ=RDWz2 (variable length records with 4 byte or 2 byte prefixes).

demo with supplied test file

 #1. cd $UV      <-- change to /home/uvadm

 #2. uvcopy varstat1,fili1=d0ebc/testRDW,filo2=rpts/testRDW_recsizes
     ===============================================================

 #3. vi rpts/testRDW_recsizes   <-- inspect report
     ========================

sample report

 varstat1  2007/12/19_18:10:29  record-sizes in dat1/testRDW
 tbl#001 pg#001     -argument-
 line#  count    %  record-sizes

     1       1  12  00020
     2       3  37  00024
     3       2  25  00028
     4       1  12  00032
     5       1  12  00040

             8*100   *TOTAL*

create reports for all files in directory

 #1. cd $CMPDATA     <-- change to file compare superdir

 #2. mkdir tmp1      <-- make tmp1 subdir if not existing
 #2a. rm -f tmp1/*   <-- OR remove all files if tmp1 already exists

 #3. uvcopyx varstat1 d0ebc tmp1 uop=q0i7,rop=r0
     ===========================================
     - create table summary recsize counts for all files in d0ebc subdir
     - output reports in tmp1 with same names as datafiles in d0ebc

 #4. uvlpd12 tmp1     <-- print all reports in tmp subdir
     ============

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7X0. Variable Length RDW Files

uvcopy jobs used in Part 7

The 'uvcopy jobs' used in this section are listed on the following pages. You might need to modify them for complex variable length conversions.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7X1. Variable Length BDW/RDW Files

 # varstat1 - create summary table of record-sizes used in variable file
 #          - for variable lth records with 4 byte hdrs (standard RDW)
 #          - by Owen Townsend, Dec 12, 2007
 #Dec19/07 - replace getvr2 subrtn with file typ=RDWz4 & RDWz2 (optional)
 #
 #           ** create report for 1 file (for testing) **
 #
 # uvcopy varstat1,fili1=d0ebc/datafilename,filo2=tmp1/reportname
 # ==============================================================
 #
 #           ** create reports for all files in directory **
 #
 # 1. cd $CNVDATA     <-- change to conversion superdir
 #                     - subdir d0ebc contains EBCDIC var lth files
 #
 # 2. mkdir tmp1      <-- make tmp1 subdir if not existing
 # 2a. rm -f tmp1/*   <-- OR remove all files if tmp1 already exists
 #
 # 3. uvcopyx varstat1 d0ebc tmp1 uop=q0i7,rop=r0
 #    ===========================================
 #    - create table summary recsize counts for all files in d0ebc subdir
 #    - output reports in tmp1 with same names as datafiles in d0ebc
 #
 # 4. uvlpd12 tmp1     <-- print all reports in tmp subdir
 #    ============
 #
 #                  ** sample report **
 #
 # varstat1  2006/12/17_18:15:17  record-sizes in d0ebc/E2121656
 # tbl#001 pg#001     -argument-
 # line#  count    %  record-sizes
 #     1  10,552  16  00046
 #     2   4,451   7  00065
 #     3  23,347  37  00066
 #     4     367   0  00068
 #     5  21,010  33  00083
 #       - - - etc - - -
 #    18       3   0  00218
 #    19     441   0  00233
 #    20     813   1  00239
 #        62,115*100   *TOTAL*
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 # This job designed for variable lth records typ=RDW & typ=RDWz2
 # typ=RDW (standard RDW) - 4 byte record prefixes
 #         - binary recsize in 1st 2 bytes (3rd & 4th x'0000' unused)
 #         - binary recsize of data (includes the 4 byte prefix)
 #         - blk/rec lth binary BIG-END format, need 's'witch option for INTEL
 # typ=RDWz2 (non-std RDW) - 2 byte record prefixes (2 nulls omitted)
 #         - binary recsize does not include the 2 byte prefix
 #
 opr='$jobname - create summary table of record-sizes used in variable file'
 opr='uop=q1z0 - option defaults (message only, see uop=... in job)'
 opr='      z0 - RDW 4 byte prefix, recsize 1st 2 BIG-END binary'
 opr='      z2 - RDW 2 byte prefix, recsize 1st 2 BIG-END binary'
 uop=q1z0      # option defaults
 rop=r1        # option to prompt for report disposition
 was=a16384
 fili1=?d0ebc/input,rcs=8192,typ=RDW        #Dec19/07 typ=RDW replaces getvr
 filo1=?tmp1/$jobname.rpt,typ=LSTt,rcs=128
 @run
 #Dec19/07 - replace subrtn getvr with typ=RDW & RDWz2
 #         - for nonstd 2 byte prefix vs std 4 bytes
 # if uop=z2, append file option z2 for typ=RDWz2
        tsb    o26(1),x'02'            uop z2 for nonstd RDW ?
        skp!   1
        cata8  $fili1+180(15),'z2'
        opn    all
 #
 # begin loop to get records & build summary table of recsizes
 # - to be dumped at end of file
 man20  get    fili1,a0                 get next RDW record data
        skp>   eof                      (cc set > at EOF)
        mvn    c0(5),$rv                cnvrt binary recsize to numeric
        tblt1f1 c0(5),'record-sizes'    build table in memory
        skp    man20                    repeat get loop until EOF
 #
 # EOF - close files & end job
 eof    mvfv1  f0(80),'record-sizes in $fili1'
        tbpt1s1 filo1,f0(50)            dump table from memory to file
        cls    all
        eoj
 #
 # getvr - subrtn to get records from IBM std variable length file
 #Dec19/07 - getvr replaced by file typ=RDW & typ=RDWz2
 #         - subrtn saved (for interest) in $UV/pf/util/getvr
 ## folwng instrns modified (see above) for typ=RDW
 ##man20  bal    getvr                    set rgstrs a & b to next record
 ##       mvn    c0(5),a0(2bs)            cnvrt binary recsize to numeric

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7X2. Variable Length BDW/RDW Files

 # varfix11 - convert RDW variable length file test Aug 24/05
 #          - convert RDW variable length file to fixed length records
 #          - prior to 2nd job that will convert EBCDIC to ASCII
 #            OR compare 2 files uvcmpVA1/uvcmpVE1 (fixlth max 4096)
 #          - by Owen Townsend, UV Software
 #          - originally for Laval, Dec 2005
 #
 #Dec19/07 - replace subrtn getvr with typ=RDW & add option z2
 #           for nonstd 2 byte prefix vs std 4 bytes (binary recsize 1st 2)
 #         - subrtn code preserved in $UV/pf/util/getv for interest
 #
 # Enhanced Oct 2007 for Sungard, options added
 # - a1=translate to ASCII, h1=cnvt hdr to numerics, r4096=fixed output recsize
 # - b1=convert nulls to ASCII blanks, t1=terminate with LineFeed
 #
 #                 ** Operating Instructions **
 #
 # uvcopy varfix11,fili1=d0ebc/infile,filo1=d1ebc/outfile,uop=a1h1r4096
 # ====================================================================
 #  - convert data file from variable to fixed 4096 byte records
 #
 #  option a1 - translate from EBCDIC to ASCII
 #  option h1 - replace 4 byte binary recsize hdr with recsize numerics
 #  option r4096 - output fixed records 4096 bytes
 #               - large enough to hold largest variable lth record
 #
 # This 1 uvcopy job 'varfix11' can be used for all data files
 # - to convert from variable to fixed
 # - drops 4 byte variable recsize headers, or retain converted to numerics
 # - null fill records to fixed recsize spcfd by option r
 #
 #                       ** uses for varfix11 **
 #
 # Convert variable length to fixed length to facilitate:
 #
 # 1. CONVERTING datafiles from EBCDIC to ASCII (preserving packed/binary)
 #    - could convert back to variable length after conversion
 #    - see examples in VSEDATA.doc
 #
 # 2. COMPARING 2 datafiles
 #    - see script uvcmpVE1 to compare 2 varlth EBCDIC datafiles
 #    - this varfix11 used to convert both files to fixed max lth in tmp1/tmp2
 #    - followed by utility 'uvcmp1' to compare & create report in rpts/...
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 #                  ** sample variable-length RDW files **
 #
 # uvhd dat1/testRDWe a    <-- display demo file EBCDIC varlth RDWz4
 # ====================      - option 'a' to show char lines in ASCII
 #                             (zones & digits lines show EBCDIC codes)
 #
 #                      10        20        30        40        50        60
 # r#        1 0123456789012345678901234567890123456789012345678901234567890123
 #           0 ....DELL10 - Dell Inc.  ....HP0010 - Hewlett Packard....IBM010 -
 #             0100CCDDFF464C8994C984440100CDFFFF464C8A98AA4D8898980200CCDFFF46
 #             080045331000045330953B000C008700100008563533071321940C0092401000
 #          64  International Business Machines....MFC010 - Micro Focus COBOL
 #             4C9A8998A899894CAA898AA4D888898A0200DCCFFF464D88994C98AA4CDCDD44
 #             0953595139651302429552204138955200004630100004939606634203626300
 #         128 ....MS0010 - Microsoft Corp.....REDH10 - Red Hat Linux  ....SUN0
 #             0100DEFFFF464D8899A98A4C99940100DCCCFF464D884C8A4D89AA440200EEDF
 #             0C0042001000049396266303697B0C0095481000095408130395470004002450
 #         192 10 - Sun Microsystems Ltd   ....UVSI10 - UV Software Inc.
 #             FF464EA94D8899AAAA89A4DA84440200EEECFF464EE4E98AA8984C984444
 #             10000245049396282354203340000000452910000450266361950953B000
 #
 # uvhd dat0/testRDWe z4a1  <-- option 'z4' to process RDWz4 files
 # =======================    - 'a1' to translate char line to ASCII
 #                            - hexadecimal zones & digits show EBCDIC values
 #
 #                      10        20        30        40        50        60
 # r#        1 0123456789012345678901234567890123456789012345678901234567890123
 #           0 ....DELL10 - Dell Inc.
 #             0100CCDDFF464C8994C98444
 #             080045331000045330953B00
 #
 # Note - uvhd displays 1st record & prompts for commands
 #      - enter w999 to write all records & q to quit:
 #
 # --> w999a1t6r64 <-- Write all records to the tmp/... subdir
 #                   - filename date/time stamped ex: tmp/testRDWe_yymmdd_hhmmssW
 #                   - On Nov 17/07 3:30 PM might be: tmp/testRDWe_071117_153059W
 #         a1        - translate output records to ASCII
 #           t6      - insert LineFeed terminators after last non-blank
 #                     (t6=t2+t4, t2=LineFeed, t4=insert after last non-blank)
 #             r64   - max size for output records
 #                                  - 'y7' to insert CR+LF after last non-blank
 # --> q    <-- quit uvhd
 #
 # cat tmp/*59W       <-- display output text records
 # ============         - use *59W to save a lot of keystrokes
 #
 # DELL10 - Dell Inc.
 # HP0010 - Hewlett Packard
 # IBM010 - International Business Machines
 # MFC010 - Micro Focus COBOL
 # MS0010 - Microsoft Corp.
 # REDH10 - Red Hat Linux
 # SUN010 - Sun Microsystems Ltd
 # UVSI10 - UV Software Inc.
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 opr='$jobname - convert variable length file to fixed length records'
 opr='uop=a0b0h1r4096t0z4 - option defaults'
 opr='    a0              - no translate (for EBCDIC packed/binary fields)'
 opr='    a1              - translate from EBCDIC to ASCII'
 opr='      b0            - do NOT convert nulls to blanks'
 opr='      b1            - DO convert nulls to ASCII blanks'
 opr='      b2            - DO convert nulls to EBCDIC blanks'
 opr='        h0          - drop the 4 byte binary record-size headers'
 opr='        h1          - replace 4 byte binary recsize hdr with numeric chars'
 opr='          r4096     - output fixed records 4096 bytes'
 opr='          r8192     - max output fixed size 8192 bytes'
 opr='               t0   - do NOT insert LineFeed in last byte of record'
 opr='               t1   - DO insert LineFeed in last byte of record'
 opr='                 z4 - RDW 4 byte prefix, recsize 1st 2 BIG-END binary'
 opr='                 z2 - RDW 2 byte prefix, recsize 1st 2 BIG-END binary'
 uop=q1a0b0h1r4096t0z4   # option defaults
 was=a32768b32768c32768  # increase areas a,b,c from dflt 1024
 fili1=?d0ebc/filename,rcs=8192,typ=RDWz4   # uop z2 will change to option z2
 filo1=?d1ebc/filename,rcs=8192,typ=RSF
 @run
 #Dec19/07 - replace subrtn getvr with typ=RDWz4 & typ=RDWz2
 #         - typ=RDWz2 for nonstd 2 byte prefix vs std 4 bytes
 # if uop=z2, change file option z4 to z2
        tsb    o26(1),x'02'            uop z2 for nonstd RDW ?
        skp!   1
        cata8  $fili1+180(15),'z2'
 #
        opn    all                     open files
        mvn    $rf,$uopbr              load fixed outsize in rgstr 'f'
        mvn    $rg,$rf                 transfer to $rg
        sub    $rg,1                   -1 byte to insert LF in last byte
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 # begin loop to read variable length records until EOF
 # - using get typ=RDW to get var lth records depending on size in prefix
 # - write fixed length records padded to max 8192 with EBCDIC blanks
 ##man20  bal    getvr     <--Dec19/07 change to get file typ=RDW
 man20  get    fili1,b0                get next record depending on varlth prfx
        skp>   eof
 #
 # test option to translate EBCDIC to ASCII
        tsb    o1(1),x'01'             trnslt EBCDIC to ASCII ?
        skp!   1
        tra    b0($rf8192)
 #
 # presume option h1 to replace 4 byte recsize hdr with numerics
 ##     mvn    c0(4),a0(2bs)  <--Dec19/07 chg to $rv
        mvn    c0(4),$rv               cnvt binary recsize to ASCII numerics
        tsb    o1(1),x'01'             trnslt EBCDIC to ASCII ?
        skp=   1
        tre    c0(4)                   no - trnslt ASCII recsize to EBCDIC
        mvc    c4($rf8192),b0          move data to follow recsize
 #
 # test option h0/h1 drop recsizehdr or replace with numerics ?
        tsb    o8(1),x'01'             replace recsize hdr w numerics ?
        skp=   1
        mvc    c0($rf8192),b0          no - drop recsize hdr
 #
 # test option b1 to convert nulls to ASCII blanks
 man40  tsb    o2(1),x'01'             convert nulls to ASCII blanks ?
        skp!   1
        rep    c0($rf8192),x'00',x'20'

 # test option b2 to convert nulls to EBCDIC blanks
        tsb    o2(1),x'02'             convert nulls to EBCDIC blanks ?
        skp!   1
        rep    c0($rf8192),x'00',x'40'
 #
 # test option t1 to insert Line-Feed Terminator in last byte of record
        tsb    o20(1),x'01'            insert LF in last byte ?
        skp!   1
        mvc    cg0(1),x'0A'            Yes - insert LF in last byte
 #
 # write current record to output file & return to get next
 man48  put    filo1,c0($rf8192)       write record to file#1 out
        skp    man20                   return to get next record
 #
 # EOF - close files & end job
 eof    cls    all
        eoj
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 # getvr - subrtn to get RDW variable length records
 #Dec19/07 - getvr subrtn no longer required
 #         - getvr replaced by file typ=RDW & typ=RDWz2
 #         - this subrtn saved (for interest) in $UV/pf/util/getvr
 #
 # - RDW format with no BDW
 # - record length in 1st 2 bytes of 4 with last 2 x'0000'
 # - record length includes data length + 4 bytes for RDW prefix
 # - blk/rec lth in binary big end format, need 's'witch option for INTEL
 #
 # - read record into area 'a' (1st 4 bytes will be RDW)
 # - sets rgstr 'a' to record length (including the 4 byte RDW)
 # - sets rgstr 'b' to record length (excluding the 4 byte RDW)
 # - return with cc > at EOF
 #
 # note - see alternate job BDW2LST1 to handle BDW+RDW
 #      - includes subrtn 'getvbr' to process BDW+RDW
 #
 # read 4 byte RDW to get record length (-4 for following data)
 getvr   get    fili1,a0(4)          read 4 byte block lth prefix
         skp>   getvr9
         cmc    a2(2),x'0000'        verify RDW (bytes 3&4 null)
         skp!   getvre1
         add    $ca9,1               count blocks (in case errmsg)
         mvn    $ra,a0(2bs)          store rec lth (switch ends INTEL)
         mvn    $rb,$ra              copy to $rb for subtract 4
         sub    $rb,4                -4 for following data length
         skp<=  getvre1              (err if 0 or negative)
 #
 # now read the data following 4 byte RDW
 getvr4  get    fili1,a4($rb8192)    read block (lth in rgstr c)
         skp>   getvr9
 #
 # exit with cc = for normal, or cc > for EOF
 getvr8  ret=
 getvr9  ret>
 #
 # error rtn for invalid block or record prefix
 getvre1 msgv1 'rec prefix 3&4 not x0000, reclth=$ra'
         mvf    g0(80),'1st 32 bytes='
         hxc    g13(64),a0(32)       convert 1st 32 bytes to hex
         msg    g0(80)               display block prefix
         can    'unrecoverable error - job cancelled'

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_8 DATAcnv1 - Mainframe Data file Conversion

Part 8 - Verifying Conversions & Generating Test Files

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8A1. DATAcnv1 - Special Projects

converting EBCDIC back to ASCII

 Part_3 documented generating & executing uvcopy jobs to convert EBCDIC to ASCII
 - 1 job at a time. Part_6 documented the same thing for ALL copybooks & ALL
 data files in a dirctory.

We will now show you how to generate & execute jobs to convert the ASCII files back to EBCDIC (& then compare the reconverted files to the original mainframe files).

You might do this to verify your conversions &/or you might need the ASCII to EBCDIC conversions to return some files to the maifnrame if you can not switch all applications over to unix/linux at the same time.

We could generate the ASCII to EBCDIC jobs using 'uvdata31' the opposite of 'uvdata51' that we used to generate EBCDIC to ASCII, but it is much better to use 'uvdata55' to convert our already generated EBCDIC-->ASCII jobs to ASCII-->EBCDIC jobs.

Besides saving steps, the big advantage is that this method saves having to duplicate any manual coding required for files with multiple Record Types. With the addition of uvdata55, pfx4,& uvcmp1all, our flow charts will then be:

Generate jobs: EBCDIC-->ASCII & ASCII-->EBCDIC

 cpys -------> maps --------> pfx1 --------> pfx2 ---------> pfx3 -------> pfx4
      cobmap1       uvdata51       uvdata52       copy/edit       uvdata55

Execute jobs: EBCDIC-->ASCII, ASCII-->EBCDIC,& compare

 Main -----> d1ebc ----------> d2asc ----------> d3ebc --------> rptcmp
 frame FTP   uvcopyxx 'pfx3/*'  uvcopyxx 'pfx4/*'    uvcmp1all   mismatch rpts

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8A2. converting EBCDIC data-files back to ASCII

generate ASCII-->EBCDIC jobs - 1 file at a time

 #0. Login as cnvdata --> /home/cnvdata
     ================

 #0a. OR Login as yourself if you have a cnvdata dir with required subdirs
      ====================

 #0b. cd cnvdata      <-- change into conversion superdir
      ==========

 #1. mkdir pfx4       <-- make subdir for ASCII-->EBCDIC jobs
     ==========           (if not already existing)

 #2. uvcopy uvdata55,fili1=pfx3/citytax1,filo1=pfx4/citytax1
     =======================================================

generated uvcopy job to convert ASCII to EBCDIC

 # citytax1 - uvcopy job, ASCII to EBCDIC preserve packed, fix signs        #01
 opr='citytax1 - uvcopy code generated from copybook: citytax1 '            #02
 uop=q0                                                                     #03
 was=a33000b33000                                                           #04
 fili1=?d2asc/citytax1,rcs=00128,typ=RSF                                    #05
 filo1=?d3ebc/citytax1,rcs=00128,typ=RSF                                    #06
 @run                                                                       #07
        opn    all                                                          #08
 loop   get    fili1,a0                                                     #09
        skp>   eof                                                          #10
        mvc    b0(00128),a0       move rec to outarea before field prcsng   #11
        tre    b0(00128)          translate entire outarea to EBCDIC        #12
 #      ---                <-- insert R/T tests here for redefined records  #13
        mvc    b88(19),a88             pns post-date:face-value             #14
        trt    b107(9),$trtsae          ns maint-tax                        #15
 put1   put    filo1,b0                                                     #16
        skp    loop                                                         #17
 eof    cls    all                                                          #18
        eoj                                                                 #19

Compare this ASCII to EBCDIC job with the original EBCDIC to ASCII job job listed on page '3B2'. Aside from directory & file names, there are only 2 instructions different as follows:

        tra    b0(00128)          translate entire outarea to ASCII         #12
        tre    b0(00128)          translate entire outarea to EBCDIC        #12

        trt    b107(9),$trtsea     ns maint-tax                             #15
        trt    b107(9),$trtsae          ns maint-tax                        #15

Note that $trtsea is a table to translate Signs Ebcdic to Ascii and that $trtsae is a table to translate Signs Ascii to Ebcdic

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8A3. converting EBCDIC data-files back to ASCII

execute generated job to convert ASCII to EBCDIC

 #4a. uvcopy pfx4/citytax1,fili1=d2asc/citytax1,filo1=d3ebc/citytax1
      ==============================================================

 #4b. uvcopy pfx4/citytax1   <-- easier (filenames default as shown above)
      ====================

sample input ASCII

 #3. uvhd d2asc/citytax1 r128a  <-- display ASCII input file
     =========================

                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 10139923  GEORGE BROWN             1250 EAST PENDER STREET  VANC
             3333333322444544245454222222222222233332445525444452555445225444
             101399230075F275022F7E0000000000000125005134005E45203425540061E3
          64 OUVER           BCV5L1W1......W.........xRL00014626q980601190156
             4554522222222222445343530888005880008800754333333337333333333333
             F5652000000000002365C171029C0072C0047C0182C000146261980601190156
                                                                ^--- ASCII -1

sample output EBCDIC

 #4. uvhd d3ebc/citytax1 r128   <-- display EBCDIC output file
     ==========================

                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 10139923  GEORGE BROWN             1250 EAST PENDER STREET  VANC
             FFFFFFFF44CCDDCC4CDDED4444444444444FFFF4CCEE4DCDCCD4EEDCCE44ECDC
             1013992300756975029665000000000000012500512307554590239553005153
          64 OUVER           BCV5L1W1.bi...Wb....g...yR<00014626J980601190156
             DEECD44444444444CCEFDFEF0888005880008800754FFFFFFFFDFFFFFFFFFFFF
             645590000000000023553161029C0072C0047C0182C000146261980601190156
                                                                ^--- EBCDIC -1

We have shown record #2 because it has a negative zoned sign at byte 115 (the units position of the field in 107-115).

In ASCII a -1 is a 'q' or x'71'. The normal translate EBCDIC to ASCII (tre) would translate it to an EBCDIC 'q' x'98' which would be wrong, so the 'trt' instruction is used with special translate table '$trtsae' to translate Signs Ascii to Ebcdic. This table translates the 'q' to a 'J' (-1 in EBCDIC).

See more details about zoned sign conversion on page '3C2'

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8B1. converting EBCDIC data-files back to ASCII

Verifying Data Conversions

Pages '3C1' thru '3C6' documented converting EBCDIC to ASCII.

Pages '8A1' thru '8A3' documented converting ASCII to EBCDIC.

You could use the ASCII to EBCDIC conversion to verify the accuracy of the EBCDIC to ASCII conversion, by comparing the output of 2nd conversion to the input of the 1st. The 3 test/demo files involved are:

We can't use the unix/linux 'diff' utility because these mainframe style data files are fixed length without the LineFeeds required by diff.

We might try the unix/linux 'cmp' utility. We will use the '-l' option, otherwise it quits at the 1st mismatch.

 cmp -l d1ebc/citytax1 d3ebc/citytax1   <-- compare 2 data files
 ======================================

Byte# file1 file2 116 301 361 372 302 362 500 303 363 628 311 371 756 300 360

The 'cmp' utility shows differences in 3 columns - byte#, 1st file byte contents, 2nd file byte contents. The byte contents are in 'octal' which is an anachronism but there seems to be no hexadecimal option.

A much better solution is to use a Vancouver Utility uvcopy job 'uvcmp1':

 uvcopy uvcmp1,fili1=d1ebc/citytax1,fili2=d3ebc/citytax1
 =========================================================

See the output report on the next page --->

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8B2. converting EBCDIC data-files back to ASCII

uvcmp1 file comparison report

 uvcmp1 - compare 2 files "*" diffs (uops=q1e0p30r256s5t500000v2x0e1r128)
 recsize  reccount   file-size  typ  filename
 1:  128        10       1,280  RSF  d1ebc/citytax1
 2:  128        10       1,280  RSF  d3ebc/citytax1
 2005/04/21_17:24:39   1         2         3         4         5         6
 f#rec#byte# 0123456789012345678901234567890123456789012345678901234567890123
 ============================================================================

 1    1    0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
             1013014000168508559800000000000000018150266550961400000000005153

          64 OUVER           BCV9S1H1 bi   Wb    g   qq 00014906A970530191809
             DEECD44444444444CCEFEFCF0888005880008000770FFFFFFFFCFFFFFFFFFFFF
             645590000000000023592181029C0072C0047C0111C000149061970530191809
                                                                *

 2    1      10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
             1013014000168508559800000000000000018150266550961400000000005153

          64 OUVER           BCV9S1H1 bi   Wb    g   qq 000149061970530191809
             DEECD44444444444CCEFEFCF0888005880008000770FFFFFFFFFFFFFFFFFFFFF
             645590000000000023592181029C0072C0047C0111C000149061970530191809
                                                                *

 1    5    0 10154688  MEL RAHAL                1110 - 625 HOWE STREET   VANC
             FFFFFFFF44DCD4DCCCD4444444444444444FFFF464FFF4CDEC4EEDCCE444ECDC
             1015468800453091813000000000000000011100006250866502395530005153

          64 OUVER           BCV6C2T6 bi   Dh           00011547I961126082351
             DEECD44444444444CCEFCFEF0888004800000000208FFFFFFFFCFFFFFFFFFFFF
             645590000000000023563236029C0048C0000C0153C000115479961126082351
                                                                *

 2    5      10154688  MEL RAHAL                1110 - 625 HOWE STREET   VANC
             FFFFFFFF44DCD4DCCCD4444444444444444FFFF464FFF4CDEC4EEDCCE444ECDC
             1015468800453091813000000000000000011100006250866502395530005153

          64 OUVER           BCV6C2T6 bi   Dh           000115479961126082351
             DEECD44444444444CCEFCFEF0888004800000000208FFFFFFFFFFFFFFFFFFFFF
             645590000000000023563236029C0048C0000C0153C000115479961126082351
                                                                *

The differences shown are not a problem - they are caused by translating zoned signs from EBCDIC to ASCII & back, because there are no + zones in ASCII. The original + zones in EBCDIC x'C_' are lost (not needed) in ASCII where normal numeric zones x'3_' are the only + zones. When we convert back we get the normal numeric zones in EBCDIC x'F_', which cause the mismatches (but these are not errors).

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8C1. converting EBCDIC data-files back to ASCII

generate ASCII-->EBCDIC jobs - for ALL data-files

Actually, the instructions to generate & execute jobs for for ALL data-files are shorter & simpler than for 1 at a time.

 #0. Login as cnvdata --> /home/cnvdata
     ================
 #0a. OR Login as yourself if you have a cnvdata dir with required subdirs
      ====================
 #0b. cd cnvdata      <-- change into conversion superdir
      ==========

 #1. mkdir pfx4       <-- make subdir for ASCII-->EBCDIC jobs
     ==========           (if not already existing)

 #2. uvcopyx uvdata55 pfx3 pfx4 uop=q0i7   <-- generate ALL jobs
     ===================================

Execute ALL jobs to convert ALL data-files

 #3. uvcopyxx 'pfx4/*'                     <-- execute ALL jobs
     =================

Compare ALL re-converted files to original EBCDIC files

 #4. mkdir rptcmp           <-- make subdir for mismatch reprots
     ============

 #5. uvcmp1all d1ebc d3ebc  <-- compare all original files to reconverted files
     =====================

 #6a. vi rptcmp/*          <-- inspect all mismatch reports
      ===========

 #6b. uvlpd12 rptcmp       <-- print all mismatch reports
      ==============

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8D1. Verifying EBCDIC to ASCII Conversion

using uvhdcob to verify numeric fields

You can use the 'v' command of uvhdcob to verify numeric fields (packed & unpacked) in the converted data files. Here is an example for d2asc/citytax1. First we need to update the file with some intentional errors:

Update file with BAD data

 #1. Login --> your homedir

 #2. cdc  <-- alias cdc='cd $CNVDATA' (CNVDATA=$HOME/cnvdata)
     ===    - change to data conversion superdir to perform data conversion

 #3. uvhdcob d2asc/citytax1 maps/citytax1 u  <-- Update citytax1 demo file
     ======================================    - create intentional errs
     - uvhdcob displays 1st record (as follows) & prompts for comamnds

 rec#       1 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 folio                        0   9    10130140
 002 name                        10  34    JOHN HENRY
 003 address                     35  59    1815 BOWEN ROAD
 004 city                        60  79    VANCOUVER
 005 province                    80  81    BC
 006 zip                         82  87    V9S1H1
 007 post-date                   88  91pns 0082898C
 008 land-value                  92  96pns 000057828C
 009 improv-value                97 101pns 000004870C
 010 face-value                 102 106pns 000171710C
 011 maint-tax                  107 115 ns 000149061
 012 purchase-date              116 121    970530
 013 filler001                  122 127    191809

     --> 3              <-- advance to record#3
     --> u 92(1),x'EE'  <-- Update 1st byte of 'land-value' with bad Packed data
     --> u 107(1),'X'   <-- Update 1st byte of 'maint-tax' with bad Numeric data

     --> 6              <-- advance to record#6
     --> u 88(1),x'EE'  <-- Update 1st byte of 'post-date' with bad Packed data
     --> u 115(1),'X'   <-- Update last byte of 'maint-tax' with bad Numeric data

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8D2. Verifying EBCDIC to ASCII Conversion

On the prior page we updated d2asc/citytax1 with BAD data in Packed & Numeric fields in records #3 & #6. Now we will demo the uvhdcob 'v' verify command.

using uvhdcob to verify numeric fields

 #4. uvhdcob d2asc/citytax1 maps/citytax1 u  <-- Update citytax1 demo file
     =============================================
     -->     <-- uvhdcob displays 1st record (not shown) & prompts for command
     --> v   <-- enter 'v' to search/verify (until bad numeric fields found)
               - will display rec#3 (1st record with bad data) as follows:

 rec#       3 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 014 folio                        0   9    10147615
 015 name                        10  34    BONNIE SMITH
 016 address                     35  59    44430 YALE ROAD WEST
 017 city                        60  79    VANCOUVER
 018 province                    80  81    BC
 019 zip                         82  87    V2P6J1
 020 post-date                   88  91pns 0121395C
 021 land-value                  92  96pns EE0039623C            <-- Bad Packed
 022 improv-value                97 101pns 000000000C
 023 face-value                 102 106pns 000051914C
 024 maint-tax                  107 115 ns X00037692   583030303337363932<-BadN
 025 purchase-date              116 121    950601
 026 filler001                  122 127    201031

      v --> 3 recs verified, 3 ERRs found in rec# 3, 1st at byte# 92
      2 packed & 1 numeric bad signs/digits, 0 unprintables in char fields

 ---> vv  <-- 'vv' command to continue verify

 rec#       6 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 014 folio                        0   9    10173320
 015 name                        10  34    SAM SMITH
 016 address                     35  59    2381 ROGERS AVE
 017 city                        60  79    VANCOUVER
 018 province                    80  81    BC
 019 zip                         82  87    V3K5Y2
 020 post-date                   88  91pns EE50498C              <-- Bad Packed
 021 land-value                  92  96pns 000060600C
 022 improv-value                97 101pns 000000000C
 023 face-value                 102 106pns 000169086C
 024 maint-tax                  107 115 ns 00015620X   303030313536323058<-BadN
 025 purchase-date              116 121    970530
 026 filler001                  122 127    191810

      vv --> 3 recs verified, 3 ERRs found in rec# 6, 1st at byte# 88
      2 packed & 1 numeric bad signs/digits, 0 unprintables in char fields

 ---> vv  <-- 'vv' command to continue verify

      vv --> 4 records verified, No errors found
      *** EOF reached (enter -1 to see last rec)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8E1. Verifying EBCDIC to ASCII Conversion

genverify1 - generate jobs to Verify Packed/Numeric Fields

Verifying Numeric & Packed fields is the best way to detect any discrepancy between the copybook layouts & the actual data - especially effective for files with packed fields, where the unique sign codes must be in the last byte of each packed field.

We will demonstrate using the 'citytax1' file shown on the previous page where we used 'uvhdcob' to Verify Packed & Numeric fields.

Why do we need 'genverify1' if we can use uvhdcob ? - because uvhdcob displays & verifies ONLY the 1st field of any OCCURS. uvhdcob would not be conclusive for files such as custmas1 which has occurs 12 for this year & last year sales.

Generate jobs to verify All numeric fields in All files

 #1. Login --> your homedir

 #2. cdc  <-- alias cdc='cd $CNVDATA' (CNVDATA=$HOME/cnvdata)
     ===    - change to data conversion superdir to perform data conversion

 #3. mkdir pfv1 pfv2 pfv3 rptver            <-- make subdirs for outputs
     ===========================

 #4. uvcopy cobmap1,fili1=cpys/citytax1,filo1=maps/citytax1
     ======================================================
     - convert copybook to 'cobmap' (record layout)

 #5. uvcopy genverify1,fili1=maps/citytax1,filo1=pfv1/citytax1
     =========================================================
     - convert cobmap to uvcopy job to verify numeric/packed fields

Alternative 'uvcopyx' script to verify All files

 #4a. uvcopyx cobmap1 cpys maps uop=q0i7p0
      ====================================
      - convert all COBOL copybooks to layouts

 #5a. uvcopyx genverify1 maps pfv1 uop=q0i7f0
      =======================================
      - convert all cobmaps to data verify jobs

 #6. uvcopy uvdata52,fili1=ctl/ctlfile54,fild2=pfv1,fild3=pfv2,uop=r1s1t0
     ====================================================================
     - insert datafilenames (vs copybook names)
     - see options explained on page '6D4'

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8E2. genverify1 - generate jobs to Verify Packed/Numeric Fields

sample cobmap Input to genverify1

 cobmap1  start-end bytes for cobol record fields    200306092029  pg# 0001
 cpys/citytax1                  citytax1           RCSZ=00128  bgn  end  lth typ
  01  citytax1.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-3.          0088 0091  004pns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  x(00006).                 0116 0121  006
      10 filler001             pic  x(00006).                 0122 0127  006
 *RCSZ=00128                                                       0128

sample genverify1 Output job to verify numeric fields

 # uvcopy job to verify packed or zoned numeric fields
 opr='jobname=JOBNAME   - pfv2 name=datafilename'
 opr='copybook=citytax1 - maps/pfv1 name=copybookname'
 uop=q0,was=a33000t80000
 fili1=?d2asc/citytax1,rcs=00128,typ=RSF
 filo1=?rptver/citytax1,rcs=128,typ=LSTt
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        vnft1  a88(4p),1,'post-date'
        vnf    a92(5p),1,'land-value'
        vnf    a97(5p),1,'improv-value'
        vnf    a102(5p),1,'face-value'
        vnf    a107(9z),1,'maint-tax'
        wtbe   filo1,t0(80),t0(80)
        skp    loop
 #
 eof    cls    all
        eoj

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8E3. genverify1 - generate jobs to Verify Packed/Numeric Fields

Execute generated job to Verify Numeric fields

 #6. uvcopy pfv1/citytax1   <-- execute job to verify Packed/Numeric fields
     ====================

 #7. cat rptver/citytax1    <-- display report (created in rptver/citytax1)
     ===================

 lth(p=packed)   hex-representation   numeric-data rec# field-name
 =================================================================

 05p                     EE0039623C                 003 land-value
 09              583030303337363932       X00037692 003 maint-tax

 04p                       EE50498C                 006 post-date
 09              303030313536323058       00015620X 006 maint-tax

uvhdcob rec# 003 - for comparison

 rec#       3 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 014 folio                        0   9    10147615
 015 name                        10  34    BONNIE SMITH
 016 address                     35  59    44430 YALE ROAD WEST
 017 city                        60  79    VANCOUVER
 018 province                    80  81    BC
 019 zip                         82  87    V2P6J1
 020 post-date                   88  91pns 0121395C
 021 land-value                  92  96pns EE0039623C            <-- Bad Packed
 022 improv-value                97 101pns 000000000C
 023 face-value                 102 106pns 000051914C
 024 maint-tax                  107 115 ns X00037692   583030303337363932<-BadN
 025 purchase-date              116 121    950601
 026 filler001                  122 127    201031

1. The genverify1 reports have very little description, but are very compact, which is important when you are generating error reports for big files.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8F1. Verifying conversions by Verifying Numeric fields

genverify1 - ex#2 to demo occurs verify

Example#1 (citytax1) had no occurs fields. For example#2 we will use 'custmas1' which has occurs fields for 12 months sales this year & last. Here are the illustrated procedures to generate & execute the 'genverify1' job for custmas1.

 #1. Login --> your homedir

 #2. cdc  <-- alias cdc='cd $CNVDATA' (CNVDATA=$HOME/cnvdata)
     ===    - change to data conversion superdir to perform data conversion

 #3. uvcopy cobmap1,fili1=cpys/custmas1,filo1=maps/custmas1
     ======================================================
     - convert copybook to 'cobmap' (record layout)

cobmap (record layout) for custmas1

 cobmap1  start-end bytes for cobol record fields    200902200736  pg# 0001
 cpys/custmas1.cpy                                 RCSZ=00256  bgn  end  lth typ
 * custmas1 - cobol copy book for custmas1 file (see uvtrain.
      10 cm-num           pic  9(6).                          0000 0005  006 n  6
      10 cm-delete        pic  x(4).                          0006 0009  004
      10 cm-name          pic  x(25).                         0010 0034  025
      10 cm-adrs          pic  x(25).                         0035 0059  025
      10 cm-city          pic  x(16).                         0060 0075  016
      10 filler001        pic  x.                             0076 0076  001
      10 cm-prov          pic  x(2).                          0077 0078  002
      10 filler002        pic  x.                             0079 0079  001
      10 cm-postal        pic  x(10).                         0080 0089  010
      10 cm-phone         pic  x(12).                         0090 0101  012
      10 cm-contact       pic  x(18).                         0102 0119  018
      10 cm-thisyr-sales  pic  s9(7)v99 comp-3 occurs 12.     0120 0124  005pns 9
      10 cm-lastyr-sales  pic  s9(7)v99 comp-3 occurs 12.     0180 0184  005pns 9
      10 filler003        pic  x(16).                         0240 0255  016
 *RCSZ=00256

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8F2. Verifying conversions by Verifying Numeric fields

uvhdcob Verify custmas1

We want to demonstrate that 'genverify1' generates uvcopy jobs that will verify ALL packed/numeric fields vs 'uvhdcob' which can only verify the 1st field of any 'occurs'.

So, before we demo 'genverify1' (on the next page), we will 1st show you the 'uvhdcob' verify on this page.

 #4. uvhdcob d2asc/cutsmas1 maps/custmas1   <-- execute uvhdcob for custmas1
     ====================================

 --> v  <-- search & Verify character,numeric,& packed fields

 rec#      10 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 cm-num                       0   5 n  15468X      313534363858      <-BadN
 002 cm-delete                    6   9
 003 cm-name                     10  34    TAURUS RESOURCES
 004 cm-adrs                     35  59    1110 - 625 HOWE STREET
 005 cm-city                     60  75    VANCOUVER
 006 filler001                   76  76
 007 cm-prov                     77  78    B.          42EE              <-BadC
 008 filler002                   79  79
 009 cm-postal                   80  89    V6C2T6
 010 cm-phone                    90 101    604 685 8565
 011 cm-contact                 102 119    MEL RAHAL
 012 cm-thisyr-sales       012  120 124pns EE0123456C            <-- Bad Packed
 013 cm-lastyr-sales       012  180 184pns 000000000C
 014 filler003                  240 255    C 19980331

      v --> 10 recs verified, 4 ERRs found in rec# 10, 1st at byte# 0
      2 packed & 1 numeric bad signs/digits, 1 unprintables in char fields

--> vv <-- continue Verify

 rec#      20 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 cm-num                       0   5 n  2343XX      323334335858      <-BadN
 002 cm-delete                    6   9
 003 cm-name                     10  34    TERRITORIAL REWIND
 004 cm-adrs                     35  59    BOX 1648
 005 cm-city                     60  75    WHITEHORSE
 006 filler001                   76  76
 007 cm-prov                     77  78    YK
 008 filler002                   79  79
 009 cm-postal                   80  89    Y1A...      593141EEEEEE      <-BadC
 010 cm-phone                    90 101    403-512-3948
 011 cm-contact                 102 119
 012 cm-thisyr-sales       012  120 124pns 000000000C
 013 cm-lastyr-sales       012  180 184pns 1E2E3E4E5E            <-- Bad Packed
 014 filler003                  240 255      20021130

      vv --> 10 recs verified, 10 ERRs found in rec# 20, 1st at byte# 0
      5 packed & 2 numeric bad signs/digits, 3 unprintables in char fields

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8F3. Verifying conversions by Verifying Numeric fields

Generate uvcopy verify job for custmas1

 #5. uvcopy genverify1,fili1=maps/custmas1,filo1=pfv1/custmas1
     =========================================================
     - convert cobmap to uvcopy job to verify numeric/packed fields

 # uvcopy job to verify packed or zoned numeric fields
 opr='jobname=JOBNAME   - pfv2 name=datafilename'
 opr='copybook=custmas1 - maps/pfv1 name=copybookname'
 uop=q0,was=a33000t80000
 fili1=?d2asc/custmas1,rcs=00256,typ=RSF
 filo1=?rptver/custmas1,rcs=128,typ=LSTt
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        vnft1  a0(6z),1,'cm-num'
        vnf    a120(5p),12,'cm-thisyr-sales'
        vnf    a180(5p),12,'cm-lastyr-sales'
        wtbe   filo1,t0(80),t0(80)
        skp    loop
 eof    cls    all
        eoj

Execute Verify job for custmas1

 #5. uvcopy pfv1/custmas1     <-- execute verify job for custmas1
     ====================

 #6. uvlp12 rptver/custmas1   <-- print report
     ======================

 lth(p=packed)   hex-representation   numeric-data rec# occurs# field-name
 =================================================================

 06                    313534363858          15468X 010 cm-num
 05p                     EE0123456C                 010 01 cm-thisyr-sales
 05p-                    E01234567D                 010 12 cm-lastyr-sales

 06                    323334335858          2343XX 020 cm-num
 05p                     EE0012345C                 020 03 cm-thisyr-sales
 05p                     1E2E3E4E5E                 020 01 cm-lastyr-sales
 05p                     00000000EE                 020 12 cm-lastyr-sales

1. Compare the errors reported here by the genverify job to those reported by uvhdcob on the previous page.
   
   
2. genverify reports errors in rec# 010 occurs# 12 & in rec# 020 occurs 03 & 12 which were missed by uvhdcob since it only verifies the 1st field of occurs.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8G1. Verifying Conversions by Accumulating All Numeric fields

genacum2 - generate jobs to accum ALL numeric fields

'genacum2' is a Vancouver Utility to generate uvcopy jobs to accumulate all numeric fields (packed/zoned/binary) in 1 or all data files. The jobs are generated automatically from the 'cobmap' record layout (which is generated from the COBOL copybook).

You might use these reports to verify data file conversions, OR to get a 2nd opinion on suspicious looking totals on your COBOL reports.

We will demonstrate using the 'custmas1' copybook & datafile, which has occurs fields for 12 months sales this year & last. Here are the illustrated procedures to generate & execute the 'genacum2' job for custmas1.

 #1. Login --> your homedir

 #2. cdc  <-- alias cdc='cd $CNVDATA' (CNVDATA=$HOME/cnvdata)
     ===    - change to data conversion superdir to perform data conversion

 #3. mkdir pfa1 pfa2 pfa3   <-- make subdirs for genacum2 jobs
     ====================

Directories Required

 /home/cnvdata
 :-----cpys/          <-- copybooks
 :     :----custmas1    - copybook 'custmas1' for datafile 'custmas1'
 :-----d2asc/         <-- data files (previously converted to ASCII)
 :     :-----custmas1   - datafile 'custmas1'
 :-----maps/          <-- record layouts (cobmaps) created from copybooks
 :     :-----custmas1   - cobmap 'custmas1'
 :-----pfa1/          <-- uvcopy jobs generated from cobmaps by 'genacum2'
 :     :-----custmas1   - uvcopy job 'custmas1' to add all fields in datafile
 :-----pfa2/          <-- reserved in case you use uvdata52 insert datafilenames
 :-----pfa3/          <-- uvcopy jobs copied here before execution
 :     :                - modified if required (if multiple record types)
 :-----rptacu/        <-- reports written here
 :     :-----custmas1   - reports named same as copybook & datafile
                          (run uvdata52 to change to datafilenames)

1. This is similar to prior conversions (EBCDIC to ASCII or data validate)
   
   
2. We will create subdirs pfa1,pfa2,pfa3 for the generated 'acum' jobs using the same filename for copybook,datafile,cobmap,& uvcopy acum job which simplifies the generation scripts & operating instructions
   
   
3. We can run the generated job from pfa1 (still named same as copybook) The pfa1/... job expects input datafile in d2asc/ & writes report to rptacu/
   
   
4. For major projects we would setup a control file to relate the copybook names to your actual datafilenames & run 'uvdata52' which copies pfa1/... jobs to pfa2/... inserting datafilenames & renaming same as datafiles.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8G2. Verifying Conversions by Accumulating All Numeric fields

convert copybook to cobmap (prelim to genacum2)

 #4. uvcopy cobmap1,fili1=cpys/custmas1,filo1=maps/custmas1
     ======================================================
     - convert copybook to 'cobmap' (record layout)

cobmap (record layout) for custmas1

 cobmap1  start-end bytes for cobol record fields    200902200736  pg# 0001
 cpys/custmas1.cpy                                 RCSZ=00256  bgn  end  lth typ
 * custmas1 - cobol copy book for custmas1 file (see uvtrain.
      10 cm-num           pic  9(6).                          0000 0005  006 n  6
      10 cm-delete        pic  x(4).                          0006 0009  004
      10 cm-name          pic  x(25).                         0010 0034  025
      10 cm-adrs          pic  x(25).                         0035 0059  025
      10 cm-city          pic  x(16).                         0060 0075  016
      10 filler001        pic  x.                             0076 0076  001
      10 cm-prov          pic  x(2).                          0077 0078  002
      10 filler002        pic  x.                             0079 0079  001
      10 cm-postal        pic  x(10).                         0080 0089  010
      10 cm-phone         pic  x(12).                         0090 0101  012
      10 cm-contact       pic  x(18).                         0102 0119  018
      10 cm-thisyr-sales  pic  s9(7)v99 comp-3 occurs 12.     0120 0124  005pns 9
      10 cm-lastyr-sales  pic  s9(7)v99 comp-3 occurs 12.     0180 0184  005pns 9
      10 filler003        pic  x(16).                         0240 0255  016
 *RCSZ=00256

On the next page, we will run genacum2 to generate a job to add all fields in test file d2asc/custmas1 (based on the copybook converted to a cobmap as shown on the previous page).

But first (to understand the datafile), you might run 'uvhdcob' which displays the data field contents beside the COBOL copybook fieldnames.

 #5. uvhdcob d2asc/cutsmas1 maps/custmas1  <-- execute uvhdcob for custmas1
     ====================================

uvhdcob display - 1st record of d2asc/custmas1

 rec#       1 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 cm-num                       0   5 n  130140
 002 cm-delete                    6   9
 003 cm-name                     10  34    EVERGREEN MOTORS LTD.
 004 cm-adrs                     35  59    1815 BOWEN ROAD
 005 cm-city                     60  75    NANAIMO
 006 filler001                   76  76
 007 cm-prov                     77  78    BC
 008 filler002                   79  79
 009 cm-postal                   80  89    V9S1H1
 010 cm-phone                    90 101    250-754-5531
 011 cm-contact                 102 119     LARRY WRENCH
 012 cm-thisyr-sales       012  120 124pns 001234567C
 013 cm-lastyr-sales       012  180 184pns 001234567D
 014 filler003                  240 255    C 19950531

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8G3. Verifying Conversions by Accumulating All Numeric fields

Generate 1 job to accum ALL num-fields in 1 file

 #6. uvcopy genacum2,fili1=maps/custmas1,filo1=pfa1/custmas1,uop=f0
     ==============================================================
     - convert cobmap to uvcopy job to accumulate All numeric fields

console prompts for options

 uop=f2m1 - option defaults (for generated jobs)
     f0   - Input file default typ=RSF fixed lth (in generated job)
     f2   - Input typ=RDWz2 (2 byte BIG-end binary recsize hdrs)
     f4   - Input file typ=RDWz4 (4 byte hdrs, bytes 3&4 nulls)
       m1 - read copybook record layouts from maps/... in $CNVDATA
       m2 - read copybook record layouts from maps/... in $RUNLIBS
 User OPtion (uop) defaults  = q1f2m1
 -->null to accept or enter/override --> f0  <-- enter 'f0' for typ=RSF

generated job pfa1/custmas1

 # uvcopy job to accumulate all numeric fields in a datafile
 opr='jobname=JOBNAME   - pfa2 name=datafilename'
 opr='copybook=custmas1 - maps/pfa1 name=copybookname'
 uop=q0i3,rop=r1  # minimize startup msgs, prompt for report disposition
 was=a33000c200000t100000  # 33K maxrec, 200K cpybkmap, 100K acum tbl
 fili1=?d2asc/custmas1,rcs=00256,typ=RSF
 filo1=?rptacu/custmas1,rcs=128,typ=LSTt
 fili2=${CNVDATA}/maps/custmas1,rcs=128,typ=LST
 @run
        opn    all
        rtb    fili2,c0(100),c0(100)  read cobmap into area c table
        clr    t0(100000),x'00'       clear area t for accumulator table
 loop   get    fili1,a0               get next record into area 'a'
        skp>   eof
        add    $cb3,1                 count all record types
 #      ---    ---      <-- insert any Record Type tests here
        mvn    $rt,0                  init register t for current record
        mvn    $rs,0                  init register s for current record
        add    ts9800(8b),1           count 1st (or only) R/T in copybook
        bal    acum,'0,6,z,1,cm-num'
        bal    acum,'120,5,p,12,cm-thisyr-sales'
        bal    acum,'180,5,p,12,cm-lastyr-sales'
        skp    loop
 eof    bal    dump                   dump tables of fieldnames & acummulators
        cls    all
        eoj
 @pf2=acum2.sub

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8G4. Verifying Conversions by Accumulating All Numeric fields

Execute job to acum all numeric fields in custmas1

 #7. uvcopy pfa1/custmas1     <-- execute job to add all numeric fields custmas1
     ====================

 #8. uvlp12 rptacu/custmas1   <-- print report
     ======================

sample report

 genacum2  RCSZ=00256 DataFile=d2asc/custmas1
 2009/02/26_09:22:03  copybook=cpys/custmas1.cpy
 line# Field-Name                 occurs start lth type         field-total

    1  cm-num                             0005 006  n            7,259,353
    2  cm-thisyr-sales                12  0124 005 pns           99,697.93
    3  cm-lastyr-sales                12  0184 005 pns           45,440.85

          Record count        32 **      Hash-Total             21,773,231 **

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8G5. Verifying Conversions by Accumulating All Numeric fields

generate jobs for ALL files

The above procedures were for 1 file at a time. For major projects we would want to generate jobs for all copybooks & all datafiles. Actually the operating instructions are shorter & easier to key.

 #1. Login --> your homedir

 #2. cdc  <-- alias cdc='cd $CNVDATA' (CNVDATA=$HOME/cnvdata)
     ===    - change to data conversion superdir to perform data conversion

 #3. uvcopyx cobmap1 cpys maps uop=q0i7p0
     ====================================
     - convert COBOL copybooks to layouts (field start/end on right)

 #4. uvcopyx genacum2 maps pfa1 uop=q0i7f0
     =====================================
     - convert cobmaps to uvcopy jobs to accumulate all numeric fields
     - I/O datafiles coded as copybook names

 #5. vi ctl/ctlfile54  <-- edit control file to relate copybooks to datafiles
     ================

      # ctlfile54 - sample control file to relate copybooknames & datafilenames
      ar/customer.master       cpy=custmas1 rcs=00256 typ=RSF
      gl/account.master_       cpy=acntmas  rcs=00128 typ=RDWz2
      ap/vendor.master         cpy=vendmas  rcs=00256 rcs=257 keys=0(10)

 #6. uvcopy uvdata52,fili1=ctl/ctlfile54,fild2=pfa1,fild3=pfa2,uop=r1s1t0
     ====================================================================
     - insert datafilenames (vs copybook names)
     - option r1 = override fili1 recsize with recsize from control file
              s1 = set input typ=... from control file
              t0 = do not change output file type (typ=LSTt from genverify1)

 #7. cp pfa2/* pfa3
     ==============
     - copy completed uvcopy data conversion jobs to execution subdir

 #8. vi pfa3/*          <-- edit for files with multiple record Types
     =========            - need to insert R/T test code & skip to
                            corresponding bank of auot generated code

 #9. uvcopyxx 'pfa3/*'  <-- execute all jobs to acum all num-fields in all files
     =================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8H1. DATAcnv1 - Convert EBCDIC to ASCII with COBOL copybooks

gentest1 - generate test data files based on COBOL copybooks

'gentest1' will generate uvcopy jobs to generate test data files based on the COBOL copybooks. The cobol copybooks are 1st converted to the 'cobmap' record layouts by the 'cobmap1' utility.

The generated uvcopy jobs are named after the copybooks and they call sub- routines to generate data for each field. Each picture field in the copybook generates a 'bal' to a subrtn, depending on field type (type X, packed, binary, signed/unsigned numeric). For example:

   bal  genx,'a00035b0025','address'    <-- sample call to subrtn
   =================================

This calls subrtn 'genx' for a type X field starting at byte 35 (option a00035) that is 25 bytes long (option b0025), whose fieldname is 'address'.

Here is a summary of the subrtns called & a description of the actions performed for the various field types:

Calling subrtns keeps the generated uvcopy jobs short since most of the field processing code is in these sub-routines. The subrtn module is called via the following code, which you can see at the end of the sample job listed on the next page.

@pf2=gentest1.sub <-- code to include subfunctions in uvcopy jobs

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8H2. gentest1 - generate test data files based on COBOL copybooks

gentest1 - sample input

 cobmap1  start-end bytes for cobol record fields    200408220851  pg# 0001
 cpys/citytax1                  citytax1           RCSZ=00128  bgn  end  lth typ
 * citytax1 copybook, from signscan1 + edit meaningful names
  01  citytax1.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-3.          0088 0091  004pns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  x(00006).                 0116 0121  006
      10 filler001             pic  x(00006).                 0122 0127  006
 *RCSZ=00128                                                       0128

sample output

 opr='JOBNAME citytax1 - generated by cobmap1 & gentest1'
 opr='uop=c100   - option "c" default record output count = 100'
 opr='           - may change at prompt (ex: enter "c2000" for 2000 records)'
 uop=q1c100      # option defaults
 was=a9000b9000
 filo1=?tmp/citytax1,rcs=00128,typ=RSF
 @run
        opn    all
        bal    initj                 inits for job (setup A-Z 0-9 patterns)
 # begin loop to generate fields & output each record
 loop   bal    initr                 inits for record (counts records,etc)
        clr    b0(00128),' '         init output area to all blanks
 #      ---                          <-- insert R/T tests if redef records
        bal    genx,'a00000b0010','folio'
        bal    genx,'a00010b0025','name'
        bal    genx,'a00035b0025','address'
        bal    genx,'a00060b0020','city'
        bal    genx,'a00080b0002','province'
        bal    genx,'a00082b0006','zip'
        bal    genp,'a00088b0004','post-date'
        bal    genp,'a00092b0005','land-value'
        bal    genp,'a00097b0005','improv-value'
        bal    genp,'a00102b0005','face-value'
        bal    gens,'a00107b0009','maint-tax'
        bal    genx,'a00116b0006','purchase-date'
        bal    genx,'a00122b0006','filler001'
 #
 put1   put    filo1,b0              output current record
        cmn    $ca1,$uopbc           reached output Count ?
        skp<   loop
 eof    cls    all
        eoj
 @pf2=gentest1.sub

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8H3. gentest1 - generate test data files based on COBOL copybooks

Op-Instrns for citytax1 demo file

 1. uvcopy cobmap1,fili1=cpys/citytax1,filo1=maps/citytax1
    ======================================================
           - convert the COBOL copybook into a 'cobmap' (record layout)

 2. uvcopy gentest1,fili1=maps/citytax1,filo1=pfg1/citytax1
    =======================================================
           - generate the uvcopy job from the cobmap

 3. uvcopy pfg1/citytax1,filo1=tmp/citytax1  <-- execute the generated job
    =======================================

 3a. uvcopy pfg1/citytax1    <-- same (output file defaults as shown above)
     ====================

console displays for option changes

User OPtion (uop) defaults = q1c100 null to accept or re-specify (1 or more) --> c500 <-- may change default

 tmp/citytax1 = default filo01 - null accept or reenter -->   <-- null accept

 citytax1:040823:105849: EOF filo01 0 rds, 100 wrts, 12800 size; tmp/citytax1

 4. uvhd filename=tmp/citytax1 r128          <-- examine the output
    ===============================

Please see the 'uvhd' output display shown on the next page --->

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8H4. gentest1 - generate test data files based on COBOL copybooks

Here is the 'uvhd' command repeated from the previous page, but with option 's7' added to space out the display to make it easier to read.

 4. uvhd filename=tmp/citytax1 r128s7          <-- examine the output
    =================================

output from citytax1 demo file

                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123

           0 folio     name                     address                  city
             6666622222666622222222222222222222266676772222222222222222226677
             6FC9F00000E1D500000000000000000000014425330000000000000000003949

          64                 przip   ...................000000001purchafiller
             2222222222222222777672220001000010000100001333333333777666666667
             000000000000000002A90000000C0000C0000C0000C00000000105238169CC52

                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123

         128 AAAAAAAAAABBBBB000002BBBBBBBBBBBBBBCCCCC000002CCCCCCCCCCCCCCDDDD
             4444444444444443333334444444444444444444333333444444444444444444
             1111111111222220000022222222222222233333000002333333333333334444

          64 D000002DDDDDDDDDEEFFFFFF...,....,....,....,000000002GGGGGGHHHHHH
             4333333444444444444444440002000020000200002333333333444444444444
             400000244444444455666666000C0000C0000C0000C000000002777777888888

                      10        20        30        40        50        60
 r#        3 0123456789012345678901234567890123456789012345678901234567890123

         256 AAAAAAAAAABBBBB000003BBBBBBBBBBBBBBCCCCC000003CCCCCCCCCCCCCCDDDD
             4444444444444443333334444444444444444444333333444444444444444444
             1111111111222220000032222222222222233333000003333333333333334444

          64 D000003DDDDDDDDDEEFFFFFF...=....=....=....=00000000sGGGGGGHHHHHH
             4333333444444444444444440003000030000300003333333337444444444444
             400000344444444455666666000D0000D0000D0000D000000003777777888888

Please relate this output to the copybook & the rules listed previously:

• every 10th record (1st, 11th,etc) will have the field names inserted
  into type 'X' alphanumeric fields (not numeric or packed fields).
• otherwise, type 'X' fields will be filled with 'A's, 'B's, etc.
• type 'X' fields longer than 15 bytes, have the current record# inserted in
  bytes 5-10.
• unpacked fields will receive the current record# if longer than 5 bytes
• otherwise unpacked fields will be filled with '1's, '2's, etc.
• packed fields will receive the current record#
• packed & signed fields in every 3rd record will be negative
• note the x'0000003D' packed fields & the '00000000s' signed field (s=-3)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8H5. gentest1 - generate test data files based on COBOL copybooks

'gentestA' script

The 'gentestA' script is provided to make it easier to run the gentest1 utility. The script includes both the cobmap1 & gentest1 uvcopy jobs, reducing the operating instructions to 1 line:

 gentestA citytax1      - generates uvcopy job --> pfg1/citytax1
 =================

 uvcopy pfg1/citytax1,filo1=tmp/citytax1  <-- execute the generated job
 =======================================

Op-Instns for all files in directory

 #1. uvcopyx cobmap1 cpys maps uop=q0i7p0  <-- create cobmaps for all copybooks
     ====================================

 #2. uvcopyx gentest1 maps pfg1 uop=q0i7   <-- create data gen jobs for all maps
     ===================================

 #3. uvcopyX 'pfg1/*'                      <-- execute all data generate jobs
     ================

You might want to generate all test data generation jobs as shown in #1 & #2 above, but it is unlikely you would want to execute all at once because they would be generated in the 'tmp' subdir & all default to 100 records.

The uvcopy code for gentest1 & gentest1.sub is not shown here or on the web site documentation, but if you have Vancouver Utilities installed, you may find the code as follows:

 /home/uvadm/pf/IBM/gentest1  - uvcopy code to generate the uvcopy jobs
                                to generate the test data files from copybooks

 /home/uvadm/pf/IBM/gentest1.sub - uvcopy code for the sub-routines called
                                   by the generated uvcopy jobs

Note that the generated jobs would be in your sub-directories. If you had followed the recommended setup procedures (setting up userid 'cnvdata'), this would be:

/home/cnvdata/pfg1/xxxxxx

Each of the generated jobs loads the subrtns via the code found on the last line of each generated job, which is:

 @pf2=gentest1.sub          <-- loads the subrtn code

The subrtns will be loaded from /home/uvadm/pf/IBM/gentest1.sub. If you want to modify the subrtns, you should 1st copy to your own 'pf' directory & place your directory ahead of the 'uvadm' pf directory in the PFPATH variable that uvcopy uses to load uvcopy job code. Do not modify in the uvadm pf directory, because any later update install of Vancouver Utilities would overwrite your modified code.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_9 DATAcnv1 - Convert EBCDIC to ASCII with COBOL copybooks

Part 9 - Summary of Data conversion Scripts - Contents

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

9A1. DATAcnv1 - Converting Mainframe Data Files from EBCDIC to ASCII

Summary of scripts used in DATAcnv1.doc

Only a few '*' are listed on the following pages, but you can view or print any of them from /home/uvadm/sf/IBM/...

 *gencnvD1 - generate all conversion jobs (for sites with no COBOL copybooks)
           - signscan1, sign2cpy1, cobmap1, uvdata51, genpipe1, sqlcreate1

 *gencnvD2 - generate all conversion jobs from COBOL copybook
           - cobmap1, uvdata51, genpipe1, sqlcreate1 (Oracle)

 *cobmapA - convert 1 COBOL copybook to cobmap (record layout), adding
           field start,end,length,& type on the right hand side of the copybook.
           The cobmap record layout is the input to the following scripts.

 *gencnvA - generate 1 uvcopy job to translate EBCDIC file to ASCII,
         - preserving packed fields & correcting any zoned decimal signs
           from EBCDIC conventions to ASCII conventions.

 *uvcmp1all - compare all files in 2 subdirs
            - use to compare reconverted EBCDIC files to original mainframe files

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

9B1. DATAcnv1 - Converting Mainframe Data Files from EBCDIC to ASCII

Summary of uvcopy jobs used in DATAcnv1.doc

No uvcopy jobs are listed here, but you can view or print any of them from /home/uvadm/sf/IBM/...

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

9C1. scripts used for Data file conversion

cobmapA - generate record layout from COBOL copybook

 #!/bin/ksh
 # cobmapA - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # cobmapA - generate record layout from COBOL copybook
 #
 # This script makes it easier to run cobmap1 uvcopy job
 # /home/uvadm/sf/cobmapA - script files Vancouver Utility subdir 'sf'
 # /home/uvadm/pf/cobmapA - uvcopy jobs in subdir 'pf' (parameter files)
 #
 if [[ ! -f cpys/"$1" ]]; then
    echo "usage: cobmapA copybookname "
    echo "       ==================== "
    echo " - arg1 must be a file within subdir 'cpys' (specify filename only)"
    echo " - output will be written into subdir 'maps' with same filename"
    exit 1; fi
 #
 uvcopy cobmap1,fili1=cpys/$1,filo1=maps/$1,uop=q0i7p0
 #====================================================
 #
 echo "record layout created, you may view or print as follows:"
 echo "more maps/$1     - display output"
 echo "uvlp12 maps/$1   - print output (at 12cpi with UV laser printer script)"
 #
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

9D1. scripts used for Data file conversion

gencnvA - generate uvcopy job to convert EBCDIC to ASCII

 #!/bin/ksh
 # gencnvA - Generate uvcopy job to convert 1 data file EBCDIC to ASCII
 #         - see /home/uvadm/doc/MVSDATA.doc or VSEDATA.doc
 #         - or www.uvsoftware.ca/mvsdata.htm or vsedata.htm
 #
 # gencnvA must run from conversion superdir containing subdirs cpys,maps,pfx1
 # (cpys=copybooks, maps=layouts from copybooks, pfx1=uvcopy jobs from cobmaps)
 #
 # gencnvA script executes 2 uvcopy jobs:
 # 1. cobmap1  - copy cpys to maps, generating record layouts (cobmaps)
 # 2. uvdata51 - generates uvcopy skeleton2 job from cobmap record layouts
 #
 # Note difference between scripts gencnvA & gencnvB
 # - gencnvA executes only cobmap1 & uvdata51 (not uvdata52), creating output
 #   in pfx1, that assumes datafilenames are same as copybooknames
 # - gencnvB executes an extra step (uvdata52) which replaces copybooknames
 #   with actual datafilenames using a control file to relate them
 #
 echo "gencnvA using: cpys,maps,pfx1"
 if [[ ! -f "cpys/$1" ]]; then
    echo "usage: gencnvA CopyBookName"
    echo "       ===================="
    echo " - arg1 ($1) not found in cpys subdir"
    exit 9; fi
 #
 uvcopy cobmap1,fili1=cpys/$1,filo1=maps/$1,uop=q0i7p0
 #====================================================
 #
 uvcopy uvdata51,fili1=maps/$1,filo1=pfx1/$1,uop=q0i7
 #===================================================
 #
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

9E1. scripts used for Data file conversion

gencnvD2 - generate all conversion jobs for unix/linux

 # gencnvD2 - generate conversion jobs for Unix/Linux systems
 #          - cobmap1,uvdata51,genpipe1,sqlcreate1
 #          - assumes COBOL copybook available in subdir cpys
 #          - alternate script gencnvD1 will scan data to create copybook
 #          - by Owen Townsend, Feb 24/2004
 #          - see DATAcnv1.doc or www.uvsoftware.ca/datacnv1.htm
 #
 # gencnvD2 filename            <-- command format
 # =================
 #
 # gencnvD2 citytax1            <-- example (from DATAcnv1.doc)
 # =================
 #
 f="$1"; r="$2";
 #
 if [[ -f cpys/"$f" ]]; then :
    else echo "arg1 not a file"
         echo "usage:   gencnvD2 filename"
         echo "         ================="
         echo "example: gencnvD2 citytax1"
         echo "         ================="
         exit 1; fi
 #
 uvcopy cobmap1,fili1=cpys/$f,filo1=maps/$f,uop=q0p0
 #==================================================
 uvcopy uvdata51,fili1=maps/$f,filo1=pfx1/$f,uop=q0
 #=================================================
 uvcopy genpipe1,fili1=maps/$f,filo1=pfp1/$f,uop=q0,rop=r0
 #========================================================
 uvcopy sqlcreate1,fili1=maps/$f,filo1=sqls/create_$f.sql,filo2=sqls/load_$f.ctl
 #==============================================================================
 exit 0
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

9E2. DATAcnv1 - Batch scripts for DOS window under Windows

gencnvD2.bat - generate all conversion jobs for Windows

 @echo off
 rem gencnvD2 - generate conversion jobs for Unix/Linux systems
 rem          - cobmap1,uvdata51,genpipe1,sqlcreate1
 rem          - assumes COBOL copybook available in subdir cpys
 rem          - alternate script gencnvD1 will scan data to create copybook
 rem          - by Owen Townsend, Feb 24/2004
 rem          - see EBCDICjobs.doc or www.uvsoftware.ca/ebcdicjobs.htm
 rem
 rem gencnvD2 filename       <-- command format
 rem =================
 rem
 rem gencnvD2 citytax1       <-- example (from EBCDICjobs.doc)
 rem =================
 rem
 if "%1" == "" goto error
 if exist dat1ebc\"%1" goto process
 :error
    echo "arg1 not a file"
    echo "usage:   gencnvD2 filename"
    echo "         ================="
    echo "example: gencnvD2 citytax1"
    echo "         ================="
    goto end
 :process
 uvcopy cobmap1,fili1=cpys/%1,filo1=maps/%1,uop=q0p0
 uvcopy uvdata51,fili1=maps/%1,filo1=pfx1/%1,uop=q0
 uvcopy genpipe1,fili1=maps/%1,filo1=pfp1/%1,uop=q0,rop=r0
 uvcopy sqlcreate1,fili1=maps/%1,filo1=sqls/create_%1.sql,filo2=sqls/load_%1.ctl
 :end

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

9F1. scripts used for Data file conversion

uvcmp1all - compare all data files in 2 directories

 #!/bin/ksh
 # uvcmp1all - script to compare all files in 2 directories
 #           - by Owen Towsnend, Feb 2010 at Everis
 # calls 'uvcmp1' for fixed length files w/o LineFeeds
 # - shows record pairs in vertical hexadecimal
 #   with mismatches flagged with '*'s
 # uses Indexed control file to get record size (ctl/ctlfile4I)
 # - same control file as used for EBCDIC to ASCII conversion
 #
 echo "uvcmp1all - compare all files in 2 directories, using uvcmp1"
 echo "- to show record pairs in vertical hexadecimal with '*' mismatches"
 echo "- for files with packed/binary & no LineFeeds (use diff for text files)"
 d1="$1"; d2="$2";
 if [[ -d "$1" && -d "$2" ]]; then :
    else echo "usage: uvcmp1all dir1 dir2 [options]"
         echo "       ============================="
         echo "ex1:   uvcmp1all d2asc.old d2asc"
         echo "       ========================="
         echo "ex2:   uvcmp1all d1ebc d3ebc y1"
         echo "       ========================"
         echo "- option y1 for EBCDIC not req'd if dirname contains *ebc*"
         exit 99; fi
 echo "uvcmp1 $d1/... $d2/..."
 echo "- will write mismatch reports to subdir rptcmp/..."
 reply=x;
 until [[ "$reply" = "y" || "$reply" = "n" ]]
    do echo "- clear all old mismatch reports from rptcmp/... y/n ?"
       read reply; done
 if [[ "$reply" == "y" ]]; then rm -f rptcmp/*; echo "rptcmp/* removed"; fi
 x=0;y=0;
 for d1f in $d1/*
   { let x=x+1
     f=$(basename $d1f)
     if [[ $f == *.idx ]]; then continue; fi
     if [[ ! -f $d2/$f ]]; then
        echo "bypassing nomatch in $d2 for $d1/$f"
        continue; fi
     #
     # use uvcopy job to get recsize & key options from control file
     # - will write to: /tmp/${LOGNAME}.uvcmp
     # remove any '.dat' (not coded in control file)
     fx=${f%.dat}
     uvcopy uvcmpget2,filr1=${CNVDATA}/ctl/ctlfile4I\
 ,filo1=/tmp/${LOGNAME}.uvcmp,arg1=$fx
     if (( $? != 0))
        then echo "datafilename not in control file"; exit 2;fi
     #
     # open the file written by uvcopy & retrieve options into $variables
     exec 3< /tmp/${LOGNAME}.uvcmp   # open file written by uvcopy
     read -u3 copyname ctlops other  # read file contents into variables
     exec 3<&-                       # close file
     #
     # if dirname contains 'ebc' add option y1 to trnslt chars to ASCII
     if [[ $d1 == *ebc* ]]; then ctlops=${ctlops}y1; fi
     #
     echo "file# $x $d1/$f $d2/$f"
     uvcopy uvcmp1,fili1=$d1/$f,fili2=$d2/$f,rop=r0,uop=q0i7$ctlops
     #=============================================================
     if (( $? != 0)); then ((y+=1)); fi
   }
 echo "$y mismatches of $x total files compared in $d1 & $d2"
 echo "vi rptcmp/*      <-- view all mismatch reports"
 echo "==========="
 echo "uvlpd12 rptcmp   <-- print all mismatch reports"
 echo "=============="
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Permuted Index of Keywords from ** Headings **

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z