Data File Conversion - Related Documents

      JCLcnv1demo.htm - DEMO conversions, sample JCL, scripts, executions
      JCLcnv2real.htm - comprehensive instructions for REAL conversions
      JCLcnv3aids.htm - conversion AIDS (cross-refs,tips,mass changes,etc)
      JCLcnv4gdg.htm  - GDG file handler from UV Software
      DATAcnv1.htm    - Data file conversion - Original Documentation <--This Doc**
                        - datafilenames same as copybooknames for easy conversion
      DATAcnv2.htm    - Data File Conversion - Comprehensive & Complete
                        - datafilenames mainframe standards aaaaa.bbbbb.cccc,etc

DATAcnv1 - Contents by Part


Part_0 - Mainframe DATA Conversion Plans
- Plan A, convert Sequential & Indexed files retaining existing layouts
  to get off the mainframe as soon as possible
- Plan B, provides conversion of flat files to RDBMS tables
- Plan C, follow Plan A & use Plan B for desired files on your schedule
  (after initial conversion to get off the mainframe asap).

Part_1 - Introduction & Overview
- Summary of utilities used in this book
- setting up subdirs required & loading test/demo files
- using uvhd to investigate data files to be converted
- Recommendations for converting your own files

Part_2 - scanning data to determine record layout
- signscanA, sign2cpyA, cobmapA
- use only when COBOL copybook not available
- most sites using these utilities will already have the copybooks
- skip Part 2 & start in Part 3 when COBOL copybooks available

Part_3 - EBCDIC to ASCII translation with packed field preservation
- Illustration of EBCDIC data file & corresponding COBOL copybook
- GENERATING uvcopy jobs to convert data files
  (cobmap1, cobmapA, uvdata51, gencnvA)
- Sample generated job with explanations
- Executing uvcopy jobs to convert EBCDIC to ASCII & preserve packed
- Character translation & packed field preservation
- Zoned sign conversion from EBCDIC to ASCII conventions
- Manual changes required (if multiple record types)
- Demo job after inserting record type test instructions
- 'uvhdcob' displays data beside the COBOL copybook fieldnames

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

DATAcnv1.doc - Convert EBCDIC to ASCII - Contents (continued)


Part_4 - Converting data to pipe-delimited text files for loading RDBMS
- Packed/binary/numeric fields convert to zero suppressed numeric fields
  with leading separate signs & decimal points inserted.
- GENERATING uvcopy jobs to convert data to a '|' pipe delimited text
  code listing for sample generated job
- EXECUTING generated jobs to convert data to pipe delimited text
  listings of test/demo input/output files
- MODIFYING generated jobs to convert data to pipe delimited text
- Options for field delimiters, input EBCDIC vs ASCII, RISC vs Intel
- 'listpipe1' lists '|' delimited output file to verify conversion.
  (copybook fieldnames listed beside the data field contents)
- 'sqlcreate1' generates SQL scripts to create & load table
- script 'gencnvD2' generates all conversion jobs when copybook present
  (cobmap1, uvdata51, genpipe3, genfixd3, sqlcreate1).
- 'genrfmA' generate jobs to reformat records based on 2 copybooks
  example to unpack all packed/binary fields
  copy/rename alternate copybook with all comp-3 & comp-4 removed
- 'genfixd2' generate jobs to convert pipe|delimited to fixed-field
- 'genpipe1' expand copybooks with item occurs, demo for armaster
- 'cpyoccurs1' expand copybooks for 'group occurs', demo for custran1

Part_5 - Special situations requiring manual changes to generated jobs
- Multi-Record-Type files (copybooks with redefined records)
  require manual coding to test the record type & skip to auto gen code
- Converting Multi Record Type files & Splitting to Separate files
- manual changes for 'occurs' not allowed in SQL databases
- Generating SQL Loader script to load Relational databases

Part_6 - Conversions for real datafilenames (vs same as copybooknames parts3,4,5)
  Generating & Executing ALL jobs to convert ALL data filesin directory
  vs '1 file at a time' documented in previous parts
- recommended for high volume file conversions that need to be repeated
  during testing & final conversion
- Generating a 'control-file' to relate copybooks to datafilenames
- The control file can also supply Indexed file types & key locations
- uvcopy uvdata52 inserts real datafilenames by control-file lookuip
  as it copies pfx1/... jobs (generated by uvdata51) to pfx2/...
- Then copy from pfx2 to pfx3 & modify for files with multi record types
- always execute jobs from pfx3 to allow for later control file addition
- test files provided so you can run demos to get familiar with procedures
Note
  • see DATAcnv2.htm for more comprehenive conversion of ALL data-files
    in directory, using control file to relate data-file-names to copybook-names.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

DATAcnv1.doc - Convert EBCDIC to ASCII - Contents (continued)


Part_7 - test EBCDIC/ASCII Translate tables used by Vancouver Utilities
  and conversions from Fixed-field to Delimited & back to Fixed-field
- bytes256 - test-file for translates & delimits 256 bytes x'00'-x'FF'
- Ebytes - conversion reults considering bytes256 as an EBCDIC file
- Abytes - conversion reults considering bytes256 as an ASCII file
- uvcmpFA1 compare original fixed-field to reconverted from delimited
  hex display record pair with '*'s flagging differences
- ebc2asc.c & asc2ebc.c EBCDIC/ASCII translate tables
- trantestEAE - alternate test Vancouver Utilities EBCDIC/ASCII translates.
- uvcopy job generates 256 lines showing EBCDIC/ASCII codes for each byte
- makebytes256 - uvcopy job to create 256 bytes with values x'00' - x'FF'
- trtneut.c - neutral translate table, C source code & uvhd hex representation

Part_8 - Variable Length Record Files
- RDW (Record Descriptor Word) variable length files
- option 'quote site RDW' FTP from mainframe (operating on unix/linux)
- option 'LOCSITE RDW' FTP to Unix/Linux (operating on mainframe)
- option 'OUTFIL FNAMES=SORTOUT,FTOV' SORT option to convert Fixed to Variable
- investigating RDW files with uvhd
- converting EBCDIC RDW files to ASCII using uvhd, uvcp,& uvcopy varfix11
- creating table summaries of record sizes found in RDW files
  varstat1 - uvcopy job to table summarize record sizes in RDW files

Part_9 - Verifying Conversions & Generating Test Files
- converting ASCII files back to EBCDIC to compare to original
- data file comparison utility 'uvcmp1'
- Verifying Data Conversion with 'uvhdcob' to verify Numeric fields
- Verifying Data Conversion with 'genverify1'
  (generate uvcopy jobs to Verify all Packed & Numeric fields)
- generate jobs to accumulate all numeric fields in a file
  'genacum2' generates uvcopy jobs from the copybook record layout
  (could use to verify conversions or check COBOL report totals)
- Generating test data files automatically from COBOL copybooks

Part_10 - summary of scripts used for data conversions
- Korn shell scripts for unix/linux

Copyright 1993-2021, UV Software Inc.

 Owen Townsend, UV Software, 4667 Hoskins Rd., North Vancouver BC, V7K2R3
      Tel: 604-980-5434 Fax: 604-980-5404
      mailto:owen@uvsoftware.ca
      https://www.uvsoftware.ca

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_0 mainframe DATA conversion Plans

DATA Conversion Alternatives

UV Software offers 2 strategies for converting mainframe data files for use on Unix/Linux systems. We are concerned here with the Sequential & Indexed files used on the mainframe, not with existing mainframe DBMS's. Software for converting mainframe DBMS tables is usually supplied by the vendor.

UV Software supplies powerful utilities to convert the Sequential & Indexed files, automatically from the COBOL copybooks, allowing for complex files, with multiple record types, occurs, etc.

Plan 'A' is to retain existing record layouts, which allows you to convert quickly, since COBOL program logic needs no changes.

Plan 'B' is to convert data files to pipe delimited text files for loading RDBMS tables. Any packed/binary fields are unpacked & edited with signs & decimal points as required for loading RDBMS tables. We also automatically generate SQL Loader scripts from the COBOL copybooks.

Plan A (layouts unchanged)

  1. Existing Record layouts are retained, allowing you to convert quickly, since COBOL program logic requires no changes.

  2. Character fields are translated from EBCDIC to ASCII, but packed/binary fields are preserved unchanged, as required for Micro Focus COBOL on Unix/Linux.

  3. UV Software provides utilities to generate the 'data conversion jobs' automatically from the COBOL copybook record layouts.

  4. The auto generated conversion jobs usually require no manual changes before execution, but changes are required for files with multiple record types.

  5. UV Software's conversion techniques have been proven to handle complex data files with multiple record types based on several different fields that could not be handled by other vendors.

  6. The data conversion jobs for hundreds of data files can all be executed with 1 command, which is a big advantage, since the data conversion will usually be repeated several times during a 6 month or year long conversion. Of course the important repeat is on the 'go live' weekend.

See sample conversions for plan A (layouts unchanged, preserve packed fields) at https://www.uvsoftware.ca/datacnv1.htm#Part_3

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part 0 DATA Conversion Alternatives

plan B - for RDBMS's

  1. The 2nd plan is to convert the data files to pipe delimited text files for loading RDBMS tables (for DB2, Oracle, SQL server, etc).

  2. Any packed/binary fields are unpacked & edited with signs & decimal points as required for loading RDBMS tables.

  3. UV Software provides utilities for generating the 'pipe delimit jobs' automatically from the COBOL copybook record layouts.

  4. We also automatically generate the 'SQL Loader control tables' from the COBOL copybooks.

  5. Note that mainframe files with multiple record types must be split into separate tables, since RDBMS tables do not allow multiple record types. UV Software's techniques can easily handle this situation.

  6. RDBMS tables do not allow 'occurs'. UV Software's techniques make it easy to append a sequence number on such fields, or split the occurs data out to a separate table, inserting record ID fields as required.

See sample conversions for plan B (unpack to delimited files to load RDBMSs) at https://www.uvsoftware.ca/datacnv1.htm#Part_4

plan C = Plan A + Plan B on your timetable

We recommend plan A to get off the mainframe as quickly as possible. Then you can convert files to RDBMS tables depending on your priorities & timetable.

related documentation

 SQLconvert.htm - convert flat files to pipe delimited for Microsoft SQL server
                  - driven by COBOL copybooks
                  - also generate scripts to create & load tables
                  - generates jobs for all copybooks & all data files
                    (vs this DATAcnv1.doc for 1 copybook/file at a time)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_1 Data Conversion Preparation

Part1 - Contents


1A1. Data Conversion Overview
1A3. Summary of utilities used in this book

1B1. test/demo files on uvadm.zip distribution archive
customer installation of mvstest1, 2, or 3
copy desired files to separate login or to your homedir

1B2. export RUNLIBS/RUNDATA/CNVDATA in .bash_profile
depending on which of mvstest1,2,3 desired
aliases for easy change to testlibs1,testdata1,cnvdata1

1B3. Setup for DataFile Conversion
1B4. conversion subdirs illustrated (dtree report)
1B5. subdir contents & intended use

1C1. Data file investigation with uvhd
uvhd d1ebc/citytax1 r128a
 1D1. Using 'uvhd' to investigate EBCDIC files that you wish to convert to ASCII.
     - determine/verify record size & look for packed fields & zoned signs
     - look for complex files (multi record types, variable length records, etc)

1E1. Recommendations for converting your own files

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1A1. Data Conversion Preparation

DATAcnv1.doc - OVERVIEW

The power & flexibility of these procedures are necessarily accompanied by some underlying complexity, but we want you to know at the outset, that there are master scripts to simplify these procedures.

We recommend you follow the step by step procedures to convert the test/demo file & perhaps your first data file. This will help you understand how these conversions work & allow you to modify the procedures for complex files (multi record types or variable length records).

For most files, you can use the 'gencnvD1' or 'gencnvD2' master scripts to generate all conversion jobs with 1 command. Converting any 1 data file is then simplified to the following (using 'citytax1' demo file for illustration):


 #1. Copy the data file to the expected input subdir
     (supplied demo file dat1/citytax1 may be copied to cnvdata1/d1ebc).

 #2. gencnvD1 citytax1 r128d4 <-- generate All conversion jobs for demo file
     ========================

 #3. uvcopy pfx3/citytax1     <-- execute EBCDIC to ASCII conversion
     ====================       - from d1ebc/citytax1 to d2asc/citytax1
                                  (preserves packed, fixes zoned signs)

 #4. uvcopy pfp3/citytax1     <-- execute conversion to pipe delimited text
     ====================       - from d2asc/citytax1 to d4pipe/citytax1

 #5. Converted outputs are left in following subdirs:
 #5a. d2asc/citytax1    - ASCII file (same layout as input)
 #5b. d4pipe/citytax1   - all text pipe delimited file (for loading RDB's)
 #5c. sqlTC1/citytax1   - script to Create database Table
 #5d. sqlTL1/citytax1   - script to Load database Table

If you already have the COBOL copybook (datafile record description), you would use master script 'gencnvD2' which saves the 1st 2 of the 6 steps in gencnvD1 (scanning the data file & generating the copybook).

These jobs are simplified by setting up several subdirs within a superdir, which allows us to maintain the same file name as we generate jobs & convert data thru the various steps.

You don't have to modify the generated jobs if you copy the data file to the expected input subdir (d1ebc), and rename same as copybook. After conversion you can copy the converted from the output subdir (d2asc), and rename back to your desired datafilename.

But is is a simple matter to modify the generated conversion jobs to specify your I/O data pathnames. You might do this for large files, especially if you need several re-conversions during your conversion project.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1A2. Data Conversion Preparation

DATAcnv1.doc - OVERVIEW

  1. DATAcnv1 (this document) is intended to help you convert EBCDIC data files whether or not the COBOL copybook is available. Other conversion documents in the Vancouver Utilities presume the COBOL copybooks are available.

  2. When the COBOL copybook is not available, we have procedures to scan the EBCDIC data to determine the record layout. 'signscanA' can determine the locations of packed fields & zoned signs which would be destroyed if you translate the entire record to ASCII.

  3. When the COBOL copybook is available, you can use it with this document, by skipping the signscanA & sign2cpyA procedures. All the data conversion procedures are driven by the copybook whether supplied or created by scanning the data.

  4. DATAcnv1 is intended to convert 1 file at a time. Other conversion documents (see below) are intended to convert entire mainframe sites with hundreds of data files. DATAcnv1 is applicable to sites that receive data files from outside sources which supply the data in varying formats with or without copybooks.

  5. The procedures in DATAcnv1 are much simpler & easier to execute than those in the other documents. However you may still need to reference the other documents for more complex situations (multiple record types, variable length records, magnetic tape input, etc).

    Other Data Conversion Documentation

https://uvsoftware.ca/jclcnv2real.htm#Part_3

UV Software Training & Support

UV Software recommends a minimum 1 week training course with the initial installation of the Vancouver Utilities. With this training, you should be able to convert most EBCDIC files. UV Software provides customers with a generous amount of free support, so please call if you can use some help with a complex file. UV Software can also be contracted to convert the unusually complex files if the problem is not easily resolved via telephone and emails.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1A3. Data Conversion Preparation

DATAcnv1 - Utilities Summarized

  1. 'uvhd' is the first tool you will use to investigate EBCDIC files that you wish to convert to ASCII. Use uvhd to determine or verify the correct the record size & to check for packed fields, zoned signs, and complex files (multiple record types, variable length records, etc). Note that you cannot use 'vi' or most other unix tools on EBCDIC files, or ASCII files with packed fields, or any file without linefeeds (most mainframe files do not have linefeeds so the entire file appears in vi as 1 long line).

  2. 'signscanA' will help you determine the record layout of EBCDIC data files when no COBOL copybook is available. Script signscanA runs uvcopy job 'signscan2' which scans for packed fields, zoned signs, date fields, and right blank filled fields (such as Name & Address). It is most to determine packed fields & zoned sign positions for the subsequent script 'gencnvA' which performs the EBCDIC to ASCII conversion.

  3. 'sign2cpyA' converts the signscanA record layout to a COBOL copybook (an industry standard method of describing record field layouts). The copybook is used by the following scripts to perform various conversions. Note that if a copybook is supplied with the EBCDIC data file, you can use it with the following scripts.

  4. 'cobmapA' converts COBOL copybooks to 'cobmaps' (record layouts), adding field start,end,length,& type on the right hand side of the copybook. List the cobmap & use the 'uvhd' utility to verify and make corrections & enhancements. This is most important if you have just created the copybook using signscanA & sign2cpyA.

  5. 'uvhdcob' displays the data beside the COBOL copybook fieldnames making it easy to see if the copybook is consistent with the data. I recommend you list out the 1st few records with both uvhd & uvhdcob to verify the layout & resolve any problems if file is supplied by outside sources.

  6. 'gencnvA' generates a uvcopy job to translate EBCDIC files to ASCII, preserving packed fields (would be destroyed by translation), and correcting any zoned decimal signs from the EBCDIC conventions to ASCII conventions.

  7. 'genpipeA' generates a uvcopy job to convert data to '|' pipe delimited ASCII text files. Packed & numeric fields will be converted to zero suppressed numeric fields with leading sign +/-. The pipe delimited format is often used for loading Relational data bases.

  8. 'listpipe1' lists a '|' delimited output file to verify conversion. The copybook fieldnames are listed beside the data field contents.

  9. 'sqlcreate1' generates SQL scripts to create & load tables

  10. 'genrfmA' generate jobs to reformat records based on 2 copybooks Example to unpack all packed/binary fields to numeric zoned fields Use genrfmA when the application for which the data file is intended does not allow packed fields.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1B1. Data Conversion Preparation

test/demo files on uvadm.zip distribution

Here are the data conversion test/demo files provided on the Vancouver Utilities distribution archive (usually uvadm.zip or uvadm.tar.gz) - assuming you have setup user uvadm & unzipped into /home/uvadm/...

      /home/uvadm/mvstest/mvstest0 - original JCL/COBOL/demo files (downgraded)
      /home/uvadm/mvstest/mvstest1 - cnvdata1 datafilenames same as copybooknames
      /home/uvadm/mvstest/mvstest2 - cnvdata2 datafilenames mainframe standards
      /home/uvadm/mvstest/mvstest3 - AIX COBOL conversions (vs Micro Focus)
      /home/uvadm/mvstest/mvstest0/testlibs0 <-- JCL/COBOL/DATA test/demo files
      /home/uvadm/mvstest/mvstest0/testdata0   - original versions DOWNGRADED
      /home/uvadm/mvstest/mvstest0/cnvdata0    - use mvstest1,mvstest2,mvstest3
    * /home/uvadm/mvstest/mvstest1/testlibs1 <-- JCL/COBOL/DATA test/demo files ***
    * /home/uvadm/mvstest/mvstest1/testdata1   - for DATAcnv1.doc
    * /home/uvadm/mvstest/mvstest1/cnvdata1      or www.uvsoftware.ca/datacnv1.htm
                                               - data-file-names same as copybook-names
      /home/uvadm/mvstest/mvstest2/testlibs2 <-- JCL/COBOL/DATA test/demo files
      /home/uvadm/mvstest/mvstest2/testdata2   - for DATAcnv2.doc
      /home/uvadm/mvstest/mvstest2/cnvdata2    - or www.uvsoftware.ca/datacnv2.htm
                                               - data-file-names IBM standards aaa.bbb.ccc,etc

customer installation of mvstest1,2

After customers install uvadm.zip/tar, they can setup a separate user login for the desired set of test files (mvstest1,2) & copy to that homedir.

 ---> cp -r /home/uvadm/mvstest/mvstest1 /home/mvstest1/. <-- for this DATAcnv1.doc
 ---> cp -r /home/uvadm/mvstest/mvstest2 /home/mvstest2/.
      ===================================================

OR copy mvstest1,2 to YOUR HOMEDIR from /home/uvadm/mvstest1,2

 ---> cp -r /home/uvadm/mvstest/mvstest1 /home/userxx/.
 ---> cp -r /home/uvadm/mvstest/mvstest2 /home/userxx/.
      ================================================

OR copy the subdirs of mvstest1,2 to YOUR HOMEDIR (example for mvstest1)

 ---> cp -r /home/uvadm/mvstest/mvstest1/cnvdata1  /home/userxx/.
 ---> cp -r /home/uvadm/mvstest/mvstest1/testlibs1 /home/userxx/.
 ---> cp -r /home/uvadm/mvstest/mvstest1/testdata1 /home/userxx/.
      ===========================================================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1B2. Data Conversion Preparation

export RUNLIBS/RUNDATA/CNVDATA in .bash_profile

Define CNVDATA,RUNDATA,RUNLIBS in .bash_profile, depending on which of mvstest1,2 you wish to test/demo - example for mvstest1 as a separate login.

      export RUNLIBS=$HOME/mvstest1/testlibs1
      export RUNDATA=$HOME/mvstest1/testdata1
      export CNVDATA=$HOME/mvstest1/cnvdata1

OR if you copied just testlibs1,testdat1,cnvdata1 to your homedir:

cp -r /home/uvadm/mvstest/mvstest1/testlibs1 $HOME/. cp -r /home/uvadm/mvstest/mvstest1/testdata1 $HOME/. cp -r /home/uvadm/mvstest/mvstest1/cnvdata1 $HOME/.

Then setup the exports in your .bash_profile as follows. This is the default in the .bash_profile supplied.

export RUNLIBS=$HOME/testlibs1 export RUNDATA=$HOME/testdata1 export CNVDATA=$HOME/cnvdata1

The supplied .bash_profile & .bashrc include the following Aliases that make it easy to change to testlibs, testdat, or cnvdata.

alias cdl='cd $RUNDATA' alias cdl='cd $RUNDATA' alias cdl='cd $RUNDATA'


 cdc   <-- easy change to $HOME/cnvdata (/home/userxx/cnvdata)
 ===

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1B3. Data Conversion Preparation

Initial Setup for DataFile Conversion

I recommend you setup a separate login user account to perform these data conversions. I suggest userid 'mvstest1' which will be used in this document. The conversion directory will then be /home/mvstest1/cnvdata1.

If you cannot setup a separate userid, you could use a subdirectory within your home directory (but this may not be advisable if there is not enough space for your projected requirements).

Setting up a separate userid might be better if you do not want to modify your current .profile. Note that you must modify the .profile to setup PATH, etc to run the Vancouver Utility scripts & uvcopy jobs. See install.htm.

Sub-directories are made for the different types of files, so we can keep the same name as the original datafile & copybook, as we convert copybooks to record layouts, to uvcopy jobs. For example our demo file is named 'citytax1' & the same name is used as we convert thru several subdirs:

 d1ebc ---------> signs ---------> cpys ---------> maps ----------> pfx1
       signscan2        sign2cpy1        cobmap1         uvdata51

Keeping the same name thruout the various subdirs greatly simplifies our operating instructions. This also means you must drop any extensions on your datafile or copybook names.

making subdirs for data conversions

Preferably setup a separate userid 'cnvdata1' to perform these conversions, But if not possible make directory 'cnvdata1' within your home dir or elsewhere. For Windows, make directory C:\cnvdata1.


 #1. login mvstest1 --> /home/mvstest1
 #2. mkdir cnvdata1
 #3. cd cnvdata1    --> /home/mvstest1/cnvdata1

 #4. cnvdatadirs       <-- script to create sub-dirs (listed on next page)
     ===========
       - - - OR - - -

 #1. login youruserid ---> yourhomedir (/home/yourid)
 #2. mkdir cnvdata1     <-- make conversion superdir within your homedir
 #3. cd cnvdata1        <-- change into conversion superdir

 #4. cnvdatadirs       <-- script to create sub-dirs (listed on next page)
     ===========

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1B4. DATAcnv1.doc - Convert EBCDIC to ASCII - INITIAL SETUP

cnvdata1 subdirs (dtree illustration)

 /home/uvadm/mvstest/mvstest1  <-- location in Vancouver Utilities distribution
 /home/mvstest1        <-- OR setup user mvstest1
 /home/userxx              OR setup cnvdata1 within your $HOME dir
 :-----cnvdata1        <-- cnvdata1 superdir for DATA conversion
 :     :-----archive     - backup to reload d1ebc/... copybooknames or datafilenames
 :     :     :-----d1ebc_cpy_names    <-- for Parts3,4,5
 :     :     :-----d1ebc_data_names   <-- for Part6
 :     :-----cpys       - copybooks
 :     :-----ctl        - control files
 :     :-----d0ebc      - EBCDIC data files, copy here 1st in case changes before d1ebc
 :     :-----d1ebc      - EBCDIC data files (named same as copybooks for part3,4,5)
 :     :-----d2asc      - ASCII data files (converted by uvcopy uvdata51)
 :     :-----d3ebc      - convert back to EBCDIC by uvdata31
 :     :-----d3asc      - reformatted data files (see genrfmA)
 :     :-----d4pipe     - pipe delimited to load DB tables (by genpipe1)
 :     :-----d5fix      - convert pipe|delimited back to fixed field (genfixd2)
 :     :-----listpipe   - pipe|delimited listed vertically with copybook fieldnames
 :     :-----maps       - copybooks with record layouts on right side
 :     :-----pfx1       - uvcopy jobs to convert EBCDIC to ASCII (gen by uvdata51)
 :     :-----pfx2       - with data filenames inserted (vs copybook names)
 :     :-----pfx3       - copied here for modify/execute
 :     :-----pfy1       - uvcopy jobs to convert ASCII to EBCDIC (gen by uvdata31)
 :     :-----pfp1       - uvcopy jobs to create pipe delimited (gen by genpipe1)
 :     :-----pfq1       - uvcopy jobs to convert pipe|delimited to fixed field (genfixd2)
 :     :-----pfg1       - uvcopy jobs to create test data via copybooks (gentest1)
 :     :-----pf         - misc uvcopy jobs
 :     :-----sf         - misc scripts
 :     :-----sqlTC1     - scripts to Create database tables (gen from copybooks)
 :     :-----sqlTL1     - scripts to Load database tables (gen from copybooks)
 :     :-----tmp        - temp files (keep working directory clean)
 :     :
 :-----testdata1       <-- testdata1 superdir for later testing
 :     :-----data1
 :     :-----joblog
 :     :-----jobmsgs
 :     :-----jobtmp
 :     :-----sysout
 :     :
 :-----testlibs1       <-- testlibs1 superdir for COBOL & JCL conversion
 :     :-----cbls       - see JCLcnv1demo.htm#3B2
 :     :-----cblx
 :     :-----cpys
 :     :-----ctl
 :     :-----jcls
 :     :-----maps
 :     :-----parms
 :     :-----pf
 :     :-----procs
 :     :-----sf
 :     :-----sfun
 :     :-----xref

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1B5. DATAcnv1.doc - Convert EBCDIC to ASCII - INITIAL SETUP

subdirs required for conversion scripts


 cnvdatadirs    <-- script to create the sub-dirs listed below
 ===========
archive
  • save test/demo files to restore to d1ebc/... for reconversions
    after running cnvdatainit to clear files generated by prior test/demos
    #1-----> cp archive/d1ebc_cpy_names/* d1ebc/. (names same as copybooks)
    OR #2--> cp archive/d1ebc_data_names/* d1ebc/. (real names aaa.bbb.ccc.etc)
  • also backup cpys/ (but not cleared by cnvdatainit)
d0ebc
  • EBCDIC files loaded here from FTP, 3480, CD, etc
  • when you need to convert variable to fixed for conversion jobs
  • optional, can load directly in d1ebc if all fixed records
d0ebc
  • original EBCDIC data files saved here before changes in d1ebc
  • copy to d1ebc before lowercase
    or convert variable-length to fixed-length for easier conversion
d1ebc
  • EBCDIC data files in fixed records ready for conversion
  • named same as copybooknames for Parts3,4,5
  • real datafile names for Part6 & DATAcnv2.doc
d2asc
  • data files converted to ASCII with same record layout
d3asc
  • data files reformatted by genrfmA (see '4I1')
    depending on 2 input/output copybooks (matching fieldnames)
  • fields may be different sizes & types (character,packed,binary)
d4pipe
  • data files converted to '|' pipe delimited format
cpys
  • COBOL copybooks generated by signscanA/sign2cpyA
    or received with data to be converted.
maps
  • record layouts generated from copybooks
    cobmaps showing start/end/length/type on right
pfx1
  • uvcopy jobs generated by gencnvA, to convert EBCDIC
    to ASCII with same record layout
pfx2
  • pfx2 reserved for outputs from uvdata52 that inserts datafilenames
    vs copybooknames coded on fili1=... & filo1=... in pfx1 jobs
  • uvdata52 uses a control file to relate datafilenames to copybooknames
Note
  • pfx2, uvdata52,& the control file are optional
  • omit if only a few files & OK to rename datafiles same as copybooks
  • then after conversion, you can rename back to the desired filename
  • also required if you have multiple data files for 1 copybook
pfx3
  • generated uvcopy jobs always copied to pfx3 before changes & execution
  • changes required for Multi Record Type files (insert R/T test code)
  • to split Multi Record Types to separate files, etc
  • copying to pfx3 protects your manual change code from
    being overwritten if jobs are regenerated (to pfx1 & pfx2)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1B6. DATAcnv1.doc - Convert EBCDIC to ASCII - INITIAL SETUP

subdirs required for conversion scripts (continued)

pfp1
  • uvcopy jobs generated by genpipe1, to convert data to
    pipe '|' delimited for loading Relational databases
pfp2
  • pfp2 reserved for outputs from uvdata54 that inserts datafilenames
    vs copybooknames coded on fili1=... & filo1=... in pfp1 jobs
  • uvdata54 uses a control file to relate datafilenames to copybooknames
  • same Notes as above for 'pfx2'
pfp3
  • generated uvcopy jobs always copied to pfp3 before changes & execution
  • for same reasons as for pfx3 above
pf
  • uvcopy jobs for special circumstances
    (processing data files with multiple record types,
    occurs depending on, variable length records, etc)
signs
  • record layout reports created by signscan2
    (not required if you have COBOL copybooks)
sqls
  • SQL loader scripts generated from copybooks
tmp
  • temporary miscellaneous intermediate files created by
    various procedures (can clean out at any time).
 ctl   - control-files (used in Part_6)
 ctl/datacpy51 - skeleton control-file generated from mainframe data-file-names
 ctl/datacpy52 - control-file with copybooknames coded beside datafilenames

additional subdir required for signscan

You will not need subdir 'signs' if you have the COBOL copybooks & do not need to scan the data (signscan2) to automatically create the copybooks. Note that signscan2 is not foolproof & may require correction.

mkdir signs
  • sign field location reports created by signscanA
    which scans EBCDIC data files in d1ebc.
  • converted to copybooks by sign2cpyA.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1B7. DATAcnv1.doc - Convert EBCDIC to ASCII - INITIAL SETUP

additional subdirs to create unpacked data records

You will need the following subdirs if you wish to create unpacked data records. The previous page assumed you wished only to convert the EBCDIC data to pipe delimited ASCII text records for loading into SQL databases.

mkdir cpyu
  • alternate versions of COBOL copybooks used for
    generating jobs to 'unpack' packed fields
    (or to reformat records for any reason).
mkdir mapu
  • record layouts generated from the 'cpyu' copybooks
    (used while generating a job to unpack records)
Note
  • as of Nov2014, above cpyu & mapu not required
  • alternate copybook now assigned a different name
  • can exist in same subdirs as cpys/... & maps/...
  • see demo on pages '4I1' - '4I5'
mkdir mapsI
  • Indexed files created from 'maps' used while generating
    jobs to unpack records.
mkdir pfr1
  • uvcopy jobs generated by genrfmA, to reformat records
  • unpacking packed/binary increasing record size as required
mkdir pfr2
  • uvcopy jobs copied from pfr1 & modified for various reasons

stay in the working-dir (not the subdirs)

As you run the various scripts & uvcopy jobs, you should stay in the working directory (/home/mvstest1/cnvdata1/ or whatever you call it). Do NOT change into subdirs, since you must be back at the working directory to run the next step.

Address the files thru 1 level of subdir for various view/print commands. For example, to display the EBCDIC to ASCII conversion job:


 more pfx1/citytax1      <-- address files thru 1 level of subdir
 ==================
 cd pfx1                 <-- do not change into & out of subdirs
 more citytax1             - would waste 3 commands vs 1
 cd ..                     - since next step requires working directory level

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1C1. DATAcnv1.doc - Convert EBCDIC to ASCII - INITIAL SETUP

Initial Data File Investigation (with uvhd)

You can demonstrate the data conversion scripts, by setting up the directories suggested on the previous pages,& loading the supplied test files.


 cd cnvdata1    - change to superdir setup on prior pages
 ==========      (with subdirs: d1ebc,d2asc,d4pipe,cpys,maps,etc)

Inspect the test/demo file - with 'uvhd'


 uvhd d1ebc/citytax1 r128as7   - inspect citytax1 demo file
 ===========================   - option 'r' (r128) specifies record size
                               - option 'a' translates characters to Ascii
                               - option 's7' spaces between scale & data below
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
             1013014000168508559800000000000000018150266550961400000000005153
          64 OUVER           BCV9S1H1..CK..Wb....g...qq.00014906A970530
             DEECD44444444444CCEFEFCF004D005880008000770FFFFFFFFCFFFFFF444444
             64559000000000002359218101320072C0047C0111C000149061970530000000

--> i3 <-- immediate print 1st 3 records == --> q <-- quit uvhd

When you have a new EBCDIC file to be converted, the first thing you need to do is investigate the file (using 'uvhd') to determine or verify the correct record size & to check for packed fields, zoned signs, and complex files (multiple record types, variable length records, etc).

Your first objective is to determine the 'CORRECT RECORD SIZE'. In this case I already knew it & specified it on the command line (see 'r128' above). If you don't specify any option 'r', the record size defaults to 256.

'uvhd' makes it easy to determine the correct record size. When displaying record #1, uvhd will suggest the next lower & next higher record-size. If you have it wrong you will see the data shifting left or right as you browse thru the file (just hit enter to advance to the next record).

Note that you cannot use 'vi' or most other unix tools on EBCDIC files, or ASCII files with packed fields, or any file without linefeeds (most mainframe files do not have linefeeds so the entire file appears in vi as 1 long line).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1D1. DATAcnv1.doc - Convert EBCDIC to ASCII with or without COBOL copybook

Investigating EBCDIC files - with 'uvhd'

When you have a new EBCDIC file to be converted, you should investigate the file using 'uvhd' since you cannot use 'vi' with EBCDIC files. Your 1st concern is to determine the correct record size or verify the size given to you by the source or copybook. Uvhd defaults record size to 256 & it is usually easy to determine record size by looking for data repetition.

Note
  • it is essential to determine the 'CORRECT RECORD SIZE'.
  • several jobs require the record size be entered as option 'r'.

'uvhd' displays data files in vertical hex. Data is displayed in 64 byte segments on 3 lines (characters, hex zones,& hex digits). Use option 'a' with EBCDIC files to translate the character line to ASCII while still showing the EBCDIC data as hex zones & hex digits.

For the citytax1 demo file we know the record size is 128 which is specified via option 'r' (r128). We will also specify option 'a' to translate the character line to ASCII. If we also specify 's2' to space between the 3 line groups of characters, zones,& digits the option string will be 'r128as2'.


 uvhd d1ebc/citytax1 r128as2
 ===========================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
             1013014000168508559800000000000000018150266550961400000000005153
          64 OUVER           BCV9S1H1..CK..Wb....g...qq.00014906A970530
             DEECD44444444444CCEFEFCF004D005880008000770FFFFFFFFCFFFFFF444444
             64559000000000002359218101320072C0047C0111C000149061970530000000

rec#=1 recs=10 rsiz=128 fsiz=1280 d1ebc/citytax1 enter command--> i3 <-- immediate print 1st 3 records

'uvhd' displays the 1st record in the file & waits for you to enter a command. Please see uvhd.htm for the complete command repertoire or enter '?' for the Help screens. In this case we entered 'i3' to immediately print the 1st 3 records.

Note the packed fields in bytes 88-105 & the zoned sign in byte 115. You can calculate byte positions by adding the scale position (from top line) to the offset of the current row of data. For example packed fields begin at scale position 24 + offset 64 = byte 88 (zero relative) = column 89 (1 relative).

If you have long records & have a large screen (WinDos allows 60 lines/screen) use the 'm' option of uvhd to records up to 1000 bytes on 1 screen, for example:

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1D2. DATAcnv1 - Convert EBCDIC to ASCII with or without COBOL copybook

Printing data files with uvhd

The 'immediate printing' (i3 above) feature of uvhd is very handy, but if it does not work on your computer, you can use the 'print' command (p3 vs i3). The 'print' command stores the formatted file in the 'tmp' subdir, named the same as the datafile with a date/time suffix. You could print separately:


 uvlp12 tmp/citytax1.yymmddhhmmssP
 =================================

For the 'i' command, uvhd calls the 'uvlp12' script (or 'uvlpr12' if WinDos). The printer for 'uvlp12' is defined by env-var 'UVLPDEST'. The WinDos script 'uvlpr12.bat' redirects output to 'LPT1'. See more on page C1 of uvhd.doc.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1D3. DATAcnv1.doc - Converting EBCDIC to ASCII

Investigating EBCDIC files - with 'uvhd' (continued)

Before we leave the discussion of using 'uvhd' with EBCDIC files, we should show you what happens if you don't supply any options to uvhd:


 uvhd d1ebc/citytax1            <-- uvhd with No Options
 ===================
                 1         2         3         4         5         6
     1 0123456789012345678901234567890123456789012345678901234567890123 <--scale
     0 ........@@....@.....@@@@@@@@@@@@@@@....@.....@....@@@@@@@@@@.... <--chars
       FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC <--zones
       1013014000168508559800000000000000018150266550961400000000005153 <--digits
    64 .....@@@@@@@@@@@..............W.........qq......................
       DEECD44444444444CCEFEFCF0888005880008000770FFFFFFFFCFFFFFFFFFFFF
       645590000000000023592181029C0072C0047C0111C000149061970530191809
   128 ........@@......@.....@@@@@@@@@@@@@....@....@......@......@@....
       FFFFFFFF44CCDDCC4CDDED4444444444444FFFF4CCEE4DCDCCD4EEDCCE44ECDC
       1013992300756975029665000000000000012500512307554590239553005153
   192 .....@@@@@@@@@@@..............W.........xRL.....................
       DEECD44444444444CCEFDFEF0888005880008800754FFFFFFFFDFFFFFFFFFFFF
       645590000000000023553161029C0072C0047C0182C000146261980601190156
  1. When no options are specified, uvhd defaults to single spacing. On the previous page we used option 's2' to double space between the 3 line groups of character,zones,& digits for each 64 byte segment.

  2. uvhd defaults to expecting ASCII, so for EBCDIC files we will get garbage. Note that this applies only to the character line, the zones & digits will be the same since they represent the data in hexadecimal. The 'a' option tells uvhd to translate the character line to ASCII. With option 'a' the 1st 64 byte portion from above would be:


 uvhd d1ebc/citytax1 a          <-- uvhd with option 'a'
 =====================
                  10        20        30        40        50        60
     1 0123456789012345678901234567890123456789012345678901234567890123
     0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC <--chars
       FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC <--zones
       1013014000168508559800000000000000018150266550961400000000005153 <--digits
  1. If you are working with EBCDIC files, you should have a good understanding of the EBCDIC & ASCII character set representations in hexadecimal. For example a '1' is x'F1' in EBCDIC & x'31' in ASCII, an 'A' is x'C1' in EBCDIC & x'41' in ASCII (the IBM code card shows you all this).

  2. You can identify the packed fields by looking for bytes with '0' zones, & x'_C' or x'_D' in the digit of the last byte of a field which is the sign '+' or '-'.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1D4. DATAcnv1.doc - Converting EBCDIC to ASCII

Investigating citytax2 with uvhd

'citytax2' is a multi-record-type file with 3 record types 'H','T',or 'P' in byte 8 (Header records, Tax Records,& Payment records).

The record size is 128 (same as citytax1), but we will use '512' in the example below to illustrate some points.


 uvhd d1ebc/citytax2 r128as3  <-- display citytax2 (see options below)
 ===========================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0         H December 31, 1998
             44444444C4C88898894FF64FFFF4444444444444444444444444444444444444
             000000008045354259031B019980000000000000000000000000000000000000
          64
             4444444444444444444444444444444444444444444444444444444444444444
             0000000000000000000000000000000000000000000000000000000000000000
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
             1013014000168508559800000000000000018150266550961400000000005153
          64 OUVER           BCV9S1H1..CK..Wb....g...qq.00014906A970530
             DEECD44444444444CCEFEFCF004D005880008000770FFFFFFFFCFFFFFF444444
             64559000000000002359218101320072C0047C0111C000149061970530000000
                      10        20        30        40        50        60
 r#        3 0123456789012345678901234567890123456789012345678901234567890123
         256 10130140P       1999...A....A....A....A....A....A....A....A....A
             FFFFFFFFD4444444FFFF00042000420004200042000420004200042000420004
             101301407000000019990001C0001C0001C0001C0001C0001C0001C0001C0001
          64 ....A....A....A.
             2000420004200042444444444444444444444444444444444444444444444444
             C0001C0001C0001C000000000000000000000000000000000000000000000000

 uvhd d1ebc/citytax2 r128as3   <-- note options explained below
 ===========================
option r128
  • record size
option a
  • translates the character line from EBCDIC to ASCII
option s3
  • spacing between records & 3 line groups (chars, zones, digits)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1E1. DATAcnv1.doc - Converting EBCDIC to ASCII

Recommendations to Convert Your Own Files

  1. First perform the 'citytax1' demo conversions illustrated in this document.

  2. You can use the demo set of subdirs for your own file conversions, or setup a similar set (superdir cnvdata1 with subdirs d1ebc,d2asc, d3asc,d4pipe,cpys,maps,pfx1,pfx2,pfx3,pfp1,pfp2,pfp3,sqls,tmp,etc).

  3. If possible name your input data file & copybook the same & keep it short to save keystrokes & minimize confusion. The uvcopy conversion jobs are generated assuming datafile name same as copybook name. Multiple separate subdirs allow you to keep the same name for the various datafiles & uvcopy jobs throughout the system.

  4. Whenever you wish to view or print your copybooks - DON'T, always view or print the corresponding 'cobmap' since it is much more meaningful with the start/end/length bytes calculated on the right side. The cobmaps are generated in the 'maps' subdir by many of the scripts, but if you have just updated with the editor, you can run 'cobmapA' to regenerate the cobmap.

  5. Whenever you wish to print your cobmaps or uvcopy jobs, use the 'uvlp12' script (or uvlpr12 for WinDos). These scripts add headings with filenames, dates, page#s, etc. See uvlist.doc for the environmental variable (UVHDPRINT) used by uvhd for its desination printer. For files with long lines such as the pipe delimited files, you can use uvlp14L, uvlp18L, etc to print Landscape at 14 cpi or more to get up to 200 chars per line. For Windows/DOS the script names are: uvlpr12, uvlpr12L, uvlpr20L, etc.

  6. When naming your files, it is usually a good idea to append a numeric digit to your filenames, since you will often find the necessity to create alternate versions. For example the test files here are named citytax1, citytax2, etc.

  7. Most of these procedures apply to 'fixed length records' that may have packed decimal fields. For variable length files see 'Part_8'.

  8. Our primary test/demo file has only 1 record type, but procedures for multi-record type files are illustrated later in sections M,N,O,P.

  9. For simple files (only 1 record type), you can use the master script 'gencnvD2' (see page '4H1'), which generates all conversion jobs with just 1 command. But you should do the step by step procedures for your 1st file or 2 until you thoroughly understand these procedures.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_2 DATAcnv1 - convert EBCDIC to ASCII without COBOL copybook

Part 2 - Scanning Data to determine Record Layout - Contents

Note
  • skip Part2 if you have the COBOL copybooks
  • don't need signscanA/sign2cpyA to automatically create them.

2A1. 'signscanA' will help you determine the record layout of EBCDIC data files
when no COBOL copybook is available. Script signscanA runs uvcopy job
'signscan2' which scans for packed fields, zoned signs, date fields, and
right blank filled fields (such as Name & Address).

2B1. Scan logic for various field types (unpacked, zoned/signed, packed)
2B4. Tips to improve scan results

2C1. 'sign2cpyA' converts the signscanA record layout to a COBOL copybook which
used by following scripts to perform various conversions.
If a copybook is supplied with the EBCDIC data file, you can start the
conversion at 'Part_3' & skip this Part_2.

2D1. 'cobmapA' converts the copybook automatically created by signscanA &
sign2cpyA into a record layout, which may then need manual corrections
since the signscan method is not perfect.

2E1. 'sign2mapA' script combines signscan2, sign2cpy1,& cobmap1
- the easy way to generate the cobmap from the data file with 1 command.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2A1. DATAcnv1 - Convert EBCDIC to ASCII without COBOL copybook

signscan2/signscanA - Op-Instrns

Please see page '1B4' for our operating environment, changed into cnvdata1 superdir with various subdirs (d1ebc,d2asc,...,signs,...,etc). Note that most of these procedures are uvcopy jobs which have long commands, but we have provided scripts to allow shorter commands. By convention the uvcopy jobs have a digit suffix (1) & scripts have an UPPER case alpha (A).


 #1a. uvcopy signscan2,fili1=d1ebc/citytax1,filo1=signs/citytax1,uop=r128
      ===================================================================

 #1b. signscanA citytax1 r128  <-- use script equivalent for shorter command
      =======================    - ksh for unix/linux or .bat file for Win/Dos

signscan2 option selection screen

 uop=r0000s0t500z5b2x10f1 - option defaults
     r????              - record size must agree with data file
     r8192              - record size max (increase via was= below)
          s0            - skip count before testing starts
            t500        - terminate count (end testing)
                z5      - Zoned field minimum lth for detection
                  b2    - pic x fields Ended by 2 EBCDIC blanks 0x4040
                    x10 - consolidate pic x fields with start bytes
                          within 10 of 1st of contiguous pic x fields
                  f1 - only 1 file output (final report)
                  f3 - 2nd file output before start byte dups reduced
                       to entry with Highest occurrences
                  f7 - 3rd file output (pack&zoned only) for debug
 121101:144530:signscan2: uvcopy ver=20121015 pf=/home/uvadm/pf/IBM/signscan2
 uvcopy LNX L64  license=20121015_99V_930630 site=UV_Software
 User OPtion (uop) defaults  = q1r0s0t500z5b2x10f1r128
 --> null to accept or re-specify (1 or more) --> r128   <-- ENTER REC SIZE
 121101:144531:signscan2: EOF fili01 rds=10 size=1280: d1ebc/citytax1s
 121101:144531:signscan2: EOF fili01 rds=10 size=1280: d1ebc/citytax1s
 121101:144531:signscan2: EOF filo01 wrts=20 size=690: signs/citytax1s
 EOJ, Output File written to: signs/citytax1s
 default command = null, OR enter: more,print,uvlpr12,edit,null -->

You must enter at least the record size option 'r128' in this case. Note the other option defaults & enter any you wish to change. For example if your date format was yymmdd & you wanted to test the 1st 5000 records you would specify options as --> r128d4t5000

The options are shown on the 3rd line of the output report (see next page) and in this example they would appear as:


 # default options=q1b2c1d0e99f80g33r0s0t200y4z15r128d4t5000
 ================================================***********

The default options on the left are overridden by any options you enter which appear here on the right (underlined by '*'s vs '='s).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2A2. DATAcnv1 - Convert EBCDIC to ASCII without COBOL copybook

signscan2/signscanA - Op-Instrns (repeated from previous page)


 #1a. uvcopy signscan2,fili1=d1ebc/citytax1,filo1=signs/citytax1,uop=r128
      ===================================================================

 #1b. signscanA citytax1 r128  <-- use script equivalent for shorter command
      =======================    - allows options for record-size, etc
NOTE
  • the option selection screen (see prior page) is omitted on this page
    to make room for illustrations of the 1st record of the input file
    and the entire output report.

 uvhd d1ebc/citytax1 r128a  <-- uvhd option 'a' (translate chars to ASCII)
 =========================

sample input file (uvhd display)

                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
             1013014000168508559800000000000000018150266550961400000000005153
          64 OUVER           BCV9S1H1..CK..Wb....g...qq.00014906A970530
             DEECD44444444444CCEFEFCF004D005880008000770FFFFFFFFCFFFFFF444444
             64559000000000002359218101320072C0047C0111C000149061970530000000

sample output report


 #2a. vi signs/citytax1         <-- inspect output with vi
      =================

 #2b. uvlp12 signs/citytax1     <-- list output with uvlp12
      =====================         (uvlpr12 for Windows/DOS batch script)
 # signscan2 - Determine EBCDIC data file layout 2012/11/01_14:42:14
 # recsize=00128,datafile=d1ebc/citytax1s
 # options=q1r0s0t500z5b2x10f1r128
 # total-records=00000010,records-sampled=00000010
 #start(length)   end   count (p=packed,z=+/- signs {A-I,}J-R)
 .00000(00008z) 00007 0000010
 .00008(00002x) 00009 0000000
 .00010(00025x) 00034 0000010
 .00035(00005z) 00039 0000002
 .00040(00020x) 00059 0000009
 .00060(00020x) 00079 0000010
 .00080(00008x) 00087 0000010
 .00088(00004b) 00091 0000010
 .00092(00005p) 00096 0000006
 .00097(00005p) 00101 0000010
 .00102(00005p) 00106 0000010
 .00107(00009z) 00115 0000008
 .00116(00006z) 00121 0000008
 .00122(00006x) 00127 0000000
 .00128(00000x) 00127 0000010

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2A3. DATAcnv1 - Convert EBCDIC to ASCII without COBOL copybook

                   ** signscan2 report field-layout
           1         2         3         4
 01234567890123456789012345678901234567890
 # signscan2 - Determine EBCDIC data file layout 2012/11/01_14:42:14
 # recsize=00128,datafile=d1ebc/citytax1s
 # options=q1r0s0t500z5b2x10f1r128
 # total-records=00000010,records-sampled=00000010
 #start(length)   end   count (p=packed,z=+/- signs {A-I,}J-R)
 .00000(00008z) 00007 0000010
 .00008(00002x) 00009 0000000
 .00010(00025x) 00034 0000010
 .00035(00005z) 00039 0000002
 .00040(00020x) 00059 0000009
 .00060(00020x) 00079 0000010
 .00080(00008x) 00087 0000010
 .00088(00004b) 00091 0000010
 .00092(00005p) 00096 0000006
 .00097(00005p) 00101 0000010
 .00102(00005p) 00106 0000010
 .00107(00009z) 00115 0000008
 .00116(00006z) 00121 0000008
 .00122(00006x) 00127 0000000
 .00128(00000x) 00127 0000010
 00(1) - '.' identifies an active entry (vs '#' comment)
 01(5)  - end position of signed field
 07(5)  - start position of signed field
 13(5)  - length of signed field
 18(1)  - field type (p,z,x,d) see descriptions below:
 21(7)  - count of records with this signed field detected

signscan2 field types detected

 'p'   - packed fields detected by scan for x'0C' or x'0D'
       - reliable since thee codes would not otherwise occur in EBCDIC files
 'z'   - zoned fields detected by scan for contiguous numerics 5 or more
       - may have units zoned signs x'C0' or x'D0' '{' or '}' +0 or -0
         or 'A'-'I' or 'J'-'R' for +1 thru +9 or -1 thru -9
       - you will have to split contiguous zoned fields if no signs
 'x'   - character field detected by scanning for blank fill on right side
       - detects fields such as name & address
       - field start assumed to be end of prior field which might not be
         correct if no packed, zoned, or other blank filled on left
 'b'   - 'binary' assumed if field starting with < x'40' does not have following
         valid packed sign within 10 bytes

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2B1. DATAcnv1 - Convert EBCDIC to ASCII without COBOL copybook

scanning for Packed & Signed fields

We are primarily concerned with accurate EBCDIC to ASCII file conversion preserving any packed fields & correcting any EBCDIC zoned signs to ASCII equivalents. Negatives '}JKLMNOPQR' to 'pqrstuvwxy', and positives '{ABCDEFGHI' to '0123456789'.

We only need to care about the accurate detection of packed & signed fields (type 'p' & 'z'). We don't need to worry about accurate boundaries for type 'x' fields since the entire record will be translated from EBCDIC to ASCII & then the packed & signed fields will be overlaid.

But if you also want to generate an accurate copybook (that reflects the application field boundaries), then you can modify the signscan results before conversion to copybook (or modify the copybook fields after).

After conversion to copybook (by sign2cpy1), you can assign meaningful names if desired. For the 1st 3 fields of the signscan2 results above, the default names would be f00000 (8 bytes), f00008 (2 bytes), & f00010 (25 bytes). Using your application knowledge, you might rename as 'rollno', 'flags',& 'owner-name'.

But remember if all you want is an accurate conversion, you don't need to do any alphanumeric field splitting & renaming. You only need to care about the accurate detection of packed & signed fields (types 'p' & 'z').

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2B2. DATAcnv1 - Convert EBCDIC to ASCII without COBOL copybook

signscan2 watch-out-for situations

'signscan2' will detect all packed & zoned signed fields that are logically possible to detect. However there are some data patterns where it is impossible to detect the boundaries for certain. You should understand & watch for the following possible situations.

#1 - unpacked signed fields without +0 or -0

Unpacked signed fields are detected by the sign codes in the units position.

--> Signed +0 thru +9 are x'C0' thru x'C9' displayed as '{ABCDEFGHI'. --> Signed -0 thru -9 are x'D0' thru x'D9' displayed as '}JKLMNOPQR'

See the EBCDIC file display on page '2A2', using uvhd with option 'a' to translate the character line to ASCII, but still displaying the zones & digits in hexadecimal.

+0 x'C0' '{' and -0 x'D0' '}' are unique but A-I & J-R are not unique.

#2 - unsigned fields prior to signed fields

If an unsigned field immediately precedes a signed field, we will not know where the signed field ends. For example consider the following file:

 product#  price   qty    amount
        unsigned signed    signed
 hammer  0000495000000A000000049E   <-- 4.95 * 1 =  4.95
 wrench  0000995000000C000000298E   <-- 9.95 * 3 = 29.85

The amount field is easily determined to be 9 digits long since the preceding field is also a signed field with unique codes in the units position. However we don't know for certain how long the 'qty' field is, since the preceding field is unsigned & has no unique codes in the zone of the units.

The good news here is that it does not matter, since the translations of unsigned & signed fields are the same except for the units position of the signed fields. In this example, signscan2 would report the price & qty as 1 signed field 14 digits long.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2B3. DATAcnv1 - Convert EBCDIC to ASCII without COBOL copybook

#3 - packed field Right boundary

Packed fields are identified by the sign codes in the right hand byte:

--> Negative fields will have x'0D' thru x'9D' in the right hand byte. --> Positive fields will have x'0C' thru x'9C' or x'0F' thru x'9F'.

x'_C' & x'_D' are unique, but some of the x'_F' bytes are not unique. x'4F' is ']', x'5F' is '^', x'6F' is '?', x'7F' is '"'.

If a packed field only had 1 of these last 4 codes in the sign byte (in all records tested), we could not uniquely identify it as a packed sign byte. But this is highly unlikely. We might increase our terminate count to test more records in the file.

#4 - packed field left boundary

The left boundary of a packed field is determined by scanning left until we reach a byte whose zone or digit is not 0 thru 9. This could be the sign byte of another packed field or any unpacked numeric, alpha, or special character.

If we encounter a x'40', we would not know for sure whether it is 2 digits of the packed field 4 & 0, or it could be an EBCDIC character blank. But if we encounter 2 blanks (x'4040'), we will assume this is the end of the packed field (a small possibility of ending too soon).

If a packed field always had 1 x'40' on its left side, we might report the length as 1 longer than it should be.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2B4. DATAcnv1 - Convert EBCDIC to ASCII without COBOL copybook

Tips to Improve Scan Results

  1. Adjust the scan terminate count at the prompt by signscan2. Option 't' defaults to t200. You might increase if you suspect that the record data is not uniform thruout the file. For example, the negative unpacked signed fields may be further down the file. This would only be significant if the +0's were just '0' x'F0' vs '{' x'C0'.

  2. You might set the max length for unpacked signed fields using option 'z' (default 'z50'). If you knew the actual max signed field length, you could prevent unsigned fields being considered part of max length signed fields.

  3. Probably the best way to verify the scan results is to continue these procedures until you get to 'uvhdcob'.


    2C1. sign2cpy1 - convert sign report to copybook
    2D1. cobmap1 - convert copybook to cobmap
    3D1. uvhdcob - display data field contents beside copybook field names

See the uvhdcob example on page '3D1'. The original auto assigned fieldnames (f00000, f00008, f00010, etc) have already been replaced with meaningful names, but that is not necessary to check field boundaries.

Using uvhdcob, you can quickly spot any packed or signed fields with incorrect boundaries.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2C1. DATAcnv1 - Convert EBCDIC to ASCII without COBOL copybook

sign2cpy1 - convert signscan2 report to COBOL copybook

Pre-requisite is uvcopy job 'signscan2' which created sign report file by scanning an EBCDIC data file for packed & zoned signs. This job will convert the sign report to a COBOL copybook which may be used as input for several Vancouver Utility jobs (see further below).


 #1a. uvcopy sign2cpy1,fili1=signs/citytax1,filo1=cpys/citytax1
      =========================================================
 #1b. sign2cpyA citytax1      <-- use script equivalent for shorter command
      ==================        - ksh for unix/linux or .bat file for Win/Dos

sample input (from signscan2)

 # signscan2 - Determine EBCDIC data file layout 2012/11/01_14:42:14
 # recsize=00128,datafile=d1ebc/citytax1s options=q1r0s0t500z5b2x10f1r128
 # total-records=00000010,records-sampled=00000010
 #start(length)   end   count (p=packed,z=+/- signs {A-I,}J-R)
 .00000(00008z) 00007 0000010
 .00008(00002x) 00009 0000000
 .00010(00025x) 00034 0000010
 .00035(00005z) 00039 0000002
 .00040(00020x) 00059 0000009
 .00060(00020x) 00079 0000010
 .00080(00008x) 00087 0000010
 .00088(00004b) 00091 0000010
 .00092(00005p) 00096 0000006
 .00097(00005p) 00101 0000010
 .00102(00005p) 00106 0000010
 .00107(00009z) 00115 0000008
 .00116(00006z) 00121 0000008
 .00122(00006x) 00127 0000000
 .00128(00000x) 00127 0000010
 ******* OUTPUT: copybook for: citytax1s
 ******* generated from signscan2 report 2012/11/01_14:42:21
        01  citytax1s.
            10 f00000                pic s9(00008).
            10 f00008                pic  x(00002).
            10 f00010                pic  x(00025).
            10 f00035                pic s9(00005).
            10 f00040                pic  x(00020).
            10 f00060                pic  x(00020).
            10 f00080                pic  x(00008).
            10 f00088                pic  x(00004).
            10 f00092                pic s9(00009) comp-3.
            10 f00097                pic s9(00009) comp-3.
            10 f00102                pic s9(00009) comp-3.
            10 f00107                pic s9(00009).
            10 f00116                pic s9(00006).
            10 f00122                pic  x(00006).

Before you attempt to make corrections & enhancements to the copybook, you should run cobmap1/cobmapA which converts the copybook to a 'cobmap' showing field start/end/length/type on the right side. Please see next page --->

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2D1. DATAcnv1 - Convert EBCDIC to ASCII with or without COBOL copybook

cobmap1/cobmapA - convert copybook to record-layout


 #1a. uvcopy cobmap1,fili1=cpys/citytax1,filo1=maps/citytax1
      ======================================================

 #1b. cobmapA citytax1       <-- use script equivalent for shorter command
      ================         - ksh for unix/linux or .bat file for Win/Dos

 #2a. more maps/citytax1       - display copybook map for review & enhancement
      ==================

 #2b. uvlp12 maps/citytax1     - or list copybook map with UV script for lasers
      ====================       (uvlpr12 for Windows/DOS batch script)

sample cobmap output report - from signscan2

 cobmap1  COBOL record layout            2012/11/01_14:42:22  pg# 0001
 cpys/citytax1s                 citytax1s          RCSZ=00128  bgn  end  lth typ
 * copybook for: citytax1s
 * generated from signscan2 report 2012/11/01_14:42:21
  01  citytax1s.
      10 f00000                pic s9(00008).                 0000 0007  008 ns 8
      10 f00008                pic  x(00002).                 0008 0009  002
      10 f00010                pic  x(00025).                 0010 0034  025
      10 f00035                pic s9(00005).                 0035 0039  005 ns 5
      10 f00040                pic  x(00020).                 0040 0059  020
      10 f00060                pic  x(00020).                 0060 0079  020
      10 f00080                pic  x(00008).                 0080 0087  008
      10 f00088                pic  x(00004).                 0088 0091  004
      10 f00092                pic s9(00009) comp-3.          0092 0096  005pns 9
      10 f00097                pic s9(00009) comp-3.          0097 0101  005pns 9
      10 f00102                pic s9(00009) comp-3.          0102 0106  005pns 9
      10 f00107                pic s9(00009).                 0107 0115  009 ns 9
      10 f00116                pic s9(00006).                 0116 0121  006 ns 6
      10 f00122                pic  x(00006).                 0122 0127  006
 *RCSZ=00128

 #3. vi cpys/citytax1      - modify the generated copybook (see next page)
     ================

Since the following conversions are based upon this automatically scan generated copybook, you should inspect it & if possible modify it to improve the results of the following data file conversions.

Please see the example on the next page --->

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2D2. DATAcnv1 - Convert EBCDIC to ASCII with or without COBOL copybook

sample signscan2 cobmap report (BEFORE enhancement)

  01  citytax1s.
      10 f00000                pic s9(00008).                 0000 0007  008 ns 8
      10 f00008                pic  x(00002).                 0008 0009  002
      10 f00010                pic  x(00025).                 0010 0034  025
      10 f00035                pic s9(00005).                 0035 0039  005 ns 5
      10 f00040                pic  x(00020).                 0040 0059  020
      10 f00060                pic  x(00020).                 0060 0079  020
      10 f00080                pic  x(00008).                 0080 0087  008
      10 f00088                pic  x(00004).                 0088 0091  004
      10 f00092                pic s9(00009) comp-3.          0092 0096  005pns 9
      10 f00097                pic s9(00009) comp-3.          0097 0101  005pns 9
      10 f00102                pic s9(00009) comp-3.          0102 0106  005pns 9
      10 f00107                pic s9(00009).                 0107 0115  009 ns 9
      10 f00116                pic s9(00006).                 0116 0121  006 ns 6
      10 f00122                pic  x(00006).                 0122 0127  006
 *RCSZ=00128

Enhancing the scan-generated copybook


 #3. vi cpys/citytax1      - modify the copybook (not the cobmap)
     ================
  1. Split type 'x' areas to known components
  2. Assign meaningful names
  3. Correct unreliable zoned signs (flagged with '?' on the signscan2 report) These were detected by A-I & J-R in numeric fields (vs signs { & } +0 & -0 which are more reliable).

    AFTER editing fieldnames & rerun cobmapA

 cobmap1  COBOL record layout            2012/12/04_14:55:51  pg# 0001
 cpys/citytax1                  citytax1           RCSZ=00128  bgn  end  lth typ
 * citytax1 copybook for uv demos www.uvsoftware.ca/datacnv1.
  01  citytax1.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-4.          0088 0091  004bns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  x(00006).                 0116 0121  006
      10 filler001             pic  x(00006).                 0122 0127  006
 *RCSZ=00128

Note that you should plan your changes using the 'cobmap' listing, but be sure to modify the copybook (not the cobmap). Then rerun the cobmapA script to regenerate & re-list the cobmap for the new copybook to verify that your changes result in the same record size.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2E1. DATAcnv1 - Convert EBCDIC to ASCII with or without COBOL copybook

sign2mapA - signscan2+sign2cpy1+cobmap1

'sign2mapA' script combines signscan2, sign2cpy1,& cobmap1, the easy way to generate the cobmap from the data file with 1 command.


 sign2mapA citytax1    <-- generate cobmap from datafile
 ==================
 #!/bin/ksh
 # sign2mapA - Korn shell script from UVSI stored in: /home/uvadm/sf/IBM/
 # sign2mapA - scan EBCDIC data file for packed/zoned signs, dates,& blanks
 #           - to determine record layout to assist conversion to ASCII
 #           - then convert signscan2 report to COBOL copybook
 #           - then convert copybook to cobmap
 #             (record layout start/end/lth on right side)
 #
 # This script combines scripts signscanA, sign2cpyA, & cobmapA
 # - makes it easier to run uvcopy jobs signscan2, sign2cpy1,& cobmap1
 #
 if [[ -f d1ebc/"$1" && -n "$2" ]]; then :
    else echo "usage: sign2mapA datafilename recsize"
         echo "       =============================="
         echo " - argg1 must be datafile in d1ebc/..., arg2 must be rec-size"
         echo " - output copybook in cpys/..., cobmap in maps/..."
         exit 1; fi
 #
 uvcopy signscan2,fili1=d1ebc/$1,filo1=signs/$1,uop=r$2
 #=====================================================
 uvcopy sign2cpy1,fili1=signs/$1,filo1=cpys/$1,uop=r$2
 #====================================================
 uvcopy cobmap1,fili1=cpys/$1,filo1=maps/$1,rop=r1
 #================================================
 exit 0
 #

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_3 Generating uvcopy jobs to convert data files

Part 3 - Generating uvcopy jobs to convert data files - Contents

Note
  • Start here if you have COBOL copybooks
  • don't need Part2 to scan data & create copybooks
Part3
  • test/demo conversions for datafilenames same as copybooknames
  • easier conversions for training, real datafilenames in Part6
  • after running cnvdatainit to clear files generated by prior test/demos
    #1-----> cp archive/d1ebc_cpy_names/* d1ebc/. <-- for Part3,4,5
    OR #2--> cp archive/d1ebc_data_names/* d1ebc/. <-- for Part6

3A1. EBCDIC to ASCII translation with packed field preservation
- uvcopy jobs & scripts
3A2.  Directories required & pre-requisites
- datafiles renamed same as copybooknames makes conversions easy to run
- convert all filenames to lower case & remove extensions
3A3.  Illustration - EBCDIC data file & corresponding COBOL copybook

3B1. Preparation to run or re-run test/demo conversions (for self-training)
cnvdatainit - to clear previous test/demo outputs
Restore d1ebc/... from archive/d1ebc_cpy_names or archive/d1ebc_data_names
cp archive/d1ebc_cpy_names/* d1ebc/. <-- for Part3,4,5 (vs Part6)

3B2. 'Generating' uvcopy jobs to convert EBCDIC to ASCII & preserve packed
 using uvcopy jobs (cobmap1 & uvdata51) OR script (gencnvA)
- alternate uvdata51p & gencnvAp to ensure packed sign x'_C' for AIX COBOL
3B3.  Operating Instructions to generate the conversion jobs from copybooks
3B4.  Sample generated job with explanations
3B5.  Alternative conversion for AIX COBOL with uvdata51p (vs uvdata51) to ensure
 packed field + signs x'_C' vs existing x'_F' signs in mainframe files

3C1. 'Executing' uvcopy jobs to convert EBCDIC to ASCII & preserve packed
3C2. Character translation & packed field preservation
3C3. Zoned sign conversion from EBCDIC to ASCII conventions
3C4. Manual changes required (if multiple record types)
3C5. Demo job after inserting record type test instructions

3D1. 'uvhdcob' displays the data beside the COBOL copybook fieldnames making
it easy to see if the copybook is consistent with the data. I recommend
you list out the 1st few records with both uvhd & uvhdcob to verify the
layout & resolve any problems if file is supplied by outside sources.

3E1. gencnvE - convert ASCII data files back to EBCDIC
- opposite of gencnvA
      uvcmpFE1 - might then compare converted file to the original EBCDIC file
               - uvcmpFE1 is 1 of several scripts using the 'uvcmp1' utility
               - see doc in CMPjobs.htm

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3A1. Converting Mainframe Data Files from EBCDIC to ASCII

Generating uvcopy jobs to convert data files - Introduction

The Vancouver Utilities have jobs to convert mainframe data files for use on Unix, Linux, or Windows.

A 'uvcopy job' is generated (for each data file) automatically from the COBOL copybook. The uvcopy job has 1 instruction to translate the entire record from EBCDIC to ASCII. Then we generate instructions to preserve any packed/binary fields which are the same on unix/linux/windows.

We assume here that the data files are intended for use by Micro Focus COBOL which provides for packed/binary fields exactly the same as on the mainframe.

'uvcopy jobs' to generate EBCDIC to ASCII converisons

cobmap1
  • converts COBOL copybooks to record layouts
  • used as input to the next step (uvdata51)
uvdata51
  • generates uvcopy jobs to translate EBCDIC files to ASCII,
  • preserving packed fields & correcting any zoned decimal signs
uvdata51p
  • alternate to ensure packed sign x'_C' for AIX COBOL
  • generates 1 mvn instruction per packed field
    (vs uvdata51 which groups packed fields together in 1 mvc instrn)

'scripts' to generate EBCDIC to ASCII converisons

Note
  • here are the script versions of the above 'uvcopy jobs'
  • they are easier to run (commands require fewer keystrokes)
cobmapA
  • converts COBOL copybooks to record layouts
gencnvA
  • generates uvcopy jobs to translate EBCDIC files to ASCII,
gencnvAp
  • ensure packed sign x'_C' for AIX COBOL

super scripts to generate ALL jobs

gencnvD2
  • script to generate ALL uvcopy jobs (from COBOL copybook)
    (cobmap1, uvdata51, genpipe1, genfixd2, sqlcreate1)
  • do not need genpipe1 & sqlcreate1 to just convert EBCDIC to ASCII
    but generating these jobs wont hurt & you might use them later
gencnvD2p
  • alternate version using uvdata51p to ensure packed field signs
    x'_C' positive or x'_D' negative (converts any x' F' to x'_C')
Note
  • I suggest you first use the step by step method for better understanding
  • then use the 'gencnvD2' to generate all jobs with 1 command
  • see gencnvD2 Operating Instructions on page '4H1'

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3A2. converting data files EBCDIC to ASCII, preserving packed

cnvdata1 subdirs (dtree illustration)

 /home/userxx          <-- conversions within your $HOME dir
 :-----cnvdata1        <-- cnvdata1 superdir for DATA conversion
 :     :-----archive     - backup to reload d1ebc/... copybooknames or datafilenames
 :     :     :-----d1ebc_cpy_names     <-- test/demo files for Parts3,4,5
 :     :     :     :-----armaster
 :     :     :     :-----arsales
 :     :     :     :-----artrans
 :     :     :     :-----citytax1
 :     :     :     :-----citytax2
 :     :     :     :-----glmaster
 :     :     :     :-----gltran
 :     :     :     :-----vendormas
 :     :     :     :-----vendorpaymas
 :     :     :-----d1ebc_data_names    <-- test/demo files for Part6
 :     :     :     :-----ar.customer.armaster
 :     :     :     :-----ar.customer.arsales
 :     :     :     :-----ar.customer.artrans
 :     :     :     :-----ar.customer.phone.list
 :     :     :     :-----city.tax.assess
 :     :     :     :-----city.tax.payments
 :     :     :     :-----gl.account.glmaster.g0001v00
 :     :     :     :-----gl.account.glmaster.g0002v00
 :     :     :     :-----gl.account.gltran.g0001v00
 :     :     :     :-----gl.account.gltran.g0002v00
 :     :     :     :-----vendor.master.names
 :     :     :     :-----vendor.master.payments
 :     :-----cpys       - copybooks
 :     :-----ctl        - control files
 :     :-----d0ebc      - EBCDIC data files, copy here 1st in case changes before d1ebc
 :     :-----d1ebc      - EBCDIC data files (named same as copybooks for part3,4,5)
 :     :-----d2asc      - ASCII data files (converted by uvcopy uvdata51)
 :     :-----d3ebc      - convert back to EBCDIC by uvdata31
 :     :-----d3asc      - reformatted data files (see genrfmA)
 :     :-----d4pipe     - pipe delimited to load DB tables (by genpipe1)
 :     :-----d5fix      - convert pipe|delimited back to fixed field (genfixd2)
 :     :-----listpipe   - pipe|delimited listed vertically with copybook fieldnames
 :     :-----maps       - copybooks with record layouts on right side
 :     :-----pfx1       - uvcopy jobs to convert EBCDIC to ASCII (gen by uvdata51)
 :     :-----pfx2       - with data filenames inserted (vs copybook names)
 :     :-----pfx3       - copied here for modify/execute
 :     :-----pfy1       - uvcopy jobs to convert ASCII to EBCDIC (gen by uvdata31)
 :     :-----pfp1       - uvcopy jobs to create pipe delimited (gen by genpipe1)
 :     :-----pfq1       - uvcopy jobs to convert pipe|delimited to fixed field (genfixd2)
 :     :-----pfg1       - uvcopy jobs to create test data via copybooks (gentest1)
 :     :-----pf         - misc uvcopy jobs
 :     :-----sf         - misc scripts
 :     :-----signs      - record layouts created by signscan2
 :     :-----sqlTC1     - scripts to Create database tables (gen from copybooks)
 :     :-----sqlTL1     - scripts to Load database tables (gen from copybooks)
 :     :-----tmp        - temp files (keep working directory clean)
 :     :-----ctl        - control-files (Part_6 for real datafilenames vs copybooknames)
 :     :     :-----datacpy51 - skeleton control-file generated from mainframe datafilenames
 :     :     :-----datacpy52 - control-file with copybooknames coded beside datafilenames

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3A3. converting data files EBCDIC to ASCII, preserving packed

Pre-Requisites

Datafiles are renamed the same as copybooknames to make conversion scripts easy to run. You only need to specify 1 filename to the conversion scripts. Various conversion jobs can be generated with the same name because they are in different subdirs. See 'Part_6' to convert real mainframe datafilenames aaa.bbb.ccc,etc

  1. data file stored in d1ebc/...
  2. data file renamed same as copybook (with any extension removed)
  3. copybook stored in cpys/... (with any extension removed)

    translate filenames to Lower case & remove extensions

Note
  • this has already been done for the test/demo files supplied in mvstest1
    (/home/uvadm/mvstest1/cnvdata1/d1ebc/...) copied to:
  • /home/mvstest1/cnvdata1/d1ebc/... OR /home/userxx/cnvdata1/d1ebc/...

 #1. renameL cpys        <-- convert all copybooknames to Lower case
     ============

 #2. renameL d1ebc       <-- enssure Lower case
     =============

 #3. mv d1ebc/datafilename d1ebc/copybookname
     ========================================
     - change datafilenames to match copybooknames (without .cpy extension)

Illustrating EBCDIC to ASCII conversion

We will illustrate the conversion using our demo data file 'd1ebc/citytax1' and the corresponding COBOL copybook 'cpys/citytax1.cpy'. We need to use the 'uvhd' utility to display the data file since it is EBCDIC with packed fields.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3A4. converting data files EBCDIC to ASCII, preserving packed

sample input file (uvhd display)


 uvhd d1ebc/citytax1 r128a  <-- uvhd option 'a' (translate chars to ASCII)
 =========================
                  10        20        30        40        50        60
 r#    1 0123456789012345678901234567890123456789012345678901234567890123
       0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
         FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
         1013014000168508559800000000000000018150266550961400000000005153
      64 OUVER           BCV9S1H1..CK..Wb....g...qq.00014906A970530
         DEECD44444444444CCEFEFCF004D005880008000770FFFFFFFFCFFFFFF444444
         64559000000000002359218101320072C0047C0111C000149061970530000000

corresponding COBOL copybook (with layout on right)

 cobmap1  COBOL record layout            2012/02/19_09:30:13  pg# 0001
 mf/cpys/citytax1.cpy          citytax1           RCSZ=00128  bgn  end  lth typ
  01  citytax1.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-4.          0088 0091  004bns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  x(00006).                 0116 0121  006
      10 filler001             pic  x(00006).                 0122 0127  006
 *RCSZ=00128

uvcopy data conversion job - generated from copybook

 # citytax1 - uvcopy job, EBCDIC to ASCII, preserve packed, fix signs
 opr='citytax1 - uvcopy code generated from copybook: citytax1.cpy '
 was=a33000b33000
 fili1=?d1ebc/citytax1,rcs=00128,typ=RSF
 filo1=?d2asc/citytax1,rcs=00128,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
        mvc    b88(4),a88              bns post-date
        mvc    b92(15),a92             pns land-value:face-value
        trt    b107(9),$trtsea          ns maint-tax
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3B1. Preparation to run or Re-Run test/demo conversions

We assume you have installed Vancouver Utilities on your computer & copied the test/demo files from /home/uvadm/mvstest/mvstest1/* to separate userid mvstest1 or to subdir ccnvdata1 within your homedir (see page '1B1' & 1B2).


 cp -r /home/uvadm/mvstest/mvstest1/cnvdata1  /home/userxx/.
 ===========================================================

Alternativly, UV Software can provide you with a guest userid/password, so you can test the data file conversions without having to install on your own computer.

Login as mvstest1, yourself, or ssh guest@uvsoftware.ca


 #0a. login mvstest1 --> /home/mvstest1
 #0b. cd cnvdata1    --> /home/mvstest1/cnvdata1
       ---------------- OR ---------------
 #0a. login youruserid ---> yourhomedir (/home/yourid)
 #0b. cd cnvdata1        <-- change into conversion dir in your homedir
       ---------------- OR ---------------
 #0a. ssh uvsoft99@uvsoftware.ca  <-- login with assigned userid/password
 #0b. cd cnvdata1        <-- change into conversion dir on UV Software
 #0c. cdc                <-- cdc alias for 'cd cnvdata1'

Preparation to Re-Run test/demo conversions

If you have already run the test/demo conversions & you or others wish to re-run, you will want to clear the outputs of prior test/demo conversions, for easier inspection of your new outputs & avoid confusion with prior outputs.


 #1. cnvdatainit   <-- run script to clear any old outputs
     ===========     - d2asc,d4pipe,pfx1,pfx2,pfx3,pfp1,etc
                     - see script listed on page '10K1' or in $UV/sf/IBM/...

 #2. rm -f d1ebc/*  <-- clear d1ebc/... in case retest Part3 after Part6
     =============

 #3. cp archive/d1ebc_cpy_names/* d1ebc/.  <-- for Part3,4,5
     ====================================
      - refresh d1ebc/... files named same as copybooks (for Parts3,4,5)
Note
  • later for Part6, we will copy the real datafilenames to d1ebc/...

 #3a. cp archive/d1ebc_data_names/* d1ebc/. <-- Alternative for Part6
      =====================================   - do NOT do for Parts3,4,5

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3B2. converting data files EBCDIC to ASCII, preserving packed

Generating uvcopy job to convert EBCDIC to ASCII

          uvcopy cobmap1         uvcopy uvdata51
 copybook --------------> cobmap ---------------> uvcopy job (EBCDIC to ASCII)
 cpys/citytax1.cpy        maps/citytax1            pfx1/citytax1
 We will present 2 ways to generate the EBCDIC to ASCII conversion job
 - run the 2 uvcopy jobs separately (cobmap1 & uvdata51), OR run the script
 (gencnvA) which runs the 2 jobs within the script.

1st - convert the copybook to a cobmap (record layout)


 #1a. uvcopy cobmap1,fili1=cpys/citytax1.cpy,filo1=maps/citytax1
      ==========================================================
      - convert any 1 copybook to a 'cobmap' (record layout)
         - - - OR (shorter command, same effect) - - -

 #1b. cobmapA citytax1.cpy        <-- script to convert any 1 copybook to cobmap
      ====================          - same as above, but shorter command

copybook converted to cobmap (layout on right)

 cobmap1  COBOL record layout            2012/02/19_09:30:13  pg# 0001
 mf/cpys/citytax1.cpy          citytax1           RCSZ=00128  bgn  end  lth typ
 * citytax1 copybook
  01  citytax1.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-4.          0088 0091  004bns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  x(00006).                 0116 0121  006
      10 filler001             pic  x(00006).                 0122 0127  006
 *RCSZ=00128

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3B3. Converting Mainframe Data Files from EBCDIC to ASCII

Generating uvcopy jobs to convert data files


 #2a. uvcopy uvdata51,fili1=maps/citytax1,filo1=pfx1/citytax1  <-- uvcopy command
      =======================================================
         - - - OR (shorter command, same output & more) - - -

 #2b. gencnvA citytax1.cpy     <-- script equivalent of above (shorter command)
      ====================       - ksh for unix/linux or .bat file for Windows/Dos
                                 - see script listed on page '10C2'
Note
  • script 'gencnvA' is the only command you need to generate the uvcopy jobs
  • includes copybook conversion (don't need to run cobmap1 on page above)
Note
  • use uvdata51p or gencnvAp to ensure packed signs x'_C' for AIX COBOL

uvdata51/gencnvA output - uvcopy job to convert EBCDIC to ASCII

 # citytax1 - uvcopy job, EBCDIC to ASCII, preserve packed, fix signs
 opr='citytax1 - uvcopy code generated from copybook: citytax1.cpy '
 uop=q0
 was=a33000b33000
 fili1=?d1ebc/citytax1,rcs=00128,typ=RSF
 filo1=?d2asc/citytax1,rcs=00128,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
        mvc    b88(4),a88              bns post-date
        mvc    b92(15),a92             pns land-value:face-value
        trt    b107(9),$trtsea          ns maint-tax
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

Notes re uvcopy job to convert EBCDIC to ASCII

The uvcopy jobs are automatically generated by gencnvA/uvdata51 based on the copybook record definitions which define the locations of packed & zoned signed fields. These jobs are generated from a template into which we insert filenames, record sizes,& the instructions to preserve the packed fields & correct the zoned signs.

In the 'citytax1' example there were only 2 instructions inserted to preserve packed & correct signs, because the 4 packed fields were contiguous. But for some real life data files, there could be hundreds of packed & zoned signed fields. The number of mvc/trt instructions inserted depends on how contiguous the packed/zoned fields are. See detailed explanations on the next page --->

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3B4. Generating uvcopy jobs to convert DATA files

generated uvcopy job to convert EBCDIC to ASCII


 #01 # uvcopy job to translate EBCDIC to ASCII, preserve packed, fix zoned signs
 #02 opr='jobname=JOBNAME   - pfx2 name=datafilename'
 #03 opr='copybook=citytax1 - pfx1 name=copybookname'
 #04 uop=q0,was=a33000b33000
 #05 fili1=?d1ebc/citytax1,rcs=00128,typ=RSF
 #06 filo1=?d2asc/citytax1,rcs=00128,typ=RSF
 #07 @run
 #08        opn    all
 #09 loop   get    fili1,a0
 #10        skp>   eof
 #11        mvc    b0(00128),a0       move rec to outarea before field prcsng
 #12        tra    b0(00128)          translate entire outarea to ASCII
 #13 #      ---                <-- insert R/T tests here for redefined records
 #14        mvc    b88(4),a88              bns post-date
 #15        mvc    b92(15),a92             pns land-value:face-value
 #16        trt    b107(9),$trtsea          ns maint-tax
 #17 put1   put    filo1,b0
 #18        skp    loop
 #19 eof    cls    all
 #20        eoj

The uvcopy job generated by 'gencnvA/uvdata51' is re-listed above with line#s so we can give a brief explanation of the more relevant instructions. See all instructions documented in uvcopy3.htm.

 Lines 5&6 declare the I/O (fili1/filo1) filenames. The generation has
       inserted the correct filenames & record sizes.
 Line 8 (opn) opens all files.
 Line 9 (get) is the beginning of the record processing loop. 'get' reads
      the next record from fili1 into area 'a' starting at byte '0' (a0).
 Line 10 (skp>) will skip to eof at End Of File, because the 'get' will set
      the condition code '>' when there are no more records.
 Line 11 (mvc) moves the record to output area 'b' starting at byte '0' for
      a length of 128 bytes(recsize), from input area 'a' starting at byte '0'.
 Line 12 (tra) translates to ASCII, entire record in area 'b' from byte '0'.
 Line 13 is a #comment indicating where you can add code to test Record Types.
 Line 14 (mvc) moves the binary field from the input area 'a' beginning at
      byte 88 to the output area 'b' beginning at byte 88 for a length of (4),
 Line 15 (mvc) moves the packed fields from the input area 'a' beginning at
      byte 92 to the output area 'b' beginning at byte 92 for a length of (15),
      which is the total of 3 packed fields (5+5+5).
 Line 16 (trt) translates the zoned signed field b107(9) using translate
      table 'trtsea', which translates Ebcdic zoned signs to Ascii zoned signs.
 Line 17 (put) writes to the output file from area 'b' starting from byte '0'
      for the record length declared on the filo1= (line 6).
 Line 18 'skp' returns to get the next record & repeat loop until EOF

Line 19 closes the files & line 20 ends the job.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3B5. uvcopy jobs to convert EBCDIC to ASCII

ensure packed signs x'_C' for AIX COBOL

Use alternates uvdata51p or gencnvAp to ensure packed signs x'_C' for AIX COBOL vs uvdata51/gencnvA which do not change any existing x'_F' signs. But uvdata51p generates 1 mvn instruction per packed field vs uvdata51 which groups all contiguous packed fields together in 1 mvc instruction.


 #2a. uvcopy uvdata51p,fili1=maps/citytax1,filo1=pfx1/citytax1  <-- uvcopy command
      ========================================================

 #2b. gencnvAp citytax1.cpy     <-- script equivalent of above (shorter command)
      =====================

sample generated by uvdata51p vs uvdata51


 #01 # uvcopy job to translate EBCDIC to ASCII, preserve packed, fix zoned signs
 #02 opr='jobname=JOBNAME   - pfx2 name=datafilename'
 #03 opr='copybook=citytax1 - pfx1 name=copybookname'
 #04 uop=q0,was=a33000b33000
 #05 fili1=?d1ebc/citytax1,rcs=00128,typ=RSF
 #06 filo1=?d2asc/citytax1,rcs=00128,typ=RSF
 #07 @run
 #08        opn    all
 #09 loop   get    fili1,a0
 #10        skp>   eof
 #11        mvc    b0(00128),a0       move rec to outarea before field prcsng
 #12        tra    b0(00128)          translate entire outarea to ASCII
 #13 #      ---                <-- insert R/T tests here for redefined records
 #14        mvc    b88(4),a88              bns post-date
 #15        mvn    b92(5p),a92(5p)          pns land-value
 #16        mvn    b97(5p),a97(5p)          pns improv-value
 #17        mvn    b102(5p),a102(5p)        pns face-value
 #18        trt    b107(9),$trtsea          ns maint-tax
 #19 put1   put    filo1,b0
 #20        skp    loop
 #21 eof    cls    all
 #22        eoj

compare job generated by uvdata51p vs uvdata51

Compare this job to the job listed on the previous page. It is the same except that 3 separate 'mvn' instructions are generated for the 3 packed fields vs uvdata51 which combined the 3 contiguous 5 byte packed fields into 1 x 15 byte 'mvc' instruction.

uvdata51p is provided to ensure packed field signs are x'_C' vs x'_F' existing in many packed fields in files transferred from a mainframe. This was provided for AIX COBOL vs Micro Focus COBOL.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3C1. uvcopy jobs to convert EBCDIC to ASCII

allowing for modifications before Execution


 #3. cp pfx1/citytax1 pfx3    <-- copy auto-generated job to pfx3
     =====================      - before any manual changes & execution

 #4. vi pfx3/citytax1         <-- make any manual changes required
     ================             (if multiple record types)
Note
  • pfx2 reserved (for Part_6) to insert real datafilenames vs copybooknames
  • so uvcopy jobs always executed from pfx3/...

Executing generated job to convert EBCDIC to ASCII


 #1a. uvcopy pfx3/citytax1,fili1=d1ebc/citytax1,filo1=d2asc/citytax1
      ==============================================================

 #1b. uvcopy pfx3/citytax1   <-- easier (filenames default as shown above)
      ====================

sample input EBCDIC


 #2. uvhd d1ebc/citytax1 r128a  <-- display EBCDIC input file
     =========================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
             1013014000168508559800000000000000018150266550961400000000005153
          64 OUVER           BCV9S1H1..CK..Wb....g...qq.00014906A970530
             DEECD44444444444CCEFEFCF004D005880008000770FFFFFFFFCFFFFFF444444
             64559000000000002359218101320072C0047C0111C000149061970530000000

sample output ASCII


 #3. uvhd d2asc/citytax1 r128   <-- display ASCII output file
     ========================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             3333333322444424445522222222222222233332445442544422222222225444
             1013014000AF8E085E29000000000000000181502F75E02F14000000000061E3
          64 OUVER           BCV9S1H1..C...W.........qq.000149061970530
             455452222222222244535343004D005880008000770333333333333333222222
             F5652000000000002369318101320072C0047C0111C000149061970530000000

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3C2. Converting Mainframe Data Files from EBCDIC to ASCII

Notes re EBCDIC to ASCII conversion

  1. Note that the character fields have been translated from EBCDIC to ASCII For example the 1st record byte '1' is x'F1' in the EBCDIC record, but it is x'31' in the ASCII record.

  2. The reason we can read the character lines in the uvhd EBCDIC display is because we specified option 'a' (r128a) to translate the character line of the EBCDIC file display (vs the uvhd ASCII display which omitted option 'a').

  3. Note that the packed fields are preserved in the ASCII file, because the conversion job (see prior page) moved them (unchanged) from input to output after the translation of the entire record (which would have destroyed them).

  4. Note the zoned sign at byte 115 (51+64) has been corrected from 'A' (x'C1') in EBCDIC to '1' (x'31') in ASCII. See more explanation on the next page -->

    converting Zoned signs

Zoned (unpacked) signs must be converted from the mainframe codes to the codes accepted by Micro Focus COBOL on Unix/Linux/Windows systems.

Notes

  1. Without identifying & correcting signed fields, you would simply get the EBCDIC alpha codes translated to their ASCII alpha codes, which would not be meaningful to Micro Focus COBOL on Unix/Linux/Windows.

  2. The 'trt' instruction with translate table '$trtsea' is used to translate EBCDIC zoned signs to ASCII zoned signs, for example:


    trt   b107(9),$trtsea
    =====================
  1. The above instruction translates the entire field, but it is only the units position (where the sign is) that needs the translation. (IE - the translate does not change any digits, only the alpha sign codes)

  2. The fact that ASCII signs appear as lower case alphas 'pqrstuvwxy' is simply a byproduct of them having x'7_' zones.

  3. There is no separate set of codes for +digits on unix, x'3_' zones are the only + zones required.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3C3. Converting Mainframe Data Files from EBCDIC to ASCII

Here is a sample sign conversion based on record #8 of the demo file citytax1. I am showing only bytes 64-128 of these 128 byte records & have replaced some bytes surrounding the signed field in bytes 107-115 with '...'. Note that 'R' EBCDIC -9 x'D9' is converted to 'y' ASCII -9 x'79'.

          64 OUVER           ALT6C4J2.Ri....Y@..........00000909R............
             DEECD44444444444CDEFCFDF0589000570000000090FFFFFFFFDFFFFFFFFFFFF
             645590000000000013363412029C0039C0000C0099C000009099980601190157
          64 OUVER           ALT6C4J2.R.....Y|..........00000909y............
             4554522222222222445343430589000570000000090333333337333333333333
             F5652000000000001C4634A2029C0039C0000C0099C000009099980601190157

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3C4. uvcopy jobs to convert EBCDIC to ASCII

Manual Changes Required ??

Manual changes may be required to the auto-generated uvcopy job, before you get the optimum results. The most common reason is to allow for multiple record types (with different layouts).

I recommend you do NOT modify the auto-generated jobs in subdir 'pfx1', but first copy them to subdir 'pfx3' before making any manual changes. This also protects your manual changes in pfx3 in case of regenerating (after changes to the copybook.).

Note
  • pfx2 is reserved in case you use uvdata52 to insert datafilenames
  • see discussion on pages '6B3' & '6C6'

Regardless of whether changes are required or not, I suggest you always copy the jobs from pfx1 to pfx3, before executing (from pfx3). Often you wont realize you need modifications until after you inspect the output data.


 #3.  cp pfx1/citytax1 pfx3   - copy auto-generated job to pfx3
      =====================     before any manual changes &/or execution

 #4.  vi pfx3/citytax1        - make any manual changes required
      ================          (record type tests, see example page '5B2')

sample manual change - Record-Type test

The most common change required is to insert code to test record types for files with redefined records (different layouts in the same file). Our 'citytax1' test file does not have multiple record types and the generated job requires no changes before execution.

But for illustration purposes, let us suppose it had a header record with no packed or zoned fields. It is a simple matter to insert 2 instructions to test for this & bypass the packed/zoned instructions which would destroy parts of our all character header record. We will assume that the header record is identified with an 'H' in byte 8 (zero relative).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3C5. uvcopy jobs to convert EBCDIC to ASCII

demo job after inserting Record-Type test code

 # citytax1 - uvcopy job, EBCDIC to ASCII, preserve packed, fix signs
 opr='citytax1 - uvcopy code generated from copybook: citytax1.cpy '
 uop=q0
 was=a33000b33000
 fili1=?d1ebc/citytax1,rcs=00128,typ=RSF
 filo1=?d2asc/citytax1,rcs=00128,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #------------------------------------------------------------------------
 #      ---                <-- insert R/T tests here for redefined records
        cmc    b8(1),'H'          Header record ? (no packed/zoned fields)
        skp=   put1               ifso - bypass packed/zoned mvc/trt
 #------------------------------------------------------------------------
        mvc    b88(4),a88              bns post-date
        mvc    b92(15),a92             pns land-value:face-value
        trt    b107(9),$trtsea          ns maint-tax
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj
 We have inserted instructions to compare (cmc) byte 8 to 'H' & if =, to skip
 to the 'put1' label, bypassing the mvc/trt which would destroy part of our
 header record. I have also added the '#------' lines to emphasize our inserted
 instructions, but you would not need to bother with these.

See the Operating Instructions to execute the uvcopy job on the next page --->

Note that we did not need to modify the 'citytax1' job before executing since the citytax1 datafile does not in fact have any header record.

For an example of a multi record type file (with 3 record types) & the coding changes required, please see the 'citytax2' demo beginning on page '5A1'.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3D1. Converting Mainframe Data Files from EBCDIC to ASCII

uvhdcob - list data fields beside copybook field names

'uvhdcob' is similar to 'uvhd', but also lists the copybook fieldnames beside the data field contents. You must have run cobmap1/cobmapA to convert the copybook to the 'cobmap' required by uvhdcob. To display our citytax1 demo file, we will specify option 'r128' for recsize + option 'a' to translate the characters line to ASCII.


     uvhdcob datafile cobmap [options]       <-- command format
     =================================

 #1. uvhdcob d1ebc/citytax1 maps/citytax1 a  <-- display citytax1 EBCDIC file
     ======================================      'a' option translate to ASCII
 d1ebc/citytax1 rec#=1 recs=10 rsz=128 fsiz=1280 fptr=0
 cobmap=maps/citytax1 today=0306092049 lastmod=02120515
 rec#       1 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 folio                        0   9    10130140
 002 name                        10  34    JOHN HENRY
 003 address                     35  59    1815 BOWEN ROAD
 004 city                        60  79    VANCOUVER
 005 province                    80  81    BC
 006 zip                         82  87    V9S1H1
 007 post-date                   88  91pns 0082898C
 008 land-value                  92  96pns 000057828C
 009 improv-value                97 101pns 000004870C
 010 face-value                 102 106pns 000171710C
 011 maint-tax                  107 115 ns 00014906A
 012 purchase-date              116 121    970530
 013 filler001                  122 127    191809
 rec#=1 of 10,null=next,r#=rec,s=search,u=update,p=print,i=iprint,w=write
 ,t=tally,c=chkseq,e=exit to uvhd,q=quit,?=help -->
 enter command --> p1    <-- print 1st record
 enter command --> q     <-- quit

Note you can use uvhdcob on either the EBCDIC file or the ASCII file. For the EBCDIC file, specify option 'a' to translate to ASCII (as above). For the ASCII file, omit the 'a' option. You might want to specify option 'm' to increase the 'maximum' lines per screen if you have records > 320 bytes.


 #1a. uvhdcob d2asc/citytax1 maps/citytax1 m60  <-- ASCII data + fieldnames
      ========================================    - 60 lines/scrn or recsize

 #1b. --> i3f1   <-- immediate print 1st 3 records, formfeed each page
          ====

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3E1. Converting Mainframe Data Files from EBCDIC to ASCII

convert ASCII files back to EBCDIC


 #1a. mkdir pfy1           <-- make subdir for ASCII to EBCDIC jobs
      ==========

 #1b. mkdir d3ebc          <-- make subdir for output datafile
      ===========

 #2. gencnvE citytax1.cpy  <-- generate ASCII to EBCDIC conversion job
     ====================

 #3. uvcopy pfy1/citytax1  <-- execute ASCII to EBCDIC job
     ====================

 #4. uvcmpFE1 d1ebc/citytax1 d3ebc/citytax1 r128y1
     =============================================
     - compare original EBCDIC file to file converted to ASCII & back to EBCDIC
     - see uvcmpFE1 script documented at CMPjobs.htm#1F1
     - 'r128' is option for record-sizes
     - 'y1' is option to translate character line (of vertical hexadecimal)
       to ASCII (so you can read it on the unix/linux machines)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_4 converting fixed-field records to pipe delimited

Part 4 - Contents


4A1. Jobs/Scripts to generate pipe delimit & SQL table create/load
4A2. Directories required for pipe-delimit & SQL table create/load
4A3. Illustration converting data to pipe-delimited text files for loading RDBMS
Packed/binary/numeric fields converted to zero suppressed numeric fields
with leading separate signs & decimal points inserted.

genpipe3/genfixd3 new jobs to delimit/undelimit

In Oct 2014, we developed genpipe3/genfixd3 (alternates to genpipe1/genfixd2), to remove the restriction of 100 bytes max field length (& also faster). The new jobs (genpipe3/genfixd3) are documented begining on pages '4M1' & '4N1'. The original jobs (genpipe1/genfixd2) are documented first as follows and this documentation contains several items relevant to both methods.

genpipe1/genfixd2 older jobs to delimit/undelimit


4B1. 'genpipe1' utility to convert fixed-field data to a '|' pipe delimited text.
- also see genpipe3 '4M1' to increase max field length from 100 to 32,767
4B2. - EBCDIC to ASCII conversion before pipe delimit (could combine)
4B3. - Command line instruction & arguments for genpipe1 (pipe delimit job)
4B3. - OPtions for genpipe1, prompts to allow change
4B5. - listing of generated job to create pipe|delimited file
4B6. - combining EBCDIC to ASCII & pipe|delimit to save execution time
- also see genfixd3 '4N1' to convert pipe delimited back to fixed layout

4C1. 'Executing' the generated jobs to convert data to pipe delimited text.
4C2. listings of test/demo input/output files

4D1. 'Modifying' the generated jobs to convert data to pipe delimited text.
- options for different field delimiters, etc
4D2.  Option to input EBCDIC file directly
4D3.  Option s0/s1 for Intel or RISC machines

4E1. 'listpipe1' lists a '|' delimited output file to verify conversion.
The copybook fieldnames are listed beside the data field contents.

4F1. Converting Data Files to ASCII Delimited for loading DB tables
- generate SQL scripts to create & load table
4F2.  Oracle SQL demo - CREATE & LOAD table
4F3.  Oracle 'batch-file' to create table
4F4.  Oracle SQL script to LOAD table
4F5.  create & load table using SQL scripts
- using SQL*PLUS to verify table loaded OK
- using SQL script to list table rows

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part 4 - Contents (continued)


4G1. script 'gencnvD1' to generate all conversion jobs including the scan data
 jobs that assist in creating copybooks for sites that do not have copybooks
 (signscan2, sign2cpy1, cobmap1, uvdata51, genpipe1, sqlcreate1).
- gencnvD1 is NOT practical, scanning for layout usually needs changes

4H1. script 'gencnvD2' generates all conversion jobs for any 1 copybook
 (cobmap1, uvdata51, genpipe3, genfixd3, sqlcreate1).
- updated Oct2014 to run genpipe3 (vs genpipe1)
- this script saves time (all jobs generated with minimal effort)
- usually need no manual changes (unless file has multiple record types)

4H4. script 'gencnvD9' generates ALL conversion jobs for ALL copybooks in cpys/
- great time saver when you have many files to convert
 script 'uvcopypfx1' executes pfx1/... & pfp1/... jobs for 1 file
 script 'uvcopypfx2' same as uvcopypfx1, but includes uvhd & uvhdcob displays
 script 'uvcopypfx9' executes pfx1/... & pfp1/... jobs for ALL files

4I1. 'genrfmA' generate jobs to reformat records based on 2 copybooks
 Example to unpack all packed/binary fields to numeric zoned fields
- Use genrfmA when the application for which the data file is intended
  does not allow packed fields.

4J1. Convrting Delimited format back to Fixed-field format
'genfixd2' generates jobs to convert 'delimited' format to 'fixed field'
formats described by COBOL copybooks. This job is useful when you need to
convert delimited files from DB2 'export's (on UNIX) to fixed field format
that was created by DB2 'unload's (on the mainframe). Note that mainframe
'UNLOAD' creates 'Fixed-field', but unix 'EXPORT' creates 'Delimited'.

4J2. We will demonstrate using 'citytax1' demo copybook/cobmap & data file.
You could repeat this demo in your homedir assuming Vancouver Utilities
installed & your profile updated as described at
'https://uvsoftware.ca/jclcnv1demo.htm#Part_1'

4J3. maps/citytax1 - sample copybook/map to demo genfixd2 & genpipe1
4J4. pfq1/citytax1 - sample job to convert Delimited to Fixed-field
generated by genfixd2 from copybook/map
4J5. pfp1/citytax1 - sample job to convert Fixed-field to Delimited
generated by genpipe1 from copybook/map

4J6. Copy pfq1 & pfp1 jobs to pfq2 & pfp2 before any modifiactions
Modify pfp2/citytax1 subdirs for this demo

4J7. Execute job to convert 'pipe|delimited' to 'Fixed-field'
d4pipe/citytax1 - sample input Pipe|Delimited
d5fix/citytax1 - sample output Fixed-Field record

4J8. Execute COBOL program to process Fixed-Field format
- we will simply copy file for this demo
 Execute job to convert 'Fixed-Field' to 'pipe|delimited'
 d6fix/citytax1 - sample input Fixed-Field record
 d7pipe/citytax1 - sample output Pipe|Delimited
 Execute 'diff' to prove delimited output same as original input

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part 4 - Contents (continued)


4J9. Demo BIG-end/little-end binary & file compare utility 'uvcmp1'
- Re-execute delimited to fixed with option 's0' to create Little-End
- Compare to prior Big-end file
- compare report highlights differences via '*'s & vertical hex

4J10. Understanding BIG-end & Little-end binary
- illustrate converting hex to decimal, proving same value as input

4K1. genpipe1 will expand 'item occurs' to pipe|delimit occurs fields
- demo using copybook armaster.cpy with occurs 12 for monthly sales

4L1. cpyoccurs1 will expand group occurs for genpipe1 pipe|delimit
- demo using copybook custran1 with group occurs 5 for 3 pics

genpipe3/genfixd3 new jobs to delimit/undelimit

In Oct 2014, we developed genpipe3/genfixd3 (alternates to genpipe1/genfixd2), to remove the restriction of 100 bytes max field length (& also faster).


4M1. 'genpipe3' utiltity to convert data to a '|' pipe delimited text.
- Op. Instrns. & options
4M2.  sample copybook/cobmap input maps/citytax1
- field type conversion rules
4M3.  sample generated job pfp1/citytax1
4M4.  sample job execution & results
- input d2asc/citytax1 (uvhd display)
- output d4pipe/citytax1 (pipe|delimited text, numeric fields edited)

4N1. 'genfixd3' utiltity to convert delimited files back to fixed-field
- Op. Instrns. & options
4N2.  sample copybook/cobmap input maps/citytax1
- field type conversion rules
4N3.  sample generated job pfp1/citytax1
4N4.  sample job execution & results
- input d4pipe/citytax1 (pipe|delimited text, numeric fields edited)
- output d5fix/citytax1 (uvhd display)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4A1. Converting Data Files to ASCII Delimited for loading DB tables

uvcopy jobs to generate pipe delimit & SQL table create/load

cobmap1
  • converts COBOL copybooks to record layouts
  • used as input to the next step (uvdata51)
uvdata51
  • generates uvcopy jobs to translate EBCDIC files to ASCII,
  • preserving packed fields & correcting any zoned decimal signs
uvdata51p
  • alternate to ensure packed sign x'_C' for AIX COBOL
  • generates 1 mvn instruction per packed field
    (vs uvdata51 which groups packed fields together in 1 mvc instrn)
genpipe1
  • reads the cobmap record layout
  • generates uvcopy job to create pipe-delimited file to load SQL tables
  • packed/binary fields unpacked & edited with separate signs & decimal
    points as required for loading RDBMSs.
sqlcreate1
  • generate SQL scripts to create & load table

scripts to generate pipe delimit & SQL table create/load

Note
  • here are the script versions of the above 'uvcopy jobs'
  • they are easier to run (commands require fewer keystrokes)
cobmapA
  • converts COBOL copybooks to record layouts
gencnvA
  • generates uvcopy jobs to translate EBCDIC files to ASCII,
    and to create 'pipe|delimited' format
gencnvAp
  • ensure packed sign x'_C' for AIX COBOL
genpipeA
  • generates uvcopy job to create pipe-delimited file to load SQL tables
  • also generates pipe|delimited back to Fixed-field (as of Feb2012)
sqlcreateA
  • generate SQL scripts to create & load table
gencnvD2
  • super-script to generate All data conversion jobs for any 1 file
  • all jobs named same as input copybook, but in different subddirs
    (pfx1, pfp1, pfq1, pfy1, pfg1, sqlTC1, sqlTL1)
  • see listing on page '10D1'

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4A2. Converting Data Files to ASCII Delimited for loading DB tables

super script to generate ALL jobs for 1 copybook

gencnvD2
  • script to generate ALL uvcopy jobs (from COBOL copybook)
    (cobmap1, uvdata51, genpipe3, genfixd3, sqlcreate1)
  • updated Oct2014 to run genpipe3/genfixd3 (vs genpipe1/genfixd2)
  • do not need genpipe1/3 & sqlcreate1 to just convert EBCDIC to ASCII
    but generating these jobs wont hurt & you might use them later

 gencnvD2 citytax1.cpy  <-- script to generate all jobs + batch-file + loader-file
 =====================    - from the demo copybook 'cpys/citytax1'
Note
  • I suggest you first use the step by step method for better understanding
  • then use the 'gencnvD2' to generate all jobs with 1 command
  • see gencnvD2 Operating Instructions on page '4H1'

super script to generate ALL jobs for ALL copybooks

gencnvD9
  • generates ALL conversion jobs for ALL copybooks in cpys/
  • generates jobs in ppfx1, pfp1, pfy1, pfq1, pfg1, sqlTC1, sqlTL1
  • great time saver when you have many files to convert

super script to EXECUTE multiple jobs for 1/ALL datafiles


 uvcopypfx1 citytax1 <-- executes pfx1/citytax1 & pfp1/citytax1 for d1ebc/citytax1
 ===================   - also generates listpipe/citytax1 to check pipe delimited
                       - 1st 3 records listed vertically with copybook fieldnames

 uvcopypfx2 citytax1 <-- same as above, but also creates uvhd & uvhdcob displays
 ===================   - in subdirs uvhd_d1ebc, uvhd_d2asc, uvhdcob (d2asc only)
                         (makes subdirs if not present)

super script to EXECUTE ALL jobs for ALL datafiles


 uvcopypfx9 all  <-- executes ALL pfx1/... & pfp1/... for ALL datafiles d1ebc/...
 ==============    - also generates listpipe/... to check pipe delimited
                   - 1st 3 records listed vertically with copybook fieldnames

script 'copysub2' to collect all jobs & data for 1 file

copysub2
  • script to gather all jobs & data for 1 file, from subdirs to 1 outdir
  • appends _subdir names on filename, allowing cohabitation in 1 dir
  • might use to send results to other parties for inspection
Note
  • maintaining the same filename in multiple subdirs makes generations easy
    but does make it difficult to gather results in 1 directory
    (without the copysub2 script)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4A3. Converting Data Files to ASCII Delimited for loading DB tables

Directories Required

 /home/uvadm/mvstest/mvstest1  <-- location in Vancouver Utilities distribution
 /home/mvstest1        <-- OR setup user mvstest1
 /home/userxx              OR setup cnvdata1 within your $HOME dir
 :-----cnvdata1        <-- cnvdata1 superdir for DATA conversion
 :     :-----archive     - backup to reload d1ebc/... copybooknames or datafilenames
 :     :     :-----d1ebc_cpy_names    <-- for Parts3,4,5
 :     :     :-----d1ebc_data_names   <-- for Part6
 :     :-----cpys       - copybooks
 :     :-----ctl        - control files
 :     :-----d1ebc      - EBCDIC data files
 :     :-----d2asc      - ASCII data files (converted by uvcopy uvdata51)
 :     :-----d3ebc      - convert back to EBCDIC by uvdata31
 :     :-----d3asc      - reformatted data files (see genrfmA)
 :     :-----d4pipe     - pipe delimited to load DB tables (by genpipe1)
 :     :-----d5fix      - convert pipe|delimited back to fixed field (genfixd2)
 :     :-----listpipe   - pipe|delimited listed vertically with copybook fieldnames
 :     :-----maps       - copybooks with record layouts on right side
 :     :-----pfx1       - uvcopy jobs to convert EBCDIC to ASCII (gen by uvdata51)
 :     :-----pfx2       - with data filenames inserted (vs copybook names)
 :     :-----pfx3       - copied here for modify/execute
 :     :-----pfy1       - uvcopy jobs to convert ASCII to EBCDIC (gen by uvdata31)
 :     :-----pfp1       - uvcopy jobs to create pipe delimited (gen by genpipe1)
 :     :-----pfq1       - uvcopy jobs to convert pipe|delimited to fixed field (genfixd2)
 :     :-----pfg1       - uvcopy jobs to create test data via copybooks (gentest1)
 :     :-----pf         - misc uvcopy jobs
 :     :-----sf         - misc scripts
 :     :-----signs      - record layouts created by signscan2
 :     :-----sqlTC1     - scripts to Create database tables (gen from copybooks)
 :     :-----sqlTL1     - scripts to Load database tables (gen from copybooks)
 :     :-----tmp        - temp files (keep working directory clean)

Pre-Requisites

  1. data file already converted to ASCII & stored in d2asc/...

  2. data file renamed same as copybook (with any extension removed)

  3. copybook stored in cpys/... (with any extension removed)

Note
  • option 'a1' will generate pipe delimit jobs to input EBCDIC directly
  • but you would have to make manual changes if any packed fields
  • you might do this if you have very large files (to save time & storage)
  • see discussion later on page '4D2'

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4A4. Converting Data Files to ASCII Delimited for loading DB tables

sample data file with packed data


 uvhd d2asc/citytax1 r128    <-- display sample input data with 'uvhd'
 ========================      - note packed fields in bytes 88-106
                               - will show only 1st 2 of 10 records in the file
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             3333333322444424445522222222222222233332445442544422222222225444
             1013014000AF8E085E29000000000000000181502F75E02F14000000000061E3
          64 OUVER           BCV9S1H1......W.........qq.000149061970530191809
             4554522222222222445353430888005880008000770333333333333333333333
             F56520000000000023693181029C0072C0047C0111C000149061970530191809
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 10139923  GEORGE BROWN             1250 EAST PENDER STREET  VANC
             3333333322444544245454222222222222233332445525444452555445225444
             101399230075F275022F7E0000000000000125005134005E45203425540061E3
          64 OUVER           BCV5L1W1......W.........xRL00014626q980601190156
             4554522222222222445343530888005880008800754333333337333333333333
             F5652000000000002365C171029C0072C0047C0182C000146261980601190156

sample output - pipe delimited text

 folio|name|address|city|province|zip|post-date|land-value|land-value|improv-value|face-value|maint-tax|purchase-date|purchase-d
 10130140|JOHN HENRY|1815 BOWEN ROAD|VANCOUVER|BC|V9S1H1|+82898|+57828|+4870|+171710|+1490.61|970530| |
 10139923|GEORGE BROWN|1250 EAST PENDER ST.|VANCOUVER|BC|V5L1W1|+82898|+57828|+4878|+178524|-1462.61|980601| |
 10147615|BONNIE SMITH|44430 YALE ROAD WEST|VANCOUVER|BC|V2P6J1|+121395|+39623|+0|+51914|+376.92|950601| |
 10149304|TONY TATTERSAL|3350 S.W. MARINE DRIVE|VANCOUVER|BC|V6N3Y9|+50594|+76800|+0|+208391|+1925.03|930601| |
 10154688|MEL RAHAL|1110 - 625 HOWE STREET|VANCOUVER|BC|V6C2T6|+82898|+44880|+0|+125038|+1154.79|961126| |
 10173320|SAM SMITH|2381 ROGERS AVE|VANCOUVER|BC|V3K5Y2|+50498|+60600|+0|+169086|+1562.00|970530| |
 10204700|JOHN JENSEN|12611-100 ST|VANCOUVER|AL|T8V4H2|+122396|+15720|+0|+43744|-404.40|961126| |
 10223240|BOB ANDERSON|7215 ARGYLL ROAD|VANCOUVER|AL|T6C4J2|+52899|+3597|+0|+9990|-90.99|980601| |
 10308685|TOM PERRY|3932 - 3A ST. N.W.|VANCOUVER|AL|T2E6R4|+70192|+68155|+0|+125773|+1065.10|920601| |
 10318833|HARRY SMIRNOFF|BOX 308, STN J|VANCOUVER|AL|T2A4X6|+11399|+54541|+42046|+93502|+761.46|970530| |

The output file will be an all text ASCII file and can be easily listed with the 'uvlp' scripts or other unix utilities. We will later illustrate another utility 'sqlcreate1' to generate SQL scripts to create & load tables from the pipe delimted files into Relational Data Base tables (see page '4F1').

option to generate fieldnames as 1st record

The 1st record with copybook fieldnames was generated by genpipe1 option 'w1'. Illustrated only here, option default w0 will be used on following generations.

Above also shows '+' signs on numeric signed fields by option 'e2'. Default option e1 will show only '-' signs if negative (omitting '+' if positive).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4B1. Converting Data Files to ASCII Delimited for loading DB tables

Generating uvcopy job to convert data to pipe delimited text

          uvcopy cobmap1         uvcopy genpipe1
 copybook --------------> cobmap ---------------> uvcopy job (to pipe delimit)
 cpys/citytax1.cpy        maps/citytax1           pfp1/citytax1

We will present 2 ways to generate the pipe delimit job - run the 2 uvcopy jobs separately (cobmap1 & genpipe1), OR run the script genpipeA which runs the 2 jobs within the script. The only input required by genpipeA is the COBOL copybook for the data file.


 #1. genpipeA citytax1.cpy   <-- script to generate pipe|delimit job
     =====================
Note
  • see genpipeA script listed on page '10C3'
  • effect same as running following 3 commands separately

 #1a. uvcopy cobmap1,fili1=cpys/citytax1.cpy,filo1=maps/citytax1
      ==========================================================
      - convert any 1 copybook to a 'cobmap' (record layout)

 #1b. uvcopy uvdata51,fili1=maps/citytax1,filo1=pfx1/citytax1
      =======================================================
      - convert cobmap to uvcopy job to convert EBCDIC to ASCII

 #1c. uvcopy genpipe1,fili1=maps/citytax1,filo1=pfp1/citytax1
      =======================================================
      - convert data file to pipe|delimited (unpacks/edits packed fields)

input - copybook (cobmap listed below)

 cobmap1  COBOL record layout            2012/02/19_09:30:13  pg# 0001
 mf/cpys/citytax1.cpy            citytax1           RCSZ=00128  bgn  end  lth typ
 * citytax1 copybook
  01  citytax1.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-4.          0088 0091  004bns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  x(00006).                 0116 0121  006
      10 filler001             pic  x(00006).                 0122 0127  006
 *RCSZ=00128

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4B2. Converting Mainframe Data Files from EBCDIC to ASCII

generating pipe|delimit job - step#2 convert EBCDIC to ASCII


 #1b. uvcopy uvdata51,fili1=maps/citytax1,filo1=pfx1/citytax1
      =======================================================
 # citytax1 - uvcopy job, EBCDIC to ASCII, preserve packed, fix signs
 opr='citytax1 - uvcopy code generated from copybook: citytax1.cpy '
 uop=q0
 was=a33000b33000
 fili1=?d1ebc/citytax1,rcs=00128,typ=RSF
 filo1=?d2asc/citytax1,rcs=00128,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
        mvc    b88(4),a88              bns post-date
        mvc    b92(15),a92             pns land-value:face-value
        trt    b107(9),$trtsea          ns maint-tax
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

Notes

  1. The 'uvcopy uvdata51,...' command is step#2 of generating pipe|delimited if you wish to do it the "hard way" (running conversion steps separately) vs the "easy way" (using the "genpipeA" script).

  2. This 2nd step (converting EBCDIC to ASCII) is not required if you already have the ASCII file to be converted (not a mainframe conversion situation).

    execute job to convert ASCII to EBCDIC

The generated job (listed above) would be executed as follows


 2b. uvcopy pfx1/citytax1
     ====================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4B3. Converting Data Files to ASCII Delimited for loading DB tables

generating job to create pipe-delimited file

Note
  • we will show 3 commands to generate a pipe delimit job (#3a, #3b, #3c)
  • the easiest command is the script #3c genpipeA (vs uvcopy genpipe1)
  • #2a shows all options, #1b specifies I/O files & defaults options

 #3a. uvcopy genpipe1,fili1=maps/citytax1,filo1=pfp1/citytax1\
      ========================================================
             ,arg1=delimiter,arg2=enclose-character,uop=translate-option
             ===========================================================
             ,arg1=7C,arg2=20,uop=a0   <-- default arguments & options
             =======================
                       - arg1 & arg2 must be specified in hex (if specified)
             ,arg1=7C  - delimiter '|' x'7C' (the default, no need to specify)
             ,arg1=2C  - alternate delimiter ',' x'2C' - see example page '4D1'
             ,arg1=FC  - example for umlat x'FC' (u with 2 dots above)
              arg2     - enclose char for A/N fields, numeric fields not enclosed
             ,arg2=20  - field enclose char (blank x'20' default for none)
             ,arg2=22  - alternate enclose-char " x'22' - see example page '4D1'
             ,uop=a0   - do NOT translate input data (a0/off is the default)
             ,uop=a1   - translate input from EBCDIC to ASCII (see page '4D2')

 #3b. uvcopy genpipe1,fili1=maps/citytax1,filo1=pfp1/citytax1
      =======================================================
      - only need to specify I/O files if default delimiter & no translate OK

 #3c. genpipeA citytax1.cpy     <-- easy script equivalent of above (shorter command)
      =====================       - use this if defaults OK for arg1, arg2, uop
                                  - combines cobmap1 & genpipe1

genpipe3/genfixd3 alternates to genpipe1/genfixd2

In Oct 2014, we developed genpipe3/genfixd3 (alternates to genpipe1/genfixd2), to remove the restriction of 100 bytes max field length (& also faster). The new jobs (genpipe3/genfixd3) are documented begining on pages '4M1' & '4N1'.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4B4. Converting Data Files to ASCII Delimited for loading DB tables

genpipe1 options

Note
  • genpipe1 will prompt to allow change as shown below
  • options may be specified on the comamnd line ',uop=...'
  • option prompts may be inhibited by appending ',uop=q0i7' on the command line.
 genpipe1 - generate uvcopy code to convert files to | delimited format
          - default delimiter is pipe "|", modify via arg1=xx (in hex)
          - arg1=2C for comma, arg1=FC for umlat (u with 2 dots over)
  arg2 may specify a data enclose character for A/N fields (vs numeric)
  - must be specified as 2 hex digits (arg2=22 for double quote)
 uop=a0d0e1f1m0p0s0w0 - default options
     a0         - file already translated from EBCDIC to ASCII by uvdata51
     a1         - translate data from EBCDIC to ASCII here in genpipe1 job
                - must transfer any packed field instrns from the uvdata51
                  (subdir pfx1) to this genpipe1 job (subdir pfp1)
       d0       - do not test/convert any date/time fields
       d1       - convert date/time zero/blank input to blank/null output
       d2       - convert date/time zero/blank input to zeros output
       d4       - if Macrosoft date8 - insert bal cnvrt 00000000 to |null|
         e1     - edit signed fields default: leading "-" only if negative
         e2     - edit signed fields with + if pos (as well as - if neg)
           f0   - drop fieldnames begining with "filler"
           f1   - retain fieldnames begining with "filler"
             m1 - inhibit msg "field length > 100" (filler usually not problem)
           p1   - convert invalid packed fields to blank
         s0     - auto Binary Switch (Big-END AIX/SUN, little-end Intel)
                - use option s1/s2 only if generating code for other machines
         s1     - force no switch (default on BIG-end machines (AIX/SUN)
         s2     - force switch ends (default on Little-end machines (Intel)
       w1       - generate code to write fieldnames as 1st output record
 131021:121828:genpipe1: uvcopy ver=20131008 pf=/home/uvadm/pf/IBM/genpipe1
 uvcopy LNX H64 license=20131008_99V_930630 site=UV_Software
 User OPtion (uop) defaults  = q1a0d0e1f1m0p0s0w0
 -->null to accept or enter/override -->

You may enter 1 or more options to override the defaults, for example:

 --> e2w1  <-- e2 will insert '+' signs if positive (as well as '-' if negative)
             - w1 will generate 1st record with copybook fieldnames

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4B5. Converting Data Files to ASCII Delimited for loading DB tables

genpipeA sample output - '|' pipe delimited ASCII text

 # citytax1 - uvcopy job to convert EBCDIC/ASCII data to delimited text file
 opr='citytax1 - uvcopy code generated from copybook: citytax1.cpy '
 rop=j200000r1  #Run OPtions: increase instrn storage & prompt display output
 was=a33000b33000c999000d33000e33000u16384v16384  #increase u&v if fields > 163
 fili1=?d2asc/citytax1,rcs=00128,typ=RSF
 filo1=?d4pipe/citytax1,rcs=00384,typ=LSTtd3
 @run
        opn    all
        cat    j0(4000),'folio|name|address|city|province|zip|post-date|land-value|'
        cat    j0(4000),'improv-value|face-value|maint-tax|purchase-date|filler001|'
        put    filo1,j0(4000)             write fieldnames as 1st record
 loop   get    fili1,a0
        skp>   eof
 # area a input, see 'get' instrn
 # area b translated to ASCII, in case mainframe EBCDIC file (with packed?)
 # area c data fields fixed 100 bytes apart in prep for var | delimit instrn
 # area d output, see 'put' instrn at end
        mvc    b0(00128),a0       move input area a to area b
 ###    tra    b0(00128)          uncomment for EBCDIC input (or option a1)
 #      ---                   <-- & insert instrns from uvdata51 if packed fields
 #      ---                   <-- insert R/T tests if redef records
        mvc    c0(10),b0(10)                        #1 folio
        mvc    c100(25),b10(25)                     #2 name
        mvc    c200(25),b35(25)                     #3 address
        mvc    c300(20),b60(20)                     #4 city
        mvc    c400(2),b80(2)                       #5 province
        mvc    c500(6),b82(6)                       #6 zip
        edta1  c600(9),a88(4bs),'-zzzzzzzzz9'       #7 post-date
        edta1  c700(11),a92(5p),'-zzzzzzzzz9'       #8 land-value
        edta1  c800(11),a97(5p),'-zzzzzzzzz9'       #9 improv-value
        edta1  c900(11),a102(5p),'-zzzzzzzzz9'      #10 face-value
        edta1  c1000(11),b107(9),'-zzzzzzz.99'      #11 maint-tax
        mvc    c1100(6),b116(6)                     #12 purchase-date
        mvc    c1200(6),b122(6)                     #13 filler001
        rep    c0(01300),'|','!'           #comment speedup if no '|'s in data
        varb1  d0(01300),c0(100),0013,'|'  #insert '|'s between fields & squeeze
        trt    d0(01300),$trtchr           #ensure all ascii chars x'20'-x'7E'
        clr    c0(01300),' '
 put1   put    filo1,d0
        skp    loop
 #
 eof    cls    all
        eoj
 @pf2=genpipe1.sub
Note
  • the 'tra' above is ##commented out since input already translated
  • see below to combine EBCDIC to ASCII translate with pipe delimit job
  • would save time if you have very large files

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4B6. Converting Data Files to ASCII Delimited for loading DB tables

combining EBCDIC to ASCII & pipe|delimit jobs

You could combine the 2 generated conversion jobs above - EBCDIC to ASCII (on page '4B2') & pipe|delimit (on page '4B5'). This would save execution time when you have very large files to convert.

Note that the generated pipe-delimit job contains the following ##commented out instruction:

 ##     tra    b0(00128)          uncomment for EBCDIC input (or option a1)
 #      ---                   <-- & insert instrns from uvdata51 if packed fields
 #      ---                   <-- insert R/T tests if redef records

Note that the generated EBCDIC to ASCII job contained the 3 folowing instructions (to preserve packed/binary fields & correct zoned signs).

        mvc    b88(4),a88              bns post-date
        mvc    b92(15),a92             pns land-value:face-value
        trt    b107(9),$trtsea          ns maint-tax

So you could save the EBCDIC to ASCII preliminary step if you:

The result would be as follows (showing a few surrounding instrucitons)

        mvc    b0(00128),a0       move input area a to area b
        tra    b0(00128)          uncomment for EBCDIC input (or option a1)
 #      ---                   <-- & insert instrns from uvdata51 if packed fields
 #      ---                   <-- insert R/T tests if redef records
 # following 3 instrns transferred from EBCDIC to ASCII job to save execution time
        mvc    b88(4),a88              bns post-date
        mvc    b92(15),a92             pns land-value:face-value
        trt    b107(9),$trtsea          ns maint-tax
 #
        mvc    c0(10),b0(10)                        #1 folio
        mvc    c100(25),b10(25)                     #2 name
        mvc    c200(25),b35(25)                     #3 address
        ... see all on page '4B5' ...

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4C1. Converting Data Files to ASCII Delimited for loading DB tables

Executing pipe delimit jobs - Allowing for manual change

In case manual changes are later required, I recommend you copy the jobs from subdir pfp1/... (generation) to subdir pfp2/... (for execution). This will protect any manual changes you might make in pfp2/... in case you later run mass re-generations to pfp1/...

Manual changes will be required for files with multiple record types/layouts in the same file. 'citytax1' has only 1 record type, so we will execute it now.


 #1.  cp pfp1/citytax1 pfp2   - copy auto-generated job to pfp2, before
      =====================     optional insert datafilenames vs copybooknames

 #2.  cp pfp2/citytax1 pfp3   - copy to pfp3
      =====================     before any manual changes &/or execution

 #2.  vi pfp3/citytax1        - make any manual changes required
      ================          (record type tests, see example page '5B3')

Executing jobs to pipe delimit data file


 #3a. uvcopy pfp3/citytax1,fili1=d2asc/citytax1,filo1=d4pipe/citytax1
      ===============================================================

 #3b. uvcopy pfp3/citytax1   <-- easier (filenames default as shown above)
      ====================

sample input file ASCII (or EBCDIC) with packed


 #4. uvhd d2asc/citytax1 r128
     ========================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             3333333322444424445522222222222222233332445442544422222222225444
             1013014000AF8E085E29000000000000000181502F75E02F14000000000061E3
          64 OUVER           BCV9S1H1......W.........qq.000149061970530191809
             4554522222222222445353430888005880008000770333333333333333333333
             F56520000000000023693181029C0072C0047C0111C000149061970530191809
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 10139923  GEORGE BROWN             1250 EAST PENDER STREET  VANC
             3333333322444544245454222222222222233332445525444452555445225444
             101399230075F275022F7E0000000000000125005134005E45203425540061E3
          64 OUVER           BCV5L1W1......W.........xRL00014626q980601190156
             4554522222222222445343530888005880008800754333333337333333333333
             F5652000000000002365C171029C0072C0047C0182C000146261980601190156

Only 1st 2 records of input file are shown above since we must use uvhd (vs vi/lp/uvlp12/etc) because the data contains packed fields & no linefeeds.

We will show all records of the output file on the next page, since the output data is now a text file with linefeeds & easily printed or displayed.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4C2. Converting Data Files to ASCII Delimited for loading DB tables

sample output - pipe delimited text


 #5. more d4pipe/citytax1  <-- examine output (or vi/cat/lp/uvlp14/etc)
     ====================
 10130140|JOHN HENRY|1815 BOWEN ROAD|VANCOUVER|BC|V9S1H1|+82898|+57828|+4870|+171710|+1490.61|1997/05/30|191809|
 10139923|GEORGE BROWN|1250 EAST PENDER STREET|VANCOUVER|BC|V5L1W1|+82898|+57828|+4878|+178524|-1462.61|1998/06/01|190156|
 10147615|BONNIE SMITH|44430 YALE ROAD WEST|VANCOUVER|BC|V2P6J1|+121395|+39623|+0|+51914|+376.92|1995/06/01|201031|
 10149304|TONY TATTERSAL|3350 S.W. MARINE DRIVE|VANCOUVER|BC|V6N3Y9|+50594|+76800|+0|+208391|+1925.03|1993/06/01|201512|
 10154688|MEL RAHAL|1110 - 625 HOWE STREET|VANCOUVER|BC|V6C2T6|+82898|+44880|+0|+125038|+1154.79|1996/11/26|082351|
 10173320|SAM SMITH|2381 ROGERS AVE|VANCOUVER|BC|V3K5Y2|+50498|+60600|+0|+169086|+1562.00|1997/05/30|191810|
 10204700|JOHN JENSEN|12611-100 ST|VANCOUVER|AL|T8V4H2|+122396|+15720|+0|+43744|-404.40|1996/11/26|081702|
 10223240|BOB ANDERSON|7215 ARGYLL ROAD|VANCOUVER|AL|T6C4J2|+52899|+3597|+0|+9990|-90.99|1998/06/01|190157|
 10308685|TOM PERRY|3932 - 3A ST. N.W.|VANCOUVER|AL|T2E6R4|+70192|+68155|+0|+125773|+1065.10|1992/06/01|221650|
 10318833|HARRY SMIRNOFF|BOX 308, STN J|VANCOUVER|AL|T2A4X6|+11399|+54541|+42046|+93502|+761.46|1997/05/30|191811|

The output file is now an all text ASCII file and can be easily listed with the 'uvlp' scripts or other unix utilities. The input file requires a utility such as uvhd to display the packed fields.


 #6. uvlp14L d4pipe/citytax1 e1    <-- list 1st page (e1=end after 1 page)
     ==========================

 #6a. uvlp20L d4pipe/citytax1 e1   <-- use uvlp20L for long records
      ==========================     - prints 200 chars per line Landscape
      - - - OR for WindowsDOS, use 'uvlpr...' scripts - - -

 #6. uvlpr14L d4pipe/citytax1 e1   <-- list 1st page (e1=end after 1 page)
     ===========================

 #6a. uvlpr20L d4pipe/citytax1 e1  <-- use uvlp20L for long records
      ===========================    - prints 200 chars per line Landscape

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4D1. Converting Data Files to ASCII Delimited for loading DB tables

Alternate Delimiter &/or Enclose Character

arg1 may specify an alternate field separator (vs '|' delimiter). If used, arg1 must be specified as 2 digit hexadecimal value. For example: arg1=2C defines a comma ',' & arg1=FC defines an umlat (u with 2 dots above).

arg2 may specify a character to enclose alpha-numeric data fields. If used, arg2 must be specified as 2 digit hexadecimal value. For example: arg2=22 for '"' double quote, arg2=5E for '^' caret.

Here is the command to regenerate the delimit job specifying the separator as a comma x'2C' & the data enclose character as a double quote x'22'. Followed by the command to execute the job, followed by a display of 1st few records.


 #1. uvcopy genpipe1,fili1=maps/citytax1,filo1=pfp1/citytax1a,arg1=2C,arg2=22
     ========================================================================

 #2. cp pfp1/citytax1a pfp2      - copy to standard execution pfp2 subdir
     ======================

 #4. uvcopy pfp2/citytax1a,fili1=d2asc/citytax1,filo1=d4pipe/citytax1
     ================================================================

1st few records from alternate job

 "10130140  ","JOHN HENRY               ","1815 BOWEN ROAD          ","VANCOUVER           ","BC"
 ,"V9S1H1",+82898,+57828,+4870,+171710,+1490.61,0000/00/00,"191809",
 "10139923  ","GEORGE BROWN             ","1250 EAST PENDER STREET  ","VANCOUVER           ","BC"
 ,"V5L1W1",+82898,+57828,+4878,+178524,-1462.61,0000/00/00,"190156",
 "10147615  ","BONNIE SMITH             ","44430 YALE ROAD WEST     ","VANCOUVER           ","BC"
 ,"V2P6J1",+121395,+39623,+0,+51914,+376.92,0000/00/00,"201031",
 "10149304  ","TONY TATTERSAL           ","3350 S.W. MARINE DRIVE   ","VANCOUVER           ","BC"
 ,"V6N3Y9",+50594,+76800,+0,+208391,+1925.03,0000/00/00,"201512",
 "10154688  ","MEL RAHAL                ","1110 - 625 HOWE STREET   ","VANCOUVER           ","BC"
 ,"V6C2T6",+82898,+44880,+0,+125038,+1154.79,0000/00/00,"082351",
 "10173320  ","SAM SMITH                ","2381 ROGERS AVE          ","VANCOUVER           ","BC"
 ,"V3K5Y2",+50498,+60600,+0,+169086,+1562.00,0000/00/00,"191810",

I have split the records (after "BC") so you can see the numeric fields (at the end of the record) are not enclosed in the specified enclose character. COBOL type 'x' & unpacked unsigned type '9' are enclosed, while type '9' signed or packed or binary fields are not enclosed.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4D2. Converting Data Files to ASCII Delimited for loading DB tables

EBCDIC/ASCII convert is Pre-Requisite of Pipe Delimit

Conversion from EBCDIC to ASCII is normally a pre-requisite to pipe-delimiting. For large files, it may be worth combining the EBCDIC to ASCII instructions into the pipe delimit job.

genpipe1 option to use EBCDIC file as input

You might do this to save time & space for large files where you only needed the pipe delimited output. The genpipeA script generates a uvcopy job to read the ASCII file by default. If you want the generated job to read the original EBCDIC file you could run the genpipe1 job with the 'a1' option as shown below.


 #1. uvcopy genpipe1,fili1=maps/citytax1,filo1=pfp1/citytax1,uop=a1  <-- generate
     ==============================================================

Or if already generated, you can simply uncomment the following line:

 ###    tra    b0(00128)          uncomment for EBCDIC input (or option a1)

You also need to transfer the instructions (that preserve packed fields & correct zoned signs) from the EBCDIC to ASCII conversion job to the pipe delimit job at the point marked by:

 #    ---             <-- insert instrns from uvdata51 if packed fields

From our EBCDIC to ASCII job pfx2/citytax1 listed on page '3B2', these instructions are:

      mvc    b88(19),a88             pns taxduedate:landbldgtax
      trt    b107(9),$trtsea          ns f00107
 Insert these instructions into the pipe delimit job (listed on page '4B5')
 after the '#     ----' indicator line.

The execute command would then specify the EBCDIC file as input (vs the ASCII file on the execute command on the previous page).


 #2a. uvcopy pfp2/citytax1,fili1=d1ebc/citytax1,filo1=d4pipe/citytax1
      ===========================*****===============================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4D3. Converting Data Files to ASCII Delimited for loading DB tables

option s0/s1 for BINARY fields

Note that genpipe1 generates edit instructions with or without option 's' depending on the machine architecture - BIG-end or little-end (AIX,SUN,etc or INTEL). On page '4B5' the binary field post-date was generated as follows (on my INTEL machine).

        edta1  c600(9),a88(4bs),'+zzzzzzzzz9'       #7 post-date

Prior to April 2013, you had to specify user option 's0' or 's1' depending on your machine to cause edit instructions to be generated with or without instruction option 's'. See user option 'uop=s0' or 'uop=s1'prompts on page '4B4'

Now genpipe1 tests for BIG-end/little-end coded within uvcopy when uvcopy is compiled on AIX/SUN/etc vs INTEL machines.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4E1. Converting Data Files to ASCII Delimited for loading DB tables

listpipe1 - list pipe delimited data with copybook field names


 #1a. uvcopy listpipe1,fili1=d4pipe/citytax1,fili2=maps/citytax1
                      ,filo1=listpipe/citytax1
      ==========================================================

 #1b. listpipeA citytax1    <-- script equivalent of above (shorter command)
      ==================      - ksh for unix/linux or .bat file for Win/Dos

listpipe1 OPTIONS screen

 uop=q1f1l1s3   - default options
       f1       - FormFeed after each record, f2 every 2nd rec, etc
         l1     - list data recs horizontally before vertically
           s3   - stop count default 1st 3 records in file
 User OPtion (uop) defaults  = q1f1l1s3
  null to accept or re-specify (1 or more) -->    <-- null reply lists 3 records
 report=tmp/citytax1, enter: vi/more/lp/etc
 --> more       <-- display output (shown below)

sample listpipe1 report

 listpipe1 - list '|' delimited files with copybook fieldnames
           - to verify files created by genpipe1 (see DATAcnv1.htm#Part_4)
 datafile=d4pipe/filename?  copybookmap=maps/filename?  datetime=2003/06/09_20:26:34
 SEQ# FIELDNAME              COBOL PICTURE      DATA
 datafile = d4pipe/citytax1, record# = 1, field count = 13
 10130140|JOHN HENRY|1815 BOWEN ROAD|VANCOUVER|BC|V9S1H1|+82898|+57828|+4870|+171
 710|+1490.61|1997/05/30|191809|
 001 folio                   x(00010).          10130140
 002 name                    x(00025).          JOHN HENRY
 003 address                 x(00025).          1815 BOWEN ROAD
 004 city                    x(00020).          VANCOUVER
 005 province                x(00002).          BC
 006 zip                     x(00006).          V9S1H1
 007 post-date               s9(00007) comp-4.  +82898
 008 land-value              s9(00009) comp-3.  +57828
 009 improv-value            s9(00009) comp-3.  +4870
 010 face-value              s9(00009) comp-3.  +171710
 011 maint-tax               s9(00007)v99.      +1490.61
 012 purchase-date           x(00006).          1997/05/30
 013 filler001               x(00006).          191809
  1. Only 1 record is shown here, the default is to list the 1st 3 records, which is controlled by the stop option 's3' above. (reply 's5' for 1st 5, etc).

  2. Note that the pipe delimited text record is first listed 'as is' & then listed field by field with COBOL copybook names.

  3. The pipe delimited text records could be very long if large records with many fields.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4F1. Converting Data Files to ASCII Delimited for loading DB tables

input - copybook/cobmap

 cobmap1  COBOL record layout            2012/02/19_09:30:13  pg# 0001
 mf/cpys/citytax1.cpy          citytax1           RCSZ=00128  bgn  end  lth typ
 * citytax1 copybook
  01  citytax1.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-4.          0088 0091  004bns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  x(00006).                 0116 0121  006
      10 filler001             pic  x(00006).                 0122 0127  006
 *RCSZ=00128

generate SQL scripts to create & load table


 #1. uvcopy cobmap1,fili1=cpys/citytax1.cpy,filo1=maps/citytax1
     ==========================================================
     - convert copybook to 'cobmap' (record layout)

 #2. uvcopy sqlcreate1,fili1=maps/citytax1,filo1=sqlTC1/create_citytax1.sql
     ======================================================================
                                          ,filo2=sqlTL1/load_citytax1.ctl
     ====================================================================
     - convert cobmap to SQL scripts to create & load table
      - - - OR use script 'sqlcreateA' (combines above into 1 short command)

 #12. sqlcreateA citytax1    <-- script equivalent of above (shorter command)
      ===================      - ksh for unix/linux or .bat file for Win/Dos

 #3a. more sqlTC1/create_citytax1.sql   <-- display batch-file output
      ===============================

 #3b. more sqlTL1/load_citytax1.ctl     <-- display SQL script output
      =============================

See the output files listed on the next page --->

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4F2. Oracle Database SQL test/demo

Oracle SQL demo - CREATE & LOAD table


 #1. Login demo1 --> /home/demo1
     ===========

 #2. sqlplus demo1/demo1pw     <-- start SQL*Plus
     =====================

 #3. SQL> create table citytax1 (folio char (10), name char (25),
                 address char (25), city char (20), province char (02),
                 zip char (06), post_date date, land_value number (09),
                 improv_value number (09), face_value number (09),
                 maint_tax number (09,2), purchase_date date);

 #4. SQL> describe citytax1
          =================
  Name					   Null?    Type
  ----------------------------------------- -------- ----------------------------
  FOLIO						    CHAR(10)
  NAME						    CHAR(25)
  ADDRESS					    CHAR(25)
  CITY						    CHAR(20)
  PROVINCE					    CHAR(2)
  ZIP						    CHAR(6)
  POST_DATE					    DATE
  LAND_VALUE					    NUMBER(9)
  IMPROV_VALUE					    NUMBER(9)
  FACE_VALUE					    NUMBER(9)
  MAINT_TAX					    NUMBER(9,2)
  PURCHASE_DATE					    DATE

 #5. SQL> exit    <-- exit to unix shell to run SQLLDR
          ====

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4F3. Oracle Database SQL test/demo

Oracle 'batch-file' to create table

Creating tables manually (as shown above) would be very laborious for files with many fields. UV Software provides uvcopy job 'sqlcreate1' to automatically generate an Oracle batch-file to create the table. The job also generates the SQL*LOADER script to load the table. See 'sqlcreate1' fully documented in DATAcnv1.htm#Part_4.

Oracle batch-file to create table

 /* citytax1.sql - SQL batch file to create a table */
 /*              - generated from COBOL copybook 20131021:123340 */
 /* www.uvsoftware.ca/datacnv1.htm#Part_4 & sqldemo.htm#Part_4 */
 /*                                        */
 /* sqlplus user/pass @sqlTC2/citytax1.sql */
 /* ====================================== */
 /*                                        */
 DROP TABLE citytax1;
 CREATE TABLE citytax1 (
 folio                         char     (0010) , --#001 x(00010).
 name                          char     (0025) , --#002 x(00025).
 address                       char     (0025) , --#003 x(00025).
 city                          char     (0020) , --#004 x(00020).
 province                      char     (0002) , --#005 x(00002).
 zip                           char     (0006) , --#006 x(00006).
 post_date                     number   (07)   , --#007 s9(00007) comp-4.
 land_value                    number   (09)   , --#008 s9(00009) comp-3.
 improv_value                  number   (09)   , --#009 s9(00009) comp-3.
 face_value                    number   (09)   , --#010 s9(00009) comp-3.
 maint_tax                     number   (09,2) , --#011 s9(00007)v99.
 purchase_date                 char     (0006) , --#012 x(00006).
 filler001                     char     (0006)   --#013 x(00006).
 );
 exit;

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4F4. Oracle Database SQL test/demo

Oracle SQL script to LOAD table

 -- citytax1.ctl - SQL*LOADER script to load Oracle table
 --              - generated from COBOL copybook 20131021:123340
 -- www.uvsoftware.ca/datacnv1.htm#Part_4 & sqldemo.htm#Part_4
 --
 -- sqlldr user/pass control=sqlTL2/citytax1.ctl
 -- ============================================
 --
 load data infile 'd4pipe/citytax1' into table citytax1
 fields terminated by '|' optionally enclosed by '"'
 (
 folio                         , --char     (0010) , --#001 x(00010).
 name                          , --char     (0025) , --#002 x(00025).
 address                       , --char     (0025) , --#003 x(00025).
 city                          , --char     (0020) , --#004 x(00020).
 province                      , --char     (0002) , --#005 x(00002).
 zip                           , --char     (0006) , --#006 x(00006).
 post_date                     , --number   (07)   , --#007 s9(00007) comp-4.
 land_value                    , --number   (09)   , --#008 s9(00009) comp-3.
 improv_value                  , --number   (09)   , --#009 s9(00009) comp-3.
 face_value                    , --number   (09)   , --#010 s9(00009) comp-3.
 maint_tax                     , --number   (09,2) , --#011 s9(00007)v99.
 purchase_date                 , --char     (0006) , --#012 x(00006).
 filler001                       --char     (0006)   --#013 x(00006).
 )

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4F5. Oracle Database SQL test/demo

create & load table using SQL scripts


 #1. Login demo1 --> /home/demo1
     ===========

 #2. sqlplus user/pass @sqlTC1/create_citytax1.sql
     =============================================
     - run SQL 'batch-file' to create the table

 #3. sqlldr user/pass control=sqlTL1/load_citytax1.ctl
     =================================================
     - run SQL*LOADER script to load the table

use SQL*Plus to verify table loaded OK


 #4. sqlplus demo1/demo1pw       <-- start SQL*Plus
     =====================

 #5. SQL> select * from citytax1 <-- SELECT all rows/fields from table
          ======================   - headings & 1st 2 rows shown below:
 FOLIO	   NAME 		     ADDRESS
 CITY		     PR ZIP	POST_DATE LAND_VALUE IMPROV_VALUE FACE_VALUE
  MAINT_TAX PURCHASE_D
 ============================================================================
 10130140   JOHN HENRY		     1815 BOWEN ROAD
 VANCOUVER	     BC V9S1H1	    82898      57828	     4870     171710
    1490.61 1997/05/30
 10139923   GEORGE BROWN 	     1250 EAST PENDER STREET
 VANCOUVER	     BC V5L1W1	    82898      57828	     4878     178524
   -1462.61 1998/06/01

batch-file to list table rows

 /* select_all_citytax1.sql - batch file to select all rows from citytax1  */
 /*                      - batch files useful for complex repeated selects */
 select * from citytax1;
 exit;

You could use the batch-file listed above as an alternative to logging in to SQL*PLUS to list the table rows:


 #6. SQL> exit      <-- exit from SQL*PLUS (back to Unix/Linux prompt)

 #7. sqlplus user/pass @sqlTC1/select_all_citytax1.sql
     =================================================
     - run SQL 'batch-file' to create the table

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4G1. Converting Mainframe Data Files from EBCDIC to ASCII

scripts to Generate All Data Conversion Jobs

 'gencnvD1' - script to generate all conversion jobs when copybooks ABSENT.
            - signscan2, sign2cpy1, cobmap1, uvdata51, genpipe1, sqlcreate1
 'gencnvD2' - script to generate all conversion jobs when copybooks PRESENT.
            - cobmap1, uvdata51, uvdata31, genpipe3, genfixd3, sqlcreate1
            - updated Oct2014 to run genpipe3/genfixd3 (vs genpipe1/genfixd2)
gencnvD2p
  • alternate version using uvdata51p to ensure packed field signs
    x'_C' positive or x'_D' negative (converts any x' F' to x'_C')
 'gencnvA' - script to generate most commonly used conversion jobs
           - cobmap1, uvdata51, genpipe3, genfixd3
           - updated Oct2014 to run genpipe3 (vs genpipe1)

I suggest you convert your 1st file using the individual scripts or uvcopy jobs (as described earlier in this documentation), to gain a good understanding of these procedures. Running them 1 by 1 allows you to examine the various outputs & check for any errors.

Use 'gencnvD1' when no COBOL copybook is available & you need to scan the datafile to create the copybook. Use alternate script 'gencnvD2' when the COBOL copybook is available (skips signscan2 & sign2cpy1 jobs).

The input EBCDIC datafile must be stored in the d1ebc subdir & the outputs will be named the same as the input datafile & will be stored in separate subdirs (signs, cpys, maps, pfx1,pfx2,pfx3 pfp1,pfp2,pfp3,& sqls).

Note
  • gencnvD1 is not practical, since generating copybooks is NOT perfect
  • you usually need to do manual corrections after signscan2 & sign2cpy1
  • so run signscanA & sign2cpyA separately & then run gencnvD2 (below)

related documentation

 SQLconvert.htm - convert flat files to pipe delimited for Microsoft SQL server
                  - driven by COBOL copybooks
                  - also generate scripts to create & load tables
                  - generates jobs for all copybooks & all data files
                    (vs this DATAcnv1.doc for 1 copybook/file at a time)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4H1. Converting Mainframe Data Files from EBCDIC to ASCII

gencnvD2 - Generate conversion jobs from CopyBooks

Script 'gencnvD2' runs conversion GENERATION jobs based on CopyBooks. (cobmap1, uvdata51, uvdaaata31, genpipe3, genfixd3, sqlcreate1, gentest1). gencnvD2 was updated Oct2014 to run genpipe3/genfixd3 (vs genpipe1/genfixd2) See listing of gencnvD2 on page '10D1'. See subdirs required on page '1B4'.

gencnvD2 - Operating Instructions


 #1. gencnvD2 file-name               <-- command format
     ==================

 #1a. gencnvD2 citytax1.cpy           <-- example for test/demo file
      =====================

copy to alt subdir before changes/execution

Regardless of whether changes are required or not, I suggest you always copy the jobs from pfx1 to pfx3 & from pfp1 to pfp3, before executing. Often you wont realize you need modifications until after you inspect the output data.


 #2a. cp pfx1/citytax1 pfx3   - copy auto-generated EBCDIC to ASCII job
      =====================     before any manual changes &/or execution
 #2b. cp pfy1/citytax1 pfy3   - copy auto-generated ASCII to EBCDIC job
 #2c. cp pfp1/citytax1 pfp3   - copy auto-gen pipe delimit job
 #2d. cp pfq1/citytax1 pfq3   - copy auto-gen pipe delimit to Fixed
 #2e. cp pfg1/citytax1 pfq3   - copy job to create test data from copybook

 #3a. vi pfx3/citytax1        - make any manual changes required
      ================          (for record type tests, see page '5D2')
 #3b. vi pfy3/citytax1        - make manual changes ASCII to EBCDIC job
 #3c. vi pfp3/citytax1        - make manual changes to pipe delim job
 #3d. vi pfq3/citytax1        - make manual changes to delim to fixed job
 #3e. vi pfg3/citytax1        - make manual changes to create test data job

executing jobs generated by gencnvD2


 #4. uvcopy pfx3/citytax1 <-- converts d1ebc/citytax1 to d2asc/citytax1
     ====================     (EBCDIC to ASCII, preserving packed, fixing signs)
 #5. uvcopy pfy3/citytax1 <-- converts d2asc/citytax1 to d3ebc/citytax1
     ====================     (ASCII back to EBCDIC)
 #6. uvcopy pfp3/citytax1 <-- converts d2asc/citytax1 to d4pipe/citytax1
     ====================     (converts ASCII file to delimited text file)
 #7. uvcopy pfq3/citytax1 <-- converts d4pipe/citytax1 to d5fix/citytax1
     ====================     (converts pipe|delimited to Fixed-field records)
 #8. uvcopy pfq3/citytax1 <-- generates test data file from copybook
     ====================     A/N fields A,B,C,etc numeric fields 1,2,3,etc

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4H2. Converting Mainframe Data Files from EBCDIC to ASCII

checking gencnvD2 outputs (Unix/Linux)


 #1. vi cpys/sample1    - copybook supplied
                        - must be stored in cpys/...

 #2. vi maps/sample1    - cobmap generated from copybook by cobmap1
                        - field start/end/length on right side

 #3. vi pfx1/sample1    - EBCDIC to ASCII conversion job
                          generated by uvdata51

 #4. vi pfy1/sample1    - ASCII to EBCDIC conversion job
                          generated by uvdata31

 #5. vi pfp1/sample1    - job to convert ASCII file to text delimited
                          generated by genpipe1

 #6. vi pfq1/sample1    - job to convert pipe|delimited to Fixed-field
                          generated by genfixd2

 #7. vi pfg1/sample1    - job to generate test data file from copybook
                          generated by gentest1

 #8. vi sqlTC1/sample1.sql - batch file to create table
                           - generated by sqlcreate1

 #9. vi sqlTL1/sample1.ctl   - SQL script to load table
                             - generated by sqlcreate1

create testfile from copybook


 #1. uvcopy gentest1,fili1=maps/sample1,filo1=pfg1/sample1,uop=e1
     =============================================================
     - generate job to create testfile

 #2. vi pfg1/sample1     - inspect job to crate test file
     ===============

 #3. uvcopy pfg1/sample1,filo1=d1ebc/sample1
     =======================================
     - execute job to create testfile from copybook

 #4. uvhd d1ebc/sample1 r592a   <-- inspect generated testfile
     ========================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4H3. Converting Mainframe Data Files from EBCDIC to ASCII

execute data conversion jobs


 #1. uvcopy pfx1/sample1    <-- convert EBCDIC to ASCII
     ===================

 #2. uvcopy pfy1/sample1    <-- convert ASCII to EBCDIC
     ===================      - to return file to mainframe

 #3. uvcopy pfp1/sample1    <-- create pipe|delimited data
     ===================        (to load RDBMS table)

 #4. uvcopy pfq1/sample1    <-- convert pipe|delimited back to fixed field EBCDIC
     ===================      - to return file to mainframe

 #5. listpipeA sample1      <-- convert pipe|delimited to vertical format
     =================          with copybook fieldnames

checking data files


 #1. uvhd d1ebc/sample1 r592a  - original EBCDIC input
     ========================    OR testfile generated by gentest1 above

 #2. uvhd d2asc/sample1 r592   - inspect ASCII output from pfx1/sample1
     =======================

 #3. uvhd d3ebc/sample1 r592a  - inspect EBCDIC output from pfy1/sample1
     ========================

 #4. vi d4pipe/sample1         - inspect delimited output from pfp1/sample1
     =================

 #4a. listpipeA sample1        - create vertical listing of delimited output with
      =================          copybook fieldnames beside data contents
 #4b. vi listpipe/sample1      - inspect vertical listing of delimited output
      ===================

 #5. uvhd d5fix/sample1 r592a  - inspect Fixed-field converted from pipe|delimited
     ========================

modifying script gencnvD2

Please see the gencnvD2 script listed on page '10D1' Linux, '10D2' Windows .bat. You can modify the various options as appropriate for your site. For example to use a semicolon as the delimiter, you would add ',arg1=\;' to line 38. If you want to generate 1st record with fieldnames, add ',uop=w1', for example:


 uvcopy genpipe1,fili1=maps/$f,filo1=pfp1/$f,arg1=\;,uop=w1
 ==========================================================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4H4. Converting Mainframe Data Files from EBCDIC to ASCII

super script to generate ALL jobs for ALL copybooks

gencnvD9
  • generates ALL conversion jobs for ALL copybooks in cpys/
  • generates jobs in ppfx1, pfp1, pfy1, pfq1, pfg1, sqlTC1, sqlTL1
  • great time saver when you have many files to convert

super script to EXECUTE multiple jobs for 1/ALL datafiles


 uvcopypfx1 citytax1 <-- executes pfx1/citytax1 & pfp1/citytax1 for d1ebc/citytax1
 ===================   - also generates listpipe/citytax1 to check pipe delimited
                       - 1st 3 records listed vertically with copybook fieldnames

 uvcopypfx2 citytax1 <-- same as above, but also creates uvhd & uvhdcob displays
 ===================   - in subdirs uvhd_d1ebc, uvhd_d2asc, uvhdcob (d2asc only)
                         (makes subdirs if not present)

super script to EXECUTE ALL jobs for ALL datafiles


 uvcopypfx9 all  <-- executes ALL pfx1/... & pfp1/... for ALL datafiles d1ebc/...
 ==============    - also generates listpipe/... to check pipe delimited
                   - 1st 3 records listed vertically with copybook fieldnames

script 'copysub2' to collect all jobs & data for 1 file

Generating & executing the various data conversion jobs for 1 or ALL files is made easy by maintaining the same filename in multiple subdirs, but this does make it difficult to gather the results in 1 directory. You might want to do this send the results to somebody via an email atachment.

'copysub2' will gather all jobs & data for 1 file, from multiple subdirs to one output directory. It appends _subdir names on the filename, allowing cohabitation in 1 directory.

Here is an example to gather all results for 'citytax1'. Assume userxx has just completed generating & executing his data conversion jobs in his superdir /home/userxx/cnvdata1/... with the various subdirs cpys,maps,pfx1,etc.


 #1. cd ..              <-- change above cnvdata1

 #2. mkdir tmp1         <-- make temp subdir at same level as cnvdata1

 #3. copysub2 citytax1 tmp1  <-- collects all citytax1 files into tmp1
     ======================    - made possible by appending subdir names

 #4. ls tmp1                 <-- list files collected
     =======
     citytax1_cpys citytax1_maps citytax1_d1ebc citytax1_d2asc
     citytax1_pfx1 citytax1_pfp1 ...etc...

 #5. zip -r tmp1.zip tmp1    <-- zip files for email attachment etc
     ====================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4I1. Converting Mainframe Data Files from EBCDIC to ASCII

genrfmA Operating Instructions - Reformat data records

'genrfmA' reformats data records based on old & new versions of the copybook. For example, we could unpack all packed/binary fields by copy/rename/modify the copybook removing all comp-3 & comp-4 on field pictures.

Prior to Nov2014 alternate copybook was copied to cpyu/... with same name. After Nov2014, copy/rename in same dir (cpys/citytax1u.cpy in our example).

Directories Required for genrfmA

 :---cnvdata1     <-- superdir 'cnvdata1' within your homedir or mvstest1
                                   ** Library subdirs **
 :   :---cpys/citytax1.cpy   <-- original copybook
 :   :---cpys/citytax1u.cpy  <-- copybook with 'comp-3's & 'comp-4's removed
 :   :---maps/citytax1       <-- original record layout (cobmap)
 :   :---maps/citytax1u      <-- unpacked record layout (cobmap)
 :   :---mapsI/citytax1      <-- original cobmap loaded to Indexed file for lookup
 :   :---pfr1/citytax1u      <-- uvcopy job generated by step4 of genrfmA (reform2)
                                    ** Data subdirs **
 :   :---d2asc/citytax1      <-- input data file ASCII (may have packed/binary fields)
 :   :---d3asc/citytax1u     <-- output file (all unpacked characters)

Generate job to unpack data files


 #1. Make directories as shown above
 #2. Store input files required - we will demo using following:

 #2a. d2asc/citytax1  <-- data file with packed/binary fields (to be unpacked)
      ==============
 #2b. cpys/citytax1   <-- copybook corresponding to above data file
      =============

 #3. cp cpys/citytax1.cpy cpys/citytax1u.cpy  <-- copy/rename for alternate layout
     =======================================

 #4. vi cpys/citytax1u.cpy   <-- edit copy to remove all 'comp-3's & 'comp-4's
     =====================
Note
  • #5 & #6 below optional, script genrfmA will repeat
  • might do to see the record layouts before generating reformat job
  • see cobmap listngs on the next page (before reformat job 2 pages ahead)

 #5a. uvcopy cobmap1,fili1=cpys/citytax1.cpy,filo1=maps/citytax1 <-- gen INPUT cobmap
      ==========================================================
 #5b. more maps/citytax1  <-- inspect original layout

 #6a. uvcopy cobmap1,fili1=cpys/citytax1u.cpy,filo1=maps/citytax1u <-- gen OUTPUT cobmap
      ============================================================
 #6b. more maps/citytax1u  <-- inspect unpacked layout

 #7. genrfmA citytax1 citytax1u <-- generate reformat job pfr1/citytax1
     ==========================   - will re-gen 2 pages ahead & execute 3 pages ahead

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4I2. Converting Mainframe Data Files from EBCDIC to ASCII

sample cobmap BEFORE (& after) 'comp's removed

 cobmap1  COBOL record layout            2012/12/04_14:55:51  pg# 0001
 cpys/citytax1.cpy                  citytax1           RCSZ=00128  bgn  end  lth typ
 * citytax1 copybook for uv demos www.uvsoftware.ca/datacnv1.
  01  citytax1.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-4.          0088 0091  004bns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  x(00006).                 0116 0121  006
      10 filler001             pic  x(00006).                 0122 0127  006
 *RCSZ=00128

cobmap AFTER 'comp-3's & 'comp-4's removed

 cobmap1  start-end bytes for cobol record fields    200306092029  pg# 0001
 cpys/citytax1u.cpy            citytax1u          RCSZ=00143  bgn  end  lth typ
 * citytax1u copybook
  01  citytax1u.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) .                0088 0094  007 ns 7
      10 land-value            pic s9(00009) .                0095 0103  009 ns 9
      10 improv-value          pic s9(00009) .                0104 0112  009 ns 9
      10 face-value            pic s9(00009) .                0113 0121  009 ns 9
      10 maint-tax             pic s9(00007)v99.              0122 0130  009 ns 9
      10 purchase-date         pic  x(00006).                 0131 0136  006
      10 filler001             pic  x(00006).                 0137 0142  006
 *RCSZ=00143                                                       0143

Note that the record size has increased from 128 to 143, due to unpacking. Since there was one 4 byte field (which is now 7) & three 5 byte fields (which become 9), the output is (1*3 + 3*4) = 3+12 = 15 bytes larger. The new record is 128 + 15 = 143 bytes, confirmed by the cobmap above. The conversion job option t1 (default) will add 1 more for a terminating LF.


 #7. genrfmA citytax1 citytax1u  <-- generate uvcopy job to unpack packed fields
     ==========================

Please see the genrfmA output listed on the next page --->

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4I3. Converting Mainframe Data Files from EBCDIC to ASCII


 #7. genrfmA citytax1 citytax1u  <-- generate uvcopy job to unpack packed fields
     ==========================

pfr1/citytax1u - uvcopy job to unpack packed/binary fields

 opr='JOBNAME citytax1u - generated by cobmap1,reform2,uvdata52'
 uop=q1d2m2t0z0g0n4p0s0w50x1y0q1    <-- options from reform2 gen
 was=a9000b9000
 fili1=?d2asc/citytax1,rcs=00128,typ=RSF
 filo1=?d3asc/citytax1u,rcs=00143,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        clr    b0(00143),' '             init output area to all blanks
 #      ---                          <-- insert R/T tests if redef records
        mvc    b0(88),a0(88)                 folio : zip
        bal    rfmbn,'a00088b07c07d00088e04f07','post-date'
        bal    rfmpn,'a00095b09c09d00092e05f09','land-value'
        bal    rfmpn,'a00104b09c09d00097e05f09','improv-value'
        bal    rfmpn,'a00113b09c09d00102e05f09','face-value'
        mvc    b122(9),a107(9)               maint-tax
        bal    rfmnx,'a00131b06c06d00116e06f06','purchase-date'
        mvc    b137(6),a122(6)               filler001
 #
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj
 @pf2=reform2.sub

Since this DATAcnv1.doc is intended to be the easiest to use document for file conversions, I wont go into detail here about the generated code. If you wish more explanation & discussion about possible modifications for complex files, please see REFORMjobs.htm.

Note
  • see listing of the generation script (genrfmA) on page '10H1'.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4I4. Converting Mainframe Data Files from EBCDIC to ASCII

Manual Changes Required ? (to reformat jobs)

Manual changes may be required to the auto-generated uvcopy job (usually to allow for multiple record types/layouts in the same file).

Regardless of whether changes are required or not, I suggest you always copy the jobs from pfr1 to pfr2, before executing (from pfr2). Often you wont realize you need modifications until after you inspect the output data.


 #8.  cp pfr1/citytax1u pfr2   - copy auto-generated job to pfr2
      ======================     before any manual changes &/or execution

 #9.  vi pfr2/citytax1u        - make any manual changes required
      =================          (for record type tests, see page '5D2')

execute generated job to unpack packed fields


 #10a. uvcopy pfr2/citytax1u,fili1=d2asc/citytax1,filo1=d3asc/citytax1u
       ================================================================

 #10b. uvcopy pfr2/citytax1u  <-- easier (filenames default as shown above)
       =====================

Please see next page for sample data I/O listed with uvhd --->

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4I5. Converting Mainframe Data Files from EBCDIC to ASCII

sample input file ASCII (or EBCDIC)


 #11. uvhd /home/mvstest1/cnvdata1/d2asc/citytax1 r128
      ================================================
 rec#=1 rsize=128 fptr=0 fsize=1280 rcount=10
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             3333333322444424445522222222222222233332445442544422222222225444
             1013014000AF8E085E29000000000000000181502F75E02F14000000000061E3
          64 OUVER           BCV9S1H1..C...W.........qq.000149061970530
             455452222222222244535343004D005880008000770333333333333333222222
             F5652000000000002369318101320072C0047C0111C000149061970530000000

sample output - unpacked


 #12. uvhd /home/mvstest1/cnvdata1/d3asc/citytax1u r143
      =================================================
 rec#=1 rsize=143 fptr=0 fsize=1430 rcount=10
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             3333333322444424445522222222222222233332445442544422222222225444
             1013014000AF8E085E29000000000000000181502F75E02F14000000000061E3
          64 OUVER           BCV9S1H10082898000057828000004870000171710000149
             4554522222222222445353433333333333333333333333333333333333333333
             F565200000000000236931810082898000057828000004870000171710000149
         128 061970530
             333333333222222
             061970530000000

Note that the record size has increased from 128 to 143, due to unpacking. Since there was one 4 byte field (which is now 7) & three 5 byte fields (which become 9), the output is (1*3 + 3*4) = 3+12 = 15 bytes larger.

Note option 't' to add record terminators if desired: Option t0 (default) does not append any record terminator Option t1 would add 1 byte for LF x'0A' only (unix). Option t2 would add 2 bytes for CR/LF x'0D0A' (dos/windows).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4J1. Converting Data Files from Delimited to Fixed field records

Converting Delimited to Fixed field

'genfixd2' will generate jobs to convert 'delimited formats' to 'fixed field' formats described by COBOL copybooks. This job is useful when you need to convert delimited files from DB2 'export's (on UNIX) to the fixed field format that was created by DB2 'unload's (on the mainframe). Note that mainframe 'UNLOAD' creates 'Fixed-field', but unix 'EXPORT' creates 'delimited format.

On the mainframe, you might have:

  1. UNLOADed data from z/DB2,
  2. processed with a COBOL program or DFSORT,
  3. reLOADed back into z/DB2.

Using 'genfixd2' on unix, you could:

  1. EXPORT a delimited file from AIX/DB2,
  2. convert to fixed format (with a job generated by genfixd2),
  3. process the data (with same COBOL program or similar sort as on mainframe),
  4. convert back to delimited format (with a job generated by genpipe1),
  5. IMPORT the delimited data back into AIX/DB2.

Yes, this is 5 steps vs 3 steps, but it saves you from having to rewite the COBOL program to process delimited format vs fixed format.

We will demonstrate using the 'citytax1' demo copybook/cobmap & data file. You could repeat this demo in your homedir assuming Vancouver Utilities installed & your profile updated as described at https://uvsoftware.ca/jclcnv1demo.htm#Part_1

setup & copy demo files


 #1. Login userxx --> /home/userxx
 #2. mkdir cnvdata1
 #3. cd cnvdata1 (or alias 'cdc') --> home/userxx/cnvdata1

 #4. cnvdatadirs        <-- setup subdirs (see next page)
     ===========

 #5. mkdir d6fix d7pipe pfq2 pfp2
     ============================
     - make subdirs for this demo (not made by cnvdatadirs)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4J2. Converting Delimited to Fixed field

cnvdata1/... subdirs to demo genfixd2 & genpipe1

 /home/userxx         <-- etup cnvdata1 withing your $HOME dir
 :-----cnvdata1        <-- cnvdata1 superdir for DATA conversion
 :     :-----cpys       - copybooks
 :     :-----ctl        - control files
 :     :-----d1ebc      - EBCDIC data files
 :     :-----d2asc      - ASCII data files (converted by uvcopy uvdata51)
 :     :-----d3ebc      - convert back to EBCDIC by uvdata31
 :     :-----d4pipe     - pipe delimited to load DB tables (by genpipe1)
 :     :-----d5fix      - convert pipe|delimited to fixed field (genfixd2)
 :     :--*--d6fix      - output of COBOL or SORT or cp (for this demo)
 :     :--*--d7pipe     - convert back to pipe|delimited to import back to DB2
 :     :-----maps       - copybooks with record layouts on right side
 :     :-----pfx1       - uvcopy jobs to convert EBCDIC to ASCII (gen by uvdata51)
 :     :-----pfx2       - with data filenames inserted (vs copybook names)
 :     :-----pfp1       - uvcopy jobs to create pipe delimited (gen by genpipe1)
 :     :--*--pfp2       - uvcopy jobs to create pipe delimited (gen by genpipe1)
 :     :-----pfq1       - uvcopy jobs to convert pipe|delimited to fixed field (genfixd2)
 :     :--*--pfq2       - uvcopy jobs to convert pipe|delimited to fixed field (genfixd2)
 :     :-----tmp        - temp files (keep working directory clean)
Note
  • above shows only the subdirs relevant to this export/import demo
  • script 'cnvdatadirs' sets up several more (all listed on page '1B4')
  • '*' indicates subdirs required for this demo, not made by cnvdatadirs

 #5. cp $UV/mvstest1/cnvdata1/d4pipe/citytax1 d4pipe/citytax1
     ========================================================
     - copy demo datafile, here are 1st few records
 ctx-folio|ctx-name|ctx-address|ctx-city|ctx-province|ctx-zip|ctx-post-date|ctx-land-value|ctx-imp-value|ctx-face-value|ctx-main
 10130140|JOHN HENRY|1815 BOWEN ROAD|VANCOUVER|BC|V9S1H1|82898|57828|4870|171710|14906.10|970530| |
 10139923|GEORGE BROWN|1250 EAST PENDER ST.|VANCOUVER|BC|V5L1W1|82898|57828|4878|178524|-14626.10|980601| |
 10147615|BONNIE SMITH|44430 YALE ROAD WEST|VANCOUVER|BC|V2P6J1|121395|39623|0|51914|3769.20|950601| |
 10149304|TONY TATTERSAL|3350 S.W. MARINE DRIVE|VANCOUVER|BC|V6N3Y9|50594|76800|0|208391|19250.30|930601| |
 10154688|MEL RAHAL|1110 - 625 HOWE STREET|VANCOUVER|BC|V6C2T6|82898|44880|0|125038|11547.90|961126| |
 10173320|SAM SMITH|2381 ROGERS AVE|VANCOUVER|BC|V3K5Y2|50498|60600|0|169086|15620.00|970530| |
 10204700|JOHN JENSEN|12611-100 ST|VANCOUVER|AL|T8V4H2|122396|15720|0|43744|-4044.00|961126| |
 10223240|BOB ANDERSON|7215 ARGYLL ROAD|VANCOUVER|AL|T6C4J2|52899|3597|0|9990|-909.90|980601| |
 10308685|TOM PERRY|3932 - 3A ST. N.W.|VANCOUVER|AL|T2E6R4|70192|68155|0|125773|10651.00|920601| |
 10318833|HARRY SMIRNOFF|BOX 308, STN J|VANCOUVER|AL|T2A4X6|11399|54541|42046|93502|7614.60|970530| |
Note
  • the demo file above includes the 'field names' header record generated
    by genpipe1 if option 'w1' (specify option'w0' if you do not want).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4J3. Converting Delimited to Fixed field

setup & copy demo files (continued)


 #6. cp $UV/mvstest1/cnvdata1/cpys/citytax1.cpy cpys/citytax1
     ========================================================
     - copy demo copybook (drop .cpy suffix if present)
     - here is copybook - AFTER conversion to cobmap

maps/citytax1

       * citytax1 copybook for UV demos www.uvsoftware.ca/datacnv1.htm
        01  citytax1.
            10 folio                 pic  x(00010).
            10 name                  pic  x(00025).
            10 address               pic  x(00025).
            10 city                  pic  x(00020).
            10 province              pic  x(00002).
            10 zip                   pic  x(00006).
            10 post-date             pic s9(00007) comp-4.
            10 land-value            pic s9(00009) comp-3.
            10 improv-value          pic s9(00009) comp-3.
            10 face-value            pic s9(00009) comp-3.
            10 maint-tax             pic s9(00007)V99.
            10 purchase-date         pic  x(00006).
            10 filler                pic  x(00006).

genpipe3/genfixd3 alternates to genpipe1/genfixd2

In Oct 2014, we developed genpipe3/genfixd3 (alternates to genpipe1/genfixd2), to remove the restriction of 100 bytes max field length (& also faster). The new jobs (genpipe3/genfixd3) are documented begining on pages '4M1' & '4N1'.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4J4. Converting Delimited to Fixed field

generate conversion jobs - 1 time (vs executions multi)


 #1. Login userxx --> /home/userxx
 #2. cdc --> home/userxx/cnvdata1

 #3a. gencnvD2 citytax1.cpy   <-- easiest to use script
      =====================
     - generate all data conversion jobs, all named 'citytax1'
     - runs cobmap1,uvdata51,uvdata31,genpipe3,genfixd3,gentest1,sqlcreate1
     - updated Oct2014 to run genpipe3/genfixd3 (vs genpipe1/genfixd2)

 #3b. uvcopy genfixd2,fili1=maps/citytax1,filo1=pfq1/citytax1
      =======================================================
      - generate pfq1/citytax1 to undelimit d2asc/citytax1
      - using older 'fxd' instructions generated by genfixd2
        vs newer 'ftd' instructions generated by genfixd3 (see '4N1')

pfq1/citytax1 - convert delimited to fixed

 # citytax1 - uvcopy job to convert pipe delimited to Fixed-field format
 opr='citytax1 - generated by genfixd2 from copybook: citytax1 '
 opr='uop=q1e0s1 - option defaults'
 opr='      e0   - output ASCII  (do not translate char/zoned to EBCDIC)'
 opr='      e1   - output EBCDIC (translate char/zoned fields to EBCDIC)'
 opr='        s1 - switch binary field ends if running on Intel (little-end)'
 opr='        s0 - use s0 if running on BIG-end machine (such as AIX)'
 uop=q1e0s1    # option defaults
 was=a33000b33000c999000d33000e33000
 fili1=?d4pipe/citytax1,rcs=00183,typ=LST
 filo1=?d5fix/citytax1,rcs=00128,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        fix    c0(100),a0(00183),0015,'|'
 #      ---                   <-- insert R/T tests if redef records
        clr    d0(00128),x'20'    init to x'20' or x'40' if optn e1 EBCDIC
        fxd    d0(10c),c0,'folio'
        fxd    d10(25c),c100,'name'
        fxd    d35(25c),c200,'address'
        fxd    d60(20c),c300,'city'
        fxd    d80(2c),c400,'province'
        fxd    d82(6c),c500,'zip'
        fxd    d88(4b),c600,'post-date'
        fxd    d92(5p),c700,'land-value'
        fxd    d97(5p),c800,'improv-value'
        fxd    d102(5p),c900,'face-value'
        fxd    d107(9z),c1000,'maint-tax'
        fxd    d116(6c),c1100,'purchase-date'
        fxd    d122(6c),c1200,'filler001'
 put1   put    filo1,d0
        skp    loop
 eof    cls    all
        eoj

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4J5. Converting Delimited to Fixed field

pfp1/citytax1 - convert fixed to delimited

 # citytax1 - uvcopy job to convert EBCDIC/ASCII data to delimited text file
 opr='citytax1 - uvcopy code generated from copybook: citytax1.cpy '
 rop=j200000r1  #Run OPtions: increase instrn storage & prompt display output
 was=a64000b64000c999000d64000e64000u64000v64000  #increase u&v if fields > 163
 fili1=?d2asc/citytax1,rcs=00128,typ=RSF      #<-- will change d2asc/ to d6fix/
 filo1=?d4pipe/citytax1,rcs=00384,typ=LSTtd3  #<-- will change d4pipe/ to d7pipe
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
 # area a input, see 'get' instrn
 # area b translated to ASCII, in case mainframe EBCDIC file (with packed?)
 # area c data fields fixed 100 bytes apart in prep for var | delimit instrn
 # area d output, see 'put' instrn at end
        mvc    b0(00128),a0       move input area a to area b
 ###    tra    b0(00128)          uncomment for EBCDIC input (or option a1)
 #      ---                   <-- & insert instrns from uvdata51 if packed fields
 #      ---                   <-- insert R/T tests if redef records
        mvc    c0(10),b0(10)                        #1 folio
        mvc    c100(25),b10(25)                     #2 name
        mvc    c200(25),b35(25)                     #3 address
        mvc    c300(20),b60(20)                     #4 city
        mvc    c400(2),b80(2)                       #5 province
        mvc    c500(6),b82(6)                       #6 zip
        edta1  c600(9),a88(4bs),'-zzzzzzzzz9'       #7 post-date
        edta1  c700(11),a92(5p),'-zzzzzzzzz9'       #8 land-value
        edta1  c800(11),a97(5p),'-zzzzzzzzz9'       #9 improv-value
        edta1  c900(11),a102(5p),'-zzzzzzzzz9'      #10 face-value
        edta1  c1000(11),b107(9),'-zzzzzzz.99'      #11 maint-tax
        mvc    c1100(6),b116(6)                     #12 purchase-date
        mvc    c1200(6),b122(6)                     #13 filler001
        rep    c0(01300),'|','!'           #comment speedup if no '|'s in data
        varb1  d0(01300),c0(100),0013,'|'  #insert '|'s between fields & squeeze
        trt    d0(01300),$trtchr           #ensure all ascii chars x'20'-x'7E'
        clr    c0(01300),' '
 put1   put    filo1,d0
        skp    loop
 #
 eof    cls    all
        eoj
 @pf2=genpipe1.sub

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4J6. Converting Delimited to Fixed field

modify generated jobs to insert datafilenames vs copybooknames

For my demo files datafilenames & copybooknames are the same, but you will want to modify yuor generated jobs. Always copy auto-generated job to 2nd subdir to avoid losing your edits in case jobs regenerated


 #4a. cp pfq1/citytax1 pfq2   <-- copy before edit
      =====================
 #4b. vi pfq2/citytyax1       <-- insert your datafilenames vs copybooknames
      =================           on fili1=... & filo1=...

 #5a. cp pfp1/citytax1 pfp2   <-- copy before edit
      =====================
 #5b. vi pfp2/citytyax1       <-- insert your datafilenames vs copybooknames
      =================           on fili1=... & filo1=...
Note
  • datafilenames are same as copybooknames for this demo
  • but you need to change datafile subdirs as shown below:
        fili1=?d2asc/citytax1,rcs=00128,typ=RSF
        filo1=?d4pipe/citytax1,rcs=00384,typ=LSTtd3
        fili1=?d6fix/citytax1,rcs=00128,typ=RSF
        filo1=?d7pipe/citytax1,rcs=00384,typ=LSTtd3
Note
  • you may need to add '.dat' extension on datafilenames to match
    datafilenames EXPORTed by DB2 or required on datafilenames to
    be IMPORTed to DB2.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4J7. Converting Delimited to Fixed field

Execute DB2-->delim-->fixed -->COBOL-->fixed-->delim-->DB2


 #1. Login userxx --> /home/userxx
 #2. cdc --> home/userxx/cnvdata1

 #3. Execute JCL/script calling db2 to EXPORT delimited file from DB2 table.
Note
  • We can't do this for this demo
  • We will use the delimited demo file shown on page '4C2' which was
    created by the pfp1/citytax1 job shown on page '4B5' & executed on
    page '4C1'. pfp1/citytax1 was generated by the genpipe1 utility from
    the maps/citytax1 as documented on page '4B3'.

d4pipe/citytax1 - simulation of DB2 'export'

 10130140|JOHN HENRY|1815 BOWEN ROAD|VANCOUVER|BC|V9S1H1|82898|57828|4870|171710|1490.61|970530| |
 10139923|GEORGE BROWN|1250 EAST PENDER ST.|VANCOUVER|BC|V5L1W1|82898|57828|4878|178524|-1462.61|980601| |
 10147615|BONNIE SMITH|44430 YALE ROAD WEST|VANCOUVER|BC|V2P6J1|121395|39623|0|51914|376.92|950601| |
 10149304|TONY TATTERSAL|3350 S.W. MARINE DRIVE|VANCOUVER|BC|V6N3Y9|50594|76800|0|208391|1925.03|930601| |
 10154688|MEL RAHAL|1110 - 625 HOWE STREET|VANCOUVER|BC|V6C2T6|82898|44880|0|125038|1154.79|961126| |
 10173320|SAM SMITH|2381 ROGERS AVE|VANCOUVER|BC|V3K5Y2|50498|60600|0|169086|1562.00|970530| |
 10204700|JOHN JENSEN|12611-100 ST|VANCOUVER|AL|T8V4H2|122396|15720|0|43744|-404.40|961126| |
 10223240|BOB ANDERSON|7215 ARGYLL ROAD|VANCOUVER|AL|T6C4J2|52899|3597|0|9990|-90.99|980601| |
 10308685|TOM PERRY|3932 - 3A ST. N.W.|VANCOUVER|AL|T2E6R4|70192|68155|0|125773|1065.10|920601| |
 10318833|HARRY SMIRNOFF|BOX 308, STN J|VANCOUVER|AL|T2A4X6|11399|54541|42046|93502|761.46|970530| |
  1. Convert delimited to fixed for COBOL by running 'pfq2/citytax1', which was generated by genfixd2 on page '4J4' & edited on page '4J6' if necessary (might need '.dat' on input file to match DB2 export).


 #4a. uvcopy pfq2/citytax1,uop=e0s1
      =============================
      - convert d4pipe/citytax1 to d5fix/citytax1
      - option uop=e0 to output ASCII & NOT translate to EBCDIC
      - option uop=s1 converts binary fields to BIG-end format (mainframe format)

check Output fixed - 1st data record only


 #5. uvhd d5fix/citytax1 r128
     ========================
 rec#=1 rsize=128 fptr=0 fsize=1280 rcount=10
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             3333333322444424445522222222222222233332445442544422222222225444
             1013014000AF8E085E29000000000000000181502F75E02F14000000000061E3
          64 OUVER           BCV9S1H1..C...W.........qq.001490610970530
             455452222222222244535343004D005880008000770333333333333333222222
             F5652000000000002369318101320072C0047C0111C001490610970530000000

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4J8. Converting Delimited to Fixed field

execute COBOL processing fixed records


 6. EXECUTE your COBOL program here to process d5fix/... to d6fix/...
    =================================================================
Note
  • for this demo, we will simply copy d5fix/citytax1 to d6fix/citytax1

 #7a. cp d5fix/citytax1 d6fix/citytax1
      ================================
      - copy demo file (vs executing your COBOL program)
  1. Convert COBOL fixed output back to delimited, by running pfp/citytax2 which was generated by genfixd2 on page '4J4', listed on page '4J5', and edited on page '4J6' to change subdirs on fili1=... & filo1=...


 #7a. uvcopy pfp2/citytax1
      ====================
      - convert fixed d6fix/citytax1 to delim d7pipe/citytax1

check Output delimited - 1st few records


 #8. head  d7pipe/citytax1
     =====================
 10130140|JOHN HENRY|1815 BOWEN ROAD|VANCOUVER|BC|V9S1H1|82898|57828|4870|171710|1490.61|970530| |
 10139923|GEORGE BROWN|1250 EAST PENDER ST.|VANCOUVER|BC|V5L1W1|82898|57828|4878|178524|-1462.61|980601| |
 10147615|BONNIE SMITH|44430 YALE ROAD WEST|VANCOUVER|BC|V2P6J1|121395|39623|0|51914|376.92|950601| |
 10149304|TONY TATTERSAL|3350 S.W. MARINE DRIVE|VANCOUVER|BC|V6N3Y9|50594|76800|0|208391|1925.03|930601| |
 10154688|MEL RAHAL|1110 - 625 HOWE STREET|VANCOUVER|BC|V6C2T6|82898|44880|0|125038|1154.79|961126| |
 10173320|SAM SMITH|2381 ROGERS AVE|VANCOUVER|BC|V3K5Y2|50498|60600|0|169086|1562.00|970530| |
 10204700|JOHN JENSEN|12611-100 ST|VANCOUVER|AL|T8V4H2|122396|15720|0|43744|-404.40|961126| |
 10223240|BOB ANDERSON|7215 ARGYLL ROAD|VANCOUVER|AL|T6C4J2|52899|3597|0|9990|-90.99|980601| |
 10308685|TOM PERRY|3932 - 3A ST. N.W.|VANCOUVER|AL|T2E6R4|70192|68155|0|125773|1065.10|920601| |
 10318833|HARRY SMIRNOFF|BOX 308, STN J|VANCOUVER|AL|T2A4X6|11399|54541|42046|93502|761.46|970530| |

 #9. diff d7pipe/citytax1 d4pipe/citytax1
     ====================================
     - could compare cobol output in delim format to original input delim

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4J9. Converting Delimited to Fixed field

Re-Execute to demo BIG/little-end & file compare


 #1. uvcopy pfq2/citytax1,fili1=d4pipe/citytax1,filo1=d5fix/citytax1s0,uop=s0
     ========================================================================
     - 'uop=s0' specifies little-end option on the command-line
     - OR, could enter at the options prompt, as shown below --> s0 <--
       uop=q1e0s1 - option defaults
             e0   - output ASCII  (do NOT translate char/zoned to EBCDIC)
             e1   - output EBCDIC (translate char/zoned fields to EBCDIC)
               s1 - switch binary field ends if running on Intel (little-end)
               s0 - use s0 if running on BIG-end machine (such as AIX)
       User OPtion (uop) defaults  = q1e0s1s0
       -->null to accept or enter/override --> s0 <-- ENTER option for little-end
Note
  • we explicitly specified output file 'filo1=d5fix/citytax1s0'
    to NOT overwrite the previous default output d5fix/citytax1 (BIG-end)

 #2. uvcmpFA1 d5fix/citytax1 d5fix/citytax1s0 r128
     =============================================
     - compare prior BIG-end output to this little-end output
 uvcmp1 - compare 2 files, print mismatched records, '*' flag diffs
 2012/02/19_16:15:20 uop=q1p30r256s6t500000u3x2y0q1r128
 recsize  reccount   file-size  typ Report=rptcmp/citytax1
 1:  128        10       1,280  RSF  File1=d5fix/citytax1
 2:  128        10       1,280  RSF  File2=d5fix/citytax1s0
                       1         2         3         4         5         6
 f#record#byte# 0123456789012345678901234567890123456789012345678901234567890123
 ===============================================================================
 1       1    0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
                3333333322444424445522222222222222233332445442544422222222225444
                1013014000AF8E085E29000000000000000181502F75E02F14000000000061E3
             64 OUVER           BCV9S1H1..C...W.........qq.000149061970530
                455452222222222244535343004D005880008000770333333333333333222222
                F5652000000000002369318101320072C0047C0111C000149061970530000000
                                        ****
 2       1      10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
                3333333322444424445522222222222222233332445442544422222222225444
                1013014000AF8E085E29000000000000000181502F75E02F14000000000061E3
             64 OUVER           BCV9S1H1.C....W.........qq.000149061970530
                455452222222222244535343D400005880008000770333333333333333222222
                F5652000000000002369318123100072C0047C0111C000149061970530000000
                                        ****
 ==================== EOF or StopPrint/StopRead count reached  ==============
 F1Count=10, F2Count=10, StopPrint=6, StopRead=500000
 F1Reads=10, MisMatches=10, MisMatsPrinted=6, Recsize=128

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4J10. Converting Data Files from Delimited to Fixed field records

understanding BIG-end & little-end binary

To help you understaand BIG/little end binary, we have collected relevant items from the previous paages:


4J3. copybook/map, only the binary field 'post-date' at byte 88(4b)
      10 post-date             pic s9(00007) comp-4.          0088 0091  004bns 7

4J7. 1st line of Pipe|Delimited file, note binary field value is '+82898'
 10130140|JOHN HENRY|1815 BOWEN ROAD|VANCOUVER|BC|V9S1H1|+82898|57828|4870|171710|1490.61|970530| |

4J9. file-comparison report, showing bytes 64-127 with difference in bytes 88-91
             64 OUVER           BCV9S1H1..C...W.........qq.000149061970530
                455452222222222244535343004D005880008000770333333333333333222222
                F5652000000000002369318101320072C0047C0111C000149061970530000000
                                        ****
             64 OUVER           BCV9S1H1.C....W.........qq.000149061970530
                455452222222222244535343D400005880008000770333333333333333222222
                F5652000000000002369318123100072C0047C0111C000149061970530000000
                                        ****

We can express the difference field in horizontal hex as follows:

x'000143D2'
  • BIG-end binary for decimal '+82898'
  • binary system used on mainframes & some unix (such as AIX)
  • normal arithmetic, positional values increase from right to left
x'D2430100'
  • little-end binary for decimal '+82898'
  • used on Intel architectures
  • bytes reversed, positional values increase from left to right

Can convert BIG-end binary to decimal as follows (working from right to left):

      2 * 00001 = 00002
      D * 00016 = 00208
      3 * 00256 = 00768
      4 * 04096 = 16384
      1 * 65536 = 65536
      =================
                  88898

genpipe1 option s0/s1 for BIG-end/little-end

genpipe1 generates edit instructions with or without option 's' depending on the machine architecture - BIG-end or little-end (AIX,SUN,etc or INTEL). On page '4B4' the binary field post-date was generated as follows on my INTEL.

        edta1  c600(9),a88(4bs),'+zzzzzzzzz9'       #7 post-date

See more discussion on page '4D3'.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4K1. converting data files to pipe|delimited

genpipe1 with occurs

genpipe1 will expand 'item occurs' (occurs & pic on same statement), but will not expand 'group occurs'. See following '4L1' cpyoccurs1, which would be run before cobmap1 & genpipe1.

Here we will demo genpipe1 with occurs, using the armaster copybook, which has occurs 12 for this year & last year sales. This assumes you are operating in the cnvdata1 directory with subdirs as shown on page '4A2'.


 #1. genpipeA armaster.cpy  <-- easiest way to generate all jobs required for pipe delim
     =====================      (cobmap1, uvdata51, genpipe1)
                              - easier than run separately as follows:

 #1a. uvcopy cobmap1,fili1=cpys/armaster.cpy,filo1=maps/armaster
      ==========================================================
      - generate cobmap layout (from copybook)

 #1b. uvcopy uvdata51,fili1=maps/armaster,filo1=pfx1/armaster
      ======================================================
      - generate uvcopy job to convert EBCDIC data to ASCII (from cobmap layout)
      - optional, not required if data file already ASCII

 #1c. uvcopy genpipe1,fili1=maps/armaster,filo1=pfp1/armaster
      ======================================================
      - generate uvcopy job to convert ASCII data to pipe delimited

 #2. uvcopy pfx1/armaster <-- execute uvcopy job to convert EBCDIC to ASCII
     ====================   - optional, not required if data file already ASCII

 #3. uvcopy pfp1/armaster <-- execute uvcopy job to convert ASCII to pipe delim
     ====================

cobmap for armaster to demo occurs pipe delimit

 cobmap1  COBOL record layout            2013/05/19_10:32:12  pg# 0001
 cpys/armaster.cpy                                RCSZ=00256  bgn  end  lth typ
 * armaster - cobol copy book for armaster file (see uvtrain.
      10 cm-num           pic  9(6).                          0000 0005  006 n  6
      10 cm-delete        pic  x(4).                          0006 0009  004
      10 cm-name          pic  x(25).                         0010 0034  025
      10 cm-adrs          pic  x(25).                         0035 0059  025
      10 cm-city          pic  x(16).                         0060 0075  016
      10 filler001        pic  x.                             0076 0076  001
      10 cm-prov          pic  x(2).                          0077 0078  002
      10 filler002        pic  x.                             0079 0079  001
      10 cm-postal        pic  x(10).                         0080 0089  010
      10 cm-phone         pic  x(12).                         0090 0101  012
      10 cm-contact       pic  x(18).                         0102 0119  018
      10 cm-thisyr-sales  pic  s9(7)v99 comp-3 occurs 12.     0120 0124  005pns 9
      10 cm-lastyr-sales  pic  s9(7)v99 comp-3 occurs 12.     0180 0184  005pns 9
      10 filler003        pic  x(16).                         0240 0255  016

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4K2. converting data files to pipe|delimited

uvcopy job to pipe delimit armaster w occurs

 # armaster - uvcopy job to convert EBCDIC/ASCII data to delimited text file
 opr='armaster - uvcopy code generated from copybook: armaster.cpy '
 rop=j200000r1  #Run OPtions: increase instrn storage & prompt display output
 was=a33000b33000c999000d33000e33000u16384v16384  #increase u&v if fields > 163
 fili1=?d2asc/armaster,rcs=00256,typ=RSF
 filo1=?d4pipe/armaster,rcs=00768,typ=LSTtd3
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
 # area a input, see 'get' instrn
 # area b translated to ASCII, in case mainframe EBCDIC file (with packed?)
 # area c data fields fixed 100 bytes apart in prep for var | delimit instrn
 # area d output, see 'put' instrn at end
        mvc    b0(00256),a0       move input area a to area b
 ###    tra    b0(00256)          uncomment for EBCDIC input (or option a1)
 #      ---                   <-- & insert instrns from uvdata51 if packed fields
 #      ---                   <-- insert R/T tests if redef records
        mvc    c0(6),b0(6)                          #1 cm-num
        mvc    c100(4),b6(4)                        #2 cm-delete
        mvc    c200(25),b10(25)                     #3 cm-name
        mvc    c300(25),b35(25)                     #4 cm-adrs
        mvc    c400(16),b60(16)                     #5 cm-city
        mvc    c500(1),b76(1)                       #6 filler001
        mvc    c600(2),b77(2)                       #7 cm-prov
        mvc    c700(1),b79(1)                       #8 filler002
        mvc    c800(10),b80(10)                     #9 cm-postal
        mvc    c900(12),b90(12)                     #10 cm-phone
        mvc    c1000(18),b102(18)                   #11 cm-contact
        edta1  c1100(11),a120(5p),'+zzzzzzz.99'     #12 cm-thisyr-sales occurs 12
        edta1  c1200(11),a125(5p),'+zzzzzzz.99'     #13 cm-thisyr-sales occurs 12
 #                   --- 9 fields omitted ---
        edta1  c2200(11),a175(5p),'+zzzzzzz.99'     #23 cm-thisyr-sales occurs 12
 #
        edta1  c2300(11),a180(5p),'+zzzzzzz.99'     #24 cm-lastyr-sales occurs 12
        edta1  c2400(11),a185(5p),'+zzzzzzz.99'     #25 cm-lastyr-sales occurs 12
 #                   --- 9 fields omitted ---
        edta1  c3400(11),a235(5p),'+zzzzzzz.99'     #35 cm-lastyr-sales occurs 12
 #
        mvc    c3500(16),b240(16)                   #36 filler003
        rep    c0(03600),'|','!'           #comment speedup if no '|'s in data
        varb1  d0(03600),c0(100),0036,'|'  #insert '|'s between fields & squeeze
        trt    d0(03600),$trtchr           #ensure all ascii chars x'20'-x'7E'
        clr    c0(03600),' '
 put1   put    filo1,d0
        skp    loop
 #
 eof    cls    all
        eoj
 @pf2=genpipe1.sub

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4K3. converting data files to pipe|delimited

execute uvcopy job to pipe delimit armaster occurs


 #2. uvcopy pfx1/armaster <-- execute uvcopy job to convert EBCDIC to ASCII
     ====================   - optional, not required if data file already ASCII

 #3. uvcopy pfp1/armaster <-- execute uvcopy job to convert ASCII to pipe delim
     ====================

input d2asc/armaster


 uvhd /home/mvstest1/ALLcnvdata1/d2asc/armaster
 ==============================================
 rec#=1 rsize=256 fptr=0 fsize=8192 rcount=32
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    EVERGREEN MOTORS LTD.    1815 BOWEN ROAD          NANA
             3333332222454545444244545524542222233332445442544422222222224444
             130140000056527255E0DF4F230C44E0000181502F75E02F140000000000E1E1
          64 IMO          BC V9S1H1    250-754-5531 LARRY WRENCH     ..4V|...
             4442222222222442535343222233323332333324455525544442222201357000
             9DF00000000002306931810000250D754D55310C12290725E38000000246C000
         128 .........W0....`........)X}..f3.....\.................4V}...f...
             0000000005300016000000002570063100095000000000000000013570016000
             0C0000C0270D0540C0000C0098D0263C0444C0000C0000C0000C0246D0056C00
         192 .E|...V}.......................f.....<........f.C 19950531
             0470005700000000880000000018000680001300000000694233333333222222
             35C0046D0000C0023C0000C0083C0056D0012C0000C0016D3019950531000000

ouput d4pipe/armaster

 130140| |EVERGREEN MOTORS LTD.|1815 BOWEN ROAD|NANAIMO| |BC| |V9S1H1
 |250-754-5531|LARRY WRENCH
 |+12345.67|+.00|+.00|-2573.00|+5146.00|+.00
 |-295.87|+2663.31|+4049.45|+.00|+.00|+.00
 |-12345.67|+156.60|+34.57|-45.67|+.00|+28.38
 |+.00|+81.38|-56.68|+11.23|+.00|-16.69
 |C 19950531|

Notes

  1. The 1st input record is shown by uvhd, which displays the 24 packed fields in vertical hexadecimal, so you can read the packed field contents.

  2. The 1st output record (pipe delimited) would be all 1 long line of text,but is here split onto multiple lines so you can read it in this documentation.

  3. You should be able to correlate the input packed field contents to the output pipe delimted text fields.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4L1. converting GROUP OCCURS to pipe|delimited

cpyoccurs1 for GROUP occurs

genpipe1 (see prior pages '4K1'-'4K3' expands 'item occurs' but not expand 'group occurs'. For group occurs, run 'cpyoccurs1' on the copybook before cobmap1 & genpipe1.

Here we will demo cpyoccurs1, using the custran1 copybook, which has group occurs 5 for 3 items. This assumes you are operating in the cnvdata1 directory with subdirs as shown on page '4A2'.


 #1. mkdir cpys2        <-- make subdir for output of 'cpyoccurs1'
     ===========

 #2. uvcopy cpyoccurs1,fili1=cpys/custran1,filo1=cpys2/custran1
     =========================================================
     - copy the copybook, expanding group occurs

 #3. uvcopy cobmap1,fili1=cpys2/custran1,filo1=maps/custran1
     =======================================================
     - generate cobmap layout (from copybook with expanded group occurs)

 #4. uvcopy uvdata51,fili1=maps/custran1,filo1=pfx1/custran1
     =======================================================
     - generate uvcopy job to convert EBCDIC data to ASCII (from cobmap layout)
     - optional, not required if data file already ASCII

 #5. uvcopy genpipe1,fili1=maps/custran1,filo1=pfp1/custran1
     =======================================================
     - generate uvcopy job to convert ASCII data to pipe delimited

 #2. uvcopy pfx1/custran1 <-- execute uvcopy job to convert EBCDIC to ASCII
     ====================   - optional, not required if data file already ASCII

 #3. uvcopy pfp1/custran1 <-- execute uvcopy job to convert ASCII to pipe delim
     ====================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4L2. converting GROUP OCCURS to pipe|delimited

cobmap for custran1 BEFORE group occurs expanded

 cobmap1  COBOL record layout            2013/05/19_15:56:08  pg# 0001
 cpys/custran1                                     RCSZ=00256  bgn  end  lth typ
      10 cmt-num           pic  9(6).                         0000 0005  006 n  6
      10 cmt-delete        pic  x(4).                         0006 0009  004
      10 cmt-name          pic  x(25).                        0010 0034  025
      10 cmt-adrs          pic  x(25).                        0035 0059  025
      10 cmt-city          pic  x(16).                        0060 0075  016
      10 filler001         pic  x.                            0076 0076  001
      10 cmt-prov          pic  x(2).                         0077 0078  002
      10 filler002         pic  x.                            0079 0079  001
      10 cmt-postal        pic  x(10).                        0080 0089  010
      10 cmt-phone         pic  x(12).                        0090 0101  012
      10 cmt-contact       pic  x(18).                        0102 0119  018
 *BGNOCCURSM:c-p:00005*00010=00050:00120-00169:1:
      10 cmt-trans occurs 5.
         15 cmt-tran-type  pic  x.                            0120 0120  001
         15 cmt-tran-ref   pic  9(4).                         0121 0124  004 n  4
         15 cmt-tran-amt   pic  s9(7)v99 comp-3.              0125 0129  005pns 9
 *ENDOCCURSM:c-p:00005*00010=00050:00120-00169:1:
      10 filler003         pic  x(86).                        0170 0255  086

cobmap for custran1 AFTER group occurs expanded

 cobmap1  COBOL record layout            2013/05/19_14:04:02  pg# 0001
 cpys2/custran1                                    RCSZ=00256  bgn  end  lth typ
      10 cmt-num           pic  9(6).                         0000 0005  006 n  6
      10 cmt-delete        pic  x(4).                         0006 0009  004
      10 cmt-name          pic  x(25).                        0010 0034  025
      10 cmt-adrs          pic  x(25).                        0035 0059  025
      10 cmt-city          pic  x(16).                        0060 0075  016
      10 filler001         pic  x.                            0076 0076  001
      10 cmt-prov          pic  x(2).                         0077 0078  002
      10 filler002         pic  x.                            0079 0079  001
      10 cmt-postal        pic  x(10).                        0080 0089  010
      10 cmt-phone         pic  x(12).                        0090 0101  012
      10 cmt-contact       pic  x(18).                        0102 0119  018
 *    10 cmt-trans occurs 5.
         15 cmt-tran-type001  pic  x.                         0120 0120  001
         15 cmt-tran-ref001   pic  9(4).                      0121 0124  004 n  4
         15 cmt-tran-amt001   pic  s9(7)v99 comp-3.           0125 0129  005pns 9
         15 cmt-tran-type002  pic  x.                         0130 0130  001
         15 cmt-tran-ref002   pic  9(4).                      0131 0134  004 n  4
         15 cmt-tran-amt002   pic  s9(7)v99 comp-3.           0135 0139  005pns 9
         15 cmt-tran-type003  pic  x.                         0140 0140  001
         15 cmt-tran-ref003   pic  9(4).                      0141 0144  004 n  4
         15 cmt-tran-amt003   pic  s9(7)v99 comp-3.           0145 0149  005pns 9
         15 cmt-tran-type004  pic  x.                         0150 0150  001
         15 cmt-tran-ref004   pic  9(4).                      0151 0154  004 n  4
         15 cmt-tran-amt004   pic  s9(7)v99 comp-3.           0155 0159  005pns 9
         15 cmt-tran-type005  pic  x.                         0160 0160  001
         15 cmt-tran-ref005   pic  9(4).                      0161 0164  004 n  4
         15 cmt-tran-amt005   pic  s9(7)v99 comp-3.           0165 0169  005pns 9
      10 filler003         pic  x(86).                        0170 0255  086

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4L3. converting GROUP OCCURS to pipe|delimited

uvcopy job to pipe delimit GROUP OCCURS

 # custran1 - uvcopy job to convert EBCDIC/ASCII data to delimited text file
 opr='custran1 - uvcopy code generated from copybook: custran1 '
 rop=j200000r1  #Run OPtions: increase instrn storage & prompt display output
 was=a33000b33000c999000d33000e33000u16384v16384  #increase u&v if fields > 163
 fili1=?d2asc/custran1,rcs=00256,typ=RSF
 filo1=?d4pipe/custran1,rcs=00768,typ=LSTtd3
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
 # area a input, see 'get' instrn
 # area b translated to ASCII, in case mainframe EBCDIC file (with packed?)
 # area c data fields fixed 100 bytes apart in prep for var | delimit instrn
 # area d output, see 'put' instrn at end
        mvc    b0(00256),a0       move input area a to area b
 ###    tra    b0(00256)          uncomment for EBCDIC input (or option a1)
        mvc    c0(6),b0(6)                          #1 cmt-num
        mvc    c100(4),b6(4)                        #2 cmt-delete
        mvc    c200(25),b10(25)                     #3 cmt-name
        mvc    c300(25),b35(25)                     #4 cmt-adrs
        mvc    c400(16),b60(16)                     #5 cmt-city
        mvc    c500(1),b76(1)                       #6 filler001
        mvc    c600(2),b77(2)                       #7 cmt-prov
        mvc    c700(1),b79(1)                       #8 filler002
        mvc    c800(10),b80(10)                     #9 cmt-postal
        mvc    c900(12),b90(12)                     #10 cmt-phone
        mvc    c1000(18),b102(18)                   #11 cmt-contact
        mvc    c1100(1),b120(1)                     #12 cmt-tran-type001
        mvc    c1200(4),b121(4)                     #13 cmt-tran-ref001
        edta1  c1300(11),a125(5p),'+zzzzzzz.99'     #14 cmt-tran-amt001
        mvc    c1400(1),b130(1)                     #15 cmt-tran-type002
        mvc    c1500(4),b131(4)                     #16 cmt-tran-ref002
        edta1  c1600(11),a135(5p),'+zzzzzzz.99'     #17 cmt-tran-amt002
        mvc    c1700(1),b140(1)                     #18 cmt-tran-type003
        mvc    c1800(4),b141(4)                     #19 cmt-tran-ref003
        edta1  c1900(11),a145(5p),'+zzzzzzz.99'     #20 cmt-tran-amt003
        mvc    c2000(1),b150(1)                     #21 cmt-tran-type004
        mvc    c2100(4),b151(4)                     #22 cmt-tran-ref004
        edta1  c2200(11),a155(5p),'+zzzzzzz.99'     #23 cmt-tran-amt004
        mvc    c2300(1),b160(1)                     #24 cmt-tran-type005
        mvc    c2400(4),b161(4)                     #25 cmt-tran-ref005
        edta1  c2500(11),a165(5p),'+zzzzzzz.99'     #26 cmt-tran-amt005
        mvc    c2600(86),b170(86)                   #27 filler003
        rep    c0(02700),'|','!'           #comment speedup if no '|'s in data
        varb1  d0(02700),c0(100),0027,'|'  #insert '|'s between fields & squeeze
        trt    d0(02700),$trtchr           #ensure all ascii chars x'20'-x'7E'
        clr    c0(02700),' '
 put1   put    filo1,d0
        skp    loop
 eof    cls    all
        eoj
 @pf2=genpipe1.sub

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4L4. converting GROUP OCCURS to pipe|delimited

execute uvcopy job to pipe delimit GROUP OCCURS


 #2. uvcopy pfx1/custran1 <-- execute uvcopy job to convert EBCDIC to ASCII
     ====================   - optional, not required if data file already ASCII

 #3. uvcopy pfp1/custran1 <-- execute uvcopy job to convert ASCII to pipe delim
     ====================

input d2asc/custran1


 uvhd /home/mvstest1/ALLcnvdata1/d2asc/custran1
 ==============================================
 rec#=1 rsize=256 fptr=0 fsize=8192 rcount=32
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    EVERGREEN MOTORS LTD.    1815 BOWEN ROAD          NANA
             3333332222454545444244545524542222233332445442544422222222224444
             130140000056527255E0DF4F230C44E0000181502F75E02F140000000000E1E1
          64 IMO          BC V9S1H1    250-754-5531 LARRY WRENCH     A0101..4
             4442222222222442535343222233323332333324455525544442222243333013
             9DF00000000002306931810000250D754D55310C12290725E380000010101024
         128 V|B0102...`.C0103....\ 0000.....E0105..4V}
             5743333001604333300095233330000043333013572222222222222222222222
             6C201020540C301030444C000000000C501050246D0000000000000000000000
         192
             2222222222222222222222222222222222222222222222222222222222222222
             0000000000000000000000000000000000000000000000000000000000000000

ouput d4pipe/custran1

 130140| |EVERGREEN MOTORS LTD.|1815 BOWEN ROAD|NANAIMO| |BC| |V9S1H1
 |250-754-5531| LARRY WRENCH
 |A|0101|+12345.67|B|0102|+5146.00|C|0103|+4049.45| |0000|+.00|E|0105|-12345.67| |
 132588| |GEECOE GENERATOR SERVICES|UNIT 170 - 2851 SIMPSON|RICHMOND| |BC| |V6X2R2
 |604-278-4488| HARRY LIGHT
 |A|0201|+46.80| |0000|+.00|C|0203|+37.08|D|0204|+53.60| |0000|+.00| |
 139923| |JOHNSTONE BOILER & TANKS|1250 EAST PENDER STREET|VANCOUVER| |BC| |V5L1W1
 |604-320-1845|GEORGE BROWN
 |A|0301|+15.28| |0000|+.00| |0000|+.00| |0000|+.00|E|0305|+144.96| |

Notes

  1. The 1st input record is shown by uvhd, which displays in vertical hexadecimal, so you can read the packed field contents.

  2. We are showing 1st 3 output records (pipe delimited) would be long lines of text, but here split onto multiple lines so you can see eentire records.

  3. You should be able to correlate the input packed field contents to the output pipe delimted text fields.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4M1. Converting Data Files to ASCII Delimited for loading DB tables

genpipe3 alternative to genpipe1

'genpipe3' was developed in Oct 2014 to increase the max field length from 100 allowed by 'genpipe1' to 32,767 and genpipe3 is also much faster.

demo genpipe3 using citytax1 copybook & datafile

This is similar to the genpipe1 demo on pages '4B3' - '4B5'. The easiest way to run genpipe3 is to use script 'gencnvA' or 'gencnvD2' which include the pre-requisite cobmap as well as genfixd3 (undelimit). These scripts have been updated to run the new genpipe3/genfixd3 jobs vs the old genpipe1/genfixd2 jobs (max field limit 100 bytes).


 gencnvA citytax1.cpy   <-- script runs cobmap1,uvdata51,genpipe3,genfixd3
 ====================

 gencnvD2 citytax1.cpy   <-- script runs cobmap1,uvdata51,genpipe3,genfixd3
 =====================       + sqlcreateA
Note
  • above scripts include following uvcopy jobs (+ some others)

 uvcopy cobmap1,fili1=cpys/citytax1.cpy,filo1=maps/citytax1
 ==========================================================

 uvcopy genpipe3,fili1=maps/citytax1,filo1=pfp1/citytax1
 =======================================================

genpipe3 console log options

 uop=a0e2s0   - default options
     a0       - file already translated from EBCDIC to ASCII by uvdata51
     a1       - translate data from EBCDIC to ASCII here in genpipe3 job
              - must transfer any packed field instrns from the uvdata51
                (subdir pfx1) to this genpipe3 job (subdir pfp1)
       e1     - edit signed fields with leading "-" only if negative
       e2     - edit signed fields with + if pos (as well as - if neg)
         s0   - auto Binary Switch (Big-END AIX/SUN, little-end Intel)
              - use option s1/s2 only if generating code for other machines
         s1   - force no switch (default on BIG-end machines (AIX/SUN)
         s2   - force switch ends (default on Little-end machines (Intel)
 User OPtion (uop) defaults  = q1a0e2s0
 -->null to accept or enter/override -->
 141026:172434:genpipe3: EOF fili01 rds=19 size=1257: maps/citytax1
 141026:172434:genpipe3: EOF filo01 wrts=38 size=1923: pfp1/citytax1

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4M2. Converting Data Files to ASCII Delimited for loading DB tables

sample cobmap citytax1 input to genpipe3 demo

 cobmap1  COBOL record layout            2014/10/07_12:42:02  pg# 0001
 cpys/citytax1.cpy             citytax1           RCSZ=00128  bgn  end  lth typ
 * citytax1 copybook for uv demos www.uvsoftware.ca/datacnv1.
  01  citytax1.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-4.          0088 0091  004bns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  9(00006).                 0116 0121  006 n  6
      10 filler001             pic  x(00006).                 0122 0127  006
 *RCSZ=00128

field type conversion rules


 Character pic x
 ===============
 - any bytes < space x'20' are converted to spaces
 - trailing spaces are removed
 - any delimiter (default arg1=|) in existing data is replaced by arg2=! (default)

 Numeric pic 9 (zoned, unsigned)
 ===============================
 - any non-numerics replaced by zeros '0's

 Packed, Binary, Numeric signed
 ==============================
 - converted to binary (dropping any non-digits)
 - edited with leading +/- sign
 - decimal point inserted depending on pic (...9v9...)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4M3. Converting Data Files to ASCII Delimited for loading DB tables

sample job citytax1 generated by genpipe3

 # citytax1 - uvcopy job to convert EBCDIC/ASCII data to delimited text file
 opr='citytax1 - uvcopy code generated from copybook: citytax1.cpy '
 rop=j200000    #Run OPtions: increase instrn storage
 arg1=|,arg2=!  # arg1=delimiter, arg2=delimiter replacement if in existing data
 #              # arg1=x.. for hex control chars, x09 for tab, etc
 was=a64000b64000c64000d64000e64000u64000v64000
 fili1=?d2asc/citytax1,rcs=00128,typ=RSF
 filo1=?d4pipe/citytax1,rcs=64000,typ=LSTtd3
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
 # area a input, see 'get' instrn
 # area b translated to ASCII, in case mainframe EBCDIC file (with packed?)
 # area c built field by field by 'ftd' instruction (allows > 100 bytes/field)
 #Oct2014 - vs prior fixed 100 bytes apart for var | delimit instrn
        mvc    b0(00128),a0       move input area a to area b
 ###    tra    b0(00128)          uncomment for EBCDIC input (or option a1)
 #      ---                   <-- & insert instrns from uvdata51 if packed fields
 #      ---                   <-- insert R/T tests if redef records
        ftd    b0(10c)                              #1 folio
        ftd    b10(25c)                             #2 name
        ftd    b35(25c)                             #3 address
        ftd    b60(20c)                             #4 city
        ftd    b80(2c)                              #5 province
        ftd    b82(6c)                              #6 zip
        ftd    a88(4bs),'+zzzzzz9'                  #7 post-date
        ftd    a92(5p),'+zzzzzzzz9'                 #8 land-value
        ftd    a97(5p),'+zzzzzzzz9'                 #9 improv-value
        ftd    a102(5p),'+zzzzzzzz9'                #10 face-value
        ftd    b107(9z),'+zzzzzzz.99'               #11 maint-tax
        ftd    b116(6n)                             #12 purchase-date
        ftd    b122(6c)                             #13 filler001
 put1   put    filo1,c0
        clr    c0($rc64000),' '
        skp    loop
 eof    cls    all
        eoj

arg1=x.. specify delimiter in hexadecimal

Line 4 of the generated job defines the delimiter (default arg1=|). You can change as desired & may specify in hexadecimal. For example to specify the delimiter as the 'tab' character, use 'arg1=x09').

options to inhibit dropping control characters

For each field, trailing blanks are eliminated, the delimiter is inserted after the last non-blank. Control characters (bytes < x'20') are also eliminated. You may inhibit drop ctlchars by Run option 'w1' OR instruction option 'c1'

 --> rop=w1           #<-- run option 'w1' inhibit drop ctlchars all fields
 --> ftdc1 b10(25c)   #<-- instrn option c1 inhibit drop ctlchars for 1 field

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4M4. Converting Data Files to ASCII Delimited for loading DB tables

execute generated job pfp1/citytax1


 uvcopy pfp1/citytax1
 ====================

citytax1 - uvcopy code generated from copybook: citytax1.cpy d2asc/citytax1 = default fili01 - null accept or reenter ? d4pipe/citytax1 = default filo01 - null accept or reenter ? 141026:175313:citytax1: EOF fili01 rds=10 size=1280: d2asc/citytax1 141026:175313:citytax1: EOF filo01 wrts=10 size=1051: d4pipe/citytax1

uvhd display 1st 2 records input


 uvhd d2asc/citytax1 r128
 ========================
 rec#=1 rsize=128 fptr=0 fsize=1280 rcount=10
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             3333333322444424445522222222222222233332445442544422222222225444
             1013014000AF8E085E29000F00000000000181502F75E02F14000000000061E3
          64 OUVER           BCV9S1H1..C...W.........qq.000149061970530
             455452222222222244535343004D005880008000770333333333333333222222
             F5652000000000002369318101320072C0047C0111C000149061970530000000
 rec#=2 rsize=128 fptr=128 fsize=1280 rcount=10
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 10139923  GEORGE BROWN             1250 EAST PENDER ST.     VANC
             3333333322444544245454222222222222233332445525444452552222225444
             101399230075F275022F7E0000000000000125005134005E452034E0000061E3
          64 OUVER           BCV5L1W1..C...W.........xRL00014626q980601
             455452222222222244534353004D005880008800754333333337333333222222
             F5652000000000002365C17101320072C0047C0182C000146261980601000000

pipe|delimited output file


 cat d4pipe/citytax1
 ===================
 10130140|JOHN HENRY|1815 BOWEN ROAD|VANCOUVER|BC|V9S1H1|+82898|+57828|+4870|+171710|+1490.61|970530| |
 10139923|GEORGE BROWN|1250 EAST PENDER ST.|VANCOUVER|BC|V5L1W1|+82898|+57828|+4878|+178524|-1462.61|980601| |
 10147615|BONNIE SMITH|44430 YALE ROAD WEST|VANCOUVER|BC|V2P6J1|+121395|+39623|+0|+51914|+376.92|950601| |
 10149304|TONY TATTERSAL|3350 S.W. MARINE DRIVE|VANCOUVER|BC|V6N3Y9|+50594|+76800|+0|+208391|+1925.03|930601| |
 10154688|MEL RAHAL|1110 - 625 HOWE STREET|VANCOUVER|BC|V6C2T6|+82898|+44880|+0|+125038|+1154.79|961126| |
 10173320|SAM SMITH|2381 ROGERS AVE|VANCOUVER|BC|V3K5Y2|+50498|+60600|+0|+169086|+1562.00|970530| |
 10204700|JOHN JENSEN|12611-100 ST|VANCOUVER|AL|T8V4H2|+122396|+15720|+0|+43744|-404.40|961126| |
 10223240|BOB ANDERSON|7215 ARGYLL ROAD|VANCOUVER|AL|T6C4J2|+52899|+3597|+0|+9990|-90.99|980601| |
 10308685|TOM PERRY|3932 - 3A ST. N.W.|VANCOUVER|AL|T2E6R4|+70192|+68155|+0|+125773|+1065.10|920601| |
 10318833|HARRY SMIRNOFF|BOX 308, STN J|VANCOUVER|AL|T2A4X6|+11399|+54541|+42046|+93502|+761.46|970530| |

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4N1. Converting Data Files to ASCII Delimited for loading DB tables

genfixd3 alternative to genfixd2

'genfixd3' was developed in Oct 2014 to increase the max field length from 100 allowed by 'genfixd2' to 32,767 and genfixd3 is also faster.

demo genfixd3 using citytax1 copybook & datafile

This is similar to the genfixd2 demo on pages '4J1' - '4J10'. The easiest way to run genfixd3 is to use script 'gencnvA' or 'gencnvD2' which include the pre-requisite cobmap as well as genfixd3 (undelimit). These scripts have been updated to run the new genpipe3/genfixd3 jobs vs the old genpipe1/genfixd2 jobs (max field limit 100 bytes).


 gencnvA citytax1.cpy   <-- script runs cobmap1,uvdata51,genpipe3,genfixd3
 ====================

 gencnvD2 citytax1.cpy   <-- script runs cobmap1,uvdata51,genpipe3,genfixd3
 =====================       + sqlcreateA
Note
  • above scripts include following uvcopy jobs (+ some others)

 uvcopy cobmap1,fili1=cpys/citytax1.cpy,filo1=maps/citytax1
 ==========================================================

 uvcopy genfixd3,fili1=maps/citytax1,filo1=pfq1/citytax1
 =======================================================

genfixd3 console log options

 uop=q1e0s1t0   - default options (e0s1 generated job, t0 at gen time)
       e0       - output ASCII  (do not translate char/zoned to EBCDIC)
       e1       - output EBCDIC (translate char/zoned fields to EBCDIC)
         s1     - switch binary field ends if running on Intel (little-end)
                - for mainframe files converted on Intel machines
         s0     - use s0 if running on BIGend machine (such as AIX)
  ---------- above for GENERATED job, following for generation time
             t0 - output file typ=RSF (no LineFeed)
             t1 - output file typ=RST (LineFeed in last byte of recsize)
             t2 - output file typ=RSR (LineFeed appended to recsize)
 User OPtion (uop) defaults  = q1e0s1t0
 -->null to accept or enter/override -->   <-- null accept filenames on cmd line
 141027:111042:genfixd3: EOF fili01 rds=19 size=1257: maps/citytax1
 141027:111042:genfixd3: EOF filo01 wrts=36 size=1398: pfq1/citytax1

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4N2. Converting Data Files to ASCII Delimited for loading DB tables

sample cobmap citytax1 input to genfixd3 demo

 cobmap1  COBOL record layout            2014/10/07_12:42:02  pg# 0001
 cpys/citytax1.cpy             citytax1           RCSZ=00128  bgn  end  lth typ
 * citytax1 copybook for uv demos www.uvsoftware.ca/datacnv1.
  01  citytax1.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-4.          0088 0091  004bns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  9(00006).                 0116 0121  006 n  6
      10 filler001             pic  x(00006).                 0122 0127  006
 *RCSZ=00128

field type conversion rules


 Character pic x
 ===============
 - output to field defined by copybook depending on displacement,& length
 - space filled if input data length < copybook field length
 - truncated if input data length > copybook field length

 Numeric pic 9 (zoned, unsigned)
 ===============================
 - same as Character pic x
 Same length as original field in fixed-field record that created delimited
 - assuming that delimited input was created by genpipe3
 - which converted any non-numerics to zeros '0's

 Packed, Binary, Numeric signed
 ==============================
 - converted to binary (dropping any non-digits)
 - output to field defined by copybook depending on type, displacement,& length

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4N3. Converting Data Files to ASCII Delimited for loading DB tables

sample job citytax1 generated by genfixd3

 # citytax1 - uvcopy job to convert pipe delimited to Fixed-field format
 opr='citytax1 - generated by genfixd3 from copybook: citytax1.cpy '
 opr='uop=q1e0s1 - option defaults'
 opr='      e0   - output ASCII  (do not translate char/zoned to EBCDIC)'
 opr='      e1   - output EBCDIC (translate char/zoned fields to EBCDIC)'
 opr='        s1 - switch binary field ends if running on Intel (little-end)'
 opr='        s0 - use s0 if running on BIG-end machine (such as AIX)'
 uop=q1e0s1  # User option defaults
 arg1=|      # delimiter default, arg1=x.. for hex (arg1=x09 for tab)
 was=a64000b64000c64000d64000e64000
 fili1=?d4pipe/citytax1,rcs=00220,typ=LST
 filo1=?d5fix/citytax1,rcs=00128,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
 #      ---                   <-- insert R/T tests if redef records
        clr    d0(00128),x'20'    init to x'20' or x'40' if optn e1 EBCDIC
        dtf    d0(10c),c0,'folio'
        dtf    d10(25c),c0,'name'
        dtf    d35(25c),c0,'address'
        dtf    d60(20c),c0,'city'
        dtf    d80(2c),c0,'province'
        dtf    d82(6c),c0,'zip'
        dtf    d88(4b),c0,'post-date'
        dtf    d92(5p),c0,'land-value'
        dtf    d97(5p),c0,'improv-value'
        dtf    d102(5p),c0,'face-value'
        dtf    d107(9z),c0,'maint-tax'
        dtf    d116(6z),c0,'purchase-date'
        dtf    d122(6c),c0,'filler001'
 put1   put    filo1,d0
        skp    loop
 eof    cls    all
        eoj

arg1=x.. specify delimiter in hexadecimal

Line 9 of the generated job defines the delimiter (default arg1=|). You may specify in hexadecimal. For example to specify the 'tab' character

arg1=x09 #<-- change delimiter to tab (hex '09')

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4N4. Converting Data Files to ASCII Delimited for loading DB tables

execute generated job pfq1/citytax1


 uvcopy pfq1/citytax1
 ====================
 uop=q1e0s1 - option defaults
       e0   - output ASCII  (do not translate char/zoned to EBCDIC)
       e1   - output EBCDIC (translate char/zoned fields to EBCDIC)
         s1 - switch binary field ends if running on Intel (little-end)
         s0 - use s0 if running on BIG-end machine (such as AIX)
 User OPtion (uop) defaults  = q1e0s1
 -->null to accept or enter/override -->   <-- null accept default filenames
 d4pipe/citytax1 = default fili01 - null accept or reenter ?
 d5fix/citytax1 = default filo01 - null accept or reenter ?
 141027:111144:citytax1: EOF fili01 rds=10 size=1047: d4pipe/citytax1
 141027:111144:citytax1: EOF filo01 wrts=10 size=1280: d5fix/citytax1

pipe|delimited input file


 cat d4pipe/citytax1
 ===================
 10130140|JOHN HENRY|1815 BOWEN ROAD|VANCOUVER|BC|V9S1H1|+82898|+57828|+4870|+171710|+1490.61|970530| |
 10139923|GEORGE BROWN|1250 EAST PENDER ST.|VANCOUVER|BC|V5L1W1|+82898|+57828|+4878|+178524|-1462.61|980601| |
 10147615|BONNIE SMITH|44430 YALE ROAD WEST|VANCOUVER|BC|V2P6J1|+121395|+39623|+0|+51914|+376.92|950601| |
 10149304|TONY TATTERSAL|3350 S.W. MARINE DRIVE|VANCOUVER|BC|V6N3Y9|+50594|+76800|+0|+208391|+1925.03|930601| |
 10154688|MEL RAHAL|1110 - 625 HOWE STREET|VANCOUVER|BC|V6C2T6|+82898|+44880|+0|+125038|+1154.79|961126| |
 10173320|SAM SMITH|2381 ROGERS AVE|VANCOUVER|BC|V3K5Y2|+50498|+60600|+0|+169086|+1562.00|970530| |
 10204700|JOHN JENSEN|12611-100 ST|VANCOUVER|AL|T8V4H2|+122396|+15720|+0|+43744|-404.40|961126| |
 10223240|BOB ANDERSON|7215 ARGYLL ROAD|VANCOUVER|AL|T6C4J2|+52899|+3597|+0|+9990|-90.99|980601| |
 10308685|TOM PERRY|3932 - 3A ST. N.W.|VANCOUVER|AL|T2E6R4|+70192|+68155|+0|+125773|+1065.10|920601| |
 10318833|HARRY SMIRNOFF|BOX 308, STN J|VANCOUVER|AL|T2A4X6|+11399|+54541|+42046|+93502|+761.46|970530| |

uvhd display 1st 2 records output


 uvhd d5fix/citytax1 r128
 ========================
 rec#=1 rsize=128 fptr=0 fsize=1280 rcount=10
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             3333333322444424445522222222222222233332445442544422222222225444
             1013014000AF8E085E29000F00000000000181502F75E02F14000000000061E3
          64 OUVER           BCV9S1H1..C...W.........qq.000149061970530
             455452222222222244535343004D005880008000770333333333333333222222
             F5652000000000002369318101320072C0047C0111C000149061970530000000
 rec#=2 rsize=128 fptr=128 fsize=1280 rcount=10
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 10139923  GEORGE BROWN             1250 EAST PENDER ST.     VANC
             3333333322444544245454222222222222233332445525444452552222225444
             101399230075F275022F7E0000000000000125005134005E452034E0000061E3
          64 OUVER           BCV5L1W1..C...W.........xRL00014626q980601
             455452222222222244534353004D005880008800754333333337333333222222
             F5652000000000002365C17101320072C0047C0182C000146261980601000000

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_5 Special Situations Requiring Manual Changes

Part5 Special Situations Requiring Manual Changes


5A1. Multi Record Type test/demo datafile & copybook
cpys/citytax2.cpy - copybook with redefined records
d1ebc/citytax2 - uvhd hexdump of test file with multi record types

5B1. Generating uvcopy jobs to convert Multi Record Type files
- inserting manual code to test the record type & skip to the proper
  set of conversion instructions which are automatically generated
  from the copybook with redefined records.

5C1. Converting Multi Record Type files & Splitting to Separate files
- a minor change to the manual coding (above), to write separate files

5D1. Converting Separated Record Types to Pipe Delimited for loading DB tables
- 1st modify the copybook to change any occurs to separate fields
  since SQL DataBases do not allow occurs.

5E1. Generating SQL Loader script to load Relational databases
- for each of the record types split from a Multi Record Type file

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5A1. DATAcnv1 - Converting Multiple Record Type Files

citytax2 - copybook with redefined records

 cobmap1  start-end bytes for cobol record fields    200403021116  pg# 0001
 cpys/citytax2.cpy             citytax2           RCSZ=00128  bgn  end  lth typ
 * citytax2 copybook for redefined multi record type file
 * tax record (type 'T')
  01  citytax2.
      10 folio                 pic  x(00008).                 0000 0007  008
      10 rectype               pic  x.                        0008 0008  001
      10 filler001             pic  x.                        0009 0009  001
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-4.          0088 0091  004bns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  x(00006).                 0116 0121  006
      10 filler002             pic  x(00006).                 0122 0127  006
 *
 * monthly payment record (type 'P')
 *RCSZ=00128                                                       0128
  01 mthpayrec redefines citytax2.
      10 folio                 pic  x(00008).                 0000 0007  008
      10 rectype               pic  x.                        0008 0008  001
      10 filler003             pic  x(7).                     0009 0015  007
      10 tax-year              pic  9(4).                     0016 0019  004 n  4
      10 mthpayments           pic  s9(9) comp-3 occurs 12.   0020 0024  005pns 9
      10 filler004             pic  x(48).                    0080 0127  048
 *
 * file header record (type 'H')
 *RCSZ=00128                                                       0128
  01 headerrec redefines citytax2.
      10 folio                 pic  x(00008).                 0000 0007  008
      10 rectype               pic  x.                        0008 0008  001
      10 filler005             pic  x.                        0009 0009  001
      10 year-end              pic  x(30).                    0010 0039  030
      10 filler006             pic  x(88).                    0040 0127  088
 ************** end of citytax2 copybook ********************
 *RCSZ=00128                                                       0128
Note
  • this is the 'cobmap', which is much more informative than the 'copybook'.
  • the cobmap is generated from the copybook by the following command.

 uvcopy cobmap1,fili1=cpys/citytax2.cpy,filo1=maps/citytax2
 ==========================================================

 cobmapA citytax2.cpy   <-- script to simplify running cobmap1 (above)
 ====================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5A2. DATAcnv1 - Converting Multiple Record Type Files

test data file to demo Multi-Record-Type files


 uvhd d1ebc/citytax2 r128a    <-- 'a' option to translate char lines to ASCII
 =========================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0         H December 31, 1998
             44444444C4C88898894FF64FFFF4444444444444444444444444444444444444
             000000008045354259031B019980000000000000000000000000000000000000
          64
             4444444444444444444444444444444444444444444444444444444444444444
             0000000000000000000000000000000000000000000000000000000000000000
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 10130140T JOHN HENRY               1815 BOWEN ROAD          VANC
             FFFFFFFFE4DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
             1013014030168508559800000000000000018150266550961400000000005153
          64 OUVER           BCV9S1H1.bi...Wb....g...qq.00014906A970530191809
             DEECD44444444444CCEFEFCF0888005880008000770FFFFFFFFCFFFFFFFFFFFF
             645590000000000023592181029C0072C0047C0111C000149061970530191809
                      10        20        30        40        50        60
 r#        3 0123456789012345678901234567890123456789012345678901234567890123
         256 10130140P       1999...A....A....A....A....A....A....A....A....A
             FFFFFFFFD4444444FFFF00042000420004200042000420004200042000420004
             101301407000000019990001C0001C0001C0001C0001C0001C0001C0001C0001
          64 ....A....A....A.
             2000420004200042444444444444444444444444444444444444444444444444
             C0001C0001C0001C000000000000000000000000000000000000000000000000

Please relate the 3 record types here (H,T,P in byte 8 above) to the redefined records in the copybook listed on the previous page.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5B1. DATAcnv1 - Converting Multiple Record Type Files

Generate uvcopy job to convert MRT files


 #1a. uvcopy uvdata51,fili1=maps/citytax2,filo1=pfx1/citytax2  <-- uvcopy command
      ======================================================

 #1b. gencnvA citytax2.cpy     <-- script equivalent of above (shorter command)
      ====================       - ksh for unix/linux or .bat file for Win/Dos

pfx1/citytax2 - uvcopy job for multi record types

 # uvcopy job to translate EBCDIC to ASCII, preserve packed, fix zoned signs
 opr='jobname=JOBNAME   - pfx2 name=datafilename'
 opr='copybook=citytax2 - pfx1 name=copybookname'
 uop=q0,was=a33000b33000
 fili1=?d1ebc/citytax2,rcs=00128,typ=RSF
 filo1=?d2asc/citytax2,rcs=00128,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
        mvc    b88(4),a88              bns post-date
        mvc    b92(15),a92             pns land-value:face-value
        trt    b107(9),$trtsea          ns maint-tax
        skp    put1
 #      ---                                              redef, R/T test?
 typ__  mvc    b20(60),a20             pns mthpayments
        skp    put1
 #      ---                                              redef, R/T test?
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

uvcopy jobs generated from copybooks with redefined records require manual coding to test the record type & skip to the proper set of conversion instructions.

Please see this job re-listed on the next page, AFTER record type test coding has been inserted --->

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5B2. DATAcnv1 - Converting Multiple Record Type Files

Inserting code to test Record-Types

Before modifying, copy the auto generated job to an alternate subdir, for protection in case the generation job gets rerun in future.


 cp pfx1/citytax2 pfx3    <-- copy auto-generated job to alternate subdir
 =====================

 vi pfx3/citytax2         <-- modify the job as shown below:
 ================

uvcopy job AFTER manual changes

 opr='JOBNAME citytax2 - genby: cobmap1,uvdata51,uvdata52'
 uop=q0
 was=a33000b33000
 fili1=?d1ebc/citytax2,rcs=00128,typ=RSF
 filo1=?d2asc/citytax2,rcs=00128,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #------------------------------------------------------------------------
 # test Record Type & skip to appropriate instrns to preserve packed & signed
        cmc    b8(1),'H'           Header rec (no packed or signed)
        skp=   put1
        cmc    b8(1),'T'           Tax rec ?
        skp=   tax1
        cmc    b8(1),'P'           Payment rec ?
        skp=   pay1
        msg    b0(80)              show record with invalid type
        msgw   'Error Rec Type invalid - enter to bypass'
        skp    loop
 #-------------------------------------------------------------------------
 tax1   mvc    b88(4),a88              bns post-date
        mvc    b92(15),a92             pns land-value:face-value
        trt    b107(9),$trtsea          ns maint-tax
        skp    put1                 skip to output
 #
 pay1   mvc    b20(60),a20             pns mthpayments
 #      ---
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

Please compare this to the previous page (BEFORE changes for multi R/T's).


See 'uvcopy3.doc' for the complete uvcopy instruction set available.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5B3. DATAcnv1 - Converting Multiple Record Type Files

execute uvcopy job to convert Multi-Record-Type file


 #2a. uvcopy pfx3/citytax2,fili1=d1ebc/citytax2,filo1=d2asc/citytax2
      ==============================================================

 #2b. uvcopy pfx3/citytax2   <-- easier (filenames default as shown above)
      ====================

sample output data file (ASCII)


 uvhd d2asc/citytax2 r128
 ========================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0         H December 31, 1998
             2222222242466666672332233332222222222222222222222222222222222222
             00000000804535D252031C019980000000000000000000000000000000000000
          64
             2222222222222222222222222222222222222222222222222222222222222222
             0000000000000000000000000000000000000000000000000000000000000000
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 10130140T JOHN HENRY               1815 BOWEN ROAD          VANC
             3333333352444424445522222222222222233332445442544422222222225444
             1013014040AF8E085E29000000000000000181502F75E02F14000000000061E3
          64 OUVER           BCV9S1H1......W.........qq.000149061970530191809
             4554522222222222445353430888005880008000770333333333333333333333
             F56520000000000023693181029C0072C0047C0111C000149061970530191809
                      10        20        30        40        50        60
 r#        3 0123456789012345678901234567890123456789012345678901234567890123
         256 10130140P       1999...A,...A,...A,...A,...A,...A,...A,...A,...A
             3333333352222222333300042000420004200042000420004200042000420004
             101301400000000019990001C0001C0001C0001C0001C0001C0001C0001C0001
          64 ,...A,...A,...A,
             2000420004200042222222222222222222222222222222222222222222222222
             C0001C0001C0001C000000000000000000000000000000000000000000000000

Please relate these 3 ASCII data records to the original EBCDIC records shown on page '5A2'. Note that option 'a' was used on page '5A2' to translate the character line to ASCII, but not used here since this is ASCII. You should also see that the zones & digits are EBCDIC on page '5A2', but are ASCII here. For example EBCDIC blanks are x'40', but ASCII blanks are x'20'.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5C1. DATAcnv1 - Converting Multiple Record Type Files

Converting MRT files & Splitting to Separate files

Splitting different record types to separate files is a minor change to the uvcopy conversion job listed on page '5B2', since that already includes the code to test the record types. First we will copy & rename in order not to destroy our job that converts record types into 1 output file. We will append an 's' to the original jobname to signify 'split'.


 cp pfx3/citytax2 pfx3/citytax2s   <-- copy/rename previous job
 ===============================
 vi pfx3/citytax2s                 <-- modify the job as shown below:
 =================

uvcopy job AFTER changes to write separate files

 opr='citytax2s - genby: cobmap1,uvdata51 convert EBCDIC to ASCII'
 opr=' - preserving packed fields & correcting signs of zoned fields'
 opr=' - record type tests added for redefined records'
 opr=' - alternate version to split different rec types to different files'
 uop=q0
 was=a33000b33000
 fili1=?d1ebc/citytax2,rcs=00128,typ=RSF    #input EBCDIC 3 rec types
 filo1=?d2asc/citytax2a,rcs=00128,typ=RSF   #output #1 tax recs
 filo2=?d2asc/citytax2b,rcs=00128,typ=RSF   #output #2 payment recs       <--
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #------------------------------------------------------------------------
 # test Record Type & skip to appropriate instrns to preserve packed&signed
        cmc    b8(1),'H'           Header rec (no packed or signed)
        skp=   loop                drop Header rec                          <--
        cmc    b8(1),'T'           Tax rec ?
        skp=   tax1
        cmc    b8(1),'P'           Payment rec ?
        skp=   pay1
        msg    b0(80)              show record with invalid type
        msgw   'Error Rec Type invalid - enter to bypass'
        skp    loop
 #-----------------------------------------------------------------
 tax1   mvc    b88(4),a88              bns post-date
        mvc    b92(15),a92             pns land-value:face-value
        trt    b107(9),$trtsea          ns maint-tax
        put    filo1,b0                write tax rec to file#1
        skp    loop                    return to get next
 pay1   mvc    b20(60),a20             pns mthpayments
        put    filo2,b0                write payment rec to file#2          <--
        skp    loop                    return to get next
 eof    cls    all
        eoj

Please compare this to page '5B2' (before changes to write separate files). We have marked the few changes required with '<--' on the right side above.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5C2. DATAcnv1 - Converting Multiple Record Type Files

Executing file split job


 uvcopy pfx3/citytax2s   <-- convert & split MRTs to separate files
 =====================

Tax records separated to d2asc/citytax2a


 uvhd d2asc/citytax2a r128    <-- display Tax records (1st 2 of 10)
 =========================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140T JOHN HENRY               1815 BOWEN ROAD          VANC
             3333333352444424445522222222222222233332445442544422222222225444
             1013014040AF8E085E29000000000000000181502F75E02F14000000000061E3
          64 OUVER           BCV9S1H1......W.........qq.000149061970530191809
             4554522222222222445353430888005880008000770333333333333333333333
             F56520000000000023693181029C0072C0047C0111C000149061970530191809
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 10139923T GEORGE BROWN             1250 EAST PENDER STREET  VANC
             3333333352444544245454222222222222233332445525444452555445225444
             101399234075F275022F7E0000000000000125005134005E45203425540061E3
          64 OUVER           BCV5L1W1......W.........xRL00014626q980601190156
             4554522222222222445343530888005880008800754333333337333333333333
             F5652000000000002365C171029C0072C0047C0182C000146261980601190156

Payment records separated to d2asc/citytax2a


 uvhd d2asc/citytax2b r128    <-- display Payment records (1st 2 of 10)
 =========================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140P       1999...A,...A,...A,...A,...A,...A,...A,...A,...A
             3333333352222222333300042000420004200042000420004200042000420004
             101301400000000019990001C0001C0001C0001C0001C0001C0001C0001C0001
          64 ,...A,...A,...A,
             2000420004200042222222222222222222222222222222222222222222222222
             C0001C0001C0001C000000000000000000000000000000000000000000000000
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 10139923P       1999...B....B....B....B....B....B....B....B....B
             3333333352222222333300048000480004800048000480004800048000480004
             101399230000000019990002C0002C0002C0002C0002C0002C0002C0002C0002
          64 ....B....B....B.
             8000480004800048222222222222222222222222222222222222222222222222
             C0002C0002C0002C000000000000000000000000000000000000000000000000

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5D1. DATAcnv1 - Converting Multiple Record Type Files

Converting Separated Record-Types to Pipe Delimited for DB tables

In the section above, we have split the data records to separate files, and we will now split the copybook redefined records to separate copybooks, so we can use them to process the separate data files.


 vi cpys/citytax2.cpy     <-- split redefined records to separate copybooks
 ====================

 #1. write Tax record as cpys/citytax2a
 #2. write Payment record as cpys/citytax2b
 #3. ignore the Header/Date record (not required)

We will not illustrate the Tax record cpys/citytax2a, since it is the same as cpys/citytax1.cpy, already illustrated previously.

We will illustrate the Payment record, which requires special consideration since it contains recurring data defined by OCCURS (pic s9(9) occurs 12). SQL databases do not allow for 'occurs', so we must define these fields individually. The easiest way to do this is to modify the copybook & then generate the pipe delimit job.

maps/citytax2b AFTER replacing OCCURS with separate fields

 cobmap1  start-end bytes for cobol record fields    200306090839  pg# 0001
 cpys/citytax2b                 mthpayrec          RCSZ=00128  bgn  end  lth typ
 * citytax2b copybook
 * monthly payment record (type 'P')
 * - isolated out of citytax2 (multi rec type)
 *   to convert mthpay recs to pipe delimited
 * - occurs 12 manually split to 12 separate fields
 *   since occurs not allowed in sql data bases
  01 mthpayrec.
      10 folio                 pic  x(00008).                 0000 0007  008
      10 rectype               pic  x.                        0008 0008  001
      10 filler001             pic  x(7).                     0009 0015  007
      10 tax-year              pic  9(4).                     0016 0019  004 n  4
 *    10 mthpayments           pic  s9(9) comp-3 occurs 12.
 * original split to sep fields sql does not allow occurs
      10 mthpayments-01        pic  s9(9) comp-3.             0020 0024  005pns 9
      10 mthpayments-02        pic  s9(9) comp-3.             0025 0029  005pns 9
      10 mthpayments-03        pic  s9(9) comp-3.             0030 0034  005pns 9
      10 mthpayments-04        pic  s9(9) comp-3.             0035 0039  005pns 9
      10 mthpayments-05        pic  s9(9) comp-3.             0040 0044  005pns 9
      10 mthpayments-06        pic  s9(9) comp-3.             0045 0049  005pns 9
      10 mthpayments-07        pic  s9(9) comp-3.             0050 0054  005pns 9
      10 mthpayments-08        pic  s9(9) comp-3.             0055 0059  005pns 9
      10 mthpayments-09        pic  s9(9) comp-3.             0060 0064  005pns 9
      10 mthpayments-10        pic  s9(9) comp-3.             0065 0069  005pns 9
      10 mthpayments-11        pic  s9(9) comp-3.             0070 0074  005pns 9
      10 mthpayments-12        pic  s9(9) comp-3.             0075 0079  005pns 9
      10 filler002             pic  x(48).                    0080 0127  048
 ************** end of citytax2b copybook *******************
 *RCSZ=00128                                                       0128

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5D2. DATAcnv1 - Converting Multiple Record Type Files

Generate Pipe Delimit job for Payment file


 #1a. uvcopy genpipe1,fili1=maps/citytax2b,filo1=pfp1/citytax2b,uop=a0 <-hard way
      ================================================================

 #1b. genpipeA citytax2b   <-- easy script equivalent of above (shorter command)
      ==================     - no-ext(ksh) for unix/linux or .bat for Win/Dos

pfp1/citytax2b - job to pipe delimit payment file

 opr='JOBNAME citytax2b - generated by cobmap1,genpipe1,uvdata52'
 # - uvcopy job to convert & fix data fields 100 bytes apart
 #   for compress & '|' delimit (see 'var' instrn below)
 rop=j200000r1  #Run OPtions: increase instrn storage & prompt display output
 was=a33000b33000c999000d33000e33000
 fili1=?d2asc/citytax2b,rcs=00128,typ=RSFm10000
 filo1=?d4pipe/citytax2b,rcs=9000,typ=LSTtd3  #<-- DOS option d3 for CR/LF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
 # area a input, see 'get' instrn
 # area b translated to ASCII, in case mainframe EBCDIC file (with packed?)
 # area c data fields fixed 100 bytes apart in prep for var | delimit instrn
 # area d output, see 'put' instrn at end
        mvc    b0(00128),a0       move input area a to area b
 ###    tra    b0(00128)          uncomment for EBCDIC input (or option a1)
 #      ---                   <-- & insert instrns from uvdata51 if packed fields
 #      ---                   <-- insert R/T tests if redef records
        mvc    c0(8),b0(8)                          #1 folio
        mvc    c100(1),b8(1)                        #2 rectype
        mvc    c200(7),b9(7)                        #3 filler001
        mvc    c300(4),b16(4)                       #4 tax-year
        edta1  c400(11),b20(5p),'+zzzzzzzzz9'       #5 mthpayments-01
        edta1  c500(11),b25(5p),'+zzzzzzzzz9'       #6 mthpayments-02
        edta1  c600(11),b30(5p),'+zzzzzzzzz9'       #7 mthpayments-03
        edta1  c700(11),b35(5p),'+zzzzzzzzz9'       #8 mthpayments-04
        edta1  c800(11),b40(5p),'+zzzzzzzzz9'       #9 mthpayments-05
        edta1  c900(11),b45(5p),'+zzzzzzzzz9'       #10 mthpayments-06
        edta1  c1000(11),b50(5p),'+zzzzzzzzz9'      #11 mthpayments-07
        edta1  c1100(11),b55(5p),'+zzzzzzzzz9'      #12 mthpayments-08
        edta1  c1200(11),b60(5p),'+zzzzzzzzz9'      #13 mthpayments-09
        edta1  c1300(11),b65(5p),'+zzzzzzzzz9'      #14 mthpayments-10
        edta1  c1400(11),b70(5p),'+zzzzzzzzz9'      #15 mthpayments-11
        edta1  c1500(11),b75(5p),'+zzzzzzzzz9'      #16 mthpayments-12
        mvc    c1600(48),b80(48)                    #17 filler002
        var    d0(32000),c0(100),0017,'|'
        trt    d0(32000),$trtchr
        clr    c0(001700),' '
 put1   put    filo1,d0
        skp    loop
 eof    cls    all
        eoj
 @pf2=genpipe1.sub

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5D3. DATAcnv1 - Converting Multiple Record Type Files

execute generated job to create '|' pipe delimited data file


 #2a. uvcopy pfp1/citytax2b,fili1=d2asc/citytax2b,filo1=d4pipe/citytax2b
      ==================================================================
 #2b. uvcopy pfp1/citytax2b   <-- easier (filenames default as shown above)
      =====================

sample output - pipe delimited text


 #3a. more d4pipe/citytax2b       <-- display output
      =====================

 #3a. uvlp12 d4pipe/citytax2b     <-- list the output file
      =======================         (uvlpr12 for Windows/DOS batch script)
 10130140|P||1999|+412|+412|+412|+412|+412|+412|+412|+412|+412|+412|+412|+412||
 10139923|P||1999|+428|+428|+428|+428|+428|+428|+428|+428|+428|+428|+428|+428||
 10147615|P||1999|+124|+124|+124|+124|+124|+124|+124|+124|+124|+124|+124|+124||
 10149304|P||1999|+500|+500|+500|+500|+500|+500|+500|+500|+500|+500|+500|+500||
 10154688|P||1999|+300|+300|+300|+300|+300|+300|+300|+300|+300|+300|+300|+300||
 10173320|P||1999|+405|+405|+405|+405|+405|+405|+405|+405|+405|+405|+405|+405||
 10204700|P||1999|+104|+104|+104|+104|+104|+104|+104|+104|+104|+104|+104|+104||
 10223240|P||1999|+23|+23|+23|+23|+23|+23|+23|+23|+23|+23|+23|+23||
 10308685|P||1999|+301|+301|+301|+301|+301|+301|+301|+301|+301|+301|+301|+301||
 10318833|P||1999|+224|+224|+224|+224|+224|+224|+224|+224|+224|+224|+224|+224||

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5D4. DATAcnv1 - Converting Multiple Record Type Files

list pipe delimited data with copybook field names


 #1a. uvcopy listpipe1,fili1=d4pipe/citytax2b,fili2=maps/citytax2b,filo1=tmp/citytax2b
      ================================================================================

 #1b. listpipeA citytax2b    <-- script equivalent of above (shorter command)
      ===================      - ksh for unix/linux or .bat file for Win/Dos

sample listpipe1 report for Payments

 listpipe1 - list '|' delimited files with copybook fieldnames
           - to verify files created by genpipe1 (see DATAcnv1.htm#Part_4)
 datafile=d4pipe/filename?  copybookmap=maps/filename?  datetime=2003/06/09_13:05:26
 SEQ# FIELDNAME              COBOL PICTURE      DATA

datafile = d4pipe/citytax2b, record# = 1, field count = 17

 10130140|P||1999|+412|+412|+412|+412|+412|+412|+412|+412|+412|+412|+412|+412||
 001 folio                   x(00008).          10130140
 002 rectype                 x.                 P
 003 filler001               x(7).
 004 tax-year                9(4).              1999
 005 mthpayments-01          s9(9) comp-3.      +412
 006 mthpayments-02          s9(9) comp-3.      +412
 007 mthpayments-03          s9(9) comp-3.      +412
 008 mthpayments-04          s9(9) comp-3.      +412
 009 mthpayments-05          s9(9) comp-3.      +412
 010 mthpayments-06          s9(9) comp-3.      +412
 011 mthpayments-07          s9(9) comp-3.      +412
 012 mthpayments-08          s9(9) comp-3.      +412
 013 mthpayments-09          s9(9) comp-3.      +412
 014 mthpayments-10          s9(9) comp-3.      +412
 015 mthpayments-11          s9(9) comp-3.      +412
 016 mthpayments-12          s9(9) comp-3.      +412
 017 filler002               x(48).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5E1. DATAcnv1 - Converting Multiple Record Type Files

generate SQL scripts to create & load table


 #1. uvcopy cobmap1,fili1=cpys/citytax2b.cpy,filo1=maps/citytax2b
     ========================================================
     - convert copybook to 'cobmap' (record layout)

 #2. uvcopy sqlcreate1,fili1=maps/citytax2b,filo1=sqlTC1/create_citytax2b.sql
     ========================================================================
                                          ,filo2=sqlTL1/load_citytax2b.ctl
     =====================================================================
     - convert cobmap to SQL scripts to create & load table
      - - - OR use script 'sqlcreateA' (combines above into 1 short command)

 #12. sqlcreateA citytax2b    <-- script equivalent of above (shorter command)
      ====================      - ksh for unix/linux or .bat file for Win/Dos

 #3a. more sqlTC1/create_citytax2b.sql   <-- display batch-file output
      ================================

 #3b. more sqlTL1/load_citytax2b.ctl     <-- display SQL script output
      ==============================

maps/citytax2b AFTER replacing OCCURS with separate fields

 cobmap1  start-end bytes for cobol record fields    200306090839  pg# 0001
 cpys/citytax2b                 mthpayrec          RCSZ=00128  bgn  end  lth typ
 * citytax2b copybook monthly payment record (type 'P')
 * - occurs 12 manually split to 12 separate fields
  01 mthpayrec.
      10 folio                 pic  x(00008).                 0000 0007  008
      10 rectype               pic  x.                        0008 0008  001
      10 filler001             pic  x(7).                     0009 0015  007
      10 tax-year              pic  9(4).                     0016 0019  004 n  4
 *    10 mthpayments           pic  s9(9) comp-3 occurs 12.
 * original split to sep fields sql does not allow occurs
      10 mthpayments-01        pic  s9(9) comp-3.             0020 0024  005pns 9
      10 mthpayments-02        pic  s9(9) comp-3.             0025 0029  005pns 9
      10 mthpayments-03        pic  s9(9) comp-3.             0030 0034  005pns 9
      10 mthpayments-04        pic  s9(9) comp-3.             0035 0039  005pns 9
      10 mthpayments-05        pic  s9(9) comp-3.             0040 0044  005pns 9
      10 mthpayments-06        pic  s9(9) comp-3.             0045 0049  005pns 9
      10 mthpayments-07        pic  s9(9) comp-3.             0050 0054  005pns 9
      10 mthpayments-08        pic  s9(9) comp-3.             0055 0059  005pns 9
      10 mthpayments-09        pic  s9(9) comp-3.             0060 0064  005pns 9
      10 mthpayments-10        pic  s9(9) comp-3.             0065 0069  005pns 9
      10 mthpayments-11        pic  s9(9) comp-3.             0070 0074  005pns 9
      10 mthpayments-12        pic  s9(9) comp-3.             0075 0079  005pns 9
      10 filler002             pic  x(48).                    0080 0127  048
 ************** end of citytax2b copybook *******************
 *RCSZ=00128                                                       0128

See the output files listed on the next page --->

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5E2. Oracle Database SQL test/demo

Oracle 'batch-file' to create table

 /* create_citytax2b.sql - SQL batch file to create a table */
 /*                     - generated from COBOL copybook 20090220:094034 */
 /* see www.uvsoftware.ca/datacnv1.htm#Part_4 & sqldemo.htm#Part_4      */
 /*                                                                     */
 /* sqlplus user/pass @sqlTC1/create_citytax2b.sql                      */
 /* =============================================                       */
 DROP TABLE citytax2b;
 CREATE TABLE citytax2b (
 folio                         char     (0008) , --#001 x(00008).
 rectype                       char     (0001) , --#002 x.
 tax_year                      number   (04)   , --#003 9(4).
 mthpayments_01                number   (09)   , --#004 s9(9) comp-3.
 mthpayments_02                number   (09)   , --#005 s9(9) comp-3.
 mthpayments_03                number   (09)   , --#006 s9(9) comp-3.
 mthpayments_04                number   (09)   , --#007 s9(9) comp-3.
 mthpayments_05                number   (09)   , --#008 s9(9) comp-3.
 mthpayments_06                number   (09)   , --#009 s9(9) comp-3.
 mthpayments_07                number   (09)   , --#010 s9(9) comp-3.
 mthpayments_08                number   (09)   , --#011 s9(9) comp-3.
 mthpayments_09                number   (09)   , --#012 s9(9) comp-3.
 mthpayments_10                number   (09)   , --#013 s9(9) comp-3.
 mthpayments_11                number   (09)   , --#014 s9(9) comp-3.
 mthpayments_12                number   (09)     --#015 s9(9) comp-3.
 );
 exit;

Oracle SQL script to LOAD table

 -- load_citytax2b.ctl - SQL*LOADER script to load Oracle table
 --                   - generated from COBOL copybook 20081221:080752
 -- see www.uvsoftware.ca/datacnv1.htm#Part_4 & sqldemo.htm#Part_4
 --
 -- sqlldr user/pass control=sqlTL1/load_citytax2b.ctl
 -- ==================================================
 load data infile 'd4pipe/citytax2b' into table citytax2b
 fields terminated by '|' optionally enclosed by '"'
 (
 folio                         , --char     (0008) , --#001 x(00008).
 rectype                       , --char     (0001) , --#002 x.
 tax_year                      , --number   (04)   , --#003 9(4).
 mthpayments_01                , --number   (09)   , --#004 s9(9) comp-3.
 mthpayments_02                , --number   (09)   , --#005 s9(9) comp-3.
 mthpayments_03                , --number   (09)   , --#006 s9(9) comp-3.
 mthpayments_04                , --number   (09)   , --#007 s9(9) comp-3.
 mthpayments_05                , --number   (09)   , --#008 s9(9) comp-3.
 mthpayments_06                , --number   (09)   , --#009 s9(9) comp-3.
 mthpayments_07                , --number   (09)   , --#010 s9(9) comp-3.
 mthpayments_08                , --number   (09)   , --#011 s9(9) comp-3.
 mthpayments_09                , --number   (09)   , --#012 s9(9) comp-3.
 mthpayments_10                , --number   (09)   , --#013 s9(9) comp-3.
 mthpayments_11                , --number   (09)   , --#014 s9(9) comp-3.
 mthpayments_12                  --number   (09)     --#015 s9(9) comp-3.
 )

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5E3. Oracle Database SQL test/demo

create & load table using SQL scripts


 #1. Login demo1 --> /home/demo1
     ===========

 #2. sqlplus user/pass @sqlTC1/create_citytax2b.sql
     ==============================================
     - run SQL 'batch-file' to create the table

 #3. sqlldr user/pass control=sqlTL1/load_citytax2b.ctl
     ==================================================
     - run SQL*LOADER script to load the table

use SQL*Plus to verify table loaded OK


 #4. sqlplus demo1/demo1pw       <-- start SQL*Plus
     =====================

 #5. SQL> select * from citytax2b <-- SELECT all rows/fields from table
          ======================   - headings & 1st 2 rows shown below:

>>>>> to be supplied

batch-file to list table rows

 /* select_all_citytax2b.sql - batch file to select all rows from citytax2b  */
 /*                      - batch files useful for complex repeated selects */
 select * from citytax2b;
 exit;

You could use the batch-file listed above as an alternative to logging in to SQL*PLUS to list the table rows:


 #6. SQL> exit      <-- exit from SQL*PLUS (back to Unix/Linux prompt)

 #7. sqlplus user/pass @sqls/select_all_citytax2b.sql
     ================================================
     - run SQL 'batch-file' to create the table

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_6 DATAcnv1.doc - Converting EBCDIC to ASCII - Contents

Conversions for real data-file-names vs copy-book-names

Note
  • see DATAcnv2.htm for more comprehenive conversion of ALL data-files
    in a directory & using a control file to relate data-file-names
    to copybook-names.

6A1. Conversions for real data-file-names (vs copy-book-names as in Part3)
 Converting ALL files in the Directory (vs 1 at a time in previous parts).
- Introduction & overview

6B1. Generating & Executing Data File Conversions - Preparation
- VU installed, profile setup, conversion directories setup
 May demo in homedir, cnvdata1 superdir with 15 subdirs
 (cpys,ctl,d1ebc,d2asc,d4pipe,maps,pfx2,pfx2,pfx3,pfp1,tmp)
6B2.  Test/Demo files supplied
6B3.  Directories required for data file covnersions
6B4.  FTP data from Mainframe to Unix/Linux
6B5. Re-Naming files as required

6C1. Illustrated Operating Instructions (vs concise at 6H1+).
 Demo Conversion using supplied test files
- copy demo datafiles & copybooks to d1ebc/ & cpys/
- rename mainframe GDG files to VU convention & lower case

6C2. Generate control file from datafilenames in d1ebc/
- inserting copybooknames method#1 - manual edit

6C3. Inserting copybooknames method#2
- automatic if copybooknames embedded in datafilenames

6C4. Using control file - to generate data conversion jobs
- script gencnvC (runs cobmap1,uvdata51,& uvdata52)

6C5. Sample outputs for demo file gl.account.gltran
maps/gltran - cobmap (record layout) generated from copybook
pfx1/gltran - conversion job generated from cobmap
pfx2/gltran - conversion job with datafilenames (vs copybooknames)

6C6. Inspect & Edit generated jobs
- editing usually required only for files with multi-record-types

6C7. Execute jobs to convert datafiles EBCDIC to ASCII (d2asc --> d3ebc)
- execute all jobs to convert all datafiles in directory
- with 1 command 'uvcoypfxgA' (console log shown)
- repeats conversion jobs for GDG files for all members in the set

6C8. Investigate converted data files
- with uvhd &/or uvhdcob if any packed/binary fields

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Conversions for real data-file-names vs copy-book-names


6D1. Modifying control-file rcs=... typ=... Appending rcso=... typo=...
Might use to change output files to typ=LST (with LineFeeds) to
allow use of unix commands (vi,more,lp,etc), but also need to
change COBOL programs to ORGANIZATION LINE SEQUENTIAL.

6E1. Adding new files to be converted 1 at a time (vs all in directory).

6F1. Generating jobs to convert fixed layout datafiles to 'Pipe|Delmited'
- gencnvC2, script to generate pipe|delimit jobs

6F2. Sample pipe|delimit job pfp2/gl.account.gltran_

6F3. Execute pipe|delimit jobs for all files in directory

6G1. Copy converted data files from $CNVDATA to $RUNDATA for testing

6G2. Execute a few VU test/demo JCL/scripts to prove datafile conversions OK

6H1. Concise Operating Instructions
Note
  • We recommend you 1st follow the "Illustrated Operating Instructions"
    on pages 6B1 - 6C8 (using the Vancouver Utility demo files supplied)
    to get familiar with these data conversion procedures.
  • Then for your own data file conversions, you can follow the
    "Concise Opperating Instructions" on pages '6H1' - '6H5'

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6A1. DATAcnv1.doc - Converting ALL files in Directory

Conversions for real data-file-names vs copy-book-names

Up to this point, we have been converting 1 file at a time and we have been renaming the datafiles the same as the copybooks to simplify the procedures to generate the data converison jobs from the copybooks.

Using the same name for the copybook, datafile,& all generated jobs is possible because we have different subdirs for different file types (cpys/..., maps/..., pfx1/..., pfp1/... d1ebc/..., d2asc/..., d4pipe/...).

This Part 6 will show you how to perform these conversions for all files in the directory (vs one at a time documented previously), and to insert the actual data filenames in the generated jobs (vs copybooknames).

A key part of this plan is use a control file to relate the data file names to the copybook names. The control file method allows for the distinct possibility of having multiple datafiles using the same copybook.

control file created from directory of datafilenames

 ar.customer.armaster            cpy=____________ rcs=_____ typ=_____
 ar.sales.arsales                cpy=____________ rcs=_____ typ=_____
 gl.account.glmaster_            cpy=____________ rcs=_____ typ=_____ dups=003

control file after inserting copybooknames

 ar.customer.armaster            cpy=armaster.cpy rcs=_____ typ=_____
 ar.sales.arsales                cpy=arsales.cpy  rcs=_____ typ=_____
 gl.account.glmaster_            cpy=glmaster.cpy rcs=_____ typ=_____ dups=003

You will probably have to insert the copybooknames manually with the editor, but after that the conversion jobs are generated automatically from the copybooks & they can all be executed with 1 command. Exceptions would be for files with multiple record types (redefined records in the copybook). In those usually rare instances you need to add code to the generated jobs to select the appopriate bank of auto generated converison isntructions.

copybooknames embedded in datafilenames

Some sites (admittedly rare) embed the copybooknames within the datafilenames and for these sites, we have provided a job to automatically insert the copybooknames in the control file (as illustrated above). This means there would might be no manual editing to connvert the datafiles at these sites.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6B1. Converting ALL files in Directory - Illustrated Op. Instrns.

Generating jobs to use real data-file-names

Here is an illustrated demo of generating & executing all jobs to convert all data files in the test/demo directories supplied by UV Software. We suggest you perform this demo to better understand these procedures, before using them with your own datafiles.

We assume the following has been performed:

  1. Vancouver Utilities installed: https://uvsoftware.ca/install.htm#B1 - B6 OR https://uvsoftware.ca/jclcnv1demo.htm#Part_1

  2. Your login userid setup as per https://uvsoftware.ca/jclcnv1demo.htm#3C1 We will assume your homedir is /home/userxx

  3. Your profile setup as per https://uvsoftware.ca/jclcnv1demo.htm#1C1 But modify definitions of testlibs,testdata1,cnvdata1 as shown below (in case you already have the default names present)


 #3a. Login userxx --> /home/userxx
 #3b. vi .profile  OR .bash_profile
      ===========

export CNVDATA=$HOME/cnvdata1 # Conversion superdir & subdirs export RUNDATA=$HOME/testdata1 # data for testing in homedir export RUNLIBS=$HOME/testlibs1 # COBOL, copybooks,& JCL/scripts


 #4a. exit
 #4b. Login userxx --> /home/userxx    Re-login to make exports effective

 #4c. mkdir cnvdata1 testdata1 testlibs1
      ==================================
  1. Setup conversion subdirs using supplied scripts - see https://uvsoftware.ca/jclcnv2real.htm#1B1


 #5a. cdc --> alias cdc='cd $CNVDATA' --> /home/userxx/cnvdata1

 #5b. cnvdatadirs     <-- setup subdirs for data conversion
      ===========

 #6a. cdd --> /home/userxx/testdata1

 #6b. mvsdatadirs     <-- setup subdirs for JCL/script executions
      ===========

 #6a. cdl --> /home/userxx/testlibs1

 #6b. mvslibsdirs     <-- setup subdirs for copybooks,COBOL,& JCL conversion
      ===========

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6B2. Generating jobs to use real data-file-names

VU test/demo files supplied

 /home/uvadm/mvstest/mvstest1  <-- location in Vancouver Utilities distribution
 /home/mvstest1        <-- OR setup user mvstest1
 /home/userxx              OR setup cnvdata1 within your $HOME dir
 :-----cnvdata1        <-- cnvdata1 superdir for DATA conversion
 :     :-----archive     - backup to reload d1ebc/... copybooknames or datafilenames
 :     :     :-----d1ebc_cpy_names    <-- for Parts3,4,5
 :     :     :-----d1ebc_data_names   <-- for Part6
 :-----cnvdata1
 :     :-----cpys   <-- copybooks <--- copy from $UV/mvstest1/cnvdata1/cpys/*
 :     :-----ctl      - control files
 :     :-----d0ebc  <-- EBCDIC data files <-- copy $UV/mvstest1/cnvdata1/d0ebc/*
 :     :-----d1ebc    - EBCDIC data files renamed to VU conventions
 :     :-----d2asc    - ASCII data files (converted by uvcopy uvdata51)
 :     :-----d3ebc    - convert back to EBCDIC by uvdata31
 :     :-----d4pipe   - pipe delimited to load DB tables (by genpipe1)
 :     :-----maps     - copybooks with record layouts on right side
 :     :-----pfx1     - uvcopy jobs to convert EBCDIC to ASCII (gen by uvdata51)
 :     :-----pfx2     - with data filenames inserted (vs copybook names)
 :     :-----pfx3     - copied here for modify/execute
 :     :-----pfp1     - uvcopy jobs to create pipe delimited (gen by genpipe1)
 :     :-----pfp2     - pipe delim jobs with datafilenames inserted
 :     :-----pfp3     - copy here before editing for multi-record-type files
 :     :-----sqlTC1   - scripts to Create database tables (gen from copybooks)
 :     :-----sqlTL1   - scripts to Load database tables (gen from copybooks)
 :     :-----tmp      - temp files (keep working directory clean)
 :     :
 :-----testdata1
 :     :-----ctl
 :     :-----data      <-- copy d2asc/* here to test with JCL/scripts
 :     :-----jobctl
 :     :-----joblog
 :     :-----jobmsgs
 :     :-----jobtimes
 :     :-----jobtmp
 :     :-----sysout
 :     :
 :-----testlibs1
 :     :-----cbl0,cbl1,cbl2,cbls,cblx
 :     :-----cpy0,cpy1,cpy2,cpys
 :     :-----ctl
 :     :-----jcl0,jcl1,jcl2,jcl3,jcls
 :     :-----parm0,parms
 :     :-----proc0,procs

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6B3. Generating jobs to use real data-file-names

Preparation to Run conversions for real datafilenames

If you have already run the test/demo conversions & you or others wish to re-run, you will want to clear the outputs of prior test/demo conversions, for easier inspection of your new outputs & avoid confusion with prior outputs.


 #1. cnvdatainit   <-- run script to clear any old outputs
     ===========     - d2asc,d4pipe,pfx1,pfx2,pfx3,pfp1,etc
                     - see script listed on page '10K1' or in $UV/sf/IBM/...

 #2. rm -f d1ebc/*  <-- clear d1ebc/... in case retest Part3 after Part6
     =============

 #3. cp archive/d1ebc_data_names/* d1ebc/.  <-- for Part6 real datafilenames
     ====================================
      - refresh d1ebc/... files named same as copybooks (for Parts3,4,5)
     archive
     |-- d1ebc_cpy_names/          <-- copybooknames for Part3,4,5
     |   |-- armaster
     |   |-- arsales
     |   |-- artrans
     |   |-- citytax1
     |   |-- citytax2
     |   |-- glmaster
     |   |-- gltran
     |   |-- vendormas
     |   |-- vendorpaymas
     |   `
     `-- d1ebc_data_names/         <-- datafilenames for Part6 & DATAcnv2.doc
         |-- ar.customer.armaster
         |-- ar.customer.arsales
         |-- ar.customer.artrans
         |-- ar.customer.phone.list
         |-- city.tax.assess
         |-- city.tax.payments
         |-- gl.account.glmaster.g0001v00
         |-- gl.account.glmaster.g0002v00
         |-- gl.account.gltran.g0001v00
         |-- gl.account.gltran.g0002v00
         |-- vendor.master.names
         `-- vendor.master.payments

Generating ALL jobs to convert ALL datafiles EBCDIC to ASCII

 cpys-------->maps---------->pfx1-------------->pfx2----------->pfx3--------->
     cobmap1       uvdata51         uvdata52          copy/edit       execute
             convert copybooks  change copybooknames
               to uvcopy jobs     to datafilenames

Converting ALL data files EBCDIC to ASCII

 Mainframe---->d0ebc---------->d1ebc------------->d2asc------->$RUNDATA/data
           FTP   rename GDG/lower   EBCDIC->ASCII      copy files to test dirs

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6B4. Converting ALL files in Directory - Illustrated Op. Instrns.

FTP data from Mainframe to Unix/Linux

For demos in Part 6, we will use testfiles supplied with Vancouver Utilities, but for your covnersion, you will need to FTP files from the mainframe & convert as illustrated with the demo files in the rest of this Part 6.

If you had no packed/binary fields you could FTP in TEXT MODE which automatically translates to ASCII, but the CR/LF makes records 2 bytes longer, & you either have to remove to match COBOL programs, or modify program SELECTs to 'ORGANIZATION LINE SEQUENTIAL' which allows for the CR/LF & also allows you to use unix system utilities (vi,more,lp,cat,etc) that required LineFeeds.

But most mainframes have a lot of files with packed/binary fields & it is probably best to FTP all files in BINARY MODE & use the VU conversion methods, which generate data conversion jobs automatically from the COBOL copybooks, to preserve the packed/binary fields (same for unix COBOLs as mainframe COBOLs). Note that unpacked signed fields require zoned sign conversion. Mainframe units byte positives are coded '{ABCDEFGHI' & negatives '}JKLMNOPQR'. Unix COBOL positives are coded just '0123456789' & negatives 'pqrstuvwxy'. The VU methods aallow for this & an alternate version could be used with FTP text/ASCII files.

               Variable-> fixed  BCDIC->ASCII          copy/rename
 MAINFRAME -----> d0ebc ------> d1ebc ----------> d2asc ---------> $RUNDATA/data
    FTP/BINARY   ASCII to d2asc  pfx1/uvcopy jobs       cp/renameL

 #1. Login --> your homedir
     =====
 #2. cdc   --> $HOME/$CNVDATA
     ===
 #3. cd d0ebc   <-- change into the data subdir
     ========

 #4. ftp xxx.xxx.xxx.xxx   <-- FTP to mainframe IP#
     ===================
 #4a. userid--> .....      <-- enter userid
 #4b. passwd--> .....      <-- enter password
 #4c. ascii                <-- translate EBCDIC to ASCII & insert CR/LF
      ---OR---                 (ascii is usually the default)
 #4c. binary               <-- ensure binary transfer
 #4d. quote site RDW       <-- required if any variable length files
 #4e. cd ...               <-- change to data files directory ??
 #4f. get 'XXX.XXX.XXX'    <-- get desired data files
 #4g. get '...etc....'       - 'single quotes' may be required

 #5. Might need to change permissions on files FTP'd from mainframe

 #5a. chmod 664 *          <-- files should be 664
      ===========

See '8A2' for 'LOCSITE RDW' option to FTP to unix via JCL on mainframe

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6B5. Converting ALL files in Directory

Re-Naming files as required

On some systems FTP'd files may be enclosed in single quotes, may be UPPER case, may have embedded blanks or '$' signs. ON unix we need to remove single quotes, remove any embedded blanks or '$' signs, & lower case is strongly recommended.


 #5. mv  \'XXX.XXX.XXX\'  xxx.xxx.xxx   <-- the hard way
     ================================     - see #6. the easy way

Imagine how awkward manual renaaming via 'mv' commands would be if you have hundreds of files to be renamed. Vancouver Utilities has many 'rename' scripts to make mass changes to filenames easy.


 #6a. renameQQ data   <-- easy way to remove Quotes from All files in subdir
      =============

 #6b. renameL data    <-- easy way to translate names to Lower case
      ============

 #6c. renameBU data   <-- easy way to convert Blanks to '_' Underscores
      =============

 #6d. renameDU data   <-- easy way to convert '$' signs to '_' Underscores
      =============

VU JCL/script filename standards

Renaming GDG files

After converting GDG files, you must rename them using the VU GDG file naming conventions - appending '_000001' for the 1st generation. For example:


 cp $CNVDATA/d2asc/AR.CUSTOMER.MASTER(0) $RUNDATA/data/ar.customer.master_000001
 ===============================================================================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6B6. Converting ALL files in Directory

FTP from mainframe - ASCII or EBCDIC/binary

If your data files do not have packed/binary fields, you can use FTP ASCII vs binary which requires UV Software DATA conversions driven by copybooks to translate only the character fields, but preserve the packed/binary fields which would be destroyed by EBCDIC to ASCII translation.

But FTP ASCII adds 2 bytes (Carriage-Return & Line-Feed) to each record, which means the record-sizes will not agree with the converted COBOL programs & utilities in JCL/scripts (SORT,IDCAMS,IEBGENER,etc).

You can use the ASCII files with CR/LF if you change the COBOL program SELECTs from 'organization RECORD sequential' to 'organization LINE sequential' and you do not have to change the record size in the COBOL programs.

The Vancouver Utilities (uvsort,uvcp,etc) can also accept the ASCII files if you change 'typ=RSF' to 'typ=LST' which means the the record size is determiend by the LF (LineFeed) & allows the record-size to remain the same as long as there are not more than the 2 extra bytes for CR/LF.

If the record size was any greater the record would be split into 2 records. You could increase the record sizes in the JCL to allow a saftey margin.

Here is the JCL/script for a SORT of 50 byte fixed length records, with the original #commented out & the changes on a following line + a note about the change. It is a good idea to make notes such as these, for other team members (and youself when you come back to it at a later time).


 #3------------------------------------------------------------------------------
 #Sep08/13/OT - change file type & rec-size for FTP ASCII files & LINE sequential
 # uvsort "fili1=$SORTIN,typ=RSF,rcs=50,filo1=$SORTOUT,typ=RSF,rcs=50,keys=(0,5,c,a)"
 uvsort "fili1=$SORTIN,typ=LSTt,rcs=80,filo1=$SORTOUT,typ=LSTt,rcs=80,keys=(0,5,c,a)"
 #4------------------------------------------------------------------------------

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6C1. Converting ALL files in Directory - Illustrated Op. Instrns.

Demo illustrated conversion using supplied test files


 #0a. Login userxx --> /home/userxx

 #0b. cdc --> alias cdc='cd $CNVDATA' --> /home/userxx/cnvdata1
  1. Copy EBCDIC files to be converted to $CNVDATA/d0ebc/... For your conversion you will FTP files from the mainframe. For this demo you can copy the supplied demo files as follows:


 #1a. cp $UV/mvstest1/cnvdata1/d0ebc/* d0ebc
      ======================================
      - copy VU supplied test data files to your homedir/cnvdata1/d0ebc/

sample EBCDIC files in d0ebc/...

      AR.CUSTOMER.ARMASTER
      AR.SALES.ARSALES
      GL.ACCOUNT.GLMASTER.G0001V00   <-- GDG files in mainframe format
      GL.ACCOUNT.GLMASTER.G0002V00
      GL.ACCOUNT.GLMASTER.G0003V00
      GL.ACCOUNT.GLTRAN
      GL.ACCOUNT.GLTRAN.G0001V00
      GL.ACCOUNT.GLTRAN.G0002V00
      GL.ACCOUNT.GLTRAN.G0003V00

 #1b. cp $UV/mvstest1/cnvdata1/cpys/* cpys
      ====================================
      - copy VU supplied copybooks to your homedir/cnvdata1/cpys/

sample copybooks in cpys/...

      -rw-rw-r-- 1 mvstest1 apps  720 Aug 18 13:50 armaster.cpy
      -rw-rw-r-- 1 mvstest1 apps  687 Aug 18 13:53 arsales.cpy
      -rw-rw-r-- 1 mvstest1 apps  791 Aug 18 13:54 citytax1.cpy
      -rw-rw-r-- 1 mvstest1 apps 1795 Aug 18 13:51 citytax2.cpy
      -rw-rw-r-- 1 mvstest1 apps  499 Aug 18 13:52 glmaster.cpy
      -rw-rw-r-- 1 mvstest1 apps  548 Aug 24 11:03 gltran.cpy

 #2. cp d0ebc/* d1ebc/   <-- copy to d1ebc before rename (save original)
     =================

 #3a. renameGDG d1ebc    <-- rename GDG files from mainframe convention to UV
      ===============        .G0001V00 -> _000001, .G0002V00 -> _000002, etc

 #3b. renameL d1ebc      <-- rename to lowercase
      =============

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6C2. Converting ALL files in Directory - Illustrated Op. Instrns.

d1ebc/... after copy & rename

      ar.customer.armaster
      ar.sales.arsales
      gl.account.glmaster_000001    <-- GDG files in VU/GDG format
      gl.account.glmaster_000002
      gl.account.glmaster_000003
      gl.account.gltran             <-- non-gdg file
      gl.account.gltran_000001
      gl.account.gltran_000002      <-- GDG files
      gl.account.gltran_000003

generate control file from datafilenames in d1ebc/


 #4. uvcopy data2cpy51,fild1=d1ebc,filo2=ctl/datacpy51
     =================================================
     - create ctl/datacpy51 from all datafilenames in d1ebc/...
     - reducing GDG filenames to 1 entry per GDG set

 #4a. uvcopy data2cpy51  <-- same as above, filenames default as shown above
      =================
      --> cat <-- reply 'cat' at the prompt for output file viewing

control file output ctl/datacpy51 (from this job)

 ar.customer.armaster            cpy=____________ rcs=_____ typ=_____
 ar.sales.arsales                cpy=____________ rcs=_____ typ=_____
 citytax1                        cpy=____________ rcs=_____ typ=_____
 citytax2                        cpy=____________ rcs=_____ typ=_____
 gl.account.glmaster_            cpy=____________ rcs=_____ typ=_____ dups=003
 gl.account.gltran               cpy=____________ rcs=_____ typ=_____
 gl.account.gltran_              cpy=____________ rcs=_____ typ=_____ dups=003

insert copybooknames method#1 - manual edit


 #5. cp ctl/datacpy51 ctl/datacpy52
     ==============================
     - copy/rename to prevent edited file overwrite if data2cpy51 rerun

 #6. vi ctl/datacpy52      <-- edit/insert copybook names as shown below
     ================

after edit/insert copybooknames

 ar.customer.armaster            cpy=armaster.cpy rcs=_____ typ=_____
 ar.sales.arsales                cpy=arsales.cpy  rcs=_____ typ=_____
 citytax1                        cpy=citytax1.cpy rcs=_____ typ=_____
 citytax2                        cpy=citytax2.cpy rcs=_____ typ=_____
 gl.account.glmaster_            cpy=glmaster.cpy rcs=_____ typ=_____ dups=003
 gl.account.gltran               cpy=gltran.cpy   rcs=_____ typ=_____
 gl.account.gltran_              cpy=gltran.cpy   rcs=_____ typ=_____ dups=003

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6C3. Converting ALL files in Directory - Illustrated Op. Instrns.

insert copybooknames method#2 - data2cpy52 automatic (if)

IF copybooknames are embedded in datafilenames (at some sites), you can use 'data2cpy52' to extract copybooknames from datafilenames into the control-file. (Unique to some sites, but a great convenience).

'data2cpy52' would be run following 'data2cpy51' & it assumes that the copybookname is the last segment in the filename, following the last '.' & ended by a blank or the '_' indicating a GDG file.


 #5. uvcopy data2cpy52,fili1=ctl/datacpy51,filo1=ctl/datacpy52
     =========================================================
     - automatic insert copybooknames from datafilenames

 #5a. uvcopy data2cpy52  <-- same, files default as above
      =================

 #6. vi ctl/datacpy52    <-- inspect ctlfile with copybook names
     ================      - make corrections if auto code not perfect

ctl/datacpy52 - copybooknames extracted from datafilenames

 ar.customer.armaster            cpy=armaster.cpy rcs=_____ typ=_____
 ar.sales.arsales                cpy=arsales.cpy  rcs=_____ typ=_____
 citytax1                        cpy=citytax1.cpy rcs=_____ typ=_____
 citytax2                        cpy=citytax2.cpy rcs=_____ typ=_____
 gl.account.glmaster_            cpy=glmaster.cpy rcs=_____ typ=_____ dups=003
 gl.account.gltran               cpy=gltran.cpy   rcs=_____ typ=_____
 gl.account.gltran_              cpy=gltran.cpy   rcs=_____ typ=_____ dups=003

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6C4. Converting ALL files in Directory - Illustrated Op. Instrns.

using control file - to generate data conversion jobs


 #7. gencnvC ctl/datacpy52   <-- generate data conversion jobs
     =====================
     - script 'gencnvC' runs 3 uvcopy jobs (cobmap1, uvdata51,& uvdata52)
     - see script listed at https://uvsoftware.ca/datacnv1.htm#10E1
       ----- OR could run separately as follows -----

 #7a. uvcopyx cobmap1 cpys maps uop=q0i7
      ==================================
      - convert cobol copybooks to maps (record layouts)

 #7b. uvcopyx uvdata51 maps pfx1 uop=q0i7
      ===================================
      - generates uvcopy data conversion jobs from copybookmaps
      - codes fili1=... & filo1=... & jobnames sames as copybooknames

 #7c. cp $UV/IBM/skeleton2 pfx1
      =========================
      - copy uvcopy job skeleton for datafiles without copybooks
      - OK for files that have no packed/binary fields
      - could modify control file typ=LST (LineFeeds OK if no packed/binary)

ls -l pfx1/... jobnames same as copybooknames

      -rw-rw-r-- 1 mvstest1 apps  743 Aug 18 15:03 armaster
      -rw-rw-r-- 1 mvstest1 apps  725 Aug 18 15:03 arsales
      -rw-rw-r-- 1 mvstest1 apps  811 Aug 18 15:03 citytax1
      -rw-rw-r-- 1 mvstest1 apps 1033 Aug 18 15:03 citytax2
      -rw-rw-r-- 1 mvstest1 apps  680 Aug 18 15:03 glmaster
      -rw-rw-r-- 1 mvstest1 apps  673 Aug 18 15:03 gltran
      -rw-rw-r-- 1 mvstest1 apps  943 Aug 18 15:03 skeleton2

 #7d. uvcopy uvdata52,fili1=ctl/datacpy52,fild2=pfx1,fild3=pfx2,uop=q0i7
      ==================================================================
      - copies pfx1/* to pfx2/... inserting datafilenames (vs copybooknames)
      - also renames the jobs for the datafilenames (vs copybooknames
      - since could have multiple datafiles for same copybook (esp. for GDG)

ls -l pfx2/... jobs renamed same as datafilenames

      -rw-rw-r-- 1 mvstest1 apps  780 Aug 18 15:03 ar.customer.armaster
      -rw-rw-r-- 1 mvstest1 apps  752 Aug 18 15:03 ar.sales.arsales
      -rw-rw-r-- 1 mvstest1 apps  812 Aug 18 15:03 citytax1
      -rw-rw-r-- 1 mvstest1 apps 1034 Aug 18 15:03 citytax2
      -rw-rw-r-- 1 mvstest1 apps  717 Aug 18 15:03 gl.account.glmaster_
      -rw-rw-r-- 1 mvstest1 apps  708 Aug 18 15:03 gl.account.gltran
      -rw-rw-r-- 1 mvstest1 apps  708 Aug 18 15:03 gl.account.gltran_

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6C5. Converting ALL files in Directory - Illustrated Op. Instrns.

maps/gltran - copybook/map to generate conversion jobs

  01  gltran.
      10 glt-acntno           pic  9(00008).           0000 0007  008 n  8
      10 glt-type             pic  x(00002).           0008 0009  002
      10 glt-description      pic  x(00030).           0010 0039  030
      10 glt-amount           pic s9(00010).           0040 0049  010 ns10
      10 glt-status           pic  x(00001).           0050 0050  001
      10 glt-client           pic  x(00004).           0051 0054  004
      10 glt-province         pic  x(00002).           0055 0056  002
      10 glt-trace            pic  x(00012).           0057 0068  012
      10 glt-filler1          pic  x(00011).           0069 0079  011
 *RCSZ=00080

pfx1/gltran - sample job generated from copybook

 # uvcopy job to translate EBCDIC to ASCII, preserve packed, fix zoned signs
 opr='jobname=JOBNAME   - pfx2 name=datafilename'
 opr='copybook=gltran - pfx1 name=copybookname'
 uop=q0,was=a64000b64000
 fili1=?d1ebc/gltran,rcs=00080,typ=RSF
 filo1=?d2asc/gltran,rcs=00080,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00080),a0       move rec to outarea before field prcsng
        tra    b0(00080)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
        trt    b40(10),$trtsea          ns at-amount
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

pfx2/gl.account.gltran_ datafilenames replacing copybooknames

 # uvcopy job to translate EBCDIC to ASCII, preserve packed, fix zoned signs
 opr='jobname=gl.account.gltran_   - pfx2 name=datafilename'
 opr='copybook=gltran - pfx1 name=copybookname'
 uop=q0,was=a64000b64000
 fili1=?d1ebc/gl.account.gltran_,rcs=00080,typ=RSF
 filo1=?d2asc/gl.account.gltran_,rcs=00080,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00080),a0       move rec to outarea before field prcsng
        tra    b0(00080)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
        trt    b40(10),$trtsea          ns at-amount
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6C6. Converting ALL files in Directory - Illustrated Op. Instrns.

edit generated jobs (for multi-record-types)

It may be necessary to make some manual changes to auto-generated jobs. Most common reason is to allow for files with multiple record-types. You need to insert code to test R/T & skip to corresponding bank of instructions already automatically generated from redefined records in copybook. Here is the example from https://uvsoftware.ca/datacnv1.htm#5A1 - 5C2


 #8. cp pfx2/* pfx3  <-- copy auto-generated jobs before any editing
     ==============    - to avoid losing manual changes if generation rerun

 #9. vi pfx3/...       <-- make any manual changes required
     ===========         - as discussed above for multi-record-type files
 # uvcopy job to translate EBCDIC to ASCII, preserve packed, fix zoned signs
 opr='jobname=citytax2   - pfx2 name=datafilename'
 opr='copybook=citytax2.cpy - pfx1 name=copybookname'
 uop=q0,was=a64000b64000
 fili1=?d1ebc/citytax2,rcs=00128,typ=RSF
 filo1=?d2asc/citytax2,rcs=00128,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00128),a0       move rec to outarea before field prcsng
        tra    b0(00128)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
 # Next 9 instructions inserted manually to test multi-record-types
        cmc    b8(1),'H'          Header record
        skp=   put1
        cmc    b8(1),'T'          Tax record
        skp=   typT
        cmc    b8(1),'P'          Payment record
        skp=   typP
        msg    b0(64)             Error - show 1st 64 bytes
        msgw   'Invalid record type byte 8 not H,T,P - enter to bypass'
        skp    loop
 #
 typT   mvc    b88(4),a88              bns post-date
        mvc    b92(15),a92             pns land-value:face-value
        trt    b107(9),$trtsea         ns maint-tax
        skp    put1
 #      ---
 typP   mvc    b20(60),a20             pns mthpayments
        skp    put1
 #      ---
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6C7. Converting ALL files in Directory - Illustrated Op. Instrns.

Execute jobs to convert datafiles d2asc --> d3ebc


 #10. uvcopypfxgA pfx3 d1ebc d2asc   <-- script executes all jobs in pfx3
      ============================     - converting files from d1ebc to d2asc
      - execute all jobs in pfx3/...
      - to convert all files from d1ebc/... to d2asc/...
      - Repeats jobs for all members of GDG file groups
      - see script 'uvcopypfxgA' listed on page '10F1'
      - see console log below illustrating GDG file conversions

uvcopypfxgA console log

 jobname=ar.customer.armaster   - pfx2 name=datafilename
 copybook=armaster.cpy - pfx1 name=copybookname
 140824:103046:ar.customer.ar: uvcopy ver=20140813 pf=pfx3/ar.customer.armaster
 uvcopy LNX H64 license=140813_99V_930630 site=UV_Software
 140824:103046:ar.customer.ar: EOF fili01 rds=32 size=8192: d1ebc/ar.customer.armaster
 140824:103046:ar.customer.ar: EOF filo01 wrts=32 size=8192: d2asc/ar.customer.armaster
                     --- omitting log for 5 files ---
 jobname=gl.account.gltran_   - pfx2 name=datafilename
 copybook=gltran - pfx1 name=copybookname
 140824:103046:gl.account.glt: uvcopy ver=20140813 pf=pfx3/gl.account.gltran_
 uvcopy LNX H64 license=140813_99V_930630 site=UV_Software
 140824:103046:gl.account.glt: EOF fili01 rds=20 size=1600: d1ebc/gl.account.gltran_000001
 140824:103046:gl.account.glt: EOF filo01 wrts=20 size=1600: d2asc/gl.account.gltran_000001
 jobname=gl.account.gltran_   - pfx2 name=datafilename
 copybook=gltran - pfx1 name=copybookname
 140824:103046:gl.account.glt: uvcopy ver=20140813 pf=pfx3/gl.account.gltran_
 uvcopy LNX H64 license=140813_99V_930630 site=UV_Software
 140824:103046:gl.account.glt: EOF fili01 rds=20 size=1600: d1ebc/gl.account.gltran_000002
 140824:103046:gl.account.glt: EOF filo01 wrts=20 size=1600: d2asc/gl.account.gltran_000002
Note
  • this illustrates uvcopypfxgA repeating a conversion job for all generations
    of a GDG file - only 2 generations shown here (gl.account.gltran_000001 & 000002)
  • gencnvC generates only 1 job for a GDG file
  • script uvcopypfxgA re-executes the 1 job for all members of the GDG set

Execute All jobs in pfx3/... to convert All data files


 uvcopyxx 'pfx3/*'
 =================

Being able to kickoff all the data-file covnersions with 1 command is very handy since you may need to repeat teh conversions several times before testing complete and of course on the go-live weekend.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6C8. Converting ALL files in Directory - Illustrated Op. Instrns.

Inspect output files


 #11. l d2asc     <-- list output files converted to ASCII
      =======
 -rw-rw-r-- 1 uvtest apps  8192 Aug 27 10:23 ar.customer.armaster
 -rw-rw-r-- 1 uvtest apps  1280 Aug 27 10:23 ar.sales.arsales
 -rw-rw-r-- 1 uvtest apps  1280 Aug 27 10:23 citytax1
 -rw-rw-r-- 1 uvtest apps  2048 Aug 27 10:23 citytax2
 -rw-rw-r-- 1 uvtest apps 13952 Aug 27 10:23 gl.account.glmaster_000001
 -rw-rw-r-- 1 uvtest apps 13952 Aug 27 10:23 gl.account.glmaster_000002
 -rw-rw-r-- 1 uvtest apps 13952 Aug 27 10:23 gl.account.glmaster_000003
 -rw-rw-r-- 1 uvtest apps  1600 Aug 27 10:23 gl.account.gltran
 -rw-rw-r-- 1 uvtest apps  1600 Aug 27 10:23 gl.account.gltran_000001
 -rw-rw-r-- 1 uvtest apps  1600 Aug 27 10:23 gl.account.gltran_000002
 -rw-rw-r-- 1 uvtest apps  1600 Aug 27 10:23 gl.account.gltran_000003
  1. Check data files with uvhd &/or uvhdcob (if any packed/binary fields)


 #12a. uvhd d2asc/citytax1 r128  <-- inspect file with uvhd
       ========================
 rec#=1 rsize=128 fptr=0 fsize=1280 rcount=10
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             3333333322444424445522222222222222233332445442544422222222225444
             1013014000AF8E085E29000000000000000181502F75E02F14000000000061E3
          64 OUVER           BCV9S1H1..C...W.........qq.000149061970530
             455452222222222244535343004D005880008000770333333333333333222222
             F5652000000000002369318101320072C0047C0111C000149061970530000000

 #12b. uvhdcob d2asc/citytax1 maps/citytax1  <-- uvhdcob to see COBOL fieldnames
       ====================================
 rec#=1 rcount=10 rsize=128 fsize=1280 fptr=0
 rec#       1           occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 ctx-folio                    0    9    10130140
 002 ctx-name                    10   34    JOHN HENRY
 003 ctx-address                 35   59    1815 BOWEN ROAD
 004 ctx-city                    60   79    VANCOUVER
 005 ctx-province                80   81    BC
 006 ctx-zip                     82   87    V9S1H1
 007 ctx-post-date               88   91bns 000143D2        (82898 dcml)
 008 ctx-land-value              92   96pns 000057828C
 009 ctx-imp-value               97  101pns 000004870C
 010 ctx-face-value             102  106pns 000171710C
 011 ctx-maint-tax              107  115 ns 000149061
 012 ctx-purch-date             116  121    970530
 013 filler001                  122  127

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6D1. Converting ALL files in Directory - Illustrated Op. Instrns.

modify control file rcs/rcso & typ/typo

The control file is generated with rcs=_____ & typ=______ which is OK, because the record-size is determined by the copybook & the file type is defaulted to 'typ=RSF' (mainframe files FTP'd binary mode).

If desired, you can MODIFY rcs=_____ & typ=_____ and ADD rcso=... & typo=... I suggest you complete the 1st conversion using the control file as generated, & then as an experiment, you can modify the control file,& re-convert to note the differences in the generated jobs.

ctl/datacpy52 - demo modify rcs/typ & add rcso/typo

 ar.sales.arsales1       cpy=arsales.cpy  rcs=_____ typ=LST
 ar.sales.arsales2       cpy=arsales.cpy  rcs=_____ typ=_____ typo=LST
 ar.sales.arsales3       cpy=arsales.cpy  rcs=_____ typ=_____ typo=RST
 ar.sales.arsales4       cpy=arsales.cpy  rcs=_____ typ=_____ rcso=80 typo=RST

ar.sales.arsales1 ... typ=LST

Changing control file ar.sales.arsales1 from 'typ=_____' to 'typ=LST' would have the following BEFORE/AFTER effect in the generated job (pfx2/ar.sales.arsales1).

 fili1=?d1ebc/ar.sales.arsales,rcs=00064,typ=RSF  <-- BEFORE changing
 filo1=?d2asc/ar.sales.arsales,rcs=00064,typ=RSF      control file

fili1=?d1ebc/ar.sales.arsales,rcs=00064,typ=LST <-- AFTER change to typ=LST filo1=?d2asc/ar.sales.arsales,rcs=00064,typ=LST

You might think of doing this for files that are FTP'd in ASCII mode (vs binary) to allow for the LineFeed appended by FTP ASCII, but you would also have to disable the 'tra' (TRanslate to Ascii) instruction in the generated uvcopy job. These conversions are intended to be used when ALL files are FTP'd in binary mode to allow for packed/binary fields.

Difference between typ=RSF, typ=LST, typ=RST

typ=RSF
  • Record Sequential Fixed, No Linefeed
typ=LST
  • Line Sequential Terminated, Linefeed appended on specified rcs=...
  • If rcs=80, LineFeed would be in column 81
  • physical record size 81
typ=RST
  • Record Sequential Terminated, Linefeed in last byte of rcs=...
  • can use if you know last byte of record is unused.
typ=LSTt
  • Line Sequential Terminated, Linefeed after last non-blank
  • record size varies depending on data content (typical text file)
  • COBOL ORGANIZATION LINE SEQUENTIAL outputs these files
    (saves a lot of space in print files)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6D2. modify control file rcs=, typ=, append rcso=, typo=

ar.sales.arsales2 ... typo=LST

Adding 'typo=LST' in the control file would change output filo1 to 'typ=LST' in the generated job (pfx2/ar.sales.arsales2) as follows:

fili1=?d1ebc/ar.sales.arsales2,rcs=00064,typ=RSF filo1=?d2asc/ar.sales.arsales2,rcs=00064,typ=LST <-- control file typo=LST

You might append 'typo=LST' to change output file type to Line Sequential (LineFeed terminated) to allow use of unix utilities (vi, more, lp, etc). You would also need to change the COBOL programs using this file. You could add 1 to the record-size to allow for the extra byte (LineFeed), but much better to change the file select to ORGANIZATION LINE SEQUENTIAL (could leave record-size as is).

You would also need to change any uvsort's & uvcp's in the JCL/scripts from typ=RSF to typ=LST. BUT, you could do this automatically by copying the $CNVDATA/ctl/datacpy52 file to $RUNLIBS/ctl/add/... and reconverting the JCL/scripts using jcldata51A & jclxx51 (jclxx53 for AIX).

ar.sales.arsales3 ... typo=RST

Adding 'typo=RST' in the control file would change filo1 output to 'typ=RST' in the generated job (pfx2/ar.sales.arsales3) as follows:

fili1=?d1ebc/ar.sales.arsales3,rcs=00064,typ=RSF filo1=?d2asc/ar.sales.arsales3,rcs=00064,typ=RST <-- control file typo=RST

You could use 'typo=RST' if you know that the last byte of the record is unused. 'typ=RST' inserts a LineFeed in last byte of the specified record-size vs appending a LineFeed onto the specified record-size.

ar.sales.arsales4 ... rcso=80 typo=LST

Appending 'rcso=80 typo=RST' in the control file would have the following effect in the generated job (pfx2/ar.sales.arsales4).

 fili1=?d1ebc/ar.sales.arsales4,rcs=00064,typ=RSF
 filo1=?d2asc/ar.sales.arsales4,rcs=00080,typ=RST  <-- rcso=80 & typo=RST

This example shows you could change the output record-size, but you probably would not, because COBOL programs & uvsorts would also have to be changed.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6E1. Converting All files in Directory

generate/execute files 1 at a time vs all

This Part 6 documents generating conversion jobs for ALL files in the directory (vs previous Parts 3,4,& 5 for 1 ata time). This page will show you how you might make 1 at a time additions to the operating procedures written for all files in the directory.

Let's assume you need to add 'PY.PAYROLL.PYMASTER' file to the files already converted in this Part 6.


 #0a. Login userxx --> /home/userxx

 #0b. cdc --> alias cdc='cd $CNVDATA' --> /home/userxx/cnvdata1

1a. FTP the new data-file to $CNVDATA/d0ebc/PY.PAYROLL.PYMASTER

1b. FTP the corresponding copybook to $CNVDATA/cpys/pymaster.cpy


 #2. cp d0ebc/PY.PAYROLL.PYMASTER d1ebc/  <-- copy to d1ebc
     ===================================

 #3a. renameGDG d1ebc   <-- if GDG file, rename to UV convention
      ===============       .G0001V00 -> _000001, etc

 #3b. renameL d1ebc     <-- rename to lowercase
      =============
Note
  • renameGDG & renameL examine all files in the directory
  • but will change only those files not already changed

add new file to control file

On page '6C2', we generated the control file for all files in directory d1ebc/ but to add 1 file, we can just add a new line to the existing control file.


 #6. vi ctl/datacpy52    <-- add entry for py.payroll.pymaster
     ================
 ar.customer.armaster            cpy=armaster.cpy rcs=_____ typ=_____
 ar.sales.arsales                cpy=arsales.cpy  rcs=_____ typ=_____
 citytax1                        cpy=citytax1.cpy rcs=_____ typ=_____
 citytax2                        cpy=citytax2.cpy rcs=_____ typ=_____
 gl.account.glmaster_            cpy=glmaster.cpy rcs=_____ typ=_____ dups=003
 gl.account.gltran               cpy=gltran.cpy   rcs=_____ typ=_____
 gl.account.gltran_              cpy=gltran.cpy   rcs=_____ typ=_____ dups=003
 py.payroll.pymaster             cpy=pymaster.cpy rcs=00512 typ=RSF
Note
  • we are showing entire control file with the new addition
  • we coded rcs=00512 & typ=RSF just to draw attention to new entry
  • they would be coded from the copybook if we lef them blank

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6E2. Converting All files in Directory - Concise Op. Instrns.

generate data conversion jobs from copybook & control-file


 #7. gencnvC ctl/datacpy52   <-- generate data conversion jobs
     =====================
     - script 'gencnvC' runs 3 uvcopy jobs (cobmap1, uvdata51,& uvdata52)
     - see script listed at https://uvsoftware.ca/datacnv1.htm#10E1
Note
  • the above will re-generate conversion jobs for all files
    (not just the new file added to the control file)
  • but should do no harm, if you followed prior recommendations
    to copy all pfx2/* to pfx3/... prior to any edits & executions

edit generated jobs (if required)

You would now copy just the new generated job for the new file from pfx2 to pfx3 and edit if required (for multi-record-types).


 #8. cp pfx2/py.payroll.pymaster pfx3  <-- copy new job before any editing
     ================================

 #9. vi pfx3/py.payroll.pymster        <-- make any manual changes required
     ==========================

Execute new job to convert new datafile


 #10. uvcopy pfx3/py.payroll.pymaster
      ===============================
      - execute just 1 job in pfx3/...
      - to convert just 1 file from d1ebc/... to d2asc/...
      - jobs for GDG files are repeated depending on generations present
Note
  • this is in contrast to #10 on page '6C7', where script
    'uvcopypfxgA' was used to execute ALL jobs for all files

Alternate control-file with just 1 entry

The above procedure added the new file entry to the existing control file & re-generated conversion jobs for all datafiles. Alternatively you could make a separate control-file with just 1 entry to generate just 1 new job. (As noted above regenerating all to pfx2/... does no harm as long as you had copied pfx2/* to pfx3/... before any manual edits).


 #6a. vi ctl/datacpy52a  <-- create alternate ctlfile with just 1 new entry
      =================
      py.payroll.pymaster     cpy=pymaster.cpy rcs=00512 typ=RSF

 #7a. gencnvC ctl/datacopy52a
      =======================
      - generate just the 1 new job (as specified in aalternate ctlfile)
      - 'gencnvC' runs 3 utilities (cobmap1,uvdata51, uvdata52) see page '6C4'

 #8. #9, #10 - run the new job same as above
     =======================================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6F1. Converting ALL files in Directory - Illustrated Op. Instrns.

gencnvC2 - Generate Pipe|Delimit jobs

Script 'gencnvC2' is similar to gencnvC (EBCDIC to ASCII conversions), but generates jobs to create pipe|delimited files. 'gencnvC' is a pre-requisite, since the input to the pipe|delimit jobs is the ASCII files from gencnvC. gencnvC2 uses the same control file as gencnvC.

ctl/datacpy52 - control file of datafilenames & copybooknames

 ar.customer.armaster            cpy=armaster.cpy rcs=_____ typ=_____
 ar.sales.arsales                cpy=arsales.cpy  rcs=_____ typ=_____
 citytax1                        cpy=citytax1.cpy rcs=_____ typ=_____
 citytax2                        cpy=citytax2.cpy rcs=_____ typ=_____
 gl.account.glmaster_            cpy=glmaster.cpy rcs=_____ typ=_____ dups=003
 gl.account.gltran               cpy=gltran.cpy   rcs=_____ typ=_____
 gl.account.gltran_              cpy=gltran.cpy   rcs=_____ typ=_____ dups=003

 #1. gencnvC2 ctl/datacpy52  <-- generate pipe|delimit jobs
     ======================
 cpys-------->maps---------->pfp1-------------->pfp2----------->pfp3--------->
     cobmap1       genpipe1         uvdata52          copy/edit       execute
             convert copybooks   change copybooknames
           to pipe|delimit jobs   to datafilenames
                                  via control-file

 #2. cp pfp2/* pfp3/   <-- copy to pfp3/ before any edits
     ===============

 #3. vi pfp3/...       <-- modify for multi-record-type files (if any)
     ===========

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6F2. Converting ALL files in Directory - Illustrated Op. Instrns.

sample pipe|delimit job pfp2/gl.account.gltran_

 # gltran - uvcopy job to convert EBCDIC/ASCII data to delimited text file
 opr='gltran - uvcopy code generated from copybook: gltran '
 rop=j200000   #Run OPtions: increase instrn storage
 was=a64000b64000c999000d64000e64000u64000v64000  #increase u&v if fields > 163
 fili1=?d2asc/gl.account.gltran_,rcs=00080,typ=RSF
 filo1=?d4pipe/gl.account.gltran_,rcs=00240,typ=LSTtd3
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
 # area a input, see 'get' instrn
 # area b translated to ASCII, in case mainframe EBCDIC file (with packed?)
 # area c data fields fixed 100 bytes apart in prep for var | delimit instrn
 # area d output, see 'put' instrn at end
        mvc    b0(00080),a0       move input area a to area b
 ###    tra    b0(00080)          uncomment for EBCDIC input (or option a1)
 #      ---                   <-- & insert instrns from uvdata51 if packed fields
 #      ---                   <-- insert R/T tests if redef records
        mvc    c0(8),b0(8)                          #1 glt-acntno
        mvc    c100(2),b8(2)                        #2 glt-type
        mvc    c200(30),b10(30)                     #3 glt-description
        edta1  c300(12),b40(10),'-zzzzzzzzz9'       #4 glt-amount
        mvc    c400(1),b50(1)                       #5 glt-status
        mvc    c500(4),b51(4)                       #6 glt-client
        mvc    c600(2),b55(2)                       #7 glt-province
        mvc    c700(12),b57(12)                     #8 glt-trace
        mvc    c800(11),b69(11)                     #9 glt-filler1
        rep    c0(00900),'|','!'           #comment speedup if no '|'s in data
        varb1  d0(00900),c0(100),0009,'|'  #insert '|'s between fields & squeeze
        trt    d0(00900),$trtchr           #ensure all ascii chars x'20'-x'7E'
        clr    c0(00900),' '
 put1   put    filo1,d0
        skp    loop
 #
 eof    cls    all
        eoj
 @pf2=genpipe1.sub

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6F3. Converting ALL files in Directory - Illustrated Op. Instrns.

Execute pipe|delimit jobs


 #4. uvcopypfxgA pfp3 d2asc d4pipe   <-- script to execute all jobs in pfp3
     =============================     - converting from d2asc/* to d4pipe/...

 #5. vi d4pipe/gl.account.gltran_000001
     ==================================
     - inspect same pipe|delimited file

sample pipe|delimited file

 00011100|T|Bank Fee|-1000| |1234|BJ|040428| |
 00011100|T|EMJ Sony Tape Drive|-123141| |1234|BC|000871040428| |
 00011100|T|Telephone|-5598| |1234|BC|000882040428| |
 00011100|T|FEDEX|-1091| |1234|BC|000870040428| |
 00011100|T|VISA payment|-400000| |1234|BC|000884040428| |
 00011100|T|Staples office supplies|-12275| |1234|BC|000874040428| |
 00011100|T|Datex office rent|-32100| |1234|BC|000885040428| |

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6G1. DATAcnv1 - Converting ALL files in Directory

Testing Converted Files

If desired you could copy your converted datafiles to $testdata1/data & execute some of the demo JCL/scripts supplied in $UV/mvstest1/testlibs1. On page '6B1' we setup testdata1 & testlibs1 (with empty subdirs). We only need to store datafiles in testdata1/data & GDG control file in testdata1/ctl.

BUT, to execute, we will need several subdirs in testlibs1 (jcls,cblx,etc). The COBOL & JCL conversions are documented extensively in https://www.uvsoftware.ca/jclcnv1demo.htm#Part_3 so we will take a shortcut here & simply copy from the supplied $UV/mvstest1/testlibs1.

copy datafiles & libraries for testing


 #1. Login --> /home/userxx

 #2. cdl --> /home/userxx/testlibs1
     ==============================

 #3. cp -r $UV/mvstest1/testlibs1/* .    (shortcut for data conversion demos)
     ================================
     - copy all supplied COBOL & JCL libraries
     - programs already compiled for Micro Focus COBOL
     - JCL/scripts converted for Micro Focus COBOL
     - you would need to convert for AIX COBOL
     - see https://www.uvsoftware.ca/jclcnv1demo.htm#Part_3

 #4. cdd --> /home/userxx/testdata1

 #5. cp /home/userxx/cnvdata1/d2asc/* data
     =====================================
     - copy datafiles (converted above 6C1 - 6C7) to $RUNDATA/data/...

 #6. cp $UV/mvstest1/testdata1/ctl/* ctl
     ==================================
     - copy supplied GDG control file gdgctl51

 #7. testdatainit
     ============
     - good idea to init $RUNDATA/jobtmp before testing
     - also reloads Indexed file version gdgctl51I.dat/.idx from gdgctl51

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6G2. DATAcnv1 - Converting ALL files in Directory

Testing Converted Files

You could run a few of the demo JCL/scripts, see more details & console logs in https://www.uvsoftware.ca/jclcnv1demo.htm#Part_4.


 #1. Login --> /home/userxx

 #2. cdl --> /home/userxx/testlibs1
     ===============================

 #3. l jcls    <-- list JCL/scripts available
     ======      - previously converted in JCLcnv1demo.htm#Part_3

 #3a. jar100.ksh - 1 step COBOL
 #3b. jar200.ksh - SORT + COBOL
 #3c. jgl100.ksh - GDG files with COBOL
 #3d. jgl200.ksh - GDG files + SORT + COBOL
 #3e. jgl230.ksh - SORT + COBOL + COBOL
 #3f. jgl320.ksh - IEBGENER + IDCAMS + IF/THEN/ELSE

 #4. joblog jar100.ksh   <-- execute a script using files converted
     =================
 140827:155237:jar100: Begin Job=jar100
 140827:155237:jar100: /home/uvtest/testlibs1/jcls/jar100.ksh
 140827:155237:jar100: Arguments: jar100.ksh
 140827:155238:jar100: RUNDATE=20140827
 140827:155238:jar100: RUNLIBS=/home/uvtest/testlibs1
 140827:155238:jar100: RUNDATA=/home/uvtest/testdata1
 140827:155238:jar100: JTMP=jobtmp/jar100
 140827:155238:jar100: SYOT=sysout/jar100
 140827:155238:jar100: LOGNAME=uvtest TESTPROD=T000
 140827:155238:jar100: HOSTNAME=uvsoft4
 140827:155238:jar100: ******** Begin Step S0010 car100 (#1) ********
 140827:155238:jar100: file: CUSTMAS=data/ar.customer.armaster fsize=8K
 140827:155238:jar100: file: NALIST=data/ar.customer.nameadrs.list100 fsize=K
 140827:155238:jar100: file: SYSOUT=sysout/jar100/jar100_s0010_sysout_140827_155238 fsize=K
 140827:155238:jar100: Executing--> cobrun -F /home/uvtest/testlibs1/cblx/car100
 140827:155238:JAR100: StepTimes: S0010 car100 Begun=15:52:38 End=15:52:38 Elapsed=00:00:00
 140827:155238:JAR100: Job Times: Begun=15:52:38 NormalEnd=15:52:38 Elapsed=00:00:00
 140827:155238:jar100: JobEnd=Normal, StepsExecuted=1, LastStep=S0010

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6H1. DATAcnv1 - Converting ALL files in Directory

Concise Operating Instructions

On pages '6C1' - '6C8', we presented the "Illustrated Operating Instructions" using VU supplied test/demo datafiles & copybooks. We hope this gave you a good understanding of these procedures.

For your own datafile conversions, you might like a more "Concise" version of the operating procedures, which is presented on the following pages. We assume the following has been performed:

  1. Vancouver Utilities installed: https://uvsoftware.ca/install.htm#B1 - B6
  2. Your login userid setup as per https://uvsoftware.ca/jclcnv1demo.htm#Part_1
  3. Your profile setup as per https://uvsoftware.ca/jclcnv1demo.htm#Part_1
  4. You have setup conversion superdirs (testlibs1, testdata1,& cnvdata1) in your homedir & copied required control files from $UV (using scripts mvslibsdirs, mvsdatadirs,& cnvdatadirs) as per https://uvsoftware.ca/jclcnv1demo.htm#1B1

    Datafile Conversion Directories Required

 /home/userxx       <-- your homedir
 :-----cnvdata1     <-- cnvdata1 superdir for DATA conversion
 :     :-----cpys   <-- copybooks         <-- FTP from mainframe
 :     :-----ctl      - control files
 :     :-----d0ebc  <-- EBCDIC data files <-- FTP from mainframe
 :     :-----d1ebc    - EBCDIC data files
 :     :-----d2asc    - ASCII data files (converted by uvcopy uvdata51)
 :     :-----d3ebc    - convert back to EBCDIC by uvdata31
 :     :-----d4pipe   - pipe delimited to load DB tables (by genpipe1)
 :     :-----maps     - copybooks with record layouts on right side
 :     :-----pfx1     - uvcopy jobs to convert EBCDIC to ASCII (gen by uvdata51)
 :     :-----pfx2     - with data filenames inserted (vs copybook names)
 :     :-----pfx3     - copied here for modify/execute
 :     :-----pfp1     - uvcopy jobs to create pipe delimited (gen by genpipe1)
 :     :-----pf       - misc uvcopy jobs
 :     :-----sf       - misc scripts
 :     :-----sqlTC1   - scripts to Create database tables (gen from copybooks)
 :     :-----sqlTL1   - scripts to Load database tables (gen from copybooks)
 :     :-----tmp      - temp files (keep working directory clean)

Generating ALL jobs to convert ALL datafiles EBCDIC to ASCII

 cpys-------->maps---------->pfx1-------------->pfx2----------->pfx3--------->
     cobmap1       uvdata51         uvdata52          copy/edit       execute
             convert copybooks  change copybooknames
               to uvcopy jobs     to datafilenames

Converting ALL data files EBCDIC to ASCII

 Mainframe---->d0ebc---------->d1ebc------------->d2asc------->$RUNDATA/data
           FTP   rename GDG/lower   EBCDIC->ASCII      copy files to test dirs

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6H2. Converting All files in Directory - Concise Op. Instrns.

Concise Operating Instructions


 #0a. Login userxx --> /home/userxx

 #0b. cdc --> alias cdc='cd $CNVDATA' --> /home/userxx/cnvdata1
  1. Copy EBCDIC files to be converted to $CNVDATA/d0ebc/... For your conversion you will FTP files from the mainframe. For a demo you can copy the supplied demo files as follows:


 #1a. cp $UV/mvstest1/cnvdata1/d0ebc/* d0ebc
      ======================================
      - copy VU supplied test data files to your homedir/cnvdata1/d0ebc/
      - OR FTP mainframe datafiles for your real conversion

 #1b. cp $UV/mvstest1/cnvdata1/cpys/* cpys
      ====================================
      - copy VU supplied copybooks to your homedir/cnvdata1/cpys/
      - OR FTP mainframe copybooks for your real conversion

 #2. cp d0ebc/* d1ebc/   <-- copy to d1ebc before rename (save original)
     =================

 #3a. renameGDG d1ebc    <-- rename GDG files from mainframe convention to UV
      ===============        .G0001V00 -> _000001, .G0002V00 -> _000002, etc

 #3b. renameL d1ebc      <-- rename to lowercase
      =============

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6H3. Converting All files in Directory - Concise Op. Instrns.

generate control file from datafilenames in d1ebc/


 #4. uvcopy data2cpy51,fild1=d1ebc,filo2=ctl/datacpy51
     =================================================
     - create ctl/datacpy51 from all datafilenames in d1ebc/...
     - reducing GDG filenames to 1 entry per GDG set

 #4a. uvcopy data2cpy51  <-- same as above, filenames default as shown above
      =================

insert copybooknames method#1 - manual edit


 #5. cp ctl/datacpy51 ctl/datacpy52
     ==============================
     - copy/rename to prevent edited file overwrite if data2cpy51 rerun

 #6. vi ctl/datacpy52      <-- edit/insert copybook names as shown below
     ================

insert copybooknames method#2 - data2cpy52 automatic (if)

IF copybooknames are embedded in datafilenames (at some sites), you can use 'data2cpy52' to extract copybooknames from datafilenames into the control-file. (Unique to some sites, but a great convenience).

'data2cpy52' would be run following 'data2cpy51' & it assumes that the copybookname is the last segment in the filename, following the last '.' & ended by a blank or the '_' indicating a GDG file.


 #5. uvcopy data2cpy52,fili1=ctl/datacpy51,filo1=ctl/datacpy52
     =========================================================
     - automatic insert copybooknames from datafilenames

 #5a. uvcopy data2cpy52  <-- same, files default as above
      =================

 #6. vi ctl/datacpy52    <-- inspect ctlfile with copybook names
     ================      - make corrections if auto code not perfect

ctl/datacpy52 - copybooknames extracted from datafilenames

 ar.customer.armaster            cpy=armaster.cpy rcs=_____ typ=_____
 ar.sales.arsales                cpy=arsales.cpy  rcs=_____ typ=_____
 citytax1                        cpy=citytax1.cpy rcs=_____ typ=_____
 citytax2                        cpy=citytax2.cpy rcs=_____ typ=_____
 gl.account.glmaster_            cpy=glmaster.cpy rcs=_____ typ=_____ dups=003
 gl.account.gltran               cpy=gltran.cpy   rcs=_____ typ=_____
 gl.account.gltran_              cpy=gltran.cpy   rcs=_____ typ=_____ dups=003

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6H4. Converting All files in Directory - Concise Op. Instrns.

using control file - to generate data conversion jobs


 #7. gencnvC ctl/datacpy52   <-- generate data conversion jobs
     =====================
     - script 'gencnvC' runs 3 uvcopy jobs (cobmap1, uvdata51,& uvdata52)
     - see script listed at https://uvsoftware.ca/datacnv1.htm#10E1
       ----- OR could run separately as follows -----

 #7a. uvcopyx cobmap1 cpys maps uop=q0i7
      ==================================
      - convert cobol copybooks to maps (record layouts)

 #7b. uvcopyx uvdata51 maps pfx1 uop=q0i7
      ===================================
      - generates uvcopy data conversion jobs from copybookmaps
      - codes fili1=... & filo1=... & jobnames sames as copybooknames

 #7c. cp $UV/IBM/skeleton2 pfx1
      =========================
      - copy uvcopy job skeleton for datafiles without copybooks
      - OK for files that have no packed/binary fields
      - could modify control file typ=LST (LineFeeds OK if no packed/binary)

 #7d. uvcopy uvdata52,fili1=ctl/datacpy52,fild2=pfx1,fild3=pfx2,uop=q0i7
      ==================================================================
      - copies pfx1/* to pfx2/... inserting datafilenames (vs copybooknames)
      - also renames the jobs for the datafilenames (vs copybooknames
      - since could have multiple datafiles for same copybook (esp. for GDG)

edit generated jobs (for multi-record-types)

It may be necessary to make some manual changes to auto-generated jobs, to allow for any files with multiple record-types, for variable-length files,& corrections to for missing copybooks (job generated from skeleton2).


 #8. cp pfx2/* pfx3    <-- copy auto-generated jobs before any editing
     ==============

 #9. vi pfx3/...       <-- make any manual changes required
     ===========

Execute jobs to convert datafiles d2asc --> d3ebc


 #10. uvcopypfxgA pfx3 d1ebc d2asc
      ============================
      - execute all jobs in pfx3/...
      - to convert all files from d1ebc/... to d2asc/...
      - jobs for GDG files are repeated depending on generations present
      - see console log below illustrating GDG file conversions

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6H5. Converting All files in Directory - Concise Op. Instrns.

gencnvC2 - Generate Pipe|Delimit jobs

Script 'gencnvC2' is similar to gencnvC (EBCDIC to ASCII conversions), but generates jobs to create pipe|delimited files. 'gencnvC' is a pre-requisite, since the input to the pipe|delimit jobs is the ASCII files from gencnvC. gencnvC2 uses the same control file as gencnvC.


 #11. gencnvC2 ctl/datacpy52  <-- generate pipe|delimit jobs
      ======================
 cpys-------->maps---------->pfp1-------------->pfp2----------->pfp3--------->
     cobmap1       genpipe1         uvdata52          copy/edit       execute
             convert copybooks   change copybooknames
           to pipe|delimit jobs   to datafilenames
                                  via control-file

 #12. cp pfp2/* pfp3/   <-- copy to pfp3/ before any edits
      ===============

 #13. vi pfp3/...       <-- modify for multi-record-type files (if any)
      ===========

Execute pipe|delimit jobs


 #14. uvcopypfxgA pfp3 d2asc d4pipe   <-- script to execute all jobs in pfp3
      =============================     - converting from d2asc/* to d4pipe/...

 #15. vi d4pipe/gl.account.gltran_000001
      ==================================
      - inspect same pipe|delimited file

sample pipe|delimited file

 00011100|T|Bank Fee|-1000| |1234|BJ|040428| |
 00011100|T|EMJ Sony Tape Drive|-123141| |1234|BC|000871040428| |
 00011100|T|Telephone|-5598| |1234|BC|000882040428| |
 00011100|T|FEDEX|-1091| |1234|BC|000870040428| |
 00011100|T|VISA payment|-400000| |1234|BC|000884040428| |
 00011100|T|Staples office supplies|-12275| |1234|BC|000874040428| |
 00011100|T|Datex office rent|-32100| |1234|BC|000885040428| |

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_7 DATAcnv1.doc - Converting EBCDIC/ASCII & Delimit/Undelimit

Part 7 - Contents


7A1. bytes256 - test-file for translates & delimits

7B1. preparation for tests - make directories
- can make directory in your homedir with multiple subdirs
- get test file from $UV/dat1/bytes256
- get copybook from $UV/mf/cpys/bytes256 & convert to cobmap (layout)

7C1. Op. Instrns. to test translate EBCDIC to ASCII & back to EBCDIC

7D1. RESULTS - d1ebc(EBCDIC), d2asc(ASCII), d3ebc(back to EBCDIC)
7D2. RESULTS - compare original EBCDIC file before translate to ASCII
& final EBCDIC file reconverted from ASCII

7E1. uvcopy jobs to translate EBCDIC to ASCII & ASCII to EBCDIC
- generated automatically from copybook

7F1. Op. Instrns. to convert EBCDIC Fixed to Delimited & back to Fixed

7G1. RESULTS - pipe|delimted file d4pipe/Ebytes, ugly for vi
- use listpipeA to list record broken into 80 byte segments
  & then listing delimited fields vertically
7G2. RESULTS - uvhd hex before & after converting fixed to delimited & back
7G3. RESULTS - uvcmpFA1 comparing original fixed-field to reconverted fixed file
- hex displays with '*'s flagging differences

7H1. Op. Instrns. to test translate ASCII to EBCDIC & back to EBCDIC

7I1. RESULTS - d1ebc(EBCDIC), d2asc(ASCII), d3ebc(back to EBCDIC)
7I2. RESULTS - compare original EBCDIC file before translate to ASCII
& final EBCDIC file reconverted from ASCII

7J1. Op. Instrns. to convert EBCDIC Fixed to Delimited & back to Fixed

7K1. RESULTS - pipe|delimted file d4pipe/Abytes, ugly for vi
- use listpipeA to list record broken into 80 byte segments
  & then listing delimited fields vertically
7K2. RESULTS - uvhd hex before & after converting fixed to delimited & back
7K3. RESULTS - uvcmpFA1 comparing original fixed-field to reconverted fixed file
- hex displays with '*'s flagging differences
7K4.  Notes re differences before/after delimiting ASCII files

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part 7 - Contents - continued


7L1. Checking Data File Conversions
checking data file conversions with 'uvhd'
uvhd check ar.customer.armaster - EBCDIC file

7L2. uvhd check ar.customer.armaster - ASCII file
checking EBCDIC to ASCII conversion

7L3. compress/uncompress with 'tar'

7L4. compress/uncompress with 'zip'

7L5. Notes re: Micro Focus COBOL variable length records

7L6. checking data file conversions with 'uvhdcob'
uvhdcob check ar.customer.armaster - EBCDIC file

7L7. uvhdcob check ar.customer.armaster - ASCII file

7L8. uvhdcob check city.tax.payments - EBCDIC file
uvhdcob check city.tax.payments - ASCII file

7M1. ebc2asc.c - EBCDIC to ASCII translate table

7M2. asc2ebc.c - ASCII to EBCDIC translate table

7M3. ebc1141asc.c - ASCII to EBCDIC translate table for codepage 1141
- option rop=t1 for uvcopy tre instruction

7N1. trantestEAE - test translate EBCDIC to ASCII & back to EBCDIC
- alternate way to test Vancouver Utilities EBCDIC/ASCII translates.
- uvcopy job to translate EBCDIC to ASCII & back to EBCDIC.
- no data file input, adds binary to create each character of 256 possible
- translates each from EBCDIC to ASCII & back to EBCDIC.
- shows results in hexadecimal & character
7N2.  trantestESE uvcopy job listing that creates above report

7O1. makebytes256 - uvcopy job to create 256 bytes with values x'00' - x'FF'
7O3. alternate way to create bytes256 by compiling neutral traanslate table
& extracting the 256 bytes with unix utility 'dd'
7O4. trtneut.c - neutral translate table, C source code & uvhd hex representation

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7A1. Test Conversion EBCDIC/ASCII & Delimit/Undelimit

test-file for translates & delimits

We will use $UV/dat1/bytes256 to test EBCDIC/ASCII translates & delimit/undelimit. 'bytes256' is a file of 1 256 byte record containing all possible combinations of 8 bits from '00000000' to '11111111' or in hexadecimal from x'00' to x'FF'.

bytes256 displayed by uvhd


 uvhd /home/uvadm/dat1/bytes256 r256
 ===================================
 records=1 rsize=256 fsize=256
 rec#=1 rsize=256 fptr=0 fsize=256 rcount=1
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ................................ !"#$%&'()*+,-./0123456789:;<=>?
             0000000000000000111111111111111122222222222222223333333333333333
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
          64 @ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^abcdefghijklmnopqrstuvwxyz{|}~.
             4444444444444444555555555555555566666666666666667777777777777777
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
         128 ................................................................
             88888888888888889999999999999999AAAAAAAAAAAAAAAABBBBBBBBBBBBBBBB
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
         192 ................................................................
             CCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEFFFFFFFFFFFFFFFF
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF

We will copy $UV/dat1/bytes256, renaming as Ebytes to test EBCDIC to ASCII translates and renaming as Abytes to test ASCII to EBCDIC translates.

We will use 'Ebytes' to translate EBCDIC to ASCII & back to EBCDIC, then compare to prove the re-translated EBCDIC characters are the same as the original EBCDIC. Note that EBCDIC characters are x'40' to x'F9' & the other bytes in x'00'-x'3F' will not all compare the same, but should not matter.

We will use 'Abytes' to translate ASCII to EBCDIC & back to ASCII, then compare to prove the re-translated ASCII characters are the same as the original EBCDIC. Note that ASCII characters are x'00' to x'7F' & the other bytes in x'80'-x'FF' will not all compare the same, but should not matter.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7B1. Test Conversion EBCDIC/ASCII & Delimit/Undelimit

preparation for tests

You can see the results of my tests in $UV/mvstest1/cnvtran/... You can repeat my tests by setting up a subdir (cnvtran or whatever) in your homedir and proceeding as follows:


 #1. Login userxx --> /home/userxx

 #2. mkdir cnvtran   <-- setup directory for tests
     =============

 #3. cd cnvtran      <-- chnage into directory
     ==========

 #4. cnvdatadirs     <-- make subdirs required
     ===========
Note
  • following shows only the subdirs relevant to these translate/delimit tests
 /home/userxx         <-- setup cnvtran withing your $HOME dir
 :-----cnvtran        <-- cnvtran superdir for DATA conversion
 :     :-----cpys       - copybooks
 :     :-----d1ebc      - EBCDIC data files
 :     :-----d2asc      - ASCII data files (converted by uvcopy uvdata51)
 :     :-----d3ebc      - convert back to EBCDIC by uvdata31
 :     :-----d4pipe     - pipe delimited to load DB tables (by genpipe1)
 :     :-----d5fix      - convert pipe|delimited back to fixed field (genfixd3)
 :     :-----listpipe   - pipe|delimited listed vertically with copybook fieldnames
 :     :-----maps       - copybooks with record layouts on right side
 :     :-----pfx1       - uvcopy jobs to convert EBCDIC to ASCII (gen by uvdata51)
 :     :-----pfy1       - uvcopy jobs to convert ASCII to EBCDIC (gen by uvdata31)
 :     :-----pfp1       - uvcopy jobs to create pipe delimited (gen by genpipe1)
 :     :-----pfq1       - uvcopy jobs to convert pipe|delimited to fixed field (genfixd3)
 :     :-----tmp        - temp files (keep working directory clean)

 #5. Get supplied datafile 1 record of 256 bytes x'00' - x'FF'

 #5a. cp $UV/dat1/bytes256 d1ebc/Ebytes  <-- copy/rename to consider
      =================================      as an EBCDIC file

 #5b. cp $UV/dat1/bytes256 d2asc/Abytes  <-- copy/rename to consider
      =================================      as an ASCII file

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7B2. Test Conversion EBCDIC/ASCII & Delimit/Undelimit

get supplied copybook for bytes256


 #6. Get supplied copybook for bytes256 - 16 fields of pic x(16).

 #6a. cp cpys/bytes256 cpys/Ebytes  <-- for EBCDIC file conversions
      ============================

 #6b. cp cpys/bytes256 cpys/Abytes  <-- for ASCII file conversions
      ============================

 #7.  uvcopy cobmap1,fili1=cpys/Abytes.cpy,filo1=maps/Abytes
      ======================================================
      - convert copybook to cobmap (record layout)
        to generate various uvcopy jobs to translate & delimit
      - could omit this since it is included in following 'gencnvA' script
 cobmap1  COBOL record layout            2014/10/22_15:57:54  pg# 0001
 cpys/Abytes                   bytes256           RCSZ=00256  bgn  end  lth typ
 * bytes256 - copybook to test translations ebcdic to ascii &
 *          - by owen townsend, uv software, oct22/2014
 * copybook for datafiles bytes256, Ebytes,& Abytes
 * - bytes256 datafile is 256 bytes total from x'00' to x'FF'
 * - this copybook will describe as 16 fields of 16 bytes eac
 * use script 'gencnvA' to generate uvcopy jobs to convert:
 * - ebcdic to ascii (pfx1/...), ascii to ebcdic (pfy1/...)
 * - fixed-field to delimited (pfp1), delimited to fixed-fiel
  01 bytes256.
      10 b00-0f     pic x(16).                                0000 0015  016
      10 b10-1f     pic x(16).                                0016 0031  016
      10 b20-2f     pic x(16).                                0032 0047  016
      10 b30-3f     pic x(16).                                0048 0063  016
      10 b40-4f     pic x(16).                                0064 0079  016
      10 b50-5f     pic x(16).                                0080 0095  016
      10 b60-6f     pic x(16).                                0096 0111  016
      10 b70-7f     pic x(16).                                0112 0127  016
      10 b80-8f     pic x(16).                                0128 0143  016
      10 b90-9f     pic x(16).                                0144 0159  016
      10 ba0-af     pic x(16).                                0160 0175  016
      10 bb0-bf     pic x(16).                                0176 0191  016
      10 bc0-cf     pic x(16).                                0192 0207  016
      10 bd0-df     pic x(16).                                0208 0223  016
      10 be0-ef     pic x(16).                                0224 0239  016
      10 bf0-ff     pic x(16).                                0240 0255  016
 ******** end copybook bytes256 ****************
 *RCSZ=00256

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7C1. Test Conversion EBCDIC/ASCII & Delimit/Undelimit

Op. Instrns. translate EBCDIC to ASCII & back to EBCDIC

We will generate jobs to translate EBCDIC to ASCII & back to EBCDIC, execute, & prove the re-converted EBCDIC file same as original EBCDIC file. We will also generate pipe delimit/undelimit jobs, execute them, & prove the EBCDIC characters x'40' - x'FF' are the same.

Note
  • Here we are considering the input file (Ebytes) to be an EBCDIC file.
  • Later we will repeat similar procdedures considering input to be ASCII.
  • We will get different results since valid EBCDIC characters are x'40' - x'F9'
    but valid ASCII characters are x'20' - x'7E'

 #1. gencnvA Ebytes   <-- generates all conversion jobs
     ==============       pfx1,pfy1,pfq1,pfp1
                       - see #comments doc in $UV/sf/IBM/gencnvA

 #2. uvcopy pfx1/Ebytes  <-- execute EBCDIC to ASCII, d1ebc --> d2asc
     ==================

 #3. uvcopy pfy1/Ebytes  <-- execute ASCII to EBCDIC, d2asc --> d3ebc
     ==================

 #4. uvcmpFE1 d1ebc/Ebytes d3ebc/Ebytes r256
     =======================================
     - compare original EBCDIC file to reconverted file to see any diff
     - x'40' - x'FF' (EBCDIC characters) are the same
     - x'00' - x'3F' does have differences, but should not matter

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7C2. Test Conversion EBCDIC/ASCII & Delimit/Undelimit


 #5. use uvhd to display results in vertical hexadecimal

 #5a. uvhd d1ebc/Ebytes r256a  <-- note option 'a' show chars in ASCII
      =======================    - zones & digits remain EBCDIC codes
      --> p1  <-- capture in text file tmp/Ebytes_date/timeP original EBCDIC

 #5b. uvhd d2asc/Ebytes r256   <-- show converted to ASCII in hexadecimal
      =======================
      --> p1  <-- capture in text file tmp/Ebytes_date/timeP ASCII version

 #5c. uvhd d3ebc/Ebytes r256a  <-- note option 'a' show chars in ASCII
      =======================    - zones & digits remain EBCDIC codes
      --> p1  <-- capture in text file tmp/Ebytes_date/timeP reconverted file

 #5d. vi tmp/*   <-- investigate uvhd hex displays
     ========     - satisfy yourself reconversion same as original

 #6. OR see uvhd hex-rep files renamed & saved in subdir uvhd/...

 #6a. vi uvhd/uvhd_Ebytes_d1ebc
      =========================
 #6b. vi uvhd/uvhd_Ebytes_d2asc
      =========================
 #6c. vi uvhd/uvhd_Ebytes_d3ebc
      =========================

 #7. uvcmpFE1 d1ebc/Ebytes d3ebc/Ebytes r256
     =======================================
     - compare original EBCDIC file to reconverted file to see any diff
Note
  • see next page 'uvcmpFE1' that flags differences with '*'s.
  • you can see that the EBCDIC characters in 128-255 are the same
    before & after conversion from EBCDIC to ASCII & back to EBCDIC.
  • characters in 0-127 have many differences '*' flagged, because
    the many control character differences between EBCDIC & ASCII.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7D1. Test Conversion EBCDIC/ASCII & Delimit/Undelimit

RESULTS - d1ebc(EBCDIC), d2asc(ASCII), d3ebc(back to EBCDIC)


 uvhd d1ebc/Ebytes r256a  <-- option 'a' character line ASCII so you can read
 =======================    - zones & digits lines are EBCDIC codes
                              mvstest1/cnvtran/uvhd/uvhd_Ebytes_d1ebc
 rec#=1 rsize=256 fptr=0 fsize=256 rcount=1
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ................................................................
             0000000000000000111111111111111122222222222222223333333333333333
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
          64  ABCDEFGHI..<(+|&QRSTUVWXY!$*);^-/.cdefghi.,%_>?p.rs.vwxy`:#@'="
             4444444444444444555555555555555566666666666666667777777777777777
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
         128 .abcdefghi.......jklmnopqr.......~stuvwxyz...[...............]..
             88888888888888889999999999999999AAAAAAAAAAAAAAAABBBBBBBBBBBBBBBB
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
         192 {ABCDEFGHI......}JKLMNOPQR......\.STUVWXYZ......0123456789......
             CCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEFFFFFFFFFFFFFFFF
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF

 uvhd d2asc/Ebytes r256  <-- all 3 line groups ASCII (character/zones/digits)
 ======================      mvstest1/cnvtran/uvhd/uvhd_Ebytes_d2asc
 rec#=1 rsize=256 fptr=0 fsize=256 rcount=1
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ................................ !"#$...()*+,...01.3456.89:;..>.
             0000000700000000111111011111111122222011222220003313333033331131
             0123496F89ABCDEF0123458789ABCDEF01234A7B89ABC5670163456489AB45EA
          64  ABCDEFGHI..<(+|&QRSTUVWXY!$*);^-/.cdefghi.,%_>?p.rs.vwxy`:#@'="
             2444444444A23227255555555522223522C6666666A225337C77C77776324232
             01234567892EC8BC612345678914A9BEDF234567896C5FEF0923867890A307D2
         128 .abcdefghi.......jklmnopqr.......~stuvwxyz...[...............]..
             86666666668888889666666777999999A777777777AAA5AABBBBBBBBBBBBB5BB
             0123456789ABCDEF0ABCDEF012ABCDEF0E3456789AABCBEF0123456789ABCDEF
         192 {ABCDEFGHI......}JKLMNOPQR......\.STUVWXYZ......0123456789......
             7444444444CCCCCC7444444555DDDDDD5E55555555EEEEEE3333333333FFFFFF
             B123456789ABCDEFDABCDEF012ABCDEFC13456789AABCDEF0123456789ABCDEF

 uvhd d3ebc/Ebytes r256a  <-- option 'a' character line ASCII so you can read
 =======================      mvstest1/cnvtran/uvhd/uvhd_Ebytes_d3ebc
 rec#=1 rsize=256 fptr=0 fsize=256 rcount=1
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ................................ !"#$...()*+,...01.3456.89:;..>.
             000030201020000011113312113211114577522245546222FF3FFFF3FF753363
             012375E7655BCDEF0123CD6689F7CDEF0AFBB567DDCEBDEF0123456789AECDEF
          64  ABCDEFGHI..<(+|&QRSTUVWXY!$*);^-/.cdefghiw,%_>?p.rs.vwxy`:#@'="
             4CCCCCCCCC4444445DDEEEEEEE5555556668888888A66666979A7AAAA7777777
             0123456789ABCDEF0892345678ABCDEF01234567896BCDEF7192456789ABCDEF
         128 .abcdefghi.......jklmnopqr.......~stuvwxyz...[...............]..
             88888888888888889999999999999999AAAAAAAAAAAAAAAABBBBBBBBBBBBBBBB
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
         192 {ABCDEFGHI......}JKLMNOPQR......\.STUVWXYZ......0123456789......
             CCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEFFFFFFFFFFFFFFFF
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7D2. Test Conversion EBCDIC/ASCII & Delimit/Undelimit

compare reconverted EBCDIC to original EBCDIC


 4. uvcmpFE1 d1ebc/Ebytes d3ebc/Ebytes r256
    ======================================= rptcmp/Ebytes_d1ebc_d3ebc
 uvcmp1 - compare 2 files, print mismatched records, '*' flag diffs
 2014/12/24_20:19:46 uop=q1p30r256s6t1000000u3x2y0q1y1r256u0
 recsize  reccount   file-size  typ Report=rptcmp/Ebytes
 1:  256         1         256  RSF  File1=d1ebc/Ebytes
 2:  256         1         256  RSF  File2=d3ebc/Ebytes
                       1         2         3         4         5         6
 f#record#byte# 0123456789012345678901234567890123456789012345678901234567890123
 ===============================================================================
 1       1    0 ................................ !"#$...()*+,...01.3456.89:;..>.
                0000000000000000111111111111111122222222222222223333333333333333
                0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
                    * * ***         ** *  **    *****   *****   ** **** ****  *
             64  ABCDEFGHI..<(+|&QRSTUVWXY!$*);^-/.cdefghi.,%_>?p.rs.vwxy`:#@'="
                4444444444444444555555555555555566666666666666667777777777777777
                0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
                 *********       *********         ********     * ** ****
            128 .abcdefghi.......jklmnopqr.......~stuvwxyz...[...............]..
                88888888888888889999999999999999AAAAAAAAAAAAAAAABBBBBBBBBBBBBBBB
                0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
            192 {ABCDEFGHI......}JKLMNOPQR......\.STUVWXYZ......0123456789......
                CCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEFFFFFFFFFFFFFFFF
                0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
 2       1      ................................ !"#$...()*+,...01.3456.89:;..>.
                000030201020000011113312113211114577522245546222FF3FFFF3FF753363
                012375E7655BCDEF0123CD6689F7CDEF0AFBB567DDCEBDEF0123456789AECDEF
                    * * ***         ** *  **    *****   *****   ** **** ****  *
             64  ABCDEFGHI..<(+|&QRSTUVWXY!$*);^-/.cdefghiw,%_>?p.rs.vwxy`:#@'="
                4CCCCCCCCC4444445DDEEEEEEE5555556668888888A66666979A7AAAA7777777
                0123456789ABCDEF0892345678ABCDEF01234567896BCDEF7192456789ABCDEF
                 *********       *********         ********     * ** ****
            128 .abcdefghi.......jklmnopqr.......~stuvwxyz...[...............]..
                88888888888888889999999999999999AAAAAAAAAAAAAAAABBBBBBBBBBBBBBBB
                0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
            192 {ABCDEFGHI......}JKLMNOPQR......\.STUVWXYZ......0123456789......
                CCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEFFFFFFFFFFFFFFFF
                0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
 ==================== EOF or StopPrint/StopRead count reached  ==============
 F1Count=1, F2Count=1, StopPrint=6, StopRead=1000000
 F1Reads=1, MisMatches=1, MisMatsPrinted=1, Recsize=256

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7E1. Test Conversion EBCDIC/ASCII & Delimit/Undelimit

generated uvcopy job to translate EBCDIC to ASCII

 # uvcopy job to translate EBCDIC to ASCII, preserve packed, fix zoned signs
 opr='jobname=JOBNAME   - pfx2 name=datafilename'
 opr='copybook=Ebytes - pfx1 name=copybookname'
 uop=q0,was=a64000b64000
 fili1=?d1ebc/Ebytes,rcs=00256,typ=RSF
 filo1=?d2asc/Ebytes,rcs=00256,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00256),a0       move rec to outarea before field prcsng
        tra    b0(00256)          translate entire outarea to ASCII
 #      ---                <-- insert R/T tests here for redefined records
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

generated uvcopy job to translate ASCII to EBCDIC

 # Ebytes - uvcopy job, ASCII to EBCDIC preserve packed, fix signs
 opr='Ebytes - uvcopy code generated from copybook: Ebytes'
 uop=q0
 was=a33000b33000
 fili1=?d2asc/Ebytes,rcs=00256,typ=RSF
 filo1=?d3ebc/Ebytes,rcs=00256,typ=RSF
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        mvc    b0(00256),a0       move rec to outarea before field prcsng
        tre    b0(00256)          translate entire outarea to EBCDIC
 #      ---                <-- insert R/T tests here for redefined records
 put1   put    filo1,b0
        skp    loop
 eof    cls    all
        eoj

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7F1. Test Conversion EBCDIC/ASCII & Delimit/Undelimit

Op. Instrns. EBCDIC Fixed to Delimited & back to Fixed

Using input d2asc/Ebytes created above in translate tests

Note
  • Our delimited files are always ASCII, we probably do NOT need
    to create EBCDIC delimited files for FTP back to mainframe.

 #1. uvcopy pfp1/Ebytes,fili1=d2asc/Ebytes,filo1=d2pipe/Ebytes
     =========================================================
     - convert ASCII fixed-field (origianlly EBCDIC) to ASCII pipe|delimited

 #1a. uvcopy pfp1/Ebytes     <-- same, files default as above
      ==================

 #2. vi d4pipe/Ebytes        <-- inspect pipe|delimited file with vi
     ================          - ugly, try listpipe below

 #3. listpipeA Ebytes       <-- create vertical listing of pipe|delim fields
     ================

 #3a. vi listpipe/Ebytes     <-- view listpipe results
      ==================

 #4. uvcopy pfq1/Ebytes,fili1=d2pipe/Ebytes,filo1=d5fix/Ebytes
     =========================================================
     - convert ASCII pipe|delimited back to fixed-field ASCII

 #4a. uvcopy pfq1/Ebytes     <-- same, files default as above
      ==================

 #5. uvhd d5fix/Ebytes r256  <-- inspect results re-converted back to fixed-field
     ======================

 #6. uvcmpFA1 d2asc/Ebytes d5fix/Ebytes r256
     =======================================
     - compare original fixed-field to reconverted fixed-field from pipe|delimited

 #7. vi rptcmp/Ebytes        <-- view compare results
     ================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7G1. Test Conversion EBCDIC/ASCII & Delimit/Undelimit

delimit & undelimit results


 #2. vi d4pipe/Ebytes    <-- inspect pipe|delimited file with vi
     ================      - not good, try listpipe below
Note
  • Looking at the pipe|delimited file is ugly, and not very relevant
    since we are using a test file of all byte codes vs meaningful data.
  • It is more relevant to see the next page conversion back to fixed field
    to see which codes are cleared wwhen delimiting character fields

 #3. listpipeA Ebytes       <-- create vertical listing of pipe|delim fields
     ================

 #3a. vi listpipe/Ebytes     <-- view listpipe results
      ==================
 listpipe1 - list '|' delimited files (from genpipe1) with copybook fieldnames
 DataFile=d4pipe/Ebytes, RecSize=00256, RecCount=1
 CopyBook=maps/Ebytes, DateTime=2014/12/24_20:16:59
 Seq# FieldName       OffSet COBOL picture      DataFieldContents
 DataFile = d4pipe/Ebytes, Record# = 1 of 1, Fields = 16
         | | !"#$   ()*+,|01 3456 89:;  >| ABCDEFGHI .<(+!|&QRSTUVWXY!$*);^|-/ cd
 efghi ,%_>?|p rs vwxy`:#@'="| abcdefghi| jklmnopqr| ~stuvwxyz   [|             ]
 |{ABCDEFGHI|}JKLMNOPQR|\ STUVWXYZ|0123456789|
 001 b00-0f                        000 x(16).
 002 b10-1f                        016 x(16).
 003 b20-2f                        032 x(16).                 !"#$   ()*+,
 004 b30-3f                        048 x(16).                01 3456 89:;  >
 005 b40-4f                        064 x(16).                 ABCDEFGHI .<(+!
 006 b50-5f                        080 x(16).                &QRSTUVWXY!$*);^
 007 b60-6f                        096 x(16).                -/ cdefghi ,%_>?
 008 b70-7f                        112 x(16).                p rs vwxy`:#@'="
 009 b80-8f                        128 x(16).                 abcdefghi
 010 b90-9f                        144 x(16).                 jklmnopqr
 011 ba0-af                        160 x(16).                 ~stuvwxyz   [
 012 bb0-bf                        176 x(16).                             ]
 013 bc0-cf                        192 x(16).                {ABCDEFGHI
 014 bd0-df                        208 x(16).                }JKLMNOPQR
 015 be0-ef                        224 x(16).                \ STUVWXYZ
 016 bf0-ff                        240 x(16).                0123456789

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7G2. Test Conversion EBCDIC/ASCII & Delimit/Undelimit

compare BEFORE conversion to delimited & back


 uvhd d2asc/Ebytes r256
 ======================  mvstest1/cnvtran/uvhd/uvhd_Ebytes_d2asc
 rec#=1 rsize=256 fptr=0 fsize=256 rcount=1
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ................................ !"#$...()*+,...01.3456.89:;..>.
             0000000700000000111111011111111122222011222220003313333033331131
             0123496F89ABCDEF0123458789ABCDEF01234A7B89ABC5670163456489AB45EA
          64  ABCDEFGHI..<(+|&QRSTUVWXY!$*);^-/.cdefghi.,%_>?p.rs.vwxy`:#@'="
             2444444444A23227255555555522223522C6666666A225337C77C77776324232
             01234567892EC8BC612345678914A9BEDF234567896C5FEF0923867890A307D2
         128 .abcdefghi.......jklmnopqr.......~stuvwxyz...[...............]..
             86666666668888889666666777999999A777777777AAA5AABBBBBBBBBBBBB5BB
             0123456789ABCDEF0ABCDEF012ABCDEF0E3456789AABCBEF0123456789ABCDEF
         192 {ABCDEFGHI......}JKLMNOPQR......\.STUVWXYZ......0123456789......
             7444444444CCCCCC7444444555DDDDDD5E55555555EEEEEE3333333333FFFFFF
             B123456789ABCDEFDABCDEF012ABCDEFC13456789AABCDEF0123456789ABCDEF

compare AFTER conversion to delimited & back


 uvhd d5fix/Ebytes r256
 ======================  mvstest1/cnvtran/uvhd/uvhd_Ebytes_d5fix
 rec#=1 rsize=256 fptr=0 fsize=256 rcount=1
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0                                  !"#$   ()*+,   01 3456 89:;  >
             2222222222222222222222222222222222222222222222223323333233332232
             000000000000000000000000000000000123400089ABC0000103456089AB00E0
          64  ABCDEFGHI .<(+!&QRSTUVWXY!$*);^-/ cdefghi ,%_>?p rs vwxy`:#@'="
             2444444444223222255555555522223522266666662225337277277776324232
             01234567890EC8B1612345678914A9BEDF034567890C5FEF0023067890A307D2
         128  abcdefghi       jklmnopqr       ~stuvwxyz   [               ]
             2666666666222222266666677722222227777777772225222222222222222522
             01234567890000000ABCDEF0120000000E3456789A000B000000000000000D00
         192 {ABCDEFGHI      }JKLMNOPQR      \ STUVWXYZ      0123456789
             7444444444222222744444455522222252555555552222223333333333222222
             B123456789000000DABCDEF012000000C03456789A0000000123456789000000
Note
  • you can see that the EBCDIC characters in 128-255 are the same
    before & after conversion to delimited & back
  • control characters in 0-31 < x'20' are converted to spaces x'20'
    when converting to delimited
  • not easy to confirm special chars in 64-127 same or not
  • see next page 'uvcmpFA1' that flags differences with '*'s

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7G3. Test Conversion EBCDIC/ASCII & Delimit/Undelimit

compare fixed-field before delimit to fixed-field after undelimit


 uvcmpFA1 d2asc/Ebytes d5fix/Ebytes r256
 =======================================  rptcmp/Ebytes_d2asc_d5fix
 uvcmp1 - compare 2 files, print mismatched records, '*' flag diffs
 2014/12/24_20:20:31 uop=q1p30r256s6t1000000u3x2y0q1r256u0
 recsize  reccount   file-size  typ Report=rptcmp/Ebytes
 1:  256         1         256  RSF  File1=d2asc/Ebytes
 2:  256         1         256  RSF  File2=d5fix/Ebytes
                       1         2         3         4         5         6
 f#record#byte# 0123456789012345678901234567890123456789012345678901234567890123
 ===============================================================================
 1       1    0 ................................ !"#$...()*+,...01.3456.89:;..>.
                0000000700000000111111011111111122222011222220003313333033331131
                0123496F89ABCDEF0123458789ABCDEF01234A7B89ABC5670163456489AB45EA
                ********************************     ***     ***  *    *    ** *
             64  ABCDEFGHI..<(+|&QRSTUVWXY!$*);^-/.cdefghi.,%_>?p.rs.vwxy`:#@'="
                2444444444A23227255555555522223522C6666666A225337C77C77776324232
                01234567892EC8BC612345678914A9BEDF234567896C5FEF0923867890A307D2
                          *    *                  *       *      *  *
            128 .abcdefghi.......jklmnopqr.......~stuvwxyz...[...............]..
                86666666668888889666666777999999A777777777AAA5AABBBBBBBBBBBBB5BB
                0123456789ABCDEF0ABCDEF012ABCDEF0E3456789AABCBEF0123456789ABCDEF
                *         *******         *******         *** *************** **
            192 {ABCDEFGHI......}JKLMNOPQR......\.STUVWXYZ......0123456789......
                7444444444CCCCCC7444444555DDDDDD5E55555555EEEEEE3333333333FFFFFF
                B123456789ABCDEFDABCDEF012ABCDEFC13456789AABCDEF0123456789ABCDEF
                          ******          ****** *        ******          ******
 2       1                                       !"#$   ()*+,   01 3456 89:;  >
                2222222222222222222222222222222222222222222222223323333233332232
                000000000000000000000000000000000123400089ABC0000103456089AB00E0
                ********************************     ***     ***  *    *    ** *
             64  ABCDEFGHI .<(+!&QRSTUVWXY!$*);^-/ cdefghi ,%_>?p rs vwxy`:#@'="
                2444444444223222255555555522223522266666662225337277277776324232
                01234567890EC8B1612345678914A9BEDF034567890C5FEF0023067890A307D2
                          *    *                  *       *      *  *
            128  abcdefghi       jklmnopqr       ~stuvwxyz   [               ]
                2666666666222222266666677722222227777777772225222222222222222522
                01234567890000000ABCDEF0120000000E3456789A000B000000000000000D00
                *         *******         *******         *** *************** **
            192 {ABCDEFGHI      }JKLMNOPQR      \ STUVWXYZ      0123456789
                7444444444222222744444455522222252555555552222223333333333222222
                B123456789000000DABCDEF012000000C03456789A0000000123456789000000
                          ******          ****** *        ******          ******
 ==================== EOF or StopPrint/StopRead count reached  ==============
 F1Count=1, F2Count=1, StopPrint=6, StopRead=1000000
 F1Reads=1, MisMatches=1, MisMatsPrinted=1, Recsize=256

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7H1. Test Conversion ASCII/EBCDIC & Delimit/Undelimit

test translate ASCII to EBCDIC & back to EBCDIC

We will generate jobs to translate ASCII to EBCDIC & back to ASCII, execute, & prove the re-converted ASCII file same as original ASCII file. We will also generate pipe delimit/undelimit jobs, execute them, & prove the ASCII characters x'20' - x'7E' are the same.

Note
  • The procedures are little different than the above where we considered
    the input file (Ebytes) to be an EBCDIC file, but the results look
    quite different here when we consider the input file (Abytes) to be ASCII.
  • Because EBCDIC characters are x'40' - x'F9', but ASCII characters are
    x'20' - x'7E'

 #1. gencnvA Abytes   <-- generates all conversion jobs
     ==============       pfx1,pfy1,pfq1,pfp1
                        - see #comments doc in $UV/sf/IBM/gencnvA

 #2. uvcopy pfy1/Abytes  <-- execute ASCII to EBCDIC, d2asc --> d3ebc
     ==================

 #3. uvcopy pfx1/Abytes,fili1=d3ebc/Abytes,filo1=d4asc/Abytes
     ========================================================
     - execute EBCDIC to ASCII, d3ebc --> d4asc
     - must specify fili1=d3ebc/... & filo1=d4asc/..., since
       defaults are fili1=d1ebc/... & filo1=d2asc/...

 #4. uvcmpFA1 d2asc/Abytes d4asc/Abytes r256
     =======================================
     - compare original EBCDIC file to reconverted file to see any diff
     - x'20' - x'7E' (EBCDIC characters) are the same
     - x'00' - x'1F' & x'7F' - x'FF' have differences, but should not matter
     - could modify translate tables to make more similar if required

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7H2. Test Conversion ASCII/EBCDIC & Delimit/Undelimit


 #5. use uvhd to display results in vertical hexadecimal

 #5a. uvhd d2asc/Abytes r256   <-- show ASCII input file in hexadecimal
      =======================
      --> p1  <-- capture in text file tmp/Abytes_date/timeP ASCII input

 #5b. uvhd d3ebc/Abytes r256a  <-- show EBCDIC output file in hexadecimal
      =======================    - note option 'a' show chars in ASCII
                                 - zones & digits remain EBCDIC codes
      --> p1  <-- capture in text file tmp/Abytes_date/timeP EBCDIC output

 #5c. uvhd d4asc/Abytes r256   <-- show reconverted ASCII file in hexadecimal
      ======================
      --> p1  <-- capture in text file tmp/Abytes_date/timeP reconverted file

 #5d. vi tmp/*   <-- investigate uvhd hex displays
      ========     - satisfy yourself reconversion same as original
                   - at least in the ASCII character range x'20' - x'7E'

 #6. OR see uvhd hex-rep files renamed & saved in subdir uvhd/...

 #6a. vi uvhd/uvhd_Abytes_d2asc
      =========================
 #6b. vi uvhd/uvhd_Abytes_d3ebc
      =========================
 #6c. vi uvhd/uvhd_Abytes_d4asc
      =========================

 #7. uvcmpFA1 d2asc/Abytes d4asc/Abytes r256
     =======================================
     - compare original ASCII to after conversion to EBCDIC & back to ASCII

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7I1. Test Conversion ASCII/EBCDIC & Delimit/Undelimit

RESULTS - d2asc(ASCII), d3ebc(EBCDIC), d4asc(ASCII)


 uvhd d2asc/Abytes r256      mvstest1/cnvtran/uvhd/uvhd_Abytes_d2asc
 ======================
 rec#=1 rsize=256 fptr=0 fsize=256 rcount=1
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ................................ !"#$%&'()*+,-./0123456789:;<=>?
             0000000000000000111111111111111122222222222222223333333333333333
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
          64 @ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^abcdefghijklmnopqrstuvwxyz{|}~.
             4444444444444444555555555555555566666666666666667777777777777777
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
         128 ................................................................
             88888888888888889999999999999999AAAAAAAAAAAAAAAABBBBBBBBBBBBBBBB
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
         192 ................................................................
             CCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEFFFFFFFFFFFFFFFF
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF

 uvhd d3ebc/Abytes r256a     mvstest1/cnvtran/uvhd/uvhd_Abytes_d3ebc
 =======================
 rec#=1 rsize=256 fptr=0 fsize=256 rcount=1
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ................................ !"#$%&'()*+,-./0123456789:;<=>?
             000032221020000011113332113211114577565745546646FFFFFFFFFF754766
             01237DEF655BCDEF0123CD2689F7CDEF0AFBBC0DDDCEB0B10123456789AECEEF
          64 @ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^abcdefghijklmnopqrstuvwxyz{|}~.
             7CCCCCCCCCDDDDDDDDDEEEEEEEEAEB567888888888999999999AAAAAAAAC4DA0
             C12345678912345678923456789D0DFD9123456789123456789234567890F017
         128 .abcdefghi.......jklmnopqr.......~.tuvwxyz...[...............]..
             88888888888888889999999999999999AA4AAAAAAAAAAAAABBBBBBBBBBBBBBBB
             0123456789ABCDEF0123456789ABCDEF01A3456789ABCDEF0123456789ABCDEF
         192 {A.CDEFG........}JKLMNOPQR......\.STUVWXYZ......0123456789......
             CC6CCCCC77CCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEFFFFFFFFFFFFFFFF
             0123456741ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF

 uvhd d4asc/Abytes r256      mvstest1/cnvtran/uvhd/uvhd_Abytes_d4asc
 ======================
 rec#=1 rsize=256 fptr=0 fsize=256 rcount=1
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ................................ !"#$%&'()*+,-./0123456789:;<=>?
             0000000000000000111111111111111122222222222222223333333333333333
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
          64 @ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^abcdefghijklmnopqrstuvwxyz{|}~.
             4444444444444444555555555555555566666666666666667777777777777777
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
         128 ................................................................
             88888888888888889999999999999999AAAAAAAAAAAAAAAABBBBBBBBBBBBBBBB
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
         192 ................................................................
             CCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEFFFFFFFFFFFFFFFF
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7I2. Test Conversion ASCII/EBCDIC & Delimit/Undelimit

compare reconverted ASCII to original ASCII


 #7. uvcmpFA1 d2asc/Abytes d4asc/Abytes r256
     =======================================
 uvcmp1 - compare 2 files, print mismatched records, '*' flag diffs
 2014/12/24_20:24:34 uop=q1p30r256s6t1000000u3x2y0q1r256u0
 recsize  reccount   file-size  typ Report=rptcmp/Abytes
 1:  256         1         256  RSF  File1=d2asc/Abytes
 2:  256         1         256  RSF  File2=d4asc/Abytes
                       1         2         3         4         5         6
 f#record#byte# 0123456789012345678901234567890123456789012345678901234567890123
 ===============================================================================
 1       1    0 ................................ !"#$%&'()*+,-./0123456789:;<=>?
                0000000000000000111111111111111122222222222222223333333333333333
                0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
             64 @ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^abcdefghijklmnopqrstuvwxyz{|}~.
                4444444444444444555555555555555566666666666666667777777777777777
                0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
            128 ................................................................
                88888888888888889999999999999999AAAAAAAAAAAAAAAABBBBBBBBBBBBBBBB
                0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
                 *********       *********       * *******   *               *
            192 ................................................................
                CCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEFFFFFFFFFFFFFFFF
                0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
                ** *****        **********      * ********      **********
 2       1      ................................ !"#$%&'()*+,-./0123456789:;<=>?
                0000000000000000111111111111111122222222222222223333333333333333
                0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
             64 @ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^abcdefghijklmnopqrstuvwxyz{|}~.
                4444444444444444555555555555555566666666666666667777777777777777
                0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
            128 .abcdefghi.......jklmnopqr.......~.tuvwxyz...[...............]..
                86666666668888889666666777999999A7A7777777AAA5AABBBBBBBBBBBBB5BB
                0123456789ABCDEF0ABCDEF012ABCDEF0E2456789AABCBEF0123456789ABCDEF
                 *********       *********       * *******   *               *
            192 {A.CDEFG........}JKLMNOPQR......\.STUVWXYZ......0123456789......
                74C44444CCCCCCCC7444444555DDDDDD5E55555555EEEEEE3333333333FFFFFF
                B123456789ABCDEFDABCDEF012ABCDEFC13456789AABCDEF0123456789ABCDEF
                ** *****        **********      * ********      **********
 ==================== EOF or StopPrint/StopRead count reached  ==============
 F1Count=1, F2Count=1, StopPrint=6, StopRead=1000000
 F1Reads=1, MisMatches=1, MisMatsPrinted=1, Recsize=256

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7J1. Test Conversion ASCII/EBCDIC & Delimit/Undelimit

test ASCII Fixed to Delimited & back to Fixed

Using input d2asc/Abytes created above in translate tests

Note
  • This (considering input file to be ASCII) is similar to the above
    where we considered the origninal input file to be EBCDIC.
  • Our delimited files are always ASCII, we probably do not need
    to create EBCDIC delimited files for FTP back to mainframe.

 #1. uvcopy pfp1/Abytes,fili1=d2asc/Abytes,filo1=d2pipe/Abytes
     =========================================================
     - convert ASCII fixed-field to ASCII pipe|delimited

 #1a. uvcopy pfp1/Abytes  <-- same, files default as above
      ==================

 #2. uvcopy pfq1/Abytes,fili1=d4pipe/Abytes,filo1=d5fix/Abytes
     =========================================================
     - convert ASCII pipe|delimited back to fixed-field ASCII

 #2a. uvcopy pfq1/Abytes  <-- same, files default as above
      ==================

 #3. listpipeA Abytes     <-- create vertical listing of pipe|delim fields
     ================

 #3a. vi listpipe/Abytes  <-- view results
      ==================

 #4. uvcmpFA1 d2asc/Abytes d5fix/Abytes r256
     =======================================
     - compare original fixed-field to reconverted fixed-field from pipe|delimited

 #4a. vi rptcmp/Abytes    <-- view results
      ================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7K1. Test Conversion ASCII/EBCDIC & Delimit/Undelimit

delimit & undelimit results


 #2. vi d4pipe/Abytes    <-- inspect pipe|delimited file with vi
     ================      - not good, try listpipe below
Note
  • Looking at the pipe|delimited file is ugly, and not very relevant
    since we are using a test file of all byte codes vs meaningful data.
  • It is more relevant to see the next page conversion back to fixed field
    to see which codes are cleared wwhen delimiting character fields

 #3. listpipeA Abytes       <-- create vertical listing of pipe|delim fields
     ================

 #3a. vi listpipe/Abytes     <-- view listpipe results
      ==================
 listpipe1 - list '|' delimited files (from genpipe1) with copybook fieldnames
 DataFile=d4pipe/Abytes, RecSize=00256, RecCount=1
 CopyBook=maps/Abytes, DateTime=2014/12/24_20:27:16
 Seq# FieldName       OffSet COBOL picture      DataFieldContents
 DataFile = d4pipe/Abytes, Record# = 1 of 1, Fields = 16
  | | !"#$%&'()*+,-./|0123456789:;<=>?|@ABCDEFGHIJKLMNO|PQRSTUVWXYZ[\]^_|`abcdefg
 hijklmno|pqrstuvwxyz{!}~ | | | | | | | | |
 001 b00-0f                        000 x(16).
 002 b10-1f                        016 x(16).
 003 b20-2f                        032 x(16).                 !"#$%&'()*+,-./
 004 b30-3f                        048 x(16).                0123456789:;<=>?
 005 b40-4f                        064 x(16).                @ABCDEFGHIJKLMNO
 006 b50-5f                        080 x(16).                PQRSTUVWXYZ[\]^_
 007 b60-6f                        096 x(16).                `abcdefghijklmno
 008 b70-7f                        112 x(16).                pqrstuvwxyz{!}~
 009 b80-8f                        128 x(16).
 010 b90-9f                        144 x(16).
 011 ba0-af                        160 x(16).
 012 bb0-bf                        176 x(16).
 013 bc0-cf                        192 x(16).
 014 bd0-df                        208 x(16).
 015 be0-ef                        224 x(16).
 016 bf0-ff                        240 x(16).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7K2. Test Conversion ASCII/EBCDIC & Delimit/Undelimit

compare BEFORE conversion to delimited & back


 uvhd d2asc/Abytes r256      mvstest1/cnvtran/uvhd_Abytes_d2asc
 ======================
 rec#=1 rsize=256 fptr=0 fsize=256 rcount=1
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ................................ !"#$%&'()*+,-./0123456789:;<=>?
             0000000000000000111111111111111122222222222222223333333333333333
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
          64 @ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^abcdefghijklmnopqrstuvwxyz{|}~.
             4444444444444444555555555555555566666666666666667777777777777777
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
         128 ................................................................
             88888888888888889999999999999999AAAAAAAAAAAAAAAABBBBBBBBBBBBBBBB
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
         192 ................................................................
             CCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEFFFFFFFFFFFFFFFF
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF

compare AFTER conversion to delimited & back


 uvhd d5fix/Abytes r256      mvstest1/cnvtran/uvhd_Abytes_d5fix
 ======================
 rec#=1 rsize=256 fptr=0 fsize=256 rcount=1
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0                                  !"#$%&'()*+,-./0123456789:;<=>?
             2222222222222222222222222222222222222222222222223333333333333333
             000000000000000000000000000000000123456789ABCDEF0123456789ABCDEF
          64 @ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^abcdefghijklmnopqrstuvwxyz{!}~
             4444444444444444555555555555555566666666666666667777777777772772
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789AB1DE0
         128
             2222222222222222222222222222222222222222222222222222222222222222
             0000000000000000000000000000000000000000000000000000000000000000
         192
             2222222222222222222222222222222222222222222222222222222222222222
             0000000000000000000000000000000000000000000000000000000000000000
Note
  • control characters in 0-31 < x'20' are converted to spaces x'20'
    when converting to delimited
  • hi-bit chars in 128-255 are converted to blanks when delimiting ASCII
  • you can see that most ASCII characters in 31-63 & 64-127 are the same
    before & after conversion to delimited & back
  • BUT, there are a few differences
  • see next page 'uvcmpFA1' that flags differences with '*'s

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7K3. Test Conversion ASCII/EBCDIC & Delimit/Undelimit

compare fixed-field before delimit to fixed-field after undelimit


 uvcmpFA1 d2asc/Ebytes d5fix/Ebytes r256
 =======================================
 uvcmp1 - compare 2 files, print mismatched records, '*' flag diffs
 2014/12/24_20:27:40 uop=q1p30r256s6t1000000u3x2y0q1r256u0
 recsize  reccount   file-size  typ Report=rptcmp/Abytes
 1:  256         1         256  RSF  File1=d2asc/Abytes
 2:  256         1         256  RSF  File2=d5fix/Abytes
                       1         2         3         4         5         6
 f#record#byte# 0123456789012345678901234567890123456789012345678901234567890123
 ===============================================================================
 1       1    0 ................................ !"#$%&'()*+,-./0123456789:;<=>?
                0000000000000000111111111111111122222222222222223333333333333333
                0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
                ********************************
             64 @ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^abcdefghijklmnopqrstuvwxyz{|}~.
                4444444444444444555555555555555566666666666666667777777777777777
                0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
                                                                            *  *
            128 ................................................................
                88888888888888889999999999999999AAAAAAAAAAAAAAAABBBBBBBBBBBBBBBB
                0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
                ****************************************************************
            192 ................................................................
                CCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEFFFFFFFFFFFFFFFF
                0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
                ****************************************************************
 2       1                                       !"#$%&'()*+,-./0123456789:;<=>?
                2222222222222222222222222222222222222222222222223333333333333333
                000000000000000000000000000000000123456789ABCDEF0123456789ABCDEF
                ********************************
             64 @ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^abcdefghijklmnopqrstuvwxyz{!}~
                4444444444444444555555555555555566666666666666667777777777772772
                0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789AB1DE0
                                                                            *  *
            128
                2222222222222222222222222222222222222222222222222222222222222222
                0000000000000000000000000000000000000000000000000000000000000000
                ****************************************************************
            192
                2222222222222222222222222222222222222222222222222222222222222222
                0000000000000000000000000000000000000000000000000000000000000000
                ****************************************************************
 ==================== EOF or StopPrint/StopRead count reached  ==============
 F1Count=1, F2Count=1, StopPrint=6, StopRead=1000000
 F1Reads=1, MisMatches=1, MisMatsPrinted=1, Recsize=256

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7K4. Test Conversion ASCII/EBCDIC & Delimit/Undelimit

Notes re differences before/after delimiting ASCII

The above uvcmpFA1 comparison of ASCII files before & after pipe|delimiting shows 2 differences.

In the before record x'7C' is a '|', but in the after record we see a '!'.

This is because it would be a problem if we converted a '|' in existing data to a '|' in the output file - it would shift following fields when attempting to extract data from the pipe|delimited file.

The solution is to convert any '|'s in existing data to '!'s. This is the default, but you can specify arg2=X when you create the delimited file. For example, to change the default '!' to '@' you would run the delimt job on page '7J1' as follows:


 #1. uvcopy pfp1/Abytes,arg2=@
     =========================

The other difference is insignificant, the x'7F' DELete character is converted to space x'20'.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7L1. Checking Data File Conversions

checking data file conversions with 'uvhd'

First we will check data file conversions with 'uvhd' (vertical hexadecimal dumps of data records) & later we will check with 'uvhdcob' which shows the COBOL copybook names beside the data field contents.

You can also convert the files back to EBCDIC & compare to the original FTP. See http://uvsoftware.ca/datacnv1.htm#Part_9. Utility 'uvdat55' (page 9A2) copies conversion jobs from pfx4/* to pfx5/... reversing the translate. Then use utility 'uvcmp1' to compare the 2 EBCDIC files showing any mismatched record pairs in character & hexadecimal with differences highlighted by '*'s (see page http://uvsoftware.ca/datacnv1.htm#9B2).

uvhd check ar.customer.armaster - EBCDIC file


 #0a. Login userxx --> /home/userxx/
 #0b. cdc (alias cdc='cd $CNVDATA') --> /home/userxx/cnvdata1/

 #1. uvhd d1ebc/ar.customer.armaster r256a  <-- check EBCDIC file
     =====================================
     - option 'r' (r256) for Record size
     - option 'a' to translate character line1 to ASCII so we can read
     - zones & digits on lines 2&3 (of each 64 byte segment) show EBCDIC values
     records=32  rsize=256 fsize=8192
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    EVERGREEN MOTORS LTD.    1815 BOWEN ROAD          NANA
             FFFFFF4444CECDCDCCD4DDEDDE4DEC44444FFFF4CDECD4DDCC4444444444DCDC
             130140000055597955504636920334B000018150266550961400000000005151
          64 IMO          BC V9S1H1    250-754-5531 LARRY WRENCH     ...V@...
             CDD4444444444CC4EFEFCF4444FFF6FFF6FFFF4DCDDE4EDCDCC4444401357000
             946000000000023059218100002500754055310319980695538000000246C000
         128 .........W.....-.........X'..f.....m*..................V'...f...
             0000000005300016000000002570063100095000000000000000013570016000
             0C0000C0270D0540C0000C0098D0263C0444C0000C0000C0000C0246D0056C00
         192 ......f.........c..............f..............f.C 19950531
             000000890000000088000000001800068000130000000069C4FFFFFFFF444444
             00C0026C0000C0023C0000C0083C0056D0012C0000C0016D3019950531000000

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7L2. checking data file conversions with 'uvhd'

uvhd check ar.customer.armaster - ASCII file


 #2. uvhd data1/ar.customer.armaster r257  <-- check ASCII file
     ====================================    - Indexed IDXFORMAT1 (fixed-length)
     - option 'r257' Recsize, ONE greater for delete flag at end each record
     - option 'a' omitted since now already ASCII
     - zones & digits on lines 2&3 (of each 64 byte segment) show ASCII values
     records=32  rsize=257 fsize=8224
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    EVERGREEN MOTORS LTD.    1815 BOWEN ROAD          NANA
             3333332222454545444244545524542222233332445442544422222222224444
             130140000056527255E0DF4F230C44E0000181502F75E02F140000000000E1E1
          64 IMO          BC V9S1H1    250-754-5531 LARRY WRENCH     ..4V|...
             4442222222222442535343222233323332333324455525544442222201357000
             9DF00000000002306931810000250D754D55310C12290725E38000000246C000
         128 .........W0....`........)X}..f3.....\.................4V}...f...
             0000000005300016000000002570063100095000000000000000013570016000
             0C0000C0270D0540C0000C0098D0263C0444C0000C0000C0000C0246D0056C00
         192 ...............................f.....<........f.C 19950531
             0000008900000000880000000018000680001300000000694233333333222222
             00C0026C0000C0023C0000C0083C0056D0012C0000C0016D3019950531000000
         256 .
             0
             A

checking EBCDIC to ASCII conversion

  1. The 'a' option (show ASCII translation of EBCDIC file) of uvhd allows you to compare the character fields visually.

  2. You can see the hex zones & digits (vertically below the characters) show the true data values for EBCDIC & ASCII. For example the 1st byte in the EBCDIC record '1' is x'F1' in EBCDIC, but x'31' in ASCII & the 10th byte 'E' is x'C5' in EBCDIC, but x'45' in ASCII.

  3. The important thing is that the packed fields are unchanged as required both for COBOL on the mainframe & for Micro Focus or AIX COBOL on unix/linux. Note that the data conversion jobs translate the whole record, but then preserve the Packed/Binary fields by moving them back guided by the copybook.

  4. For example, the 1st packed field is at zero relative bytes 120-124, and is x'001234567C' for both EBCDIC & ASCII. To calculate the position of byte 120 on the 2nd line, subtract 1st line 64 bytes (120 - 64 = 56) and look down from the scale at 56 to the 2nd line.

  5. You will see that the characters above the packed fields are '...V@' for EBCDIC, but '..V4|' for ASCII - because the translations of packed field contents just happen to result in some readable characters, but most are unreadable & shown as '.'s.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7L3. checking data file conversions with 'uvhd'

uvhd check city.tax.payments - EBCDIC file


 #1. uvhd d1ebc/city.tax.payments z4a
     ================================
     - option 'z4' RDW variable length files with 4 byte record prefixes
     - option 'a' to translate character line (zones/digits show EBCDIC values)
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 .T..        H December 31, 1998
             050044444444C4C88898894FF64FFFF444444444444444444444444444444444
             0400000000008045354259031B01998000000000000000000000000000000000
          64
             44444444444444444444
             00000000000000000000
 rec#=1 rsize=84 fptr=0 fsize=1680 records=16
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
          84 .d..10130140T JOHN HENRY               1815 BOWEN ROAD
             0800FFFFFFFFE4DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444
             0400101301403016850855980000000000000001815026655096140000000000
          64 VANCOUVER           BCV9S1H1..CK..Wb....g......000149061970530
             ECDCDEECD44444444444CCEFEFCF004D005880008000770FFFFFFFFFFFFFFF44
             515364559000000000002359218101320072C0047C0111C00014906197053000
         128
             4444
             0000
 rec#=2 rsize=132 fptr=84 fsize=1680 records=16
                      10        20        30        40        50        60
 r#        3 0123456789012345678901234567890123456789012345678901234567890123
         216 .T..10130140P       1999...A....A....A....A....A....A....A....A.
             0500FFFFFFFFD4444444FFFF0004200042000420004200042000420004200042
             0400101301407000000019990001C0001C0001C0001C0001C0001C0001C0001C
          64 ...A....A....A....A.
             00042000420004200042
             0001C0001C0001C0001C
  1. First 4 bytes of each record is the RDW prefix with record-size in binary in 1st 2 bytes (bytes 3&4 unused nulls)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7L4. checking data file conversions with 'uvhd'

uvhd check city.tax.payments - ASCII file


 #2. uvhd data1/city.tax.payments v5
     ===============================
     - option 'v5' Micro Focus COBOL variable length files with 2 byte record
       prefixes (4 if record-size > 4095) & 128 byte File-Header record
     - option 'v4' would show only the data records (default if just 'v')
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 0~......170329072520.................>..........................
             3700000033333333333300000000000000000300000000000000000008000800
             0E00000017032907252000000000000000000E01000000001000000000000000
          64 ................................................................
             0000000000000000000000000000000000000000000000000000000000000000
             0000000000000000000000000000000000000000000000000000000000000000
 rec#=1 rsize=128 fptr=0 fsize=1808 records=17
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 @P        H December 31, 1998
             4522222222424666666723322333322222222222222222222222222222222222
             0000000000804535D252031C0199800000000000000000000000000000000000
          64                   ..
             22222222222222222200
             00000000000000000000
 rec#=2 rsize=84 fptr=128 fsize=1808 records=17
                      10        20        30        40        50        60
 r#        3 0123456789012345678901234567890123456789012345678901234567890123
         212 @.10130140T JOHN HENRY               1815 BOWEN ROAD          VA
             4833333333524444244455222222222222222333324454425444222222222254
             001013014040AF8E085E29000000000000000181502F75E02F14000000000061
          64 NCOUVER           BCV9S1H1..C...W.........qq.000149061970530
             44455452222222222244535343004D0058800080007703333333333333332222
             E3F5652000000000002369318101320072C0047C0111C0001490619705300000
         128   ..
             2200
             0000
 rec#=3 rsize=132 fptr=212 fsize=1808 records=17
                      10        20        30        40        50        60
 r#        4 0123456789012345678901234567890123456789012345678901234567890123
         344 @P10130140P       1999...A,...A,...A,...A,...A,...A,...A,...A,..
             4533333333522222223333000420004200042000420004200042000420004200
             00101301400000000019990001C0001C0001C0001C0001C0001C0001C0001C00
          64 .A,...A,...A,...A,..
             04200042000420004200
             01C0001C0001C0001C00

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7L5. checking data file conversions with 'uvhd'

Notes re: Micro Focus COBOL variable length records

  1. The Micro Focus Variable length prefixes uses the 1st nibble (half byte) for a record ID x'4_' identifies a DATA record (vs x'3_' for the File Header)

  2. The record size is in the next 3 nibbles, for city.tax.payments above: System file header x'_080' = 128 bytes, User file Header x'_050' = 80 bytes, Tax record x'_080' = 128 bytes, Payment record x'_050' = 80 bytes

  3. Note that record-size in the prefix does not include the prefix-size (unlike RDW files) & does not include record padding to bring slot size up to a multiple of 4 bytes

  4. For example record #2 user Header record slot size is 84 bytes - 2 byte header, 80 bytes data,& 2 bytes padding x'0000' at end record.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7L6. Checking Data File Conversions

checking data file conversions with 'uvhdcob'

Now we will check data file conversions with 'uvhdcob' which shows the COBOL copybook field-names beside the data field contents. uvhdcob makes it easy to see if anything has gone wrong in the conversion, such as the wrong COBOL copybook, which would cause the data field contents to be misaligned with the field-names.

For uvhdcob, you must specify the COBOL copybook as the 2nd argument following the datafilename & any options as the 3rd argument. For uvhdcob, you do not need to specify option 'r' (Record-size) since that is known from the copybook.

uvhdcob check ar.customer.armaster - EBCDIC file


 #1. uvhdcob d1ebc/ar.customer.armaster maps/armaster a
     ==================================================
     - can omit option 'r' (Record-size) since known from copybook
     - need option 'a' to translate character line to Ascii
 now=20170329:1051 uvhdcob d1ebc/ar.customer.armaster m25a
 version=20161001 copybook=maps/armaster
 rec#=1 rcount=32 rsize=256 fsize=8192 fptr=0
 rec#       1           occurs  bgn  end typ <------ data (hex if typ=p/b) ----->
 001 armaster                      0    0
 002 cm-cust                       0    5 n  130140
 003 cm-delete                     6    9
 004 cm-name                      10   34    EVERGREEN MOTORS LTD.
 005 cm-adrs                      35   59    1815 BOWEN ROAD
 006 cm-city                      60   75    NANAIMO
 007 filler001                    76   76
 008 cm-prov                      77   78    BC
 009 filler002                    79   79
 010 cm-postal                    80   89    V9S1H1
 011 cm-telephone                 90  101    250-754-5531
 012 cm-contact                  102  119     LARRY WRENCH
 013 cm-thisyr-sales      1/12   120  124pns 001234567C
 014 cm-lastyr-sales      1/12   180  184pns 001234567D
 015 filler003                   240  255    C 19950531
 rec#=1 rcount=32 rsize=256 fsize=8192 fptr=0

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7L7. checking data file conversions with 'uvhdcob'

uvhdcob check ar.customer.armaster - ASCII file


 #2. uvhdcob data1/ar.customer.armaster maps/armaster  <-- check ASCII file
     ================================================    - Indexed IDXFORMAT1
     - uvhdcob needs no options for fixed length ASCII files
 now=20170329:1052 uvhdcob data1/ar.customer.armaster.dat m25
 version=20161001 copybook=maps/armaster
 rec#=1 rcount=32 rsize=257 fsize=8224 fptr=0
 rec#       1           occurs  bgn  end typ <------ data (hex if typ=p/b) ----->
 001 armaster                      0    0
 002 cm-cust                       0    5 n  130140
 003 cm-delete                     6    9
 004 cm-name                      10   34    EVERGREEN MOTORS LTD.
 005 cm-adrs                      35   59    1815 BOWEN ROAD
 006 cm-city                      60   75    NANAIMO
 007 filler001                    76   76
 008 cm-prov                      77   78    BC
 009 filler002                    79   79
 010 cm-postal                    80   89    V9S1H1
 011 cm-telephone                 90  101    250-754-5531
 012 cm-contact                  102  119     LARRY WRENCH
 013 cm-thisyr-sales      1/12   120  124pns 001234567C
 014 cm-lastyr-sales      1/12   180  184pns 001234567D
 015 filler003                   240  255    C 19950531
 rec#=1 rcount=32 rsize=257 fsize=8224 fptr=0

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7L8. checking data file conversions with 'uvhdcob'

uvhdcob check city.tax.payments - EBCDIC file


 #1. uvhdcob d1ebc/city.tax.payments maps/citytax2 z4a
     =================================================
     - option 'z4' RDW variable length files with 4 byte record prefixes
     - option 'a' to translate character line (zones/digits show EBCDIC values)

----> output to be supplied

7L9. uvhdcob check city.tax.payments - ASCII file


 #2. uvhdcob data1/city.tax.payments maps/citytax2 v5
     ================================================
     - option 'v5' Micro Focus COBOL variable length files with 2 byte record
       prefixes (4 if record-size > 4095) & 128 byte File-Header record

----> output to be supplied

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7M1. Test Conversion EBCDIC/ASCII

ebc2asc.c - EBCDIC to ASCII translate table

This is the EBCDIC to ASCII translate table used by the Vancouver Utiltiies (uvcopy, uvsort, uvcp, etc).

 /* ebc2asc.c - EBCDIC to ASCII translation table     */
 /*           - Owen Townsend, UV Software, June 1995 */
 /* similar to ISO 8859-1, but defines some control chars undefined by ISO 8859*/
 unsigned char ebc2asc[256] = {
 0x00,0x01,0x02,0x03,0x04,0x09,0x06,0x7f,0x08,0x09,0x0a,0x0b,0x0c,0x0d,0x0e,0x0f,
 0x10,0x11,0x12,0x13,0x14,0x15,0x08,0x17,0x18,0x19,0x1a,0x1b,0x1c,0x1d,0x1e,0x1f,
 0x20,0x21,0x22,0x23,0x24,0x0a,0x17,0x1b,0x28,0x29,0x2a,0x2b,0x2c,0x05,0x06,0x07,
 0x30,0x31,0x16,0x33,0x34,0x35,0x36,0x04,0x38,0x39,0x3a,0x3b,0x14,0x15,0x3e,0x1a,
  ' ',0x41,0x42,0x43,0x44,0x45,0x46,0x47,0x48,0x49,0xA2, '.', '<', '(', '+', '|',
  '&',0x51,0x52,0x53,0x54,0x55,0x56,0x57,0x58,0x59, '!', '$', '*', ')', ';', '^',
  '-', '/',0xC2,0x63,0x64,0x65,0x66,0x67,0x68,0x69,0xA6, ',', '%', '_', '>', '?',
 0x70,0xC9,0x72,0x73,0xC8,0x76,0x77,0x78,0x79, '`', ':', '#', '@',0x27, '=', '"',
 0x80, 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i',0x8a,0x8b,0x8c,0x8d,0x8e,0x8f,
 0x90, 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r',0x9a,0x9b,0x9c,0x9d,0x9e,0x9f,
 0xa0, '~', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z',0xaa,0xab,0xac,'[',0xae,0xaf,
 0xb0,0xb1,0xb2,0xb3,0xb4,0xb5,0xb6,0xb7,0xb8,0xb9,0xba,0xbb,0xbc,']',0xbe,0xbf,
  '{', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I',0xca,0xcb,0xcc,0xcd,0xce,0xcf,
  '}', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R',0xda,0xdb,0xdc,0xdd,0xde,0xdf,
 0x5c,0xe1, 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z',0xea,0xeb,0xec,0xed,0xee,0xef,
  '0', '1', '2', '3', '4', '5', '6', '7', '8', '9',0xfa,0xfb,0xfc,0xfd,0xfe,0xff
 };
 /* EBCDIC   '|' - ARC x'6A', Oracle x'BB', Simotime x'4F'       */
 /* Oracle   '|' = x'BB', '[' = x'4A', ']' = x'5A', '!' = x'4F'  */
 /* Simotime '|' = x'4F', '[' = x'BA', ']' = x'BB', '!' = x'5A'  */
 /*Sep02/12 - add accented characters: A circumflex x'62'-x'C2', */
 /*           E acute x'71'-x'C9', E grave x'74'-x'C8'           */
 /*Mar14/13 - change x'4F' from ']' to '|' for Cothern           */
 /*Apr22/13 - change x'6A' from '|' to x'A6' (simotime alt '|')  */
 /*Dec24/14 - add '[' & ']' 0xad & 0xbd code page 1047 IMMD      */
 /*         - 0x4A EBCDIC cent sign to 0xA2 ASCII cent (vs '[')  */

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7M2. Test Conversion EBCDIC/ASCII

asc2ebc.c - ASCII to EBCDIC translate table

This is the ASCII to EBCDIC translate table used by the Vancouver Utiltiies (uvcopy, uvsort, uvcp, etc).

 /* asc2ebc.c - ASCII to EBCDIC translation table     */
 /*           - Owen Townsend, UV Software, June 1995 */
 /* similar to ISO 8859-1, but defines some control chars undefined by ISO 8859*/
 unsigned char asc2ebc[256] = {
 0x00,0x01,0x02,0x03,0x37,0x2d,0x2e,0x2f,0x16,0x05,0x25,0x0b,0x0c,0x0d,0x0e,0x0f,
 0x10,0x11,0x12,0x13,0x3c,0x3d,0x32,0x26,0x18,0x19,0x3f,0x27,0x1c,0x1d,0x1e,0x1f,
 0x40,0x5a,0x7f,0x7b,0x5b,0x6c,0x50,0x7d,0x4d,0x5d,0x5c,0x4e,0x6b,0x60,0x4b,0x61,
 0xf0,0xf1,0xf2,0xf3,0xf4,0xf5,0xf6,0xf7,0xf8,0xf9,0x7a,0x5e,0x4c,0x7e,0x6e,0x6f,
 0x7c,0xc1,0xc2,0xc3,0xc4,0xc5,0xc6,0xc7,0xc8,0xc9,0xd1,0xd2,0xd3,0xd4,0xd5,0xd6,
 0xd7,0xd8,0xd9,0xe2,0xe3,0xe4,0xe5,0xe6,0xe7,0xe8,0xe9,0xAD,0xe0,0xBD,0x5f,0x6d,
 0x79,0x81,0x82,0x83,0x84,0x85,0x86,0x87,0x88,0x89,0x91,0x92,0x93,0x94,0x95,0x96,
 0x97,0x98,0x99,0xa2,0xa3,0xa4,0xa5,0xa6,0xa7,0xa8,0xa9,0xc0,0x4F,0xd0,0xa1,0x07,
 0x80,0x81,0x82,0x83,0x84,0x85,0x86,0x87,0x88,0x89,0x8A,0x8B,0x8C,0x8D,0x8E,0x8F,
 0x90,0x91,0x92,0x93,0x94,0x95,0x96,0x97,0x98,0x99,0x9A,0x9B,0x9C,0x9D,0x9E,0x9F,
 0xA0,0xA1,0x4A,0xA3,0xA4,0xA5,0xA6,0xA7,0xA8,0xA9,0xAA,0xAB,0xAC,0xAD,0xAE,0xAF,
 0xB0,0xB1,0xB2,0xB3,0xB4,0xB5,0xB6,0xB7,0xB8,0xB9,0xBA,0xBB,0xBC,0xBD,0xBE,0xBF,
 0xC0,0xC1,0x62,0xC3,0xC4,0xC5,0xC6,0xC7,0x74,0x71,0xCA,0xCB,0xCC,0xCD,0xCE,0xCF,
 0xD0,0xD1,0xD2,0xD3,0xD4,0xD5,0xD6,0xD7,0xD8,0xD9,0xDA,0xDB,0xDC,0xDD,0xDE,0xDF,
 0xE0,0xE1,0xE2,0xE3,0xE4,0xE5,0xE6,0xE7,0xE8,0xE9,0xEA,0xEB,0xEC,0xED,0xEE,0xEF,
 0xF0,0xF1,0xF2,0xF3,0xF4,0xF5,0xF6,0xF7,0xF8,0xF9,0xFA,0xFB,0xFC,0xFD,0xFE,0xFF
 };
 /* Nov2002 - asc2ebc.c modified x'80'- x'FF' now neutral    */
 /*         - asc2ebc2.c created all x'40' in x'80'-x'FF'    */
 /*Sep02/12 - add accented chars: A circumflex x'C2'-x'62',  */
 /*           E acute x'C9'-x'71', E grave x'C8'-x'74'       */
 /*Oct14/14 - convert '|' x'7C' to x'4F (vs x'6A')           */
 /*           to match Mar14/13 ebc2asc x'4F' to '|' x'7C'   */
 /*Dec24/14 - change x'5B' to x'AD' '[' & x'5D' to x'BD' ']' */
 /*         - see reverse changes in ebc2asc.c               */
 /* see following notes in 'ebc2asc.c' applicable to this 'asc2ebc.c' */
 /*Mar14/13 - change x'4F' from ']' to '|' for Cothern          */
 /*Apr22/13 - change x'6A' from '|' to x'A6' (simotime alt '|') */
 /* EBCDIC   '|' - ARC x'6A', Oracle x'BB', Simotime x'4F'      */
 /* Oracle   '|' = x'BB', '[' = x'4A', ']' = x'5A', '!' = x'4F' */
 /* Simotime '|' = x'4F', '[' = x'BA', ']' = x'BB', '!' = x'5A' */
 /*Dec24/14 - add '[' & ']' 0xad & 0xbd code page 1047 IMMD     */
 /*         - see reverse changes above in this asc2ebc.c       */
 /*         - 0xA2 ASCII cent sign to 0x4A EBCDIC cent (vs '[') */

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7M3. Test Conversion EBCDIC/ASCII

ebc1141asc.c - ASCII to EBCDIC codepage 1141

 /* ebc1141asc.c - translate table, EBCDIC-1141 to ASCII ISO8859-1     */
 /*              - stored at $UV/srcf/ebc1141asc.c                     */
 /*              - Jan23/10, for Exigen, by Owen Townsend, UV Software */
 /*              - copied/renamed/modified ebc2asc.c to ebc1141asc.c   */
 /* - code page 1141 does not specify x'00' - x'3F'                    */
 /* - these codes x'00'-x'3F' same as ebc2asc.c originally from A/R    */
 unsigned char ebc1141asc[256] = {
 0x00,0x01,0x02,0x03,0x9C,0x09,0x86,0x7f,0x97,0x8D,0x0a,0x0b,0x0c,0x0d,0x0e,0x0f,
 0x10,0x11,0x12,0x13,0x9D,0x0A,0x08,0x87,0x18,0x19,0x92,0x8F,0x1c,0x1d,0x1e,0x1f,
 0x80,0x81,0x82,0x83,0x84,0x85,0x17,0x1b,0x88,0x89,0x8a,0x8b,0x8c,0x05,0x06,0x07,
 0x90,0x91,0x16,0x93,0x94,0x95,0x96,0x04,0x38,0x39,0x3a,0x3b,0x14,0x15,0x9e,0x1a,
 0x20,0xA0,0xE2,0x7B,0xE0,0xE1,0xE3,0xE5,0xE7,0xF1,0xC4,0x2E,0x3C,0x28,0x2B,0x21,
 0x26,0xE9,0xEA,0xEB,0xE8,0xED,0xEE,0xEF,0xEC,0x7E,0xDC,0x24,0x2A,0x29,0x3B,0x5E,
 0x2D,0x2F,0xC2,0x5B,0xC0,0xC1,0xC3,0xC5,0xC7,0xD1,0xF6,0x2C,0x25,0x5F,0x3E,0x3F,
 0xF8,0xC9,0xCA,0xCB,0xC8,0xCD,0xCE,0xCF,0xCC,0x60,0x3A,0x23,0xA7,0x27,0x3D,0x22,
 0xDB,'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i' ,0xAB,0xBB,0xF0,0xFD,0xFE,0xB1,
 0xB0,'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 0xAA,0xBA,0xE6,0xB8,0xC6,0xA4,
 0xB5,0xDF,'s', 't', 'u', 'v', 'w', 'x', 'y', 'z', 0xA1,0xBF,0xD0,0xDD,0xDE,0XAE,
 0xA2,0xA3,0xA5,0xB7,0xA9,0x40,0xB6,0xBC,0xBD,0xBE,0xAC,0x7C,0xAF,0xA8,0xB4,0xD7,
 0xE4, 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I',0xAD,0xF4,0xA6,0xF2,0xF3,0xF5,
 0xFC, 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R',0xB9,0xFB,0x7D,0xF9,0xFA,0xFF,
 0xD6,0xF7, 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z',0xB2,0xD4,0x5C,0xD2,0xD3,0xD5,
  '0', '1', '2', '3', '4', '5', '6', '7', '8', '9',0xB3,0xDB,0x5D,0xD9,0xDA,0x9F
 };
 /* see uvcopy job 'trantestE2A' to show results translating each of 256 characters*/
 /* - from EBCDIC to ASCII & back to EBCDIC                                        */
 /* uvcopy trantestE2A          <-- using ebc2asc.c original A/R table             */
 /* uvcopy trantestE2A,uop=t1   <-- using ebc1141asc.c (this code page 1141 table) */
 /* =========================                                                      */
 /* uvcopy trantestE2A,rop=t1   <-- OR use Run OPtion (vs User OPtion)             */
 /* =========================                                                      */
 /* export UVCOPYROP=t1         <-- could set codepage 1141 option for             */
 /* ===================             all uvcopy jobs until logout                   */

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7N1. Test Conversion EBCDIC/ASCII/EBCDIC

trantestEAE - test translate EBCDIC to ASCII & back to EBCDIC

Here is an alternate way to test Vancouver Utilities EBCDIC/ASCII translates. "trantestEAE' is a uvcopy job to translate EBCDIC to ASCII & back to EBCDIC. It requires no data file input, it adds binary to create each character of the 256 possible & translates each from EBCDIC to ASCII & back to EBCDIC.


 uvcopy trantestEAE    <-- generate 256 lines E-->A-->E
 ==================
 uvcopy trantestEAE,uop=t1  <-- option to use codepage 1141
 =========================
 uvcopy trantestEAE,rop=t1  <-- OR can use Run OPtion (vs User OPtion)
 =========================
 export UVCOPYROP=t1        <-- could set codepage 1141 option for
 ===================            all uvcopy jobs until logout
trantestEAE
  • test uvcopy translate EBCDIC to ASCII & back to EBCDIC
  • generates 256 lines translate results for each character
  • no input file required
  • writes report to tmp/trantest_results
 x'00' --> x'00' --> x'00'     c'^@' ---> c'^@' ---> c'^@'
 x'01' --> x'01' --> x'01'     c'^A' ---> c'^A' ---> c'^A'
 x'02' --> x'02' --> x'02'     c'^B' ---> c'^B' ---> c'^B'
 x'03' --> x'03' --> x'03'     c'^C' ---> c'^C' ---> c'^C'
 x'04' --> x'04' --> x'37'     c'^D' ---> c'^D' ---> c'7'
 x'05' --> x'09' --> x'05'     c'^E' ---> c'^I' ---> c'^E'
 x'06' --> x'06' --> x'2E'     c'^F' ---> c'^F' ---> c'.'
 x'07' --> x'7F' --> x'07'     c'^G' ---> c'' ---> c'^G'
 x'08' --> x'08' --> x'16'     c'^H' ---> c'^H' ---> c'^V'
 x'09' --> x'09' --> x'05'     c'^I' ---> c'^I' ---> c'^E'
 x'0A' --> x'0A' --> x'25'     c'^J' ---> c'^J' ---> c'%'
 x'0B' --> x'0B' --> x'0B'     c'^K' ---> c'^K' ---> c'^K'
 x'0C' --> x'0C' --> x'0C'     c'^L' ---> c'^L' ---> c'^L'
 x'0D' --> x'0D' --> x'0D'     c'^M' ---> c'^M' ---> c'^M'
 x'0E' --> x'0E' --> x'0E'     c'^N' ---> c'^N' ---> c'^N'
 x'0F' --> x'0F' --> x'0F'     c'^O' ---> c'^O' ---> c'^O'
 x'10' --> x'10' --> x'10'     c'^P' ---> c'^P' ---> c'^P'
 x'11' --> x'11' --> x'11'     c'^Q' ---> c'^Q' ---> c'^Q'
 x'12' --> x'12' --> x'12'     c'^R' ---> c'^R' ---> c'^R'
 x'13' --> x'13' --> x'13'     c'^S' ---> c'^S' ---> c'^S'
 x'14' --> x'14' --> x'3C'     c'^T' ---> c'^T' ---> c'<'
 x'15' --> x'15' --> x'3D'     c'^U' ---> c'^U' ---> c'='
 x'16' --> x'08' --> x'16'     c'^V' ---> c'^H' ---> c'^V'
 x'17' --> x'17' --> x'26'     c'^W' ---> c'^W' ---> c'&'
 x'18' --> x'18' --> x'18'     c'^X' ---> c'^X' ---> c'^X'
 x'19' --> x'19' --> x'19'     c'^Y' ---> c'^Y' ---> c'^Y'
 x'1A' --> x'1A' --> x'3F'     c'^Z' ---> c'^Z' ---> c'?'
 x'1B' --> x'1B' --> x'27'     c'^[' ---> c'^[' ---> c'''
 x'1C' --> x'1C' --> x'1C'     c'^\' ---> c'^\' ---> c'^\'
 x'1D' --> x'1D' --> x'1D'     c'^]' ---> c'^]' ---> c'^]'
 x'1E' --> x'1E' --> x'1E'     c'^^' ---> c'^^' ---> c'^^'
 x'1F' --> x'1F' --> x'1F'     c'^_' ---> c'^_' ---> c'^_'
 x'20' --> x'20' --> x'40'     c' ' ---> c' ' ---> c'@'
 x'21' --> x'21' --> x'5A'     c'!' ---> c'!' ---> c'Z'
 x'22' --> x'22' --> x'7F'     c'"' ---> c'"' ---> c''
 x'23' --> x'23' --> x'7B'     c'#' ---> c'#' ---> c'{'
 x'24' --> x'24' --> x'5B'     c'$' ---> c'$' ---> c'['
 x'25' --> x'0A' --> x'25'     c'%' ---> c'^J' ---> c'%'
 x'26' --> x'17' --> x'26'     c'&' ---> c'^W' ---> c'&'
 x'27' --> x'1B' --> x'27'     c''' ---> c'^[' ---> c'''
 x'28' --> x'28' --> x'4D'     c'(' ---> c'(' ---> c'M'
 x'29' --> x'29' --> x'5D'     c')' ---> c')' ---> c']'
 x'2A' --> x'2A' --> x'5C'     c'*' ---> c'*' ---> c'\'
 x'2B' --> x'2B' --> x'4E'     c'+' ---> c'+' ---> c'N'
 x'2C' --> x'2C' --> x'6B'     c',' ---> c',' ---> c'k'
 x'2D' --> x'05' --> x'2D'     c'-' ---> c'^E' ---> c'-'
 x'2E' --> x'06' --> x'2E'     c'.' ---> c'^F' ---> c'.'
 x'2F' --> x'07' --> x'2F'     c'/' ---> c'^G' ---> c'/'
 x'30' --> x'30' --> x'F0'     c'0' ---> c'0' ---> c'ð'
 x'31' --> x'31' --> x'F1'     c'1' ---> c'1' ---> c'ñ'
 x'32' --> x'16' --> x'32'     c'2' ---> c'^V' ---> c'2'
 x'33' --> x'33' --> x'F3'     c'3' ---> c'3' ---> c'ó'
 x'34' --> x'34' --> x'F4'     c'4' ---> c'4' ---> c'ô'
 x'35' --> x'35' --> x'F5'     c'5' ---> c'5' ---> c'õ'
 x'36' --> x'36' --> x'F6'     c'6' ---> c'6' ---> c'ö'
 x'37' --> x'04' --> x'37'     c'7' ---> c'^D' ---> c'7'
 x'38' --> x'38' --> x'F8'     c'8' ---> c'8' ---> c'ø'
 x'39' --> x'39' --> x'F9'     c'9' ---> c'9' ---> c'ù'
 x'3A' --> x'3A' --> x'7A'     c':' ---> c':' ---> c'z'
 x'3B' --> x'3B' --> x'5E'     c';' ---> c';' ---> c'^'
 x'3C' --> x'14' --> x'3C'     c'<' ---> c'^T' ---> c'<'
 x'3D' --> x'15' --> x'3D'     c'=' ---> c'^U' ---> c'='
 x'3E' --> x'3E' --> x'6E'     c'>' ---> c'>' ---> c'n'
 x'3F' --> x'1A' --> x'3F'     c'?' ---> c'^Z' ---> c'?'
 x'40' --> x'20' --> x'40'     c'@' ---> c' ' ---> c'@'
 x'41' --> x'41' --> x'C1'     c'A' ---> c'A' ---> c'Á'
 x'42' --> x'42' --> x'C2'     c'B' ---> c'B' ---> c'Â'
 x'43' --> x'43' --> x'C3'     c'C' ---> c'C' ---> c'Ã'
 x'44' --> x'44' --> x'C4'     c'D' ---> c'D' ---> c'Ä'
 x'45' --> x'45' --> x'C5'     c'E' ---> c'E' ---> c'Å'
 x'46' --> x'46' --> x'C6'     c'F' ---> c'F' ---> c'Æ'
 x'47' --> x'47' --> x'C7'     c'G' ---> c'G' ---> c'Ç'
 x'48' --> x'48' --> x'C8'     c'H' ---> c'H' ---> c'È'
 x'49' --> x'49' --> x'C9'     c'I' ---> c'I' ---> c'É'
 x'4A' --> x'A2' --> x'4A'     c'J' ---> c'¢' ---> c'J'
 x'4B' --> x'2E' --> x'4B'     c'K' ---> c'.' ---> c'K'
 x'4C' --> x'3C' --> x'4C'     c'L' ---> c'<' ---> c'L'
 x'4D' --> x'28' --> x'4D'     c'M' ---> c'(' ---> c'M'
 x'4E' --> x'2B' --> x'4E'     c'N' ---> c'+' ---> c'N'
 x'4F' --> x'7C' --> x'4F'     c'O' ---> c'|' ---> c'O'
 x'50' --> x'26' --> x'50'     c'P' ---> c'&' ---> c'P'
 x'51' --> x'51' --> x'D8'     c'Q' ---> c'Q' ---> c'Ø'
 x'52' --> x'52' --> x'D9'     c'R' ---> c'R' ---> c'Ù'
 x'53' --> x'53' --> x'E2'     c'S' ---> c'S' ---> c'â'
 x'54' --> x'54' --> x'E3'     c'T' ---> c'T' ---> c'ã'
 x'55' --> x'55' --> x'E4'     c'U' ---> c'U' ---> c'ä'
 x'56' --> x'56' --> x'E5'     c'V' ---> c'V' ---> c'å'
 x'57' --> x'57' --> x'E6'     c'W' ---> c'W' ---> c'æ'
 x'58' --> x'58' --> x'E7'     c'X' ---> c'X' ---> c'ç'
 x'59' --> x'59' --> x'E8'     c'Y' ---> c'Y' ---> c'è'
 x'5A' --> x'21' --> x'5A'     c'Z' ---> c'!' ---> c'Z'
 x'5B' --> x'24' --> x'5B'     c'[' ---> c'$' ---> c'['
 x'5C' --> x'2A' --> x'5C'     c'\' ---> c'*' ---> c'\'
 x'5D' --> x'29' --> x'5D'     c']' ---> c')' ---> c']'
 x'5E' --> x'3B' --> x'5E'     c'^' ---> c';' ---> c'^'
 x'5F' --> x'5E' --> x'5F'     c'_' ---> c'^' ---> c'_'
 x'60' --> x'2D' --> x'60'     c'`' ---> c'-' ---> c'`'
 x'61' --> x'2F' --> x'61'     c'a' ---> c'/' ---> c'a'
 x'62' --> x'C2' --> x'62'     c'b' ---> c'Â' ---> c'b'
 x'63' --> x'63' --> x'83'     c'c' ---> c'c' ---> c'ƒ'
 x'64' --> x'64' --> x'84'     c'd' ---> c'd' ---> c'„'
 x'65' --> x'65' --> x'85'     c'e' ---> c'e' ---> c'…'
 x'66' --> x'66' --> x'86'     c'f' ---> c'f' ---> c'†'
 x'67' --> x'67' --> x'87'     c'g' ---> c'g' ---> c'‡'
 x'68' --> x'68' --> x'88'     c'h' ---> c'h' ---> c'ˆ'
 x'69' --> x'69' --> x'89'     c'i' ---> c'i' ---> c'‰'
 x'6A' --> x'A6' --> x'A6'     c'j' ---> c'¦' ---> c'¦'
 x'6B' --> x'2C' --> x'6B'     c'k' ---> c',' ---> c'k'
 x'6C' --> x'25' --> x'6C'     c'l' ---> c'%' ---> c'l'
 x'6D' --> x'5F' --> x'6D'     c'm' ---> c'_' ---> c'm'
 x'6E' --> x'3E' --> x'6E'     c'n' ---> c'>' ---> c'n'
 x'6F' --> x'3F' --> x'6F'     c'o' ---> c'?' ---> c'o'
 x'70' --> x'70' --> x'97'     c'p' ---> c'p' ---> c'—'
 x'71' --> x'C9' --> x'71'     c'q' ---> c'É' ---> c'q'
 x'72' --> x'72' --> x'99'     c'r' ---> c'r' ---> c'™'
 x'73' --> x'73' --> x'A2'     c's' ---> c's' ---> c'¢'
 x'74' --> x'C8' --> x'74'     c't' ---> c'È' ---> c't'
 x'75' --> x'76' --> x'A5'     c'u' ---> c'v' ---> c'¥'
 x'76' --> x'77' --> x'A6'     c'v' ---> c'w' ---> c'¦'
 x'77' --> x'78' --> x'A7'     c'w' ---> c'x' ---> c'§'
 x'78' --> x'79' --> x'A8'     c'x' ---> c'y' ---> c'¨'
 x'79' --> x'60' --> x'79'     c'y' ---> c'`' ---> c'y'
 x'7A' --> x'3A' --> x'7A'     c'z' ---> c':' ---> c'z'
 x'7B' --> x'23' --> x'7B'     c'{' ---> c'#' ---> c'{'
 x'7C' --> x'40' --> x'7C'     c'|' ---> c'@' ---> c'|'
 x'7D' --> x'27' --> x'7D'     c'}' ---> c''' ---> c'}'
 x'7E' --> x'3D' --> x'7E'     c'~' ---> c'=' ---> c'~'
 x'7F' --> x'22' --> x'7F'     c'' ---> c'"' ---> c''
 x'80' --> x'80' --> x'80'     c'€' ---> c'€' ---> c'€'
 x'81' --> x'61' --> x'81'     c'' ---> c'a' ---> c''
 x'82' --> x'62' --> x'82'     c'‚' ---> c'b' ---> c'‚'
 x'83' --> x'63' --> x'83'     c'ƒ' ---> c'c' ---> c'ƒ'
 x'84' --> x'64' --> x'84'     c'„' ---> c'd' ---> c'„'
 x'85' --> x'65' --> x'85'     c'…' ---> c'e' ---> c'…'
 x'86' --> x'66' --> x'86'     c'†' ---> c'f' ---> c'†'
 x'87' --> x'67' --> x'87'     c'‡' ---> c'g' ---> c'‡'
 x'88' --> x'68' --> x'88'     c'ˆ' ---> c'h' ---> c'ˆ'
 x'89' --> x'69' --> x'89'     c'‰' ---> c'i' ---> c'‰'
 x'8A' --> x'8A' --> x'8A'     c'Š' ---> c'Š' ---> c'Š'
 x'8B' --> x'8B' --> x'8B'     c'‹' ---> c'‹' ---> c'‹'
 x'8C' --> x'8C' --> x'8C'     c'Œ' ---> c'Œ' ---> c'Œ'
 x'8D' --> x'8D' --> x'8D'     c'' ---> c'' ---> c''
 x'8E' --> x'8E' --> x'8E'     c'Ž' ---> c'Ž' ---> c'Ž'
 x'8F' --> x'8F' --> x'8F'     c'' ---> c'' ---> c''
 x'90' --> x'90' --> x'90'     c'' ---> c'' ---> c''
 x'91' --> x'6A' --> x'91'     c'‘' ---> c'j' ---> c'‘'
 x'92' --> x'6B' --> x'92'     c'’' ---> c'k' ---> c'’'
 x'93' --> x'6C' --> x'93'     c'“' ---> c'l' ---> c'“'
 x'94' --> x'6D' --> x'94'     c'”' ---> c'm' ---> c'”'
 x'95' --> x'6E' --> x'95'     c'•' ---> c'n' ---> c'•'
 x'96' --> x'6F' --> x'96'     c'–' ---> c'o' ---> c'–'
 x'97' --> x'70' --> x'97'     c'—' ---> c'p' ---> c'—'
 x'98' --> x'71' --> x'98'     c'˜' ---> c'q' ---> c'˜'
 x'99' --> x'72' --> x'99'     c'™' ---> c'r' ---> c'™'
 x'9A' --> x'9A' --> x'9A'     c'š' ---> c'š' ---> c'š'
 x'9B' --> x'9B' --> x'9B'     c'›' ---> c'›' ---> c'›'
 x'9C' --> x'9C' --> x'9C'     c'œ' ---> c'œ' ---> c'œ'
 x'9D' --> x'9D' --> x'9D'     c'' ---> c'' ---> c''
 x'9E' --> x'9E' --> x'9E'     c'ž' ---> c'ž' ---> c'ž'
 x'9F' --> x'9F' --> x'9F'     c'Ÿ' ---> c'Ÿ' ---> c'Ÿ'
 x'A0' --> x'A0' --> x'A0'     c' ' ---> c' ' ---> c' '
 x'A1' --> x'7E' --> x'A1'     c'¡' ---> c'~' ---> c'¡'
 x'A2' --> x'73' --> x'A2'     c'¢' ---> c's' ---> c'¢'
 x'A3' --> x'74' --> x'A3'     c'£' ---> c't' ---> c'£'
 x'A4' --> x'75' --> x'A4'     c'¤' ---> c'u' ---> c'¤'
 x'A5' --> x'76' --> x'A5'     c'¥' ---> c'v' ---> c'¥'
 x'A6' --> x'77' --> x'A6'     c'¦' ---> c'w' ---> c'¦'
 x'A7' --> x'78' --> x'A7'     c'§' ---> c'x' ---> c'§'
 x'A8' --> x'79' --> x'A8'     c'¨' ---> c'y' ---> c'¨'
 x'A9' --> x'7A' --> x'A9'     c'©' ---> c'z' ---> c'©'
 x'AA' --> x'AA' --> x'AA'     c'ª' ---> c'ª' ---> c'ª'
 x'AB' --> x'AB' --> x'AB'     c'«' ---> c'«' ---> c'«'
 x'AC' --> x'AC' --> x'AC'     c'¬' ---> c'¬' ---> c'¬'
 x'AD' --> x'5B' --> x'AD'     c'­' ---> c'[' ---> c'­'
 x'AE' --> x'AE' --> x'AE'     c'®' ---> c'®' ---> c'®'
 x'AF' --> x'AF' --> x'AF'     c'¯' ---> c'¯' ---> c'¯'
 x'B0' --> x'B0' --> x'B0'     c'°' ---> c'°' ---> c'°'
 x'B1' --> x'B1' --> x'B1'     c'±' ---> c'±' ---> c'±'
 x'B2' --> x'B2' --> x'B2'     c'²' ---> c'²' ---> c'²'
 x'B3' --> x'B3' --> x'B3'     c'³' ---> c'³' ---> c'³'
 x'B4' --> x'B4' --> x'B4'     c'´' ---> c'´' ---> c'´'
 x'B5' --> x'B5' --> x'B5'     c'µ' ---> c'µ' ---> c'µ'
 x'B6' --> x'B6' --> x'B6'     c'¶' ---> c'¶' ---> c'¶'
 x'B7' --> x'B7' --> x'B7'     c'·' ---> c'·' ---> c'·'
 x'B8' --> x'B8' --> x'B8'     c'¸' ---> c'¸' ---> c'¸'
 x'B9' --> x'B9' --> x'B9'     c'¹' ---> c'¹' ---> c'¹'
 x'BA' --> x'BA' --> x'BA'     c'º' ---> c'º' ---> c'º'
 x'BB' --> x'BB' --> x'BB'     c'»' ---> c'»' ---> c'»'
 x'BC' --> x'BC' --> x'BC'     c'¼' ---> c'¼' ---> c'¼'
 x'BD' --> x'5D' --> x'BD'     c'½' ---> c']' ---> c'½'
 x'BE' --> x'BE' --> x'BE'     c'¾' ---> c'¾' ---> c'¾'
 x'BF' --> x'BF' --> x'BF'     c'¿' ---> c'¿' ---> c'¿'
 x'C0' --> x'7B' --> x'C0'     c'À' ---> c'{' ---> c'À'
 x'C1' --> x'41' --> x'C1'     c'Á' ---> c'A' ---> c'Á'
 x'C2' --> x'42' --> x'C2'     c'Â' ---> c'B' ---> c'Â'
 x'C3' --> x'43' --> x'C3'     c'Ã' ---> c'C' ---> c'Ã'
 x'C4' --> x'44' --> x'C4'     c'Ä' ---> c'D' ---> c'Ä'
 x'C5' --> x'45' --> x'C5'     c'Å' ---> c'E' ---> c'Å'
 x'C6' --> x'46' --> x'C6'     c'Æ' ---> c'F' ---> c'Æ'
 x'C7' --> x'47' --> x'C7'     c'Ç' ---> c'G' ---> c'Ç'
 x'C8' --> x'48' --> x'C8'     c'È' ---> c'H' ---> c'È'
 x'C9' --> x'49' --> x'C9'     c'É' ---> c'I' ---> c'É'
 x'CA' --> x'CA' --> x'CA'     c'Ê' ---> c'Ê' ---> c'Ê'
 x'CB' --> x'CB' --> x'CB'     c'Ë' ---> c'Ë' ---> c'Ë'
 x'CC' --> x'CC' --> x'CC'     c'Ì' ---> c'Ì' ---> c'Ì'
 x'CD' --> x'CD' --> x'CD'     c'Í' ---> c'Í' ---> c'Í'
 x'CE' --> x'CE' --> x'CE'     c'Î' ---> c'Î' ---> c'Î'
 x'CF' --> x'CF' --> x'CF'     c'Ï' ---> c'Ï' ---> c'Ï'
 x'D0' --> x'7D' --> x'D0'     c'Ð' ---> c'}' ---> c'Ð'
 x'D1' --> x'4A' --> x'D1'     c'Ñ' ---> c'J' ---> c'Ñ'
 x'D2' --> x'4B' --> x'D2'     c'Ò' ---> c'K' ---> c'Ò'
 x'D3' --> x'4C' --> x'D3'     c'Ó' ---> c'L' ---> c'Ó'
 x'D4' --> x'4D' --> x'D4'     c'Ô' ---> c'M' ---> c'Ô'
 x'D5' --> x'4E' --> x'D5'     c'Õ' ---> c'N' ---> c'Õ'
 x'D6' --> x'4F' --> x'D6'     c'Ö' ---> c'O' ---> c'Ö'
 x'D7' --> x'50' --> x'D7'     c'×' ---> c'P' ---> c'×'
 x'D8' --> x'51' --> x'D8'     c'Ø' ---> c'Q' ---> c'Ø'
 x'D9' --> x'52' --> x'D9'     c'Ù' ---> c'R' ---> c'Ù'
 x'DA' --> x'DA' --> x'DA'     c'Ú' ---> c'Ú' ---> c'Ú'
 x'DB' --> x'DB' --> x'DB'     c'Û' ---> c'Û' ---> c'Û'
 x'DC' --> x'DC' --> x'DC'     c'Ü' ---> c'Ü' ---> c'Ü'
 x'DD' --> x'DD' --> x'DD'     c'Ý' ---> c'Ý' ---> c'Ý'
 x'DE' --> x'DE' --> x'DE'     c'Þ' ---> c'Þ' ---> c'Þ'
 x'DF' --> x'DF' --> x'DF'     c'ß' ---> c'ß' ---> c'ß'
 x'E0' --> x'5C' --> x'E0'     c'à' ---> c'\' ---> c'à'
 x'E1' --> x'E1' --> x'E1'     c'á' ---> c'á' ---> c'á'
 x'E2' --> x'53' --> x'E2'     c'â' ---> c'S' ---> c'â'
 x'E3' --> x'54' --> x'E3'     c'ã' ---> c'T' ---> c'ã'
 x'E4' --> x'55' --> x'E4'     c'ä' ---> c'U' ---> c'ä'
 x'E5' --> x'56' --> x'E5'     c'å' ---> c'V' ---> c'å'
 x'E6' --> x'57' --> x'E6'     c'æ' ---> c'W' ---> c'æ'
 x'E7' --> x'58' --> x'E7'     c'ç' ---> c'X' ---> c'ç'
 x'E8' --> x'59' --> x'E8'     c'è' ---> c'Y' ---> c'è'
 x'E9' --> x'5A' --> x'E9'     c'é' ---> c'Z' ---> c'é'
 x'EA' --> x'EA' --> x'EA'     c'ê' ---> c'ê' ---> c'ê'
 x'EB' --> x'EB' --> x'EB'     c'ë' ---> c'ë' ---> c'ë'
 x'EC' --> x'EC' --> x'EC'     c'ì' ---> c'ì' ---> c'ì'
 x'ED' --> x'ED' --> x'ED'     c'í' ---> c'í' ---> c'í'
 x'EE' --> x'EE' --> x'EE'     c'î' ---> c'î' ---> c'î'
 x'EF' --> x'EF' --> x'EF'     c'ï' ---> c'ï' ---> c'ï'
 x'F0' --> x'30' --> x'F0'     c'ð' ---> c'0' ---> c'ð'
 x'F1' --> x'31' --> x'F1'     c'ñ' ---> c'1' ---> c'ñ'
 x'F2' --> x'32' --> x'F2'     c'ò' ---> c'2' ---> c'ò'
 x'F3' --> x'33' --> x'F3'     c'ó' ---> c'3' ---> c'ó'
 x'F4' --> x'34' --> x'F4'     c'ô' ---> c'4' ---> c'ô'
 x'F5' --> x'35' --> x'F5'     c'õ' ---> c'5' ---> c'õ'
 x'F6' --> x'36' --> x'F6'     c'ö' ---> c'6' ---> c'ö'
 x'F7' --> x'37' --> x'F7'     c'÷' ---> c'7' ---> c'÷'
 x'F8' --> x'38' --> x'F8'     c'ø' ---> c'8' ---> c'ø'
 x'F9' --> x'39' --> x'F9'     c'ù' ---> c'9' ---> c'ù'
 x'FA' --> x'FA' --> x'FA'     c'ú' ---> c'ú' ---> c'ú'
 x'FB' --> x'FB' --> x'FB'     c'û' ---> c'û' ---> c'û'
 x'FC' --> x'FC' --> x'FC'     c'ü' ---> c'ü' ---> c'ü'
 x'FD' --> x'FD' --> x'FD'     c'ý' ---> c'ý' ---> c'ý'
 x'FE' --> x'FE' --> x'FE'     c'þ' ---> c'þ' ---> c'þ'
 x'FF' --> x'FF' --> x'FF'     c'ÿ' ---> c'ÿ' ---> c'ÿ'

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7N2. Test Conversion EBCDIC/ASCII/EBCDIC

trantestEAE - uvcopy job listing

 # trantestEAE - test translate EBCDIC to ASCII & ASCII to EBCDIC
 #             - for Vancouver Utilities uvcopy
 #             - by Owen Townsend Oct/2014
 #
 # uvcopy trantestEAE
 # ==================
 # - generates 256 lines translate results for each character
 # - see template further below
 #
 opr='$jobname - test EBCDIC to ASCII translate for Vancouver Utilities'
 opr='         - generates 256 lines translate results for each character'
 filo1=?tmp/trantestEAE_results,rcs=80,typ=LSTt
 #
 # load table of page headings
 lodv1=h0(100)
 $jobname - test uvcopy translate EBCDIC to ASCII & back to EBCDIC
          - generates 256 lines translate results for each character
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 # load table with template
 #         1         2         3         4         5         6
 #123456789012345678901234567890123456789012345678901234567890
 lodv1=k0(100)
 x'00' --> x'00' --> x'00'     c'_' ---> c'_' ---> c'_'
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 @run
         opn    all
         wtbe   filo1,h0(100),h0(100)    print headings
 #

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

 # begin loop to print 256 lines
 man20   mvc    a0(80),k0                template
         mvn    a32(1b),$ca1             current value x'00' --> x'FF'
         mvn    a42(1b),$ca1             copy value for tra
         tra    a42(1)                   translate EBCDIC to ASCII
         mvc    a52(1),a42               copy tra for tre
         tre    a52(1)                   translate ASCII to EBCDIC
         hxc    a02(2),a32(1)            convert EBCDIC char to hex rep
         hxc    a12(2),a42(1)            convert ASCII  char to hex rep
         hxc    a22(2),a52(1)            convert retran EBCDIC to hex rep
 #
 # convert control chars < x'20' to '^X' format, x'01' --> '^A' etc
         cmc    a52(1),x'20'             control char ?
         skp=>  2
         orc    a52(1),x'40'             convert x'01' to x'41' etc
         ins    a52(40),'^'              insert '^'
 #
         cmc    a42(1),x'20'             control char ?
         skp=>  2
         orc    a42(1),x'40'             convert x'01' to x'41' etc
         ins    a42(40),'^'              insert '^'
 #
         cmc    a32(1),x'20'             control char ?
         skp=>  2
         orc    a32(1),x'40'             convert x'01' to x'41' etc
         ins    a32(40),'^'              insert '^'
 #
 # write to report, increment,& return for next char until 256
         put    filo1,a0(80)             print
         add    $ca1,1                   increment to next char of 256
         cmn    $ca1,256                 end 256 ?
         skp<   man20
 #
 # End 256 chars - close file & end job
 man90   cls    all
         eoj

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7O1. Test Conversion EBCDIC/ASCII/EBCDIC

creating bytes256: x'00' - x'FF'

In our preparation above (on page '7B1'), we simply copied a supplied file from $UV/dat1/bytes256. But you could create as follows using a supplied uvcopy job 'makebytes256' (listed on the next page).


 #1. Login userxx --> /home/userxx
 #2. mkdir cnvtran   <-- setup directory for tests
 #3. cd cnvtran      <-- chnage into directory
 #4. cnvdatadirs     <-- make subdirs required
 #5a. cp $UV/dat1/bytes256 d1ebc/Ebytes  <-- copy/rename to consider
 #5b. cp $UV/dat1/bytes256 d2asc/Abytes  <-- copy/rename to consider

Instead of #5a & 5b above, you could run uvcopy job 'makebytes256'


 #5. uvcopy makebytes256,filo1=tmp/bytes256
     ======================================
 #5a. uvcopy makebytes256  <-- same, outputfile defaults as above
      ===================

 #6a. cp $UV/dat1/bytes256 d1ebc/Ebytes  <-- copy/rename to consider as EBCDIC
      =================================
 #6b. cp $UV/dat1/bytes256 d2asc/Abytes  <-- copy/rename to consider as ASCII
      =================================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7O2. Test Conversion EBCDIC/ASCII/EBCDIC

uvcopy job to create bytes256

 # makebytes256 - uvcopy job to create bytes256
 #              - 1 record file of all possible bit combinations in 1 byte
 #              - by Owen Townsend, Oct 2014
 #
 # uvcopy makebytes256,filo1=tmp/bytes256
 # ======================================
 # - generate 256 byte record containing all bit combinations in a byte
 #
 # uvcopy makebytes256  <-- same, output file defaults as above
 # ===================
 #
 opr='$jobname - generate 256 bytes containing all bit combinations in a byte'
 filo1=?tmp/bytes256,rcs=256,typ=RSF
 @run
 # use loop to increment 1 byte x'00' - x'FF', appending on output record
 # - will increment $rb (register 'b') & append to area 'b' via $rb
 # - initially binary zero when uvcopy begins
 # - note that  bb0 is 'area b' offset by 'register b'
 # - note that     (1b) is length 1 byte, but the 'b' means 'binary' vs char
 man20   mvn    bb0(1b),$rb         append current $rb value to area 'b'
         add    $rb,1               increment register 'b' ($rb)
         cmn    $rb,256             reached max value ?
         skp<   man20               repeat loop until 256
 #
 man90   opn    filo1               open outfile
         put    filo1,b0(256)       write record to outfile
         cls    filo1               close output file
         eoj                        end job

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7O3. Test Conversion EBCDIC/ASCII/EBCDIC

alternate way to create bytes256

Here is an alternate way to create bytes256 from a neutral translate table. VU supplies translate table at $UV/srcf/trtneut.c You could copy it to your homedir/tmp, compile,& extract the 256 bytes as follows:


 #1. Login userxx --> /home/userxx

 #2. mkdir src bin data  <-- make subdirs

 #3. cp $UV/srcf/trtneut.c src

 #4. cc -c trtneut.c -o bin/trtneut.o  <-- compiles to trtneut.o
     ================================

 #5. uvhd bin/trtneut.o    <-- investigate object code
     ==================      - note 64 byte header & extra training bytes

 #6. dd if=bin/trtneut.o of=data/bytes256 bs=64 skip=1 count=4
     =========================================================
     extract the 256 bytes x'00' - x'FF' from object code

 #7. uvhd data/bytes256    <-- investigate 256 bytes extracted
    ==================      - confirm 1st 64 dropped & excess dropped

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7O4. Test Conversion EBCDIC/ASCII/EBCDIC

trtneut.c - neutral translate table


 #8. vi src/trtneut.c
     ================
 /* trtneut.c - translate table - neutral, use as basis to create others */
 unsigned char trtneut[256] = {
 0x00,0x01,0x02,0x03,0x04,0x05,0x06,0x07,0x08,0x09,0x0A,0x0B,0x0C,0x0D,0x0E,0x0F,
 0x10,0x11,0x12,0x13,0x14,0x15,0x16,0x17,0x18,0x19,0x1A,0x1B,0x1C,0x1D,0x1E,0x1F,
 0x20,0x21,0x22,0x23,0x24,0x25,0x26,0x27,0x28,0x29,0x2A,0x2B,0x2C,0x2D,0x2E,0x2F,
 0x30,0x31,0x32,0x33,0x34,0x35,0x36,0x37,0x38,0x39,0x3A,0x3B,0x3C,0x3D,0x3E,0x3F,
 0x40,0x41,0x42,0x43,0x44,0x45,0x46,0x47,0x48,0x49,0x4A,0x4B,0x4C,0x4D,0x4E,0x4F,
 0x50,0x51,0x52,0x53,0x54,0x55,0x56,0x57,0x58,0x59,0x5A,0x5B,0x5C,0x5D,0x5E,0x5F,
 0x60,0x61,0x62,0x63,0x64,0x65,0x66,0x67,0x68,0x69,0x6A,0x6B,0x6C,0x6D,0x6E,0x6F,
 0x70,0x71,0x72,0x73,0x74,0x75,0x76,0x77,0x78,0x79,0x7A,0x7B,0x7C,0x7D,0x7E,0x7F,
 0x80,0x81,0x82,0x83,0x84,0x85,0x86,0x87,0x88,0x89,0x8A,0x8B,0x8C,0x8D,0x8E,0x8F,
 0x90,0x91,0x92,0x93,0x94,0x95,0x96,0x97,0x98,0x99,0x9A,0x9B,0x9C,0x9D,0x9E,0x9F,
 0xA0,0xA1,0xA2,0xA3,0xA4,0xA5,0xA6,0xA7,0xA8,0xA9,0xAA,0xAB,0xAC,0xAD,0xAE,0xAF,
 0xB0,0xB1,0xB2,0xB3,0xB4,0xB5,0xB6,0xB7,0xB8,0xB9,0xBA,0xBB,0xBC,0xBD,0xBE,0xBF,
 0xC0,0xC1,0xC2,0xC3,0xC4,0xC5,0xC6,0xC7,0xC8,0xC9,0xCA,0xCB,0xCC,0xCD,0xCE,0xCF,
 0xD0,0xD1,0xD2,0xD3,0xD4,0xD5,0xD6,0xD7,0xD8,0xD9,0xDA,0xDB,0xDC,0xDD,0xDE,0xDF,
 0xE0,0xE1,0xE2,0xE3,0xE4,0xE5,0xE6,0xE7,0xE8,0xE9,0xEA,0xEB,0xEC,0xED,0xEE,0xEF,
 0xF0,0xF1,0xF2,0xF3,0xF4,0xF5,0xF6,0xF7,0xF8,0xF9,0xFA,0xFB,0xFC,0xFD,0xFE,0xFF
 };

 #9. uvhd data/bytes256    <-- investigate 256 bytes extracted
     ==================      - confirm 1st 64 dropped & excess dropped
 rec#=1 rsize=256 fptr=0 fsize=256 rcount=1
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ................................ !"#$%&'()*+,-./0123456789:;<=>?
             0000000000000000111111111111111122222222222222223333333333333333
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
          64 @ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^abcdefghijklmnopqrstuvwxyz{|}~.
             4444444444444444555555555555555566666666666666667777777777777777
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
         128 ................................................................
             88888888888888889999999999999999AAAAAAAAAAAAAAAABBBBBBBBBBBBBBBB
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF
         192 ................................................................
             CCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEFFFFFFFFFFFFFFFF
             0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_8 DATAcnv1 - Converting Mainframe Data to Unix/Linux

Part 8 - Variable Length Files - Contents


8A1. Introduction to Variable Length RDW files
use FTP option 'quote site RDW' to transfer variable length files
from the mainframe to unix/linux.
8A2.

8B1. test/demo files supplied by UV Software
8B2. testRDWe test file to demo RDW files

8C1. Investigating RDW files with 'uvhd'

8D1. converting EBCDIC RDW files to ASCII - with 'uvhd'
- uvhd interactive utility, easy to use

8E1. converting EBCDIC RDW files to ASCII - with 'uvcp'

8F1. converting EBCDIC RDW files to ASCII - with 'uvcopy varfix11'
- varfix11 batch utility, better for high volume conversions

8G1. creating table summary stats of record sizes in variable length files
uvcopy job 'varstat1' Operating Instructions & sample report

8X0. Listings of uvcopy jobs used in Part 8
- varstat1, varfix11,
- LST2RDW1, RDW2LST1, varfix11

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8A1. Variable Length RDW (Record Descriptor Word) files

Introduction to variable length RDW files

RDW (Record Descriptor Word) variable length files are often used to FTP variable length EBCDIC files from the mainframe to unix/linux/windows systems. uvcp provides for 2 types of RDW files - standard & non-standard (option z2).

standard RDW files

Non-standard RDW files

FTP variable length files from Mainframe

We will supply the variable length data files used for the demos here, but if you are FTP'ing variable length files from your mainframe, please note that you must use FTP option 'quote site RDW' to transfer variable length files from the mainframe to unix/linux.


 #1. Login --> your homedir
 #2. cd d0ebc   <-- change into the data subdir

 #3. ftp xxx.xxx.xxx.xxx   <-- FTP to mainframe IP#
     ===================
 #3a. userid--> .....      <-- enter userid
 #3b. passwd--> .....      <-- enter password
 #3c. binary               <-- ensure binary transfer
 #3d. quote site RDW       <-- required if any variable length files
 #3e. get XXX.XXX.XXX      <-- get desired data files
 #3f. get ...etc....         - 'single quotes' may be required

'quote site RDW' is reqquired for FTP to insert the 4 byte record prefixes with the record-length in binary. Without the RDW record profixes, we can not determine where 1 record ends & the next begins.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8A2. Variable Length RDW (Record Descriptor Word) files

FTP files to Unix/Linux - using JCL on mainframe

Before FTP'ing files from mainframe to Unix/Linux, we recommend converting the Fixed-Length files to Variable-Length so we can use the RDW option. Then, when we receive the files on unix/linux, we will immediately know the record-sizes of all files & also know which were originally Fixed because their minimum & maximum record sizes will be the same. Here is a skeleton JCL that could be used to convert Fixed to Variable on the mainframe.

 //* JCL to convert Fixed-Length to Variable-Length
 //SORT001   EXEC PGM=SORT
 //SYSUDUMP  DD SYSOUT=K,HOLD=YES
 //SYSOUT    DD SYSOUT=*
 //SYSPRINT  DD SYSOUT=*
 //SORTIN    DD DSN=AAAAA.BBBBB.CCC.DDDDD,
 //             DISP=(SHR,KEEP,KEEP)
 //SORTOUT   DD DSN=AAAAA.BBBBB.CCC.DDDDD.X,
 //             DISP=(NEW,CATLG,DELETE),UNIT=SYSDA,
 //             SPACE=(TRK,(0500,0500),RLSE),
 //             DCB=(RECFM=VB,LRECL=32000)
 //SYSIN     DD *
    SORT FIELDS=COPY
    OUTFIL FNAMES=SORTOUT,FTOV
 /*
 //*Note - 'FTOV' means Fixed TO Variable
 //*     - 'VTOF' could also convert Variable TO Fixed

You could then run FTP with RDW on the mainframe with the following JCL, or you could run FTP on Unix/Linux as shown on the next page.

 //* JCL to FTP Variable-Length with RDW to Unix/Linux
 //ACHP01B   EXEC PGM=FTP,PARM='(TRAN TRTAUGE1'
 //OUTPUT    DD  SYSOUT=*
 //SYSPRINT  DD  SYSOUT=*
 //INPUT     DD  *
 30.252.99.11 21 (exit
 userid password
 cd /p2/cnv1/cnvdata1/d0ebc/
 pwd
 type i
 LOCSITE RDW
 PUT 'AAAAA.BBBBB.CCC.DDDDD.X' AAAAA.BBBBB.CCC.DDDDD
 QUIT
 /*

Also see https://uvsoftware.ca/datacnv2.htm#5C1 - 5C6, which show how you can automatically generate the JCL to convert Fixed to Variable from a control-file list of filenames to be transferred to Unix/Linux. 'genF2Vjcl1' is a Vancouver Utility which runs on unix/linux, so you need to FTP the generated JCL to the mainframe for execution.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8B1. Variable Length RDW Files

test/demo files supplied in /home/uvadm/dat1

dat1/testLST
  • text file originally used to create testRDW
dat1/testRDW
  • RDW file created from testLST
dat1/testRDWe
  • RDW file, same as above in EBCDIC

We will illustrate how to convert RDW EBCDIC files to text or fixed length files using: uvhd, uvcp,& uvcopy.

  1. use 'uvhd' when there you only have a few files to do,& when the interactive utility is preferred.

  2. use 'uvcp' when you want to write a script to convert several files and for repeatability.

  3. use 'uvcopy' when you need to do some data manipulation beyond just the simple conversion from 1 file format to another. You could write custom uvcopy jobs to convert files with packed/binary, but DATAcnv1.htm documents pre-programmed jobs to generate uvcopy jobs from COBOL copybooks.

We will 1st investigate the testRDWe file using 'uvhd'. We can not use unix tools (vi, lp, etc) since the file is EBCDIC & contains binary record sizes.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8B2. Variable Length RDW Files

testRDWe - test file to demo RDW files


 uvhd dat1/testRDWe as3 <-- examine the RDW test file
 ======================   - option 'a' translates character line to ASCII
                          - option 's3' to space between scale & 3 line groups
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ....DELL00 - Dell Inc.  ....HP0000 - Hewlett Packard....IBM000 -
             0100CCDDFF464C8994C984440100CDFFFF464C8A98AA4D8898980200CCDFFF46
             080045330000045330953B000C008700000008563533071321940C0092400000
          64  International Business Machines....MFC000 - Micro Focus COBOL
             4C9A8998A899894CAA898AA4D888898A0200DCCFFF464D88994C98AA4CDCDD44
             0953595139651302429552204138955200004630000004939606634203626300
         128 ....MS0000 - Microsoft Corp.....REDHAT - Red Hat Linux  ....SUN0
             0100DEFFFF464D8899A98A4C99940100DCCCCE464D884C8A4D89AA440200EEDF
             0C0042000000049396266303697B0C0095481300095408130395470004002450
         192 00 - Sun Microsystems Ltd   ....UVSI00 - UV Software Inc.
             FF464EA94D8899AAAA89A4DA84440200EEECFF464EE4E98AA8984C984444
             00000245049396282354203340000000452900000450266361950953B000

The entire file is only 252 bytes & contains 8 short variable length records:

You cannot display RDW files with 'vi', because vi cannot handle binary, there are no LineFeeds to separate the records, so the entire file appears as 1 long line (unless there just happened to be a x'0A' in the length field, which would be interpreted as a LineFeed by vi).

uvhd option 'z' to recognize RDW files

OPtion 'z' would tell uvhd to look for the 'RDW' record prefixes & show 1 record at a time, but we did not specify option z above, and the default is to show any file in 256 byte blocks (4 groups of 3 64 byte lines for characters, zones,& digits).

See the next page where we will specify option 'z' to show RDW files 1 record at a time. You can press enter to browse forward until EOF reached. Then you could enter '1' to return to the begining of the file.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8C1. Variable Length RDW Files

Investigating RDWz4 files with 'uvhd'


 uvhd dat1/testRDWe z4a   <-- display RDWz4 file using option 'z'
 ======================     - option 'a' to display character line in ASCII
                            - null entries browse forward until EOF reached
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ....DELL10 - Dell Inc.
             0100CCDDFF464C8994C98444
             080045331000045330953B00
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
          24 ....HP0010 - Hewlett Packard
             0100CDFFFF464C8A98AA4D889898
             0C00870010000856353307132194
                      10        20        30        40        50        60
 r#        3 0123456789012345678901234567890123456789012345678901234567890123
          52 ....IBM010 - International Business Machines
             0200CCDFFF464C9A8998A899894CAA898AA4D888898A
             0C009240100009535951396513024295522041389552
                      10        20        30        40        50        60
 r#        4 0123456789012345678901234567890123456789012345678901234567890123
          96 ....MFC010 - Micro Focus COBOL
             0200DCCFFF464D88994C98AA4CDCDD44
             00004630100004939606634203626300
                      10        20        30        40        50        60
 r#        5 0123456789012345678901234567890123456789012345678901234567890123
         128 ....MS0010 - Microsoft Corp.
             0100DEFFFF464D8899A98A4C9994
             0C0042001000049396266303697B
                      10        20        30        40        50        60
 r#        6 0123456789012345678901234567890123456789012345678901234567890123
         156 ....REDH10 - Red Hat Linux
             0100DCCCFF464D884C8A4D89AA44
             0C00954810000954081303954700
                      10        20        30        40        50        60
 r#        7 0123456789012345678901234567890123456789012345678901234567890123
         184 ....SUN010 - Sun Microsystems Ltd
             0200EEDFFF464EA94D8899AAAA89A4DA8444
             040024501000024504939628235420334000
                      10        20        30        40        50        60
 r#        8 0123456789012345678901234567890123456789012345678901234567890123
         220 ....UVSI10 - UV Software Inc.
             0200EEECFF464EE4E98AA8984C984444
             0000452910000450266361950953B000
Note
  • option 'a' translated the character line to ASCII (so you can read it)
  • But the zones & digits lines show the data is EBCDIC
  • For example the 'D' of Dell is x'C4' (vs x'44' in ASCII)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8D1. Variable Length RDW Files

converting EBCDIC RDWz4 files to ASCII - with 'uvhd'

On this page we will show you how to convert EBCDIC RDWz4 files to ASCII text using 'uvhd'. On following pages we will do the same with 'uvcp' & 'uvcopy'.

We will specify uvhd options 'za3p4y7' which mean:

z4
  • recognize RDWz4 files
a3
  • a3=a1+a2
a1
  • translate character line of display to ASCII
a2
  • translate any 'Write' command output to ASCII
p4
  • drop 4 byte prefix on write output
y7
  • y7=y1+y2+y4
y1
  • insert CR at end of record
y2
  • insert LF at end of record
y4
  • scan back to last nonblank to insert the CR/LF

 uvhd dat1/testRDWe z4a3p4y7   <-- display 1st record & wait for command
 ===========================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ....DELL10 - Dell Inc.
             0100CCDDFF464C8994C98444
             080045331000045330953B00
 ---> w99999  <-- write all records (8) to tmp/testRDWe_yymmdd_hhmmssW
                - output filename will be date/time stamped
                - On Dec 18, 20007 at 12:15 tmp/testRDWe_071218_121500W
 ---> q       <--- quit uvhd

 vi tmp/testRDWe_071218_121500W
 ==============================
 vi tmp/*00W       <-- shortcut to display desired file
 ===========

text records extracted from RDWz4 file

     DELL00 - Dell Inc.
     HP0000 - Hewlett Packard
     IBM000 - International Business Machines
     MFC000 - Micro Focus COBOL
     MS0000 - Microsoft Corp.
     REDHAT - Red Hat Linux
     SUN000 - Sun Microsystems Ltd
     UVSI00 - UV Software Inc.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8E1. Variable Length RDW Files

converting EBCDIC RDWz4 files to ASCII - with 'uvcp'

We will now use 'uvcp' to perform the conversion from EBCDIC RDWz4 to ASCII.

Instead of variable length text, we will create fixed length output records that are required for most COBOL input files. For this demo, we will convert to fixed 64 byte records & insert a LineFeed in the last byte so we can have it both ways (fixed length for COBOL, but LFs allow investigation with vi).


 uvcp "fili1=dat1/testRDWe,rcs=64,typ=RDWz4,filo1=tmp/testz4,typ=RST,tra=0(64)"
 ==============================================================================
tra=0(64)
  • translates the records to ASCII.
typ=RST
  • output 'Record Sequential Terminated' (inserts LineFeed in last byte)

confirm conversion OK with uvhd


 uvhd tmp/testz4 r64h2     <-- display output file with uvhd
 =====================       - option r64 for record size
                             - option 'h2' for hex display to see LineFeed
                             - only 1st record shown below
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 DELL10 - Dell Inc.                                             .
             4444332224666246622222222222222222222222222222222222222222222220
             45CC100D045CC09E3E000000000000000000000000000000000000000000000A

Confirmations shown by uvhd

  1. Binary record prefixes have been dropped off, because output typ=RST extracts the data record only & omits the typ=RDW record prefix.

  2. LineFeeds in last byte of each record (x'0A's).

  3. Output is ASCII 1st char 'D' is x'44' here vs x'C4' on page 'K2'&'K3' Translation performed by instruction 'tra=0(64)'.

  4. See pages uvcp.htm#M0 thru M17 for other instructions you can use to reformat data (mvc, clr, rep, tre, tru, trl, sel, del, etc...).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8F1. Variable Length RDW Files

converting EBCDIC RDW files to ASCII - with 'uvcopy varfix11'

'varfix11' is a uvcopy utility job to convert an RDW variable length file to a specified fixed length (large enough to hold largest variable record).

On your mainframe migration, you may have EBCDIC RDW & BDW/RDW files to be transferred to & converted on your unix/linux system.

For a mainframe migration you would not want to use the interactive 'uvhd' for hundreds of files. Here we will illustrate a batch utility 'varfix11' that you could use to convert varlth RDW EBCDIC files to ASCII fixed records.

Note that this job can not be used for EBCDIC files with packed/binary fields. For those you need the procedures documented in 'Part_3' of this DATAcnv1.doc or in JCLcnv2real.htm#Part_3.


 uvcopy varfix11,fili1=dat1/testRDWe,filo1=tmp/testRDWaf,uop=a1b1h1r64t1
 =======================================================================
Note
  • we have specified I/O files & options on the command line above
  • you can enter them at prompts (see below) after option explanations
 uop=a0b0h1r2048t0 - option defaults
     a0            - no translate (input EBCDIC with packed/binary fields)
     a1            - translate from EBCDIC to ASCII
       b0          - do NOT convert nulls to blanks
       b1          - DO convert nulls to ASCII blanks
       b2          - DO convert nulls to EBCDIC blanks
         h0        - drop the 4 byte binary record-size headers
         h1        - replace 4 byte binary recsize hdr with numeric chars
           r2048   - output fixed records 2048 bytes
           r8192   - max output fixed size 8192 bytes
                t0 - do NOT insert LineFeed in last byte of record
                t1 - DO insert LineFeed in last byte of record
  null to accept or re-specify (1 or more) -->
 071128:172717:varfix11: EOF fili01 rds=16 size=252: dat1/testRDWe
 071128:172717:varfix11: EOF filo01 wrts=8 size=512: tmp/testRDWaf

 cat tmp/testRDWaf       <-- display output file
 =================
 0024DELL10 - Dell Inc.
 0028HP0010 - Hewlett Packard
 0044IBM010 - International Business Machines
 0032MFC010 - Micro Focus COBOL
 0028MS0010 - Microsoft Corp.
 0028REDH10 - Red Hat Linux
 0036SUN010 - Sun Microsystems Ltd
 0032UVSI10 - UV Software Inc.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8G1. Variable Length RDW Files

creating table summary of record sizes in RDW files

'varstat1' is a uvcopy job that you can run to create a summary table of record-sizes found in all records of a variable file typ=RDWz4 or typ=RDWz2 (variable length records with 4 byte or 2 byte prefixes).

demo with supplied test file


 #1. cd $UV      <-- change to /home/uvadm

 #2. uvcopy varstat1,fili1=d0ebc/testRDW,filo2=rpts/testRDW_recsizes
     ===============================================================

 #3. vi rpts/testRDW_recsizes   <-- inspect report
     ========================

sample report

 varstat1  2007/12/19_18:10:29  record-sizes in dat1/testRDW
 tbl#001 pg#001     -argument-
 line#  count    %  record-sizes
     1       1  12  00020
     2       3  37  00024
     3       2  25  00028
     4       1  12  00032
     5       1  12  00040
             8*100   *TOTAL*

create reports for all files in directory


 #1. cd $CMPDATA     <-- change to file compare superdir

 #2. mkdir tmp1      <-- make tmp1 subdir if not existing
 #2a. rm -f tmp1/*   <-- OR remove all files if tmp1 already exists

 #3. uvcopyx varstat1 d0ebc tmp1 uop=q0i7,rop=r0
     ===========================================
     - create table summary recsize counts for all files in d0ebc subdir
     - output reports in tmp1 with same names as datafiles in d0ebc

 #4. uvlpd12 tmp1     <-- print all reports in tmp subdir
     ============
Note
  • see the varstat1 job listed at the end of this Part

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8X0. Variable Length RDW Files

uvcopy jobs used in Part 8

The 'uvcopy jobs' used in this section are listed on the following pages. You might need to modify them for complex variable length conversions.


8X1. varstat1 - create table summary of record sizes in variable length files
- listed further below

8X2. varfix11 - convert variable length BDW/RDW files to Fixed Length files
- might be used to convert mainframe EBCDIC files to ASCII
- BUT not if packed/binary present (see Part_3).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8X1. Variable Length BDW/RDW Files

 # varstat1 - create summary table of record-sizes used in variable file
 #          - for variable lth records with 4 byte hdrs (standard RDW)
 #          - by Owen Townsend, Dec 12, 2007
 #Dec19/07 - replace getvr2 subrtn with file typ=RDWz4 & RDWz2 (optional)
 #
 #           ** create report for 1 file (for testing) **
 #
 # uvcopy varstat1,fili1=d0ebc/datafilename,filo2=tmp1/reportname
 # ==============================================================
 #
 #           ** create reports for all files in directory **
 #
 # 1. cd $CNVDATA     <-- change to conversion superdir
 #                     - subdir d0ebc contains EBCDIC var lth files
 #
 # 2. mkdir tmp1      <-- make tmp1 subdir if not existing
 # 2a. rm -f tmp1/*   <-- OR remove all files if tmp1 already exists
 #
 # 3. uvcopyx varstat1 d0ebc tmp1 uop=q0i7,rop=r0
 #    ===========================================
 #    - create table summary recsize counts for all files in d0ebc subdir
 #    - output reports in tmp1 with same names as datafiles in d0ebc
 #
 # 4. uvlpd12 tmp1     <-- print all reports in tmp subdir
 #    ============
 #
 #                  ** sample report **
 #
 # varstat1  2006/12/17_18:15:17  record-sizes in d0ebc/E2121656
 # tbl#001 pg#001     -argument-
 # line#  count    %  record-sizes
 #     1  10,552  16  00046
 #     2   4,451   7  00065
 #     3  23,347  37  00066
 #     4     367   0  00068
 #     5  21,010  33  00083
 #       - - - etc - - -
 #    18       3   0  00218
 #    19     441   0  00233
 #    20     813   1  00239
 #        62,115*100   *TOTAL*
 #

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

 # This job designed for variable lth records typ=RDW & typ=RDWz2
 # typ=RDW (standard RDW) - 4 byte record prefixes
 #         - binary recsize in 1st 2 bytes (3rd & 4th x'0000' unused)
 #         - binary recsize of data (includes the 4 byte prefix)
 #         - blk/rec lth binary BIG-END format, need 's'witch option for INTEL
 # typ=RDWz2 (non-std RDW) - 2 byte record prefixes (2 nulls omitted)
 #         - binary recsize does not include the 2 byte prefix
 #
 opr='$jobname - create summary table of record-sizes used in variable file'
 opr='uop=q1z0 - option defaults (message only, see uop=... in job)'
 opr='      z0 - RDW 4 byte prefix, recsize 1st 2 BIG-END binary'
 opr='      z2 - RDW 2 byte prefix, recsize 1st 2 BIG-END binary'
 uop=q1z0      # option defaults
 rop=r1        # option to prompt for report disposition
 was=a16384
 fili1=?d0ebc/input,rcs=8192,typ=RDW        #Dec19/07 typ=RDW replaces getvr
 filo1=?tmp1/$jobname.rpt,typ=LSTt,rcs=128
 @run
 #Dec19/07 - replace subrtn getvr with typ=RDW & RDWz2
 #         - for nonstd 2 byte prefix vs std 4 bytes
 # if uop=z2, append file option z2 for typ=RDWz2
        tsb    o26(1),x'02'            uop z2 for nonstd RDW ?
        skp!   1
        cata8  $fili1+180(15),'z2'
        opn    all
 #
 # begin loop to get records & build summary table of recsizes
 # - to be dumped at end of file
 man20  get    fili1,a0                 get next RDW record data
        skp>   eof                      (cc set > at EOF)
        mvn    c0(5),$rv                cnvrt binary recsize to numeric
        tblt1f1 c0(5),'record-sizes'    build table in memory
        skp    man20                    repeat get loop until EOF
 #
 # EOF - close files & end job
 eof    mvfv1  f0(80),'record-sizes in $fili1'
        tbpt1s1 filo1,f0(50)            dump table from memory to file
        cls    all
        eoj
 #
 # getvr - subrtn to get records from IBM std variable length file
 #Dec19/07 - getvr replaced by file typ=RDW & typ=RDWz2
 #         - subrtn saved (for interest) in $UV/pf/util/getvr
 ## folwng instrns modified (see above) for typ=RDW
 ##man20  bal    getvr                    set rgstrs a & b to next record
 ##       mvn    c0(5),a0(2bs)            cnvrt binary recsize to numeric

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8X2. Variable Length BDW/RDW Files

 # varfix11 - convert RDW variable length file test Aug 24/05
 #          - convert RDW variable length file to fixed length records
 #          - prior to 2nd job that will convert EBCDIC to ASCII
 #            OR compare 2 files uvcmpVA1/uvcmpVE1 (fixlth max 4096)
 #          - by Owen Townsend, UV Software
 #          - originally for Laval, Dec 2005
 #
 #Dec19/07 - replace subrtn getvr with typ=RDW & add option z2
 #           for nonstd 2 byte prefix vs std 4 bytes (binary recsize 1st 2)
 #         - subrtn code preserved in $UV/pf/util/getv for interest
 #
 # Enhanced Oct 2007 for Sungard, options added
 # - a1=translate to ASCII, h1=cnvt hdr to numerics, r4096=fixed output recsize
 # - b1=convert nulls to ASCII blanks, t1=terminate with LineFeed
 #
 #                 ** Operating Instructions **
 #
 # uvcopy varfix11,fili1=d0ebc/infile,filo1=d1ebc/outfile,uop=a1h1r4096
 # ====================================================================
 #  - convert data file from variable to fixed 4096 byte records
 #
 #  option a1 - translate from EBCDIC to ASCII
 #  option h1 - replace 4 byte binary recsize hdr with recsize numerics
 #  option r4096 - output fixed records 4096 bytes
 #               - large enough to hold largest variable lth record
 #
 # This 1 uvcopy job 'varfix11' can be used for all data files
 # - to convert from variable to fixed
 # - drops 4 byte variable recsize headers, or retain converted to numerics
 # - null fill records to fixed recsize spcfd by option r
 #
 #                       ** uses for varfix11 **
 #
 # Convert variable length to fixed length to facilitate:
 #
 # 1. CONVERTING datafiles from EBCDIC to ASCII (preserving packed/binary)
 #    - could convert back to variable length after conversion
 #    - see examples in VSEDATA.doc
 #
 # 2. COMPARING 2 datafiles
 #    - see script uvcmpVE1 to compare 2 varlth EBCDIC datafiles
 #    - this varfix11 used to convert both files to fixed max lth in tmp1/tmp2
 #    - followed by utility 'uvcmp1' to compare & create report in rpts/...
 #

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

 #                  ** sample variable-length RDW files **
 #
 # uvhd dat1/testRDWe a    <-- display demo file EBCDIC varlth RDWz4
 # ====================      - option 'a' to show char lines in ASCII
 #                             (zones & digits lines show EBCDIC codes)
 #
 #                      10        20        30        40        50        60
 # r#        1 0123456789012345678901234567890123456789012345678901234567890123
 #           0 ....DELL10 - Dell Inc.  ....HP0010 - Hewlett Packard....IBM010 -
 #             0100CCDDFF464C8994C984440100CDFFFF464C8A98AA4D8898980200CCDFFF46
 #             080045331000045330953B000C008700100008563533071321940C0092401000
 #          64  International Business Machines....MFC010 - Micro Focus COBOL
 #             4C9A8998A899894CAA898AA4D888898A0200DCCFFF464D88994C98AA4CDCDD44
 #             0953595139651302429552204138955200004630100004939606634203626300
 #         128 ....MS0010 - Microsoft Corp.....REDH10 - Red Hat Linux  ....SUN0
 #             0100DEFFFF464D8899A98A4C99940100DCCCFF464D884C8A4D89AA440200EEDF
 #             0C0042001000049396266303697B0C0095481000095408130395470004002450
 #         192 10 - Sun Microsystems Ltd   ....UVSI10 - UV Software Inc.
 #             FF464EA94D8899AAAA89A4DA84440200EEECFF464EE4E98AA8984C984444
 #             10000245049396282354203340000000452910000450266361950953B000
 #
 # uvhd dat0/testRDWe z4a1  <-- option 'z4' to process RDWz4 files
 # =======================    - 'a1' to translate char line to ASCII
 #                            - hexadecimal zones & digits show EBCDIC values
 #
 #                      10        20        30        40        50        60
 # r#        1 0123456789012345678901234567890123456789012345678901234567890123
 #           0 ....DELL10 - Dell Inc.
 #             0100CCDDFF464C8994C98444
 #             080045331000045330953B00
 #
 # Note - uvhd displays 1st record & prompts for commands
 #      - enter w999 to write all records & q to quit:
 #
 # --> w999a1t6r64 <-- Write all records to the tmp/... subdir
 #                   - filename date/time stamped ex: tmp/testRDWe_yymmdd_hhmmssW
 #                   - On Nov 17/07 3:30 PM might be: tmp/testRDWe_071117_153059W
 #         a1        - translate output records to ASCII
 #           t6      - insert LineFeed terminators after last non-blank
 #                     (t6=t2+t4, t2=LineFeed, t4=insert after last non-blank)
 #             r64   - max size for output records
 #                                  - 'y7' to insert CR+LF after last non-blank
 # --> q    <-- quit uvhd
 #
 # cat tmp/*59W       <-- display output text records
 # ============         - use *59W to save a lot of keystrokes
 #
 # DELL10 - Dell Inc.
 # HP0010 - Hewlett Packard
 # IBM010 - International Business Machines
 # MFC010 - Micro Focus COBOL
 # MS0010 - Microsoft Corp.
 # REDH10 - Red Hat Linux
 # SUN010 - Sun Microsystems Ltd
 # UVSI10 - UV Software Inc.
 #

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

 opr='$jobname - convert variable length file to fixed length records'
 opr='uop=a0b0h1r4096t0z4 - option defaults'
 opr='    a0              - no translate (for EBCDIC packed/binary fields)'
 opr='    a1              - translate from EBCDIC to ASCII'
 opr='      b0            - do NOT convert nulls to blanks'
 opr='      b1            - DO convert nulls to ASCII blanks'
 opr='      b2            - DO convert nulls to EBCDIC blanks'
 opr='        h0          - drop the 4 byte binary record-size headers'
 opr='        h1          - replace 4 byte binary recsize hdr with numeric chars'
 opr='          r4096     - output fixed records 4096 bytes'
 opr='          r8192     - max output fixed size 8192 bytes'
 opr='               t0   - do NOT insert LineFeed in last byte of record'
 opr='               t1   - DO insert LineFeed in last byte of record'
 opr='                 z4 - RDW 4 byte prefix, recsize 1st 2 BIG-END binary'
 opr='                 z2 - RDW 2 byte prefix, recsize 1st 2 BIG-END binary'
 uop=q1a0b0h1r4096t0z4   # option defaults
 was=a32768b32768c32768  # increase areas a,b,c from dflt 1024
 fili1=?d0ebc/filename,rcs=8192,typ=RDWz4   # uop z2 will change to option z2
 filo1=?d1ebc/filename,rcs=8192,typ=RSF
 @run
 #Dec19/07 - replace subrtn getvr with typ=RDWz4 & typ=RDWz2
 #         - typ=RDWz2 for nonstd 2 byte prefix vs std 4 bytes
 # if uop=z2, change file option z4 to z2
        tsb    o26(1),x'02'            uop z2 for nonstd RDW ?
        skp!   1
        cata8  $fili1+180(15),'z2'
 #
        opn    all                     open files
        mvn    $rf,$uopbr              load fixed outsize in rgstr 'f'
        mvn    $rg,$rf                 transfer to $rg
        sub    $rg,1                   -1 byte to insert LF in last byte
 #

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

 # begin loop to read variable length records until EOF
 # - using get typ=RDW to get var lth records depending on size in prefix
 # - write fixed length records padded to max 8192 with EBCDIC blanks
 ##man20  bal    getvr     <--Dec19/07 change to get file typ=RDW
 man20  get    fili1,b0                get next record depending on varlth prfx
        skp>   eof
 #
 # test option to translate EBCDIC to ASCII
        tsb    o1(1),x'01'             trnslt EBCDIC to ASCII ?
        skp!   1
        tra    b0($rf8192)
 #
 # presume option h1 to replace 4 byte recsize hdr with numerics
 ##     mvn    c0(4),a0(2bs)  <--Dec19/07 chg to $rv
        mvn    c0(4),$rv               cnvt binary recsize to ASCII numerics
        tsb    o1(1),x'01'             trnslt EBCDIC to ASCII ?
        skp=   1
        tre    c0(4)                   no - trnslt ASCII recsize to EBCDIC
        mvc    c4($rf8192),b0          move data to follow recsize
 #
 # test option h0/h1 drop recsizehdr or replace with numerics ?
        tsb    o8(1),x'01'             replace recsize hdr w numerics ?
        skp=   1
        mvc    c0($rf8192),b0          no - drop recsize hdr
 #
 # test option b1 to convert nulls to ASCII blanks
 man40  tsb    o2(1),x'01'             convert nulls to ASCII blanks ?
        skp!   1
        rep    c0($rf8192),x'00',x'20'
 # test option b2 to convert nulls to EBCDIC blanks
        tsb    o2(1),x'02'             convert nulls to EBCDIC blanks ?
        skp!   1
        rep    c0($rf8192),x'00',x'40'
 #
 # test option t1 to insert Line-Feed Terminator in last byte of record
        tsb    o20(1),x'01'            insert LF in last byte ?
        skp!   1
        mvc    cg0(1),x'0A'            Yes - insert LF in last byte
 #
 # write current record to output file & return to get next
 man48  put    filo1,c0($rf8192)       write record to file#1 out
        skp    man20                   return to get next record
 #
 # EOF - close files & end job
 eof    cls    all
        eoj
 #

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

 # getvr - subrtn to get RDW variable length records
 #Dec19/07 - getvr subrtn no longer required
 #         - getvr replaced by file typ=RDW & typ=RDWz2
 #         - this subrtn saved (for interest) in $UV/pf/util/getvr
 #
 # - RDW format with no BDW
 # - record length in 1st 2 bytes of 4 with last 2 x'0000'
 # - record length includes data length + 4 bytes for RDW prefix
 # - blk/rec lth in binary big end format, need 's'witch option for INTEL
 #
 # - read record into area 'a' (1st 4 bytes will be RDW)
 # - sets rgstr 'a' to record length (including the 4 byte RDW)
 # - sets rgstr 'b' to record length (excluding the 4 byte RDW)
 # - return with cc > at EOF
 #
 # note - see alternate job BDW2LST1 to handle BDW+RDW
 #      - includes subrtn 'getvbr' to process BDW+RDW
 #
 # read 4 byte RDW to get record length (-4 for following data)
 getvr   get    fili1,a0(4)          read 4 byte block lth prefix
         skp>   getvr9
         cmc    a2(2),x'0000'        verify RDW (bytes 3&4 null)
         skp!   getvre1
         add    $ca9,1               count blocks (in case errmsg)
         mvn    $ra,a0(2bs)          store rec lth (switch ends INTEL)
         mvn    $rb,$ra              copy to $rb for subtract 4
         sub    $rb,4                -4 for following data length
         skp<=  getvre1              (err if 0 or negative)
 #
 # now read the data following 4 byte RDW
 getvr4  get    fili1,a4($rb8192)    read block (lth in rgstr c)
         skp>   getvr9
 #
 # exit with cc = for normal, or cc > for EOF
 getvr8  ret=
 getvr9  ret>
 #
 # error rtn for invalid block or record prefix
 getvre1 msgv1 'rec prefix 3&4 not x0000, reclth=$ra'
         mvf    g0(80),'1st 32 bytes='
         hxc    g13(64),a0(32)       convert 1st 32 bytes to hex
         msg    g0(80)               display block prefix
         can    'unrecoverable error - job cancelled'

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_9 DATAcnv1 - Mainframe Data file Conversion

Part 9 - Verifying Conversions & Generating Test Files


9A1. Converting ASCII files back to EBCDIC (for 1 file at a time)
- uvcopy job 'uvdata55'
- sample conversion for demo file 'citytax1'

9B1. Data File Comparison utility (uvcmp1)
- use to verify EBCDIC to ASCII file conversions, by converting
  ASCII file back to EBCDIC & comparing to original EBCDIC file

9C1. Generate ASCII-->EBCDIC jobs - for ALL data-files (vs 1 at a time)
- instructions are shorter & simpler than for 1 at a time

9D1. Verifying Data Conversion with 'uvhdcob'
- by verifing packed & numeric fields

9E1. Verifying Data Conversion with 'genverify1'
- generate uvcopy jobs to Verify all Packed & Numeric fields

9F1. Demonstrate 'genverify1' verifies ALL packed/numeric fields of 'occurs'
vs 'uvhdcob' which only verify the 1st field of any 'occurs'
using armaster test file which has 12 months sales for this year & last.

9G1. genacum2 - generate uvcopy jobs to accumulate all numeric fields in a file.
- generated automatically from the 'cobmap' record layout
  (which is generated from the COBOL copybook).

9H1. gentest1 - generate test data files automatically
- based on COBOL copybooks
- inserts cobol copybook fieldnames in type X fields in every 10th record
- otherwise, inserts 'A's in 1st field, 'B's in 2nd, etc
- inserts current record# in numeric & packed fields

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9A1. DATAcnv1 - Special Projects

converting EBCDIC back to ASCII

 Part_3 documented generating & executing uvcopy jobs to convert EBCDIC to ASCII
 - 1 job at a time. Part_6 documented the same thing for ALL copybooks & ALL
 data files in a dirctory.

We will now show you how to generate & execute jobs to convert the ASCII files back to EBCDIC (& then compare the reconverted files to the original mainframe files).

You might do this to verify your conversions &/or you might need the ASCII to EBCDIC conversions to return some files to the maifnrame if you can not switch all applications over to unix/linux at the same time.

We could generate the ASCII to EBCDIC jobs using 'uvdata31' the opposite of 'uvdata51' that we used to generate EBCDIC to ASCII, but it is much better to use 'uvdata55' to convert our already generated EBCDIC-->ASCII jobs to ASCII-->EBCDIC jobs.

Besides saving steps, the big advantage is that this method saves having to duplicate any manual coding required for files with multiple Record Types. With the addition of uvdata55, pfx4,& uvcmp1all, our flow charts will then be:

Generate jobs: EBCDIC-->ASCII & ASCII-->EBCDIC

 cpys -------> maps --------> pfx1 --------> pfx2 ---------> pfx3 -------> pfx4
      cobmap1       uvdata51       uvdata52       copy/edit       uvdata55

Execute jobs: EBCDIC-->ASCII, ASCII-->EBCDIC,& compare

 Main -----> d1ebc ----------> d2asc ----------> d3ebc --------> rptcmp
 frame FTP   uvcopyxx 'pfx3/*'  uvcopyxx 'pfx4/*'    uvcmp1all   mismatch rpts

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9A2. converting EBCDIC data-files back to ASCII

generate ASCII-->EBCDIC jobs - 1 file at a time


 #0a. Login as mvstest1 ---> /home/mvstest1
 #0b. Login as yourself --> your homedir
 #0c. cd $CNVDATA  <-- change to superdir with subdirs cpys,d1ebc,d2asc,etc
      ===========
 #0d. cdc          <-- same, alias cdc='cd $CNVDATA'
      ====             export CNVDATA=$HOME/cnvdata1 <-- in profile

 #1. mkdir pfx4       <-- make subdir for ASCII-->EBCDIC jobs
     ==========           (if not already existing)

 #2. uvcopy uvdata55,fili1=pfx4/citytax1,filo1=pfx5/citytax1
     =======================================================

generated uvcopy job to convert ASCII to EBCDIC

 # citytax1 - uvcopy job, ASCII to EBCDIC preserve packed, fix signs        #01
 opr='citytax1 - uvcopy code generated from copybook: citytax1.cpy '        #02
 uop=q0                                                                     #03
 was=a33000b33000                                                           #04
 fili1=?d2asc/citytax1,rcs=00128,typ=RSF                                    #05
 filo1=?d3ebc/citytax1,rcs=00128,typ=RSF                                    #06
 @run                                                                       #07
        opn    all                                                          #08
 loop   get    fili1,a0                                                     #09
        skp>   eof                                                          #10
        mvc    b0(00128),a0       move rec to outarea before field prcsng   #11
        tre    b0(00128)          translate entire outarea to EBCDIC        #12
 #      ---                <-- insert R/T tests here for redefined records  #13
        mvc    b88(4),a88              bns post-date                        #14
        mvc    b92(15),a92             pns land-value:face-value            #15
        trt    b107(9),$trtsae          ns maint-tax                        #16
 put1   put    filo1,b0                                                     #17
        skp    loop                                                         #18
 eof    cls    all                                                          #19
        eoj                                                                 #20

Compare this ASCII to EBCDIC job with the original EBCDIC to ASCII job job listed on page '3B2'. Aside from directory & file names, there are only 2 instructions different as follows:

        tra    b0(00128)          translate entire outarea to ASCII         #12
        tre    b0(00128)          translate entire outarea to EBCDIC        #12
        trt    b107(9),$trtsea     ns maint-tax                             #16
        trt    b107(9),$trtsae          ns maint-tax                        #16

Note that $trtsea is a table to translate Signs Ebcdic to Ascii and that $trtsae is a table to translate Signs Ascii to Ebcdic

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9A3. converting EBCDIC data-files back to ASCII

execute generated job to convert ASCII to EBCDIC


 #4a. uvcopy pfx4/citytax1,fili1=d2asc/citytax1,filo1=d3ebc/citytax1
      ==============================================================

 #4b. uvcopy pfx4/citytax1   <-- easier (filenames default as shown above)
      ====================

sample input ASCII


 #3. uvhd d2asc/citytax1 r128a  <-- display ASCII input file
     =========================
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 10139923  GEORGE BROWN             1250 EAST PENDER STREET  VANC
             3333333322444544245454222222222222233332445525444452555445225444
             101399230075F275022F7E0000000000000125005134005E45203425540061E3
          64 OUVER           BCV5L1W1......W.........xRL00014626q980601190156
             4554522222222222445343530888005880008800754333333337333333333333
             F5652000000000002365C171029C0072C0047C0182C000146261980601190156
                                                                ^--- ASCII -1

sample output EBCDIC


 #4. uvhd d3ebc/citytax1 r128   <-- display EBCDIC output file
     ==========================
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 10139923  GEORGE BROWN             1250 EAST PENDER STREET  VANC
             FFFFFFFF44CCDDCC4CDDED4444444444444FFFF4CCEE4DCDCCD4EEDCCE44ECDC
             1013992300756975029665000000000000012500512307554590239553005153
          64 OUVER           BCV5L1W1.bi...Wb....g...yR<00014626J980601190156
             DEECD44444444444CCEFDFEF0888005880008800754FFFFFFFFDFFFFFFFFFFFF
             645590000000000023553161029C0072C0047C0182C000146261980601190156
                                                                ^--- EBCDIC -1

We have shown record #2 because it has a negative zoned sign at byte 115 (the units position of the field in 107-115).

In ASCII a -1 is a 'q' or x'71'. The normal translate EBCDIC to ASCII (tre) would translate it to an EBCDIC 'q' x'98' which would be wrong, so the 'trt' instruction is used with special translate table '$trtsae' to translate Signs Ascii to Ebcdic. This table translates the 'q' to a 'J' (-1 in EBCDIC).

See more details about zoned sign conversion on page '3C2'

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9B1. converting EBCDIC data-files back to ASCII

Verifying Data Conversions

Pages '3C1' thru '3C6' documented converting EBCDIC to ASCII.

Pages '9A1' thru '9A3' documented converting ASCII to EBCDIC.

You could use the ASCII to EBCDIC conversion to verify the accuracy of the EBCDIC to ASCII conversion, by comparing the output of 2nd conversion to the input of the 1st. The 3 test/demo files involved are:

d1ebc/citytax1
  • original EBCDIC data file
d2asc/citytax1
  • converted to ASCII
d3ebc/citytax1
  • converted back to EBCDIC

We can't use the unix/linux 'diff' utility because these mainframe style data files are fixed length without the LineFeeds required by diff.

We might try the unix/linux 'cmp' utility. We will use the '-l' option, otherwise it quits at the 1st mismatch.


 cmp -l d1ebc/citytax1 d3ebc/citytax1   <-- compare 2 data files
 ======================================

Byte# file1 file2 116 301 361 372 302 362 500 303 363 628 311 371 756 300 360

The 'cmp' utility shows differences in 3 columns - byte#, 1st file byte contents, 2nd file byte contents. The byte contents are in 'octal' which is an anachronism but there seems to be no hexadecimal option.

A much better solution is to use a Vancouver Utility uvcopy job 'uvcmp1':


 uvcopy uvcmp1,fili1=d1ebc/citytax1,fili2=d3ebc/citytax1
 =========================================================

See the output report on the next page --->

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9B2. converting EBCDIC data-files back to ASCII

uvcmp1 file comparison report

 uvcmp1 - compare 2 files "*" diffs (uops=q1e0p30r256s5t500000v2x0e1r128)
 recsize  reccount   file-size  typ  filename
 1:  128        10       1,280  RSF  d1ebc/citytax1
 2:  128        10       1,280  RSF  d3ebc/citytax1
 2005/04/21_17:24:39   1         2         3         4         5         6
 f#rec#byte# 0123456789012345678901234567890123456789012345678901234567890123
 ============================================================================
 1    1    0 10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
             1013014000168508559800000000000000018150266550961400000000005153
          64 OUVER           BCV9S1H1 bi   Wb    g   qq 00014906A970530191809
             DEECD44444444444CCEFEFCF0888005880008000770FFFFFFFFCFFFFFFFFFFFF
             645590000000000023592181029C0072C0047C0111C000149061970530191809
                                                                *
 2    1      10130140  JOHN HENRY               1815 BOWEN ROAD          VANC
             FFFFFFFF44DDCD4CCDDE444444444444444FFFF4CDECD4DDCC4444444444ECDC
             1013014000168508559800000000000000018150266550961400000000005153
          64 OUVER           BCV9S1H1 bi   Wb    g   qq 000149061970530191809
             DEECD44444444444CCEFEFCF0888005880008000770FFFFFFFFFFFFFFFFFFFFF
             645590000000000023592181029C0072C0047C0111C000149061970530191809
                                                                *
 1    5    0 10154688  MEL RAHAL                1110 - 625 HOWE STREET   VANC
             FFFFFFFF44DCD4DCCCD4444444444444444FFFF464FFF4CDEC4EEDCCE444ECDC
             1015468800453091813000000000000000011100006250866502395530005153
          64 OUVER           BCV6C2T6 bi   Dh           00011547I961126082351
             DEECD44444444444CCEFCFEF0888004800000000208FFFFFFFFCFFFFFFFFFFFF
             645590000000000023563236029C0048C0000C0153C000115479961126082351
                                                                *
 2    5      10154688  MEL RAHAL                1110 - 625 HOWE STREET   VANC
             FFFFFFFF44DCD4DCCCD4444444444444444FFFF464FFF4CDEC4EEDCCE444ECDC
             1015468800453091813000000000000000011100006250866502395530005153
          64 OUVER           BCV6C2T6 bi   Dh           000115479961126082351
             DEECD44444444444CCEFCFEF0888004800000000208FFFFFFFFFFFFFFFFFFFFF
             645590000000000023563236029C0048C0000C0153C000115479961126082351
                                                                *

The differences shown are not a problem - they are caused by translating zoned signs from EBCDIC to ASCII & back, because there are no + zones in ASCII. The original + zones in EBCDIC x'C_' are lost (not needed) in ASCII where normal numeric zones x'3_' are the only + zones. When we convert back we get the normal numeric zones in EBCDIC x'F_', which cause the mismatches (but these are not errors).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9C1. converting EBCDIC data-files back to ASCII

generate ASCII-->EBCDIC jobs - for ALL data-files

Actually, the instructions to generate & execute jobs for for ALL data-files are shorter & simpler than for 1 at a time.


 #0a. Login as mvstest1 ---> /home/mvstest1
 #0b. Login as yourself --> your homedir
 #0c. cd $CNVDATA  <-- change to superdir with subdirs cpys,d1ebc,d2asc,etc
      ===========
 #0d. cdc          <-- same, alias cdc='cd $CNVDATA'
      ====             export CNVDATA=$HOME/cnvdata1 <-- in profile

 #1. mkdir pfx4       <-- make subdir for ASCII-->EBCDIC jobs
     ==========           (if not already existing)

 #2. uvcopyx uvdata55 pfx3 pfx4 uop=q0i7   <-- generate ALL jobs
     ===================================

Execute ALL jobs to convert ALL data-files


 #3. uvcopyxx 'pfx4/*'                     <-- execute ALL jobs
     =================

Compare ALL re-converted files to original EBCDIC files


 #4. mkdir rptcmp           <-- make subdir for mismatch reprots
     ============

 #5. uvcmp1all d1ebc d3ebc  <-- compare all original files to reconverted files
     =====================

 #6a. vi rptcmp/*          <-- inspect all mismatch reports
      ===========

 #6b. uvlpd12 rptcmp       <-- print all mismatch reports
      ==============

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9D1. Verifying EBCDIC to ASCII Conversion

using uvhdcob to verify numeric fields

You can use the 'v' command of uvhdcob to verify numeric fields (packed & unpacked) in the converted data files. Here is an example for d2asc/citytax1. First we need to update the file with some intentional errors:

Update file with BAD data


 #1. Login mvstest1 or yourself --> /home/mvstest1 or your homedir $HOME

 #2. cdc  <-- alias cdc='cd $CNVDATA' (CNVDATA=$HOME/cnvdata1)
     ===    - change to data conversion superdir to perform data conversion

 #3. uvhdcob d2asc/citytax1 maps/citytax1 u  <-- Update citytax1 demo file
     ======================================    - create intentional errs
     - uvhdcob displays 1st record (as follows) & prompts for comamnds
 rec#       1 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 folio                        0   9    10130140
 002 name                        10  34    JOHN HENRY
 003 address                     35  59    1815 BOWEN ROAD
 004 city                        60  79    VANCOUVER
 005 province                    80  81    BC
 006 zip                         82  87    V9S1H1
 007 post-date                   88  91bns 0082898C
 008 land-value                  92  96pns 000057828C
 009 improv-value                97 101pns 000004870C
 010 face-value                 102 106pns 000171710C
 011 maint-tax                  107 115 ns 000149061
 012 purchase-date              116 121    970530
 013 filler001                  122 127    191809
     --> 3              <-- advance to record#3
     --> u 92(1),x'EE'  <-- Update 1st byte of 'land-value' with bad Packed data
     --> u 107(1),'X'   <-- Update 1st byte of 'maint-tax' with bad Numeric data
     --> 6              <-- advance to record#6
     --> u 88(1),x'EE'  <-- Update 1st byte of 'post-date' with bad Binary data
     --> u 115(1),'X'   <-- Update last byte of 'maint-tax' with bad Numeric data

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9D2. Verifying EBCDIC to ASCII Conversion

On the prior page we updated d2asc/citytax1 with BAD data in Packed & Numeric fields in records #3 & #6. Now we will demo the uvhdcob 'v' verify command.

using uvhdcob to verify numeric fields


 #4. uvhdcob d2asc/citytax1 maps/citytax1 u  <-- Update citytax1 demo file
     =============================================
     -->     <-- uvhdcob displays 1st record (not shown) & prompts for command
     --> v   <-- enter 'v' to search/verify (until bad numeric fields found)
               - will display rec#3 (1st record with bad data) as follows:
 rec#       3 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 014 folio                        0   9    10147615
 015 name                        10  34    BONNIE SMITH
 016 address                     35  59    44430 YALE ROAD WEST
 017 city                        60  79    VANCOUVER
 018 province                    80  81    BC
 019 zip                         82  87    V2P6J1
 020 post-date                   88  91bns 0121395C
 021 land-value                  92  96pns EE0039623C            <-- Bad Packed
 022 improv-value                97 101pns 000000000C
 023 face-value                 102 106pns 000051914C
 024 maint-tax                  107 115 ns X00037692   583030303337363932<-BadN
 025 purchase-date              116 121    950601
 026 filler001                  122 127    201031
      v --> 3 recs verified, 3 ERRs found in rec# 3, 1st at byte# 92
      2 packed & 1 numeric bad signs/digits, 0 unprintables in char fields

---> vv <-- 'vv' command to continue verify

 rec#       6 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 014 folio                        0   9    10173320
 015 name                        10  34    SAM SMITH
 016 address                     35  59    2381 ROGERS AVE
 017 city                        60  79    VANCOUVER
 018 province                    80  81    BC
 019 zip                         82  87    V3K5Y2
 020 post-date                   88  91bns EE50498C              <-- Bad Binary
 021 land-value                  92  96pns 000060600C
 022 improv-value                97 101pns 000000000C
 023 face-value                 102 106pns 000169086C
 024 maint-tax                  107 115 ns 00015620X   303030313536323058<-BadN
 025 purchase-date              116 121    970530
 026 filler001                  122 127    191810
      vv --> 3 recs verified, 3 ERRs found in rec# 6, 1st at byte# 88
      2 packed & 1 numeric bad signs/digits, 0 unprintables in char fields

---> vv <-- 'vv' command to continue verify

      vv --> 4 records verified, No errors found
      *** EOF reached (enter -1 to see last rec)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9E1. Verifying EBCDIC to ASCII Conversion

genverify1 - generate jobs to Verify Packed/Numeric Fields

Verifying Numeric & Packed fields is the best way to detect any discrepancy between the copybook layouts & the actual data - especially effective for files with packed fields, where the unique sign codes must be in the last byte of each packed field.

We will demonstrate using the 'citytax1' file shown on the previous page where we used 'uvhdcob' to Verify Packed & Numeric fields.

Why do we need 'genverify1' if we can use uvhdcob ? - because uvhdcob displays & verifies ONLY the 1st field of any OCCURS. uvhdcob would not be conclusive for files such as armaster which has occurs 12 for this year & last year sales.

Generate jobs to verify All numeric fields in All files


 #1. Login mvstest1 or yourself --> /home/mvstest1 or your homedir $HOME

 #2. cdc  <-- alias cdc='cd $CNVDATA' (CNVDATA=$HOME/cnvdata1)
     ===    - change to data conversion superdir to perform data conversion

 #3. mkdir pfv1 pfv2 pfv3 rptver            <-- make subdirs for outputs
     ===========================

 #4. uvcopy cobmap1,fili1=cpys/citytax1.cpy,filo1=maps/citytax1
     ==========================================================
     - convert copybook to 'cobmap' (record layout)

 #5. uvcopy genverify1,fili1=maps/citytax1,filo1=pfv1/citytax1
     =========================================================
     - convert cobmap to uvcopy job to verify numeric/packed fields
Note
  • see the input/output listed on the next page --->

Alternative 'uvcopyx' script to verify All files


 #4a. uvcopyx cobmap1 cpys maps uop=q0i7p0
      ====================================
      - convert all COBOL copybooks to layouts

 #5a. uvcopyx genverify1 maps pfv1 uop=q0i7f0
      =======================================
      - convert all cobmaps to data verify jobs

 #6. uvcopy uvdata52,fili1=ctl/ctlfile54,fild2=pfv1,fild3=pfv2,uop=r1s1t0
     ====================================================================
     - insert datafilenames (vs copybook names)
     - see options explained within $UV/pf/IBM/uvdata52

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9E2. genverify1 - generate jobs to Verify Packed/Numeric Fields

sample cobmap Input to genverify1

 cobmap1  start-end bytes for cobol record fields    200306092029  pg# 0001
 cpys/citytax1.cpy             citytax1           RCSZ=00128  bgn  end  lth typ
  01  citytax1.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-4.          0088 0091  004bns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  x(00006).                 0116 0121  006
      10 filler001             pic  x(00006).                 0122 0127  006
 *RCSZ=00128                                                       0128

sample genverify1 Output job to verify numeric fields

 # uvcopy job to verify packed or zoned numeric fields
 opr='jobname=JOBNAME   - pfv2 name=datafilename'
 opr='copybook=citytax1.cpy - maps/pfv1 name=copybookname'
 arg1=JOBNAME,arg2=citytax1
 uop=q0,was=a33000t80000
 fili1=?d2asc/citytax1,rcs=00128,typ=RSF
 filo1=?rptver/citytax1,rcs=128,typ=LSTt
 lodv1=h0(100)
 DataFile=DATAFILE CopyBook=COPYBOOK $datetime
 LthPS            <--HexData--    <--NumData-- Rec# Byte FieldName-->
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 @run
        opn    all
        repf1  h0(100),'DATAFILE',$fili1
        repf4  h0(100),'COPYBOOK',$arg2
        wtbe   filo1,h0(100),h0(100)
 loop   get    fili1,a0
        skp>   eof
        vnft3  a92(5p),1,'land-value'
        vnft2  a97(5p),1,'improv-value'
        vnft2  a102(5p),1,'face-value'
        vnft2  a107(9z),1,'maint-tax'
        wtbe   filo1,t0(80),t0(80)
        skp    loop
 #
 eof    cls    all
        eoj

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9E3. genverify1 - generate jobs to Verify Packed/Numeric Fields

Execute generated job to Verify Numeric fields


 #6. uvcopy pfv1/citytax1   <-- execute job to verify Packed/Numeric fields
     ====================

 #7. cat rptver/citytax1    <-- display report (created in rptver/citytax1)
     ===================
 DataFile=d2asc/citytax1 CopyBook=citytax1 2012/10/16_09:57:01
 LthPS            <--HexData--    <--NumData-- Rec# Byte FieldName-->
 005p               EE0039623C                 0003 0092 land-value
 009              583030303337       X00037692 0003 0107 maint-tax
 005p               000169086E                 0006 0102 face-value
 009              303030313536       00015620X 0006 0107 maint-tax

uvhdcob rec# 003 - for comparison

 rec#       3 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 014 folio                        0   9    10147615
 015 name                        10  34    BONNIE SMITH
 016 address                     35  59    44430 YALE ROAD WEST
 017 city                        60  79    VANCOUVER
 018 province                    80  81    BC
 019 zip                         82  87    V2P6J1
 020 post-date                   88  91bns 0121395C
 021 land-value                  92  96pns EE0039623C            <-- Bad Packed
 022 improv-value                97 101pns 000000000C
 023 face-value                 102 106pns 000051914C
 024 maint-tax                  107 115 ns X00037692   583030303337363932<-BadN
 025 purchase-date              116 121    950601
 026 filler001                  122 127    201031
  1. The genverify1 reports have very little description, but are very compact, which is important when you are generating error reports for big files.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9F1. Verifying conversions by Verifying Numeric fields

genverify1 - ex#2 to demo occurs verify

Example#1 (citytax1) had no occurs fields. For example#2 we will use 'armaster' which has occurs fields for 12 months sales this year & last. Here are the illustrated procedures to generate & execute the 'genverify1' job for armaster.


 #1. Login mvstest1 or yourself --> /home/mvstest1 or your homedir $HOME

 #2. cdc  <-- alias cdc='cd $CNVDATA' (CNVDATA=$HOME/cnvdata1)
     ===    - change to data conversion superdir to perform data conversion

 #3. uvcopy cobmap1,fili1=cpys/armaster.cpy,filo1=maps/armaster
     ==========================================================
     - convert copybook to 'cobmap' (record layout)

cobmap (record layout) for armaster

 cobmap1  start-end bytes for cobol record fields    200902200736  pg# 0001
 cpys/armaster.cpy                                RCSZ=00256  bgn  end  lth typ
 * armaster - cobol copy book for armaster file (see uvtrain.
      10 cm-num           pic  9(6).                          0000 0005  006 n  6
      10 cm-delete        pic  x(4).                          0006 0009  004
      10 cm-name          pic  x(25).                         0010 0034  025
      10 cm-adrs          pic  x(25).                         0035 0059  025
      10 cm-city          pic  x(16).                         0060 0075  016
      10 filler001        pic  x.                             0076 0076  001
      10 cm-prov          pic  x(2).                          0077 0078  002
      10 filler002        pic  x.                             0079 0079  001
      10 cm-postal        pic  x(10).                         0080 0089  010
      10 cm-phone         pic  x(12).                         0090 0101  012
      10 cm-contact       pic  x(18).                         0102 0119  018
      10 cm-thisyr-sales  pic  s9(7)v99 comp-3 occurs 12.     0120 0124  005pns 9
      10 cm-lastyr-sales  pic  s9(7)v99 comp-3 occurs 12.     0180 0184  005pns 9
      10 filler003        pic  x(16).                         0240 0255  016
 *RCSZ=00256

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9F2. Verifying conversions by Verifying Numeric fields

uvhdcob Verify armaster

We want to demonstrate that 'genverify1' generates uvcopy jobs that will verify ALL packed/numeric fields vs 'uvhdcob' which can only verify the 1st field of any 'occurs'.

So, before we demo 'genverify1' (on the next page), we will 1st show you the 'uvhdcob' verify on this page.


 #4. uvhdcob d2asc/cutsmas1 maps/armaster   <-- execute uvhdcob for armaster
     ====================================
Note
  • uvhdcob displays the 1st record & prompts for a command
  • we will omit the 1st record display to save space

--> v <-- search & Verify character,numeric,& packed fields

 rec#      10 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 cm-num                       0   5 n  15468X      313534363858      <-BadN
 002 cm-delete                    6   9
 003 cm-name                     10  34    TAURUS RESOURCES
 004 cm-adrs                     35  59    1110 - 625 HOWE STREET
 005 cm-city                     60  75    VANCOUVER
 006 filler001                   76  76
 007 cm-prov                     77  78    B.          42EE              <-BadC
 008 filler002                   79  79
 009 cm-postal                   80  89    V6C2T6
 010 cm-phone                    90 101    604 685 8565
 011 cm-contact                 102 119    MEL RAHAL
 012 cm-thisyr-sales       012  120 124pns EE0123456C            <-- Bad Packed
 013 cm-lastyr-sales       012  180 184pns 000000000C
 014 filler003                  240 255    C 19980331
      v --> 10 recs verified, 4 ERRs found in rec# 10, 1st at byte# 0
      2 packed & 1 numeric bad signs/digits, 1 unprintables in char fields

--> vv <-- continue Verify

 rec#      20 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 cm-num                       0   5 n  2343XX      323334335858      <-BadN
 002 cm-delete                    6   9
 003 cm-name                     10  34    TERRITORIAL REWIND
 004 cm-adrs                     35  59    BOX 1648
 005 cm-city                     60  75    WHITEHORSE
 006 filler001                   76  76
 007 cm-prov                     77  78    YK
 008 filler002                   79  79
 009 cm-postal                   80  89    Y1A...      593141EEEEEE      <-BadC
 010 cm-phone                    90 101    403-512-3948
 011 cm-contact                 102 119
 012 cm-thisyr-sales       012  120 124pns 000000000C
 013 cm-lastyr-sales       012  180 184pns 1E2E3E4E5E            <-- Bad Packed
 014 filler003                  240 255      20021130
      vv --> 10 recs verified, 10 ERRs found in rec# 20, 1st at byte# 0
      5 packed & 2 numeric bad signs/digits, 3 unprintables in char fields

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9F3. Verifying conversions by Verifying Numeric fields

Generate uvcopy verify job for armaster


 #5. uvcopy genverify1,fili1=maps/armaster,filo1=pfv1/armaster
     =========================================================
     - convert cobmap to uvcopy job to verify numeric/packed fields
 # uvcopy job to verify packed or zoned numeric fields
 opr='jobname=JOBNAME   - pfv2 name=datafilename'
 opr='copybook=armaster - maps/pfv1 name=copybookname'
 uop=q0,was=a33000t80000
 fili1=?d2asc/armaster,rcs=00256,typ=RSF
 filo1=?rptver/armaster,rcs=128,typ=LSTt
 @run
        opn    all
 loop   get    fili1,a0
        skp>   eof
        vnft3  a0(6z),1,'cm-num'
        vnft2  a120(5p),12,'cm-thisyr-sales'
        vnft2  a180(5p),12,'cm-lastyr-sales'
        wtbe   filo1,t0(80),t0(80)
        skp    loop
 eof    cls    all
        eoj

Execute Verify job for armaster


 #5. uvcopy pfv1/armaster     <-- execute verify job for armaster
     ====================

 #6. uvlp12 rptver/armaster   <-- print report
     ======================

 lth(p=packed)   hex-representation   numeric-data rec# occurs# field-name
 =================================================================
 06                    313534363858          15468X 010 cm-num
 05p                     EE0123456C                 010 01 cm-thisyr-sales
 05p-                    E01234567D                 010 12 cm-lastyr-sales
 06                    323334335858          2343XX 020 cm-num
 05p                     EE0012345C                 020 03 cm-thisyr-sales
 05p                     1E2E3E4E5E                 020 01 cm-lastyr-sales
 05p                     00000000EE                 020 12 cm-lastyr-sales
  1. Compare the errors reported here by the genverify job to those reported by uvhdcob on the previous page.

  2. genverify reports errors in rec# 010 occurs# 12 & in rec# 020 occurs 03 & 12 which were missed by uvhdcob since it only verifies the 1st field of occurs.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9G1. Verifying Conversions by Accumulating All Numeric fields

genacum2 - generate jobs to accum ALL numeric fields

'genacum2' is a Vancouver Utility to generate uvcopy jobs to accumulate all numeric fields (packed/zoned/binary) in 1 or all data files. The jobs are generated automatically from the 'cobmap' record layout (which is generated from the COBOL copybook).

You might use these reports to verify data file conversions, OR to get a 2nd opinion on suspicious looking totals on your COBOL reports.

We will demonstrate using the 'armaster' copybook & datafile, which has occurs fields for 12 months sales this year & last. Here are the illustrated procedures to generate & execute the 'genacum2' job for armaster.


 #1. Login mvstest1 or yourself --> /home/mvstest1 or your homedir $HOME

 #2. cdc  <-- alias cdc='cd $CNVDATA' (CNVDATA=$HOME/cnvdata1)
     ===    - change to data conversion superdir to perform data conversion

 #3. mkdir pfa1 pfa2 pfa3   <-- make subdirs for genacum2 jobs
     ====================

Directories Required

 /home/mvstest1/cnvdata     <-- or use your homedir with directory cnvdata1
 /home/yourhomedir/cnvdata1
 :-----cpys/              <-- copybooks
 :     :----armaster.cpy    - copybook 'armaster' for datafile 'armaster'
 :-----d2asc/             <-- data files (previously converted to ASCII)
 :     :-----armaster       - datafile 'armaster'
 :-----maps/              <-- record layouts (cobmaps) created from copybooks
 :     :-----armaster       - cobmap 'armaster'
 :-----pfa1/              <-- uvcopy jobs generated from cobmaps by 'genacum2'
 :     :-----armaster       - uvcopy job 'armaster' to add all fields in datafile
 :-----pfa2/              <-- reserved in case you use uvdata52 insert datafilenames
 :-----pfa3/              <-- uvcopy jobs copied here before execution
 :     :                    - modified if required (if multiple record types)
 :-----rptacu/            <-- reports written here
 :     :-----armaster       - reports named same as copybook & datafile
                              (run uvdata52 to change to datafilenames)
  1. This is similar to prior conversions (EBCDIC to ASCII or data validate)

  2. We will create subdirs pfa1,pfa2,pfa3 for the generated 'acum' jobs using the same filename for copybook,datafile,cobmap,& uvcopy acum job which simplifies the generation scripts & operating instructions

  3. We can run the generated job from pfa1 (still named same as copybook) The pfa1/... job expects input datafile in d2asc/ & writes report to rptacu/

  4. For major projects we would setup a control file to relate the copybook names to your actual datafilenames & run 'uvdata52' which copies pfa1/... jobs to pfa2/... inserting datafilenames & renaming same as datafiles.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9G2. Verifying Conversions by Accumulating All Numeric fields

convert copybook to cobmap (prelim to genacum2)


 #4. uvcopy cobmap1,fili1=cpys/armaster.cpy,filo1=maps/armaster
     ==========================================================
     - convert copybook to 'cobmap' (record layout)

cobmap (record layout) for armaster

 cobmap1  start-end bytes for cobol record fields    200902200736  pg# 0001
 cpys/armaster.cpy                                RCSZ=00256  bgn  end  lth typ
 * armaster - cobol copy book for armaster file (see uvtrain.
      10 cm-num           pic  9(6).                          0000 0005  006 n  6
      10 cm-delete        pic  x(4).                          0006 0009  004
      10 cm-name          pic  x(25).                         0010 0034  025
      10 cm-adrs          pic  x(25).                         0035 0059  025
      10 cm-city          pic  x(16).                         0060 0075  016
      10 filler001        pic  x.                             0076 0076  001
      10 cm-prov          pic  x(2).                          0077 0078  002
      10 filler002        pic  x.                             0079 0079  001
      10 cm-postal        pic  x(10).                         0080 0089  010
      10 cm-phone         pic  x(12).                         0090 0101  012
      10 cm-contact       pic  x(18).                         0102 0119  018
      10 cm-thisyr-sales  pic  s9(7)v99 comp-3 occurs 12.     0120 0124  005pns 9
      10 cm-lastyr-sales  pic  s9(7)v99 comp-3 occurs 12.     0180 0184  005pns 9
      10 filler003        pic  x(16).                         0240 0255  016
 *RCSZ=00256

On the next page, we will run genacum2 to generate a job to add all fields in test file d2asc/armaster (based on the copybook converted to a cobmap as shown on the previous page).

But first (to understand the datafile), you might run 'uvhdcob' which displays the data field contents beside the COBOL copybook fieldnames.


 #5. uvhdcob d2asc/cutsmas1 maps/armaster  <-- execute uvhdcob for armaster
     ====================================

uvhdcob display - 1st record of d2asc/armaster

 rec#       1 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 cm-num                       0   5 n  130140
 002 cm-delete                    6   9
 003 cm-name                     10  34    EVERGREEN MOTORS LTD.
 004 cm-adrs                     35  59    1815 BOWEN ROAD
 005 cm-city                     60  75    NANAIMO
 006 filler001                   76  76
 007 cm-prov                     77  78    BC
 008 filler002                   79  79
 009 cm-postal                   80  89    V9S1H1
 010 cm-phone                    90 101    250-754-5531
 011 cm-contact                 102 119     LARRY WRENCH
 012 cm-thisyr-sales       012  120 124pns 001234567C
 013 cm-lastyr-sales       012  180 184pns 001234567D
 014 filler003                  240 255    C 19950531

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9G3. Verifying Conversions by Accumulating All Numeric fields

Generate 1 job to accum ALL num-fields in 1 file


 #6. uvcopy genacum2,fili1=maps/armaster,filo1=pfa1/armaster,uop=f0
     ==============================================================
     - convert cobmap to uvcopy job to accumulate All numeric fields

console prompts for options

 uop=f2m1 - option defaults (for generated jobs)
     f0   - Input file default typ=RSF fixed lth (in generated job)
     f2   - Input typ=RDWz2 (2 byte BIG-end binary recsize hdrs)
     f4   - Input file typ=RDWz4 (4 byte hdrs, bytes 3&4 nulls)
       m1 - read copybook record layouts from maps/... in $CNVDATA
       m2 - read copybook record layouts from maps/... in $RUNLIBS
 User OPtion (uop) defaults  = q1f2m1
 -->null to accept or enter/override --> f0  <-- enter 'f0' for typ=RSF
Note
  • specify option 'f0' to generate job with 'typ=RSF' (on fili1=... below)
  • default f2 would generate typ=RDW2
  • we specified 'f0' on both command line & at prompt only for emphasis

generated job pfa1/armaster

 # uvcopy job to accumulate all numeric fields in a datafile
 opr='jobname=JOBNAME   - pfa2 name=datafilename'
 opr='copybook=armaster.cpy - maps/pfa1 name=copybookname'
 uop=q0i3,rop=r1  # minimize startup msgs, prompt for report disposition
 was=a33000c200000t100000  # 33K maxrec, 200K cpybkmap, 100K acum tbl
 fili1=?d2asc/armaster,rcs=00256,typ=RSF
 filo1=?rptacu/armaster,rcs=128,typ=LSTt
 fili2=${CNVDATA}/maps/armaster,rcs=128,typ=LST
 @run
        opn    all
        rtb    fili2,c0(100),c0(100)  read cobmap into area c table
        clr    t0(100000),x'00'       clear area t for accumulator table
 loop   get    fili1,a0               get next record into area 'a'
        skp>   eof
        add    $cb3,1                 count all record types
 #      ---    ---      <-- insert any Record Type tests here
        mvn    $rt,0                  init register t for current record
        mvn    $rs,0                  init register s for current record
        add    ts9800(8b),1           count 1st (or only) R/T in copybook
        bal    acum,'0,6,z,1,cm-num'
        bal    acum,'120,5,p,12,cm-thisyr-sales'
        bal    acum,'180,5,p,12,cm-lastyr-sales'
        skp    loop
 eof    bal    dump                   dump tables of fieldnames & acummulators
        cls    all
        eoj
 @pf2=acum2.sub

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9G4. Verifying Conversions by Accumulating All Numeric fields

Execute job to acum all numeric fields in armaster


 #7. uvcopy pfa1/armaster     <-- execute job to add all numeric fields armaster
     ====================

 #8. uvlp12 rptacu/armaster   <-- print report
     ======================

sample report

 genacum2  RCSZ=00256 DataFile=d2asc/armaster
 2009/02/26_09:22:03  copybook=cpys/armaster.cpy
 line# Field-Name                 occurs start lth type         field-total
    1  cm-num                             0005 006  n            7,259,353
    2  cm-thisyr-sales                12  0124 005 pns           99,697.93
    3  cm-lastyr-sales                12  0184 005 pns           45,440.85
          Record count        32 **      Hash-Total             21,773,231 **

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9G5. Verifying Conversions by Accumulating All Numeric fields

generate jobs for ALL files

The above procedures were for 1 file at a time. For major projects we would want to generate jobs for all copybooks & all datafiles. Actually the operating instructions are shorter & easier to key.


 #1. Login mvstest1 or yourself --> /home/mvstest1 or your homedir $HOME

 #2. cdc  <-- alias cdc='cd $CNVDATA' (CNVDATA=$HOME/cnvdata1)
     ===    - change to data conversion superdir to perform data conversion

 #3. uvcopyx cobmap1 cpys maps uop=q0i7p0
     ====================================
     - convert COBOL copybooks to layouts (field start/end on right)

 #4. uvcopyx genacum2 maps pfa1 uop=q0i7f0
     =====================================
     - convert cobmaps to uvcopy jobs to accumulate all numeric fields
     - I/O datafiles coded as copybook names

 #5. vi ctl/ctlfile54  <-- edit control file to relate copybooks to datafiles
     ================
      # ctlfile54 - sample control file to relate copybooknames & datafilenames
      ar/customer.master       cpy=armaster rcs=00256 typ=RSF
      gl/account.master_       cpy=glmaster rcs=00128 typ=RDWz2
      ap/vendor.master         cpy=vendmas  rcs=00256 rcs=257 keys=0(10)

 #6. uvcopy uvdata52,fili1=ctl/ctlfile54,fild2=pfa1,fild3=pfa2,uop=r1s1t0
     ====================================================================
     - insert datafilenames (vs copybook names)
     - option r1 = override fili1 recsize with recsize from control file
              s1 = set input typ=... from control file
              t0 = do not change output file type (typ=LSTt from genverify1)

 #7. cp pfa2/* pfa3
     ==============
     - copy completed uvcopy data conversion jobs to execution subdir

 #8. vi pfa3/*          <-- edit for files with multiple record Types
     =========            - need to insert R/T test code & skip to
                            corresponding bank of auot generated code

 #9. uvcopyxx 'pfa3/*'  <-- execute all jobs to acum all num-fields in all files
     =================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9H1. DATAcnv1 - Convert EBCDIC to ASCII with COBOL copybooks

gentest1 - generate test data files based on COBOL copybooks

'gentest1' will generate uvcopy jobs to generate test data files based on the COBOL copybooks. The cobol copybooks are 1st converted to the 'cobmap' record layouts by the 'cobmap1' utility.

The generated uvcopy jobs are named after the copybooks and they call sub- routines to generate data for each field. Each picture field in the copybook generates a 'bal' to a subrtn, depending on field type (type X, packed, binary, signed/unsigned numeric). For example:


   bal  genx,'a00035b0025','address'    <-- sample call to subrtn
   =================================

This calls subrtn 'genx' for a type X field starting at byte 35 (option a00035) that is 25 bytes long (option b0025), whose fieldname is 'address'.

Here is a summary of the subrtns called & a description of the actions performed for the various field types:

genx
  • generates data in type X fields
  • inserts field-names in every 10th record
  • else generates 'A's in 1st field, 'B's in 2nd, 'C's in 3rd
  • inserts current record# in bytes 5-10 of fields longer than 15 bytes
gen9
  • generates data in numeric unpacked unsigned fields
  • inserts current record# in entire field, if field longer than 5 bytes
  • else, generates '1's in 1st field, '2's in 2nd, '3's in 3rd
gens
  • generates data in numeric Signed fields
  • inserts current record# in entire field
  • makes all fields in every 3rd record negative
genp
  • generates data in packed fields
  • inserts current record# in entire field
  • makes all fields in every 3rd record negative
genb
  • generates data in binary fields
  • inserts current record# in entire field
  • makes all fields in every 3rd record negative

Calling subrtns keeps the generated uvcopy jobs short since most of the field processing code is in these sub-routines. The subrtn module is called via the following code, which you can see at the end of the sample job listed on the next page.

@pf2=gentest1.sub <-- code to include subfunctions in uvcopy jobs

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9H2. gentest1 - generate test data files based on COBOL copybooks

gentest1 - sample input

 cobmap1  start-end bytes for cobol record fields    200408220851  pg# 0001
 cpys/citytax1.cpy             citytax1           RCSZ=00128  bgn  end  lth typ
 * citytax1 copybook
  01  citytax1.
      10 folio                 pic  x(00010).                 0000 0009  010
      10 name                  pic  x(00025).                 0010 0034  025
      10 address               pic  x(00025).                 0035 0059  025
      10 city                  pic  x(00020).                 0060 0079  020
      10 province              pic  x(00002).                 0080 0081  002
      10 zip                   pic  x(00006).                 0082 0087  006
      10 post-date             pic s9(00007) comp-4.          0088 0091  004bns 7
      10 land-value            pic s9(00009) comp-3.          0092 0096  005pns 9
      10 improv-value          pic s9(00009) comp-3.          0097 0101  005pns 9
      10 face-value            pic s9(00009) comp-3.          0102 0106  005pns 9
      10 maint-tax             pic s9(00007)v99.              0107 0115  009 ns 9
      10 purchase-date         pic  x(00006).                 0116 0121  006
      10 filler001             pic  x(00006).                 0122 0127  006
 *RCSZ=00128                                                       0128

sample output

 opr='JOBNAME citytax1 - generated by cobmap1 & gentest1'
 opr='uop=c100   - option "c" default record output count = 100'
 opr='           - may change at prompt (ex: enter "c2000" for 2000 records)'
 uop=q1c100      # option defaults
 was=a9000b9000
 filo1=?tmp/citytax1,rcs=00128,typ=RSF
 @run
        opn    all
        bal    initj                 inits for job (setup A-Z 0-9 patterns)
 # begin loop to generate fields & output each record
 loop   bal    initr                 inits for record (counts records,etc)
        clr    b0(00128),' '         init output area to all blanks
 #      ---                          <-- insert R/T tests if redef records
        bal    genx,'a00000b0010','folio'
        bal    genx,'a00010b0025','name'
        bal    genx,'a00035b0025','address'
        bal    genx,'a00060b0020','city'
        bal    genx,'a00080b0002','province'
        bal    genx,'a00082b0006','zip'
        bal    genb,'a00088b0004','post-date'
        bal    genp,'a00092b0005','land-value'
        bal    genp,'a00097b0005','improv-value'
        bal    genp,'a00102b0005','face-value'
        bal    gens,'a00107b0009','maint-tax'
        bal    genx,'a00116b0006','purchase-date'
        bal    genx,'a00122b0006','filler001'
 #
 put1   put    filo1,b0              output current record
        cmn    $ca1,$uopbc           reached output Count ?
        skp<   loop
 eof    cls    all
        eoj
 @pf2=gentest1.sub

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9H3. gentest1 - generate test data files based on COBOL copybooks

Op-Instrns for citytax1 demo file


 1. uvcopy cobmap1,fili1=cpys/citytax1.cpy,filo1=maps/citytax1
    ==========================================================
           - convert the COBOL copybook into a 'cobmap' (record layout)

 2. uvcopy gentest1,fili1=maps/citytax1,filo1=pfg1/citytax1
    =======================================================
           - generate the uvcopy job from the cobmap

 3. uvcopy pfg1/citytax1,filo1=tmp/citytax1  <-- execute the generated job
    =======================================

 3a. uvcopy pfg1/citytax1    <-- same (output file defaults as shown above)
     ====================

console displays for option changes

uop=c100
  • option "c" default record output count = 100
  • may change at prompt (ex: enter "c2000" for 2000 records)

User OPtion (uop) defaults = q1c100 null to accept or re-specify (1 or more) --> c500 <-- may change default

tmp/citytax1 = default filo01 - null accept or reenter --> <-- null accept

 citytax1:040823:105849: EOF filo01 0 rds, 100 wrts, 12800 size; tmp/citytax1

 4. uvhd filename=tmp/citytax1 r128          <-- examine the output
    ===============================

Please see the 'uvhd' output display shown on the next page --->

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9H4. gentest1 - generate test data files based on COBOL copybooks

Here is the 'uvhd' command repeated from the previous page, but with option 's7' added to space out the display to make it easier to read.


 4. uvhd filename=tmp/citytax1 r128s7          <-- examine the output
    =================================

output from citytax1 demo file

                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 folio     name                     address                  city
             6666622222666622222222222222222222266676772222222222222222226677
             6FC9F00000E1D500000000000000000000014425330000000000000000003949
          64                 przip   ...................000000001purchafiller
             2222222222222222777672220001000010000100001333333333777666666667
             000000000000000002A90000000C0000C0000C0000C00000000105238169CC52
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
         128 AAAAAAAAAABBBBB000002BBBBBBBBBBBBBBCCCCC000002CCCCCCCCCCCCCCDDDD
             4444444444444443333334444444444444444444333333444444444444444444
             1111111111222220000022222222222222233333000002333333333333334444
          64 D000002DDDDDDDDDEEFFFFFF...,....,....,....,000000002GGGGGGHHHHHH
             4333333444444444444444440002000020000200002333333333444444444444
             400000244444444455666666000C0000C0000C0000C000000002777777888888
                      10        20        30        40        50        60
 r#        3 0123456789012345678901234567890123456789012345678901234567890123
         256 AAAAAAAAAABBBBB000003BBBBBBBBBBBBBBCCCCC000003CCCCCCCCCCCCCCDDDD
             4444444444444443333334444444444444444444333333444444444444444444
             1111111111222220000032222222222222233333000003333333333333334444
          64 D000003DDDDDDDDDEEFFFFFF...=....=....=....=00000000sGGGGGGHHHHHH
             4333333444444444444444440003000030000300003333333337444444444444
             400000344444444455666666000D0000D0000D0000D000000003777777888888

Please relate this output to the copybook & the rules listed previously:

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9H5. gentest1 - generate test data files based on COBOL copybooks

'gentestA' script

The 'gentestA' script is provided to make it easier to run the gentest1 utility. The script includes both the cobmap1 & gentest1 uvcopy jobs, reducing the operating instructions to 1 line:


 gentestA citytax1      - generates uvcopy job --> pfg1/citytax1
 =================

 uvcopy pfg1/citytax1,filo1=tmp/citytax1  <-- execute the generated job
 =======================================

Op-Instns for all files in directory


 #1. uvcopyx cobmap1 cpys maps uop=q0i7p0  <-- create cobmaps for all copybooks
     ====================================

 #2. uvcopyx gentest1 maps pfg1 uop=q0i7   <-- create data gen jobs for all maps
     ===================================

 #3. uvcopyX 'pfg1/*'                      <-- execute all data generate jobs
     ================

You might want to generate all test data generation jobs as shown in #1 & #2 above, but it is unlikely you would want to execute all at once because they would be generated in the 'tmp' subdir & all default to 100 records.

The uvcopy code for gentest1 & gentest1.sub is not shown here or on the web site documentation, but if you have Vancouver Utilities installed, you may find the code as follows:

 /home/uvadm/pf/IBM/gentest1  - uvcopy code to generate the uvcopy jobs
                                to generate the test data files from copybooks
 /home/uvadm/pf/IBM/gentest1.sub - uvcopy code for the sub-routines called
                                   by the generated uvcopy jobs

Note that the generated jobs would be in your sub-directories. If you had followed the recommended setup procedures (setting up userid 'cnvdata1'), this would be:

/home/mvstest1/cnvdata1/pfg1/xxxxxx

Each of the generated jobs loads the subrtns via the code found on the last line of each generated job, which is:

 @pf2=gentest1.sub          <-- loads the subrtn code

The subrtns will be loaded from /home/uvadm/pf/IBM/gentest1.sub. If you want to modify the subrtns, you should 1st copy to your own 'pf' directory & place your directory ahead of the 'uvadm' pf directory in the PFPATH variable that uvcopy uses to load uvcopy job code. Do not modify in the uvadm pf directory, because any later update install of Vancouver Utilities would overwrite your modified code.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_10 DATAcnv1 - Convert EBCDIC to ASCII with COBOL copybooks

Part 10 - Summary of Data conversion Scripts - Contents

 10A1. Summary of scripts used in this book (DATAcnv1.doc)
      (gencnvD,signscanA,sign2cpyA,cobmapA,gencnvA,genrfmA,genpipeA,listpipeA)
 10B1. Summary of uvcopy jobs used in this book
      (cobmap1,uvdata51,uvdata52,genpipe1,sqlcreate1,signscan2,sign2cpy1)
Note
  • only a few of the scripts will be listed here (marked with '*')
  • you can inspect any of them in /home/uvadm/sf/IBM/...
  • none of the uvcopy jobs are listed since they are longer than scripts
  • you can inspect any of them in /home/uvadm/pf/IBM/...

10C1. *cobmapA - listing of script to convert 1 copybook to a 'cobmap'

10C2. *gencnvA - script to generate job to convert 1 data file EBCDIC to ASCII
- combines jobs to create cobmap, convert EBCDIC to ASCII,
& convert ASCII file to pipe|delimited in d4pipe/...

10D1. *gencnvD2 - script to generate all conversion jobs from 1 copybook
- cobmap, uvcopy jobs EBCDIC to ASCII, pipe-delimit,
pipe-delimit to Fixed-field, create SQL table, load SQL table

10D2. *gencnvD2.bat - same for Windows/DOS

10E1. *gencnvC - script to generate EBCDIC to ASCII jobs from directory
of filenames in d1ebc/..., using a control-file
edited with copybook names corresponding to datafilenames
(auto-generated if copybooknames embeded in datafilenames)

10F1. *uvcopypfxgA - Execute all uvcopy jobs in a directory(arg1) to convert all
files from 1 directory(arg2) to a 2nd directory(arg3)

10G2. *uvcopypfx2 - execute pfx1/..., pfp1/...,& listpipe jobs for any 1 file
- EBCDIC to ASCII & pipe|delimit jobs
- also creates uvhd & uvhdcob displays
in subdirs uvhd_d1ebc, uvhd_d2asc, uvhdcob (d2asc only)
- also see uvcopypfx9 to execute all jobs for all files

10H1. *genrfmA - generate jobs to reformat records based on 2 copybooks
example to unpack all packed/binary fields to numeric zoned fields

10I1. *copysub2 - collect related files from multiple subdirs to 1 output dir
appending subdir names, so common filenames not overwritten

10J1. *uvcmp1all - script to compare all files in 2 directories
- use to compare reconverted files to original mainframe files
- to verify conversions
& in case all applications can go live at the same time

10K1. cnvdatainit - script to clear ouputs of previous test/demos before re-runs
- clears d2asc,d4pipe,pfx1,pfx2,pfx3,pfp1,etc

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10A1. DATAcnv1 - Converting Mainframe Data Files from EBCDIC to ASCII

Summary of scripts used in DATAcnv1.doc

Only a few '*' are listed on the following pages, but you can view or print any of them from /home/uvadm/sf/IBM/...


10C1. *cobmapA - listing of script to convert 1 copybook to a 'cobmap'

10C2. *gencnvA - script to generate job to convert 1 data file EBCDIC to ASCII
- combines jobs to create cobmap, convert EBCDIC to ASCII,
& convert ASCII file to pipe|delimited in d4pipe/...
      gencnvAp - ensure packed sign x'_C' for AIX COBOL
      gencnvD1 - generate all conversion jobs (for sites with no COBOL copybooks)
               - gencnvD1 is not practical, since generating copybooks is not perfect
               - you usually need to do manual corrections after signscan2 & sign2cpy1
               - so run signscanA & sign2cpyA separately & then run gencnvD2 (below)

10D1. *gencnvD2 - generate all conversion jobs from COBOL copybook
- cobmap1, uvdata51, genpipe1, genfixd2, sqlcreate1 (Oracle)
      gencnvD2p - same for AIX COBOL (ensure packed signs x'_C' vs x'_F'

10D2. *gencnvD2.bat - same for Windows/DOS
      gencnvD9 - generates ALL conversion jobs for ALL copybooks in cpys/
               - generates jobs in pfx1, pfp1, pfy1, pfq1, pfg1, sqlTC1, sqlTL1
      gencnvE  - generate uvcopy job to translate ASCII file back to EBCDIC
               - opposite of gencnvA
               - could then compare to original using 'uvcmpFE1' (script for uvcmp1)
      signscanA - determine record layout when no COBOL copybook available
                - script signscanA runs uvcopy job signscan2 which scans for
                  packed fields, zoned signs, date fields, and right blank filled
                  fields (such as Name & Address).
      sign2cpyA - converts the signscanA record layout to a COBOL copybook which
                  is used by following scripts to perform various conversions.
      genpipeA - generates uvcopy jobs to convert data to a '|' pipe delimited
                 ASCII text file. Packed & numeric fields will be converted to zero
                 suppressed numeric fields with leading sign +/-.
               - Pipe delimited format is used for loading Relational data bases.
               - also generates pipe|delimited back to Fixed-field (as of Feb2012)
      listpipeA - by default lists the 1st 3 records of the pipe delimited file
                  first as is (1 long line), then field by field with copybook names

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10A2. Summary of scripts used in DATAcnv1.doc (continued)


10E1. *gencnvC - script to generate EBCDIC to ASCII jobs from directory
of filenames in d1ebc/..., using a control-file
edited with copybook names corresponding to datafilenames
(auto-generated if copybooknames embeded in datafilenames)

10E2. gencnvC2 - script to generate pipe|delimit jobs from directory

10F1. *uvcopypfxgA - Execute all uvcopy jobs in a directory(arg1) to convert all
files from 1 directory(arg2) to a 2nd directory(arg3)
      uvcopypfx1 - execute pfx1/..., pfp1/...,& listpipe jobs for any 1 file
                 - EBCDIC to ASCII & pipe|delimit jobs

10G2. *uvcopypfx2 - execute pfx1/..., pfp1/...,& listpipe jobs for any 1 file
- EBCDIC to ASCII & pipe|delimit jobs
- also creates uvhd & uvhdcob displays
in subdirs uvhd_d1ebc, uvhd_d2asc, uvhdcob (d2asc only)
- also see uvcopypfx9 to execute all jobs for all files
      uvcopypfx9 - execute ALL pfx1/* & pfp1/* jobs for ALL datafiles in d1ebc/*

10H1. *genrfmA - generate jobs to reformat records based on 2 copybooks
example to unpack all packed/binary fields to numeric zoned fields

10I1. *copysub2 - script to gather all jobs & data for 1 file, from subdirs to 1 outdir
- appends _subdir names on filename, allowing cohabitation in 1 dir
- might use to send results to other parties for inspection
Note
  • maintaining the same filename in multiple subdirs makes generations easy
    but does make it difficult to gather results in 1 directory
    (without the copysub2 script)

10J1. *uvcmp1all - compare all files in 2 subdirs
- use to compare reconverted EBCDIC files to original mainframe files
- use for files with packed/binary fields
& fixed length files without the LineFeeds required by unix 'diff'
      alldiff2 - run unix 'diff' for all pairs of files in 2 directories
               - great for proving mass changes OK & no unwanted changes

      alldiff2 pfx1.old pfx1  <-- creates diff reports in tmp/pfx1.dif
      ======================
      alldiff2sub - run 'alldiff2' script for all pairs of subdirs in 2 superddirs

      alldiff2 cnvdata1.old cnvdata1 tmp1
      =================================
      - create alldiff2 reports tmp1/cpys.dif, maps.dif, pfx1.dif, pfp1.dif, etc

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10B1. DATAcnv1 - Converting Mainframe Data Files from EBCDIC to ASCII

Summary of uvcopy jobs used in DATAcnv1.doc

No uvcopy jobs are listed here, but you can view or print any of them from /home/uvadm/sf/IBM/...

signscan2
  • determine record layout when no COBOL copybook available
  • scans for packed fields, zoned signs, date fields, and
    right blank filled fields (such as Name & Address).
sign2cpy1
  • converts the signscan2 record layout to a COBOL copybook which
    is used by following jobs to perform various conversions.
data2cpy51
  • creates control-file ctl/datacpy51 from directory d1ebc/*
    of EBCDIC datafilenames to be converted.
data2cpy52
  • inserts copybooknames in control-file cpy=____________
  • if copybooknames are embedded in datafilenames
    (unique to some sites)
cobmap1
  • converts COBOL copybooks to cobmaps (record layouts), adding
    field start,end,length,& type on the right hand side of the copybook.
    The cobmap record layout is the input to the following scripts.
uvdata51
  • generates uvcopy jobs to translate EBCDIC files to ASCII,
  • preserving packed fields & correcting any zoned decimal signs
    from EBCDIC conventions to ASCII conventions.
uvdata51p
  • alternate to ensure packed sign x'_C' for AIX COBOL
  • generates 1 mvn instruction per packed field
    (vs uvdata51 which groups packed fields together in 1 mvc instrn)
uvdata52
  • replace copybooknames with datafilenames, while copying pfx1/*
    to pfx2/..., guided by a control-file relating datafilenames
    to corresponding copybooknames.
genpipe1
  • generates uvcopy jobs to convert data to a '|' pipe delimited
    ASCII text file. Packed & numeric fields will be converted to zero
    suppressed numeric fields with leading sign +/-.
  • Pipe delimited format is used for loading Relational data bases.
listpipe1
  • by default lists the 1st 3 records of the pipe delimited file
    first as is (1 long line), then field by field with copybook names
sqlcreate1
  • generates batch-file & control-file to create & load table
    from the copybook/map create by cobmap1/cobmapA

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10C1. scripts used for Data file conversion

cobmapA - generate record layout from COBOL copybook

 #!/bin/ksh
 # cobmapA - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # cobmapA - generate record layout from COBOL copybook
 #
 # This script makes it easier to run cobmap1 uvcopy job
 # /home/uvadm/sf/cobmapA - script files Vancouver Utility subdir 'sf'
 # /home/uvadm/pf/cobmapA - uvcopy jobs in subdir 'pf' (parameter files)
 #
 if [[ ! -f cpys/"$1" ]]; then
    echo "usage: cobmapA copybookname [options]"
    echo "       =============================="
    echo " - arg1 must be a file within subdir 'cpys' (specify filename only)"
    echo " -      output to subdir 'maps' with same filename"
    echo " - arg2 options (example a1 to shift free-format over 6 cols)"
    exit 1; fi
 #
 uvcopy cobmap1,fili1=cpys/$1,filo1=maps/$1,uop=q0i7$2,rop=r1
 #===========================================================
 echo "record layout created, you may view or print as follows:"
 echo "more maps/$1     - display output"
 echo "uvlp12 maps/$1   - print output (at 12cpi with UV laser printer script)"
 exit 0

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10C2. scripts used for Data file conversion

gencnvA - generate uvcopy job to convert EBCDIC to ASCII

 #!/bin/ksh
 # gencnvA - Generate uvcopy jobs to convert 1 data file EBCDIC to ASCII,
 #           ASCII to EBCDIC, fixed to delimited,& delimited to fixed
 #         - see www.uvsoftware.ca/datacnv1.htm#3B2
 #Oct2014 - change genpipe1 to genpipe3 for fields > 100 bytes
 #        - change genfixd2 to genfixd3 for fields > 100 bytes
 #
 # gencnvA must run from superdir containing subdirs cpys,maps,pfx1,pfp1,pfq1,pfy1
 # The copybooks (in cpys/...) are used by following utilties to generate uvcopy jobs
 # in the following subdirs as follows:
 #
 # utility   subdir  ------- purpose -------
 # cobmap1  - maps - record layouts (cobmaps) used to generate following jobs:
 # uvdata51 - pfx1 - uvcopy jobs translate EBCDIC to ASCII
 # uvdata31 - pfy1 - uvcopy jobs translate ASCII to EBCDIC
 # genpipe3 - pfp1 - uvcopy jobs convert ASCII datafiles to delimited (default '|' pipe)
 # genfixd3 - pfq1 - uvcopy jobs convert delimited files to fixed format (as per copybook)
 #
 #Note - gencnvA intended to generate jobs to process 1 file at a time easily
 # - requires datafile to be renamed same as copybook (simplifies procedures)
 # Also see www.uvsoftware.ca/datacnv1.htm#Part_6
 # - to generate jobs for all copybooks
 # - to process all datafiles in directory (using original datafilenames)
 # - requires a 'control file' to relate copybookname to datafilename
 # - utility 'uvdata52' reads control file & may generate multiple jobs
 #   to convert multiple datafiles for 1 copybook (often the case)
 #
 echo "gencnvA using: cpys,maps,pfx1,pfp1,pfq1,pfy1"
 f1="$1";  # capture arg1 filename (should not have .cpy)
 if [[ -f cpys/"$f1" && -d maps && -d pfp1 && -d pfq1 && -d pfy1 ]]; then :
    else echo "usage: gencnvA CopyBookName"
         echo "       ===================="
         echo "arg1 ($f1) not found in subdir cpys/..."
         echo "- OR subdir missing - must have cpys,maps,pfx1,pfp1,pfq1,pfy1"
         echo "must specify any .cpy extension if present"
         echo " - but any .cpy extension removed on output files"
         exit 9; fi
 f2=${f1%.*}    # remove any extension
 #
 uvcopy cobmap1,fili1=cpys/$f1,filo1=maps/$f2,uop=q0i7p0 # copybook to cobmap
 #======================================================
 uvcopy uvdata51,fili1=maps/$f2,filo1=pfx1/$f2,uop=q0i7  # EBCDIC to ASCII
 #=====================================================
 uvcopy genpipe3,fili1=maps/$f2,filo1=pfp1/$f2,uop=q0i7  # delimit fields > 100
 #=====================================================
 uvcopy genfixd3,fili1=maps/$f2,filo1=pfq1/$f2,uop=q0i7  # undelimit fields > 100
 #=====================================================
 uvcopy uvdata31,fili1=maps/$f2,filo1=pfy1/$f2,uop=q0i7  # ASCII to EBCDIC
 #=====================================================
 exit 0
Note
  • gencnvAp same as above, but executes 'uvdata51p' (vs uvdata51)
  • to ensure that all packed decimal signs are converted to x'_C' (vs x'_F')

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10D1. scripts used for Data file conversion

gencnvD2 - generate all conversion jobs for unix/linux

 # gencnvD2 - generate all conversion jobs for any 1 copybook
 #          - cobmap1,uvdata51,uvdata31,genpipe1,genfixd2,sqlload1
 #          - by Owen Townsend, June10/2012,
 #          - see www.uvsoftware.ca/datacnv1.htm#4H1
 #Oct07/14 - change genpipe1 to genpipe3 & genfixd2 to genfixd3
 #         - to allow pipe delimit/undelimit fields > 100 bytes
 #
 # gencnvD2 - generate all jobs using sqlcreate1 to create/load tables for Oracle
 # gencnvD2p - same but calls uvdata51p to ensure packed signs x'_C' for AIX COBOL
 # gencnvD3 - same as gencnvD2, but call sqlcreate2 for SQL Server vs Oracle
 # gencnvD9 - same as gencnvD2, but generatesjobs for ALL copybooks in cpys/...
 #
 # gencnvD2 citytax1     <-- example (from DATAcnv1.doc)
 # =================
 #
 # if [[ -d "$CNVDATA" ]]; then cd $CNVDATA fi
 #Aug13/2014 - above disabled, sometimes you want to run elsewhere
 #
 fx="$1";
 f=${fx%.*};  # remove any extensions
 if [[ -f cpys/"$fx" && -d maps && -d pfx1 && -d pfy1 && -d pfp1 && -d pfq1 \
                     && -d pfg1 && -d sqlTC1 && -d sqlTL1 ]]; then :
    else echo "usage:   gencnvD2 filename"
         echo "         ================="
         echo "example: gencnvD2 custmas.cpy"
         echo "         ===================="
         echo "- arg1 must be a file in subdir cpys/..."
         echo "- any extension in cpys/...cpy will be removed for all other dirs"
         echo "- must have outdirs: maps,pfx1,pfy1,pfp1,pfq1,pfg1,sqlTC1,sqlTL1"
         exit 1; fi
 #
 uvcopy cobmap1,fili1=cpys/$fx,filo1=maps/$f,uop=q0i7p0
 #=====================================================
 uvcopy uvdata51,fili1=maps/$f,filo1=pfx1/$f,uop=q0i7
 #===================================================
 uvcopy uvdata31,fili1=maps/$f,filo1=pfy1/$f,uop=q0i7
 #===================================================
 uvcopy genpipe3,fili1=maps/$f,filo1=pfp1/$f,uop=q0i7  #Oct2014 fields > 100
 #===================================================
 uvcopy genfixd3,fili1=maps/$f,filo1=pfq1/$f,uop=q0i7  #Oct2014 fields > 100
 #===================================================
 uvcopy gentest1,fili1=maps/$f,filo1=pfg1/$f,uop=q0i7
 #===================================================
 uvcopy sqlcreate1,fili1=maps/$f,filo1=sqlTC1/$f.sql,filo2=sqlTL1/$f.ctl,uop=q0i7
 #===============================================================================
 exit 0
 #

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10D2. DATAcnv1 - Batch scripts for DOS window under Windows

gencnvD2.bat - generate all conversion jobs for Windows

 @echo off
 rem gencnvD2.bat - generate conversion jobs for Windows systems
 rem          - cobmap1,uvdata51,genpipe1,sqlload1
 rem          - assumes COBOL copybook available in subdir cpys
 rem          - by Owen Townsend, Feb 24/2004
 rem          - see EBCDICjobs.doc or www.uvsoftware.ca/datacnv1.htm
 rem
 rem gencnvD2 filename       <-- command format
 rem =================
 rem
 rem gencnvD2 citytax1       <-- example (from EBCDICjobs.doc)
 rem =================
 rem
 if "%1" == "" goto error
 if exist dat1ebc\"%1" goto process
 :error
    echo "arg1 not a file"
    echo "usage:   gencnvD2 filename"
    echo "         ================="
    echo "example: gencnvD2 citytax1"
    echo "         ================="
    goto end
 :process
 uvcopy cobmap1,fili1=cpys/%1,filo1=maps/%1,uop=q0p0
 uvcopy uvdata51,fili1=maps/%1,filo1=pfx1/%1,uop=q0
 uvcopy genpipe1,fili1=maps/%1,filo1=pfp1/%1,uop=q0,rop=r0
 uvcopy sqlcreate1,fili1=maps/%1,filo1=sqlTC1/create_%1.sql,filo2=sqlTL1/load_%1.ctl
 :end

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10E1. scripts used for Data file conversion

gencnvC - generate all jobs from directory of filenames

 #!/bin/ksh
 # gencnvC - Generate Data Conversion jobs for all copybooks in directory maps/
 #         - using control file of datafilenames & copybooknames
 #         - by Owen Townsend, UV Software, July 2014
 #         - run from $CNVDATA with subdirs: ctl,cpys,maps,pfp1,pfp2,d2asc,d4pipe
 #
 #      ** sample control file: datafilenames & copybooknames **
 #
 # ar.customer.armaster          cpy=____________ rcs=_____ typ=_____
 # ar.sales.arsales              cpy=____________ rcs=_____ typ=_____
 # citytax1                      cpy=____________ rcs=_____ typ=_____
 # gl.account.glmaster_          cpy=____________ rcs=_____ typ=_____ dups=003
 # gl.account.gltran_            cpy=____________ rcs=_____ typ=_____ dups=003
 #
 #         ** Pre-Requisite - Creating control file ctl/datacpy52 **
 #
 # Control file must have been created as follows (3 ways);
 # (1. all manually, or 2. partial automated, or 3. fully automated)
 #
 # 1. vi ctl/datacpy52    <-- all manual insert copybooknames with editor
 #    ================      - IF datafiles not on unix/linux ssystem yet
 #
 #             --------- OR 2a,2b,2c ------------
 #
 # 2a. uvcopy data2cpy51,fild1=d1ebc,filo1=ctl/data2cpy51
 #     ==================================================
 #     IF datafiles have been FTP'd to unix/linux & stored in $CNVDATA/d1ebc/...
 #     - create skeleton control file by extracting datafilenames from d1ebc/...
 #
 # 2b. cp ctl/data2cpy51 ctl/data2cpy52
 #     ================================
 #     - copy/rename so final edited filename always ctl/data2cpy52
 #
 # 2c. vi ctl/data2cpy52  <-- edit/insert copybooknames in skeleton ctlfile
 #     =================
 #
 #             --------- OR 3a,3b ------------
 #
 # 3a. uvcopy datacpy52,fili1=ctl/datacpy51,filo1=ctl/datacpy52
 #     ========================================================
 #     IF copybooknames embedded in datafilenames - UNIQUE to some sites
 #     - extract copybooknames from datafilenames, code on cpy=_____________
 #
 # 3b. vi ctl/datacpy52   <-- inspect & fix if data2cpy52 not perfect
 #     ================
 #
 #          ** ctl/datacpy52 result from method 1,2,OR 3 **
 #
 # ar.customer.armaster          cpy=armaster.cpy rcs=_____ typ=_____
 # ar.sales.arsales              cpy=arsales.cpy  rcs=_____ typ=_____
 # citytax1                      cpy=citytax1.cpy rcs=_____ typ=_____
 # gl.account.glmaster_          cpy=glmaster.cpy rcs=_____ typ=_____ dups=003
 # gl.account.gltran_            cpy=gltran.cpy   rcs=_____ typ=_____ dups=003
 #
 # 4. gencnvC ctl/datacpy52  <-- run this script to generate data convert jobs
 #    =====================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

 if [[ ! -f "$1" ]]; then
    echo "usage: gencnvC control-file"
    echo "       ===================="
    echo "example: gencnvC ctl/datacpy52"
    echo "         ====================="
    exit 91; fi
 #
 if [[ -d cpys && -d maps && -d pfx1 && -d pfx2 && -d ctl ]]; then :
    else echo "mandatory subdirs: cpys maps pfx1 pfx2 ctl"; exit 92; fi
 echo "gencnvC process ALL cpys to maps,pfx1,pfx2 "
 echo "- enter for step1, convert copybooks to record layout maps"; read
 #
 uvcopyx cobmap1 cpys maps uop=q0i7p0
 #===================================
 rmzf maps   # remove any zero maps caused by procedure copybooks
 #========
 echo "enter to generate uvcopy jobs (in pfx1/...) from copybook maps"; read
 uvcopyx uvdata51 maps pfx1 uop=q0i7
 #==================================
 #
 echo "enter to copy \$UV/pf/skeleton2 to pfx1/ for files without copybooks"; read
 cp $UV/pf/IBM/skeleton2 pfx1
 #===========================
 #
 echo "enter to copy pfx1/* to pfx2/, replacing copybooknames with datafilenames"
 echo "- using control-file of datafilenames & corresponding copybooknames"; read
 uvcopy uvdata52,fili1=$1,fild2=pfx1,fild3=pfx2,uop=q0i7f1
 #========================================================
 exit 0
 #

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

 #              ** edit generated jobs (for multi-record-types) **
 #
 # It may be necessary to make some manual changes to auto-generated jobs.
 #
 # A. Allow for any files with multiple record-types
 #    - insert code to test R/T & skip to corresponding bank of instructions
 #      already automatically generated from redefined records in copybook
 #    - see example of inserting R/T test code for citytax2 file
 #      at 'http://uvsoftware.ca/datacnv1.htm#5A1' - 5C2
 #
 # B. Allow for any files with variable length records
 #    - change file typ=RSF to typ=RDWz4
 #
 # C. Corrections for missing copybooks (generates job from uvcopy skeleton)
 #    - translates entire record to ASCII assuming no packed/binary fields
 #    - could manually insert instrucitons to preserve packed/binary fields
 #
 # Note - should copy pfx2/* jobs to pfx/... before any editing
 #      - so you do not lose manual changes if automatic generations rerun
 #
 # #6. cp pfx2/* pfx3    <-- copy auto-generted jobs before any editing
 #     ==============
 #
 # #7. vi pfx3/...       <-- make any manual changes required
 #     ===========         - as discusses above A,B,C
 #
 #         ** Execute jobs to convert datafiles d1ebc --> d2asc **
 #
 # #8. uvcopypfxgA pfx3 d1ebc d2asc
 #     ============================
 #     - exewcute all jobs in pfx3/...
 #     - to convert all files from d1ebc/... to d2asc/...
 #     - jobs for GDG files are repeated depending on generations present
 #

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10F1. scripts used for Data file conversion

uvcopypfxgA - Execute all jobs to convert all files

 #!/bin/ksh
 # uvcopypfxgA - Execute directory of uvcopy jobs
 #             - Data Conversion jobs in pfx3/... or pfp3/..., etc
 #             - converts all members of GDG file sets
 #             - by Owen Townsend, UV Software, July 2014
 #
 # Special treatment for jobs with GDG files (filenames IF by trailing '_')
 # - Re-Executes the job for all members in the data input directory
 # uvcopy jobs in pfx2/pfx3 (or pfp2/pfp3,etc) were generated as follows:
 #
 # 1. uvcopy data2cpy51,fild1=d1ebc,filo1=ctl/datacpy51
 #
 #           ** control file created from directory of datafilenames **
 #    ar.customer.armaster      cpy=____________ rcs=_____ typ=_____
 #    gl.account.glmaster_      cpy=____________ rcs=_____ typ=_____ dups=003
 #
 # 2. cp ctl/datacpy51 ctl/datacpy52  <-- copy/rename
 # 3. vi ctl/datacpy52      <-- edit with copybooknames
 #
 #               ** control file after inserting copybooknames **
 #    ar.customer.armaster      cpy=armaster.cpy rcs=_____ typ=_____
 #    gl.account.glmaster_      cpy=glmaster.cpy rcs=_____ typ=_____ dups=003
 #
 #  --- OR - if copybooknames embedded in datafilenames (at some sites)
 #         - data2cpy52 will transfer copybookname to cpy=____________
 #
 # 2a. uvcopy data2cpy52,fili1=ctl/datacpy51,filo1=ctl/datacpy52
 # 3a. vi ctl/datacpy52     <-- inspect, correct if auto not perfect
 #
 # 4. uvcopyx cobmap1 cpys maps uop=q0i7p0
 #          - convert copybooks to record-layouts
 # 5. uvcopyx uvdata51 maps pfx1 uop=q0i7
 #          - convert cobmaps to data conversion jobs in pfx1/...
 # 6. uvcopy uvdata52,fili1=ctl/datacpy52,fild2=pfx1,fild3=pfx2,uop=q0i7f1
 #          - replace copybooknames with datafilenames in pfx2/...
 #          - via control file of datafilenames & copybooknames
 #
 # 4a. gencnvC - script alternative, runs #4,5,& 6
 #
 # 7. cp pfx2/* pfx3  <-- copy to pfx3 before any edits for multi R/T files
 #
 # 8. vi pfx3/...     <-- modify jobs for any files with record redefinitions
 #
 # 8. uvcopypfxgA pfx3 d1ebc d2asc
 #    ============================
 #    - run this script to execute all data conversion jobs in pfx2/...
 #      converting files from d1ebc/... to d2asc/...
 #

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

 pfd="$1"; d1="$2"; d2="$3";
 if [[ -d "$pfd" && -d "$d1" && -d "$d2" ]]; then :
  else echo "usage: uvcopypfxgA pfjobs indir outdir"
       echo "       ==============================="
       echo "example#1: uvcopypfxgA pfx3 d1ebc d2asc"
       echo "           ============================"
       echo "example#2: uvcopypfxgA pfp3 d2asc d4pipe"
       echo "           ============================="
       exit 91; fi
 #
 echo "$0 execute jobs in $1 to convert files from $2 to $3"
 echo "- Repeats jobs for all members of GDG file groups"
 echo "- enter to proceed"; read reply
 #
 for pfduvj in $pfd/*
   { uvj=$(basename $pfduvj)
     if [[ $uvj == *_ ]]; then
        for d1fn in $d1/${uvj}*
          { fn=$(basename $d1fn)
            uvcopy $pfd/$uvj,fili1=$d1/$fn,filo1=$d2/$fn
            #===========================================
          }
     else
       fn=$uvj
       uvcopy $pfd/$uvj,fili1=$d1/$fn,filo1=$d2/$fn
       #===========================================
     fi
   }
 exit 0

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10H1. scripts used for Data file conversion

genrfmA - generate job to reformat a datafile

 #!/bin/ksh
 # genrfmA - Korn shell script from UVSI stored in: /home/uvadm/sf/IBM/
 # genrfmA - Generate uvcopy job to reformat a data file
 #           based on old & new versions of the copy-books
 #         - unpack packed fields, or reformat records for any reason
 #         - see REFORMjobs.doc or DATAcnv1.doc#4I1
 #
 # genrfmA original_copybook modified_copybook
 # ===========================================
 #
 # This script executes the multi-steps to generate the uvcopy job
 # - For use when you only have 1 or 2 files to do
 # - For mass conversions you should the 'ALL FILES IN DIRECTORY'
 #   operating instructions documented in section 'G' of REFORMjobs.doc
 #
 # Run from cnvdata superdir with subdirs: cpys,maps,mapsI,pfr1
 # Two input files must be present:
 # 1. original copybook & modified copybook in subdir cpys
 #
 # This script runs 4 uvcopy jobs:
 # 1. cobmap1 converts original copybook from cpys to cobmap in maps
 # 2. cobmap1 converts modified copybook from cpys to cobmap in maps
 # 3. reform1 loads original copybook to indexed file mapsI/original copybook
 #            with fieldnames as keys
 # 4. reform2 reads modified copybook, looks up indexed file of origional fields
 #            & writes out uvcopy job to move/unpack data fields
 #
 # Alternate similar scripts:
 #   genrfmB - executes additional step 'uvdata52', which inserts the actual
 #             datafilename (vs copybookname) using a control file
 #   genrfmD - same as genrfmB for sites with duplicate fieldnames in copybooks
 #
 cb1="$1"; cb2="$2";
 echo "genrfmA using: cpys,maps,mapsI,pfr1"
 if [[ -f cpys/$cb1 && -f cpys/$cb2 && -d maps && -d mapsI && -d pfr1 ]]
    then :
    else echo "usage: genrfmA copybook1 copybook2"
         echo "       ==========================="
         echo "- arg1 ($cb1) or arg2 ($cb2) not found in cpys"
         echo "- or subdirs missing (cpys,maps,mapsI,pfr1)"
         exit 9; fi
 #
 echo "#1 - convert original copybook to cobmap cpys/arg1 --> maps/arg1"
 echo "   - enter to proceed"; read reply
 uvcopy cobmap1,fili1=cpys/$cb1,filo1=maps/$cb1,uop=q0i7p0
 echo "#2 - convert modified copybook to cobmap cpys/arg2 --> maps/arg2"
 echo "   - enter to proceed"; read reply
 uvcopy cobmap1,fili1=cpys/$cb2,filo1=maps/$cb2,uop=q0i7p0
 echo "#3 - load original cobmap to Indexed file maps/arg1 --> mapsI/arg1"
 echo "   - enter to proceed"; read reply
 uvcopy reform1,fili1=maps/$cb1,filo1=mapsI/$cb1,uop=q0i7
 echo "#4 - generate reformat job maps/arg2 + mapsI/arg1 --> pfr1/arg2"
 echo "   - enter to proceed"; read reply
 uvcopy reform2,fili1=maps/$cb2,filo1=pfr1/$cb2,filr1=mapsI/$cb1,uop=q1
 exit 0

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10G2. scripts used for Data file conversion

uvcopypfx2 - execute pfx1, pfp1,& listpipe

 # uvcopypfx2 - execute EBCDIC to ASCII & pipe|delimit jobs for 1 file
 #            - also creates listpipe/... report datafields with copybook names
 #            - also creates uvhd & uvhdcob reports for d1ebc & d2asc file
 #            - by Owen Townsend, UV Software, Oct19/2013
 #            - see www.uvsoftware.ca/datacnv1.htm#Part_4
 # uvcopypfx1 - same, but omits uvhd & uvhdcob reports
 # uvcopypfx9 - executes ALL (vs 1) jobs in pfx1/ pfp1/ + listpipe reports
 #
 # gencnvD2 - must have been run to create conversion jobs in pfx1/ & pfx2/
 #            via cobmap1,uvdata51,uvdata31,genpipe1,genfixd2,sqlcreate1
 #            from copybook in cpys/xxx & maps/xxx
 #
 # if [[ -d "$CNVDATA" ]]; then cd $CNVDATA fi
 #Aug13/2014 - above disabled, sometimes you want to run elsewhere
 #
 echo "execute EBCDIC to ASCII & pipe|delimit jobs for 1 specified file"
 echo "- also list 1st 3 pipe delim records vertically with copybook names"
 f="$1";
 if [[ -f pfx1/"$f" && -f pfp1/"$f" && -f d1ebc/"$f" && -d d2asc && -d d4pipe ]]
    then :
    else echo "usage:   uvcopypfx2 filename"
         echo "         ==================="
         echo "example: uvcopypfx2 citytax1"
         echo "         ==================="
         echo "- arg1 must be a file in subdir d1ebc, pfx1, pfp1, etc"
         echo "- must have outdirs: d2asc, d4pipe, listpipe, uvhd_d1ebc, etc"
         exit 1; fi
 #
 uvcopy pfx1/$f,uop=q0i31
 #=======================
 uvcopy pfp1/$f,uop=q0i31
 #=======================
 uvcopy listpipe1,fili1=d4pipe/$f,filo1=listpipe/$f,uop=q0i31
 #===========================================================
 # get rec-size from maps for uvhd & create uvhd & uvhdcob reports
 test -d uvhd_d1ebc || mkdir uvhd_d1ebc
 test -d uvhd_d2asc || mkdir uvhd_d2asc
 test -d uvhdcob    || mkdir uvhdcob
 rcsz1=$(grep '\*RCSZ=' maps/$f); rcsz2=${rcsz1##*=}
 echo -e "\n\nq" | uvhd d1ebc/$f ak4r${rcsz2} >uvhd_d1ebc/$f
 echo -e "\n\nq" | uvhd d2asc/$f  k4r${rcsz2} >uvhd_d2asc/$f
 echo -e "\n\nq" | uvhdcob d2asc/$f maps/$f f1k4m500 >uvhdcob/$f
 echo "head d4pipe/$f.dat <-- inspect pipe delimited report"
 echo "more listpipe/$f   <-- inspect listpipe report (1st 3 records)"
 echo "more uvhd*/$f  <-- inspect uvhd_d1ebc,uvhd_d2asc,& uvhdcob rpts"
 exit 0
 #
Note
  • copypfx1 is same as above but omits uvhd & uvhdcob reports

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10I1. scripts used for Data file conversion

copysub2 - collect related files from multiple subdirs

 #!/bin/ksh
 # copysub2 - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # copysub2 - copy filex from multiple subdirs in a superdir to an outdir
 #          - by Owen Townsend, UV Software, Oct19/2013
 # copysub2 - copies filename specified with '*' wildcard appended
 #            to allow for any different extensions in different subdirs
 # copysub1 <-- alternate script that copies exact filename specified
 #
 # Renames filex by apending '_' + subdirname, for example:
 # Given: cnvdata/cpys/filex, cnvdata/maps/filex, cnvdata/pfx1/filex
 # 1. cd ...above... cnvdata/...
 # 2. mkdir tmp1
 # 3. copysub2 filex cnvdata tmp1
 #    ===========================
 # 4. ls tmp1 --> tmp1/filex_cpys, tmp1/filex_maps, tmp1/filex_pfx1
 #
 # Useful for collecting data conversion files from multi subdirs in cnvdata/...
 # - since same filename used in multi subdirs
 # - output files identified by subdir-name appended
 # - then might zip files & send to clients as 1 archive
 #
 echo "copy filex from multi subdirs in superdir to outdir"
 echo "- rename/append subdirs: filex_subdir1, filex_subdir2, etc"
 f="$1"; d1="$2"; d2="$3";
 if [[ -d "$2" && -d "$3" ]]; then :
 else echo "usage: copysub2 filex superdir outdir"
      echo "       =============================="
      echo " - arg1 must be a filename"
      echo " - arg2 & arg3 must be directories"
      exit 1; fi
 x=0;
 for d1sd in $d1/*
  { sd=$(basename $d1sd)
    sdf=${f}_$sd
    ls $d1/$sd/$f >/dev/null 2>/dev/null
    if ((!$?)); then
       cp $d1/$sd/$f $d2/$sdf
       let x=x+1
       echo "copy file# $x $d1/$sd/$f to $d2/$sdf"
    fi
  }
 echo "total $x files copied from $d1/.../$f to $d2/..."
 exit
 #Oct27/14 - removed '*' from $f on ls & cp aabove

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10J1. scripts used for Data file conversion

uvcmp1all - compare all data files in 2 directories

 #!/bin/ksh
 # uvcmp1all - script to compare all files in 2 directories
 #           - by Owen Towsnend, Feb 2010 at Everis
 # calls 'uvcmp1' for fixed length files w/o LineFeeds
 # - shows record pairs in vertical hexadecimal
 #   with mismatches flagged with '*'s
 # uses Indexed control file to get record size (ctl/ctlfile4I)
 # - same control file as used for EBCDIC to ASCII conversion
 #
 echo "uvcmp1all - compare all files in 2 directories, using uvcmp1"
 echo "- to show record pairs in vertical hexadecimal with '*' mismatches"
 echo "- for files with packed/binary & no LineFeeds (use diff for text files)"
 d1="$1"; d2="$2";
 if [[ -d "$1" && -d "$2" ]]; then :
    else echo "usage: uvcmp1all dir1 dir2 [options]"
         echo "       ============================="
         echo "ex1:   uvcmp1all d2asc.old d2asc"
         echo "       ========================="
         echo "ex2:   uvcmp1all d1ebc d3ebc y1"
         echo "       ========================"
         echo "- option y1 for EBCDIC not req'd if dirname contains *ebc*"
         exit 99; fi
 echo "uvcmp1 $d1/... $d2/..."
 echo "- will write mismatch reports to subdir rptcmp/..."
 reply=x;
 until [[ "$reply" = "y" || "$reply" = "n" ]]
    do echo "- clear all old mismatch reports from rptcmp/... y/n ?"
       read reply; done
 if [[ "$reply" == "y" ]]; then rm -f rptcmp/*; echo "rptcmp/* removed"; fi
 x=0;y=0;
 for d1f in $d1/*
   { let x=x+1
     f=$(basename $d1f)
     if [[ $f == *.idx ]]; then continue; fi
     if [[ ! -f $d2/$f ]]; then
        echo "bypassing nomatch in $d2 for $d1/$f"
        continue; fi
     #
     # use uvcopy job to get recsize & key options from control file
     # - will write to: /tmp/${LOGNAME}.uvcmp
     # remove any '.dat' (not coded in control file)
     fx=${f%.dat}
     uvcopy uvcmpget2,filr1=${CNVDATA}/ctl/ctlfile4I\
 ,filo1=/tmp/${LOGNAME}.uvcmp,arg1=$fx
     if (( $? != 0))
        then echo "datafilename not in control file"; exit 2;fi
     #

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

     # open the file written by uvcopy & retrieve options into $variables
     exec 3< /tmp/${LOGNAME}.uvcmp   # open file written by uvcopy
     read -u3 copyname ctlops other  # read file contents into variables
     exec 3<&-                       # close file
     #
     # if dirname contains 'ebc' add option y1 to trnslt chars to ASCII
     if [[ $d1 == *ebc* ]]; then ctlops=${ctlops}y1; fi
     #
     echo "file# $x $d1/$f $d2/$f"
     uvcopy uvcmp1,fili1=$d1/$f,fili2=$d2/$f,rop=r0,uop=q0i7$ctlops
     #=============================================================
     if (( $? != 0)); then ((y+=1)); fi
   }
 echo "$y mismatches of $x total files compared in $d1 & $d2"
 echo "vi rptcmp/*      <-- view all mismatch reports"
 echo "==========="
 echo "uvlpd12 rptcmp   <-- print all mismatch reports"
 echo "=============="
 exit 0

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

10K1. scripts used for Data file conversion

cnvdatainit - clear prior outputs before test/demo reruns

 # cnvdatainit - clear outputs of prior test/demos before re-runs
 #             - by Owen Townsend, Feb2007, update Aug2021
 #
 # Clear files that will be created when you re-run test/demo conversions
 #  - see www.uvsoftware.ca/datacnv1.htm#3B1
 # Assuming - Vancouver Utilities installed at /home/uvadm/...
 #  - you have setup userid mvstest1 & changed to /home/mvstest1/cnvdata1
 #  - copied /home/uvadm/mvstest/mvstest1/* to /home/mvstest1/...
 #    (OR copy mvstest1/* to your own homedir & setup exports in .bash_profile)
 #  - have already run some conversions & now want to rerun with output dirs cleared
 #  - after running this cnvdatainit, restore EBCDIC datafile input as follows:
 #    #1 ----> cp archive/d1ebc_cpy_name/* d1ebc/. <-- datafilenames same as copybooknames
 #    OR#2--> cp archive/d1ebc_data_name/* d1ebc/. <-- real datafilenames aaa.bbb.ccc.etc
 #      #2 for Part_6 of DATAcnv1.doc or for all conversions in DATAcnv2.doc
 #
 # OR, you have converted some of your own files & already saved results
 #  - and you now want to convert a new set of data files
 #  - run this cnvdatainit script (clears all data subdirs & generated uvcopy jobs)
 #  - copy new set of data files to d0ebc (or d1ebc if already lowercase names)
 #  - copy new set of copybooks to cnvdata1/cpys/...
 #
 echo "clear CNVDATA subdirs before regen uvcopy jobs & reconvert data files"
 echo "clear d0ebc,d1ebc,d2asc,d3ebc,maps,pfx1,pfx2,pfx3,etc,tmp,tmp1,etc"
 reply=x;
 until [[ "$reply" = "y" || "$reply" = "n" ]]
    do echo "- clear CNVDATA subdirs before reconvert y/n ?"
       read reply; done
 if [[ "$reply" != "y" ]]; then echo "$0 aborted"; exit; fi
 cd $CNVDATA
 rm -f d[0-9]*/*             # clear d0ebc,d1ebc,d2asc,d3asc,d4pipe,d5fix
 rm -f listpipe/*
 rm -f maps/*                # clear record layouts (converted from cpys/...)
 rm -f pf[a-z][0-9]/*        # clear pfx1,pfx2,etc,pfy1,pfp1,etc
 rm -f tmp*/*                # clear tmp,tmp1,tmp2,etc files
 echo "cleared: d0ebc,d1ebc,d2asc,d3ebc,maps,pfx1,pfx2,pfx3,etc,tmp,tmp1,etc"
 exit 0
 #

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Visitor Counters for ThisYear and LastYear