Data Utilities - Contents


Part_0 - Introduction & Overview
- test files supplied for you to copy to your homedir
- step by step operating instructions to run the test/demos
- then try out on your own files

Part_1 - preparations for Vancouver Utilities self-training
- setup your profile & copy testfiles to subdirs in your homedir
- test files provided by UV Software to demo Vancouver Utilities
- customer master files: custmas0 (EBCDIC) & custmas1 (ASCII)
- sales detail files: sales0 (EBCDIC) & sales1 (ASCII)
- vendor master file: vendormas3 sample of RDW variable-length file
- these files used in all parts of this document

Part_2 - 'uvhd' file investigation utility (free download)
- display any file in vertical hexadecimal
- ideal for mainframe EBCDIC files transferred to unix/linux
  (with packed/binary fields & no LineFeeds required by most unix tools)
- commands to browse,search,select,update,translate,etc

Part_3 - 'uvhdcob' companion utility to 'uvhd'
- displays COBOL copybook fieldnames beside data field contents
- similar commands to browse,search,select,update
- + 'verify' using copybook defs of character/numeric/numeric fields
- may correct bad data: zero fill numeric/packed fields with blanks,
  and convert unprintable bytes in character fields to '.' periods

Part_4 - uvcp, uvsort,& uvlist - data file reformat, sort,& list utilities
- a few examples to give you an idea of their capabilities
- full documentation available in separate manuals
- uvcp.htm, uvsort.htm,& uvlist.htm

Part_5 - FTP mainframe data files to Unix/Linux
- must use binary mode if any packed/binary fields
- may use ASCII mode if all unpacked character fields
- ASCII mode advantage automatic translate EBCDIC to ASCII & inserts
  LineFeeds which allows use of unix tools (vi,lp,cat,mopre,etc)
- disadvantage increases record sizes, need to modify programs to match
- also need to correct zone signs in unpacked numeric fields
  from mainframe conventions to Micro Focus COBOL conventions
- using uvcp to change record lengths & file types & zoned signs
- detecting wrong-sized records in FTP'd files
- selecting desired record-sizes from FTP ASCII files
- EBCDIC to ASCII Data Conversion allowing packed/binary
- these more complex data conversions not covered in detail here
- references given to other documents with the operating instructions

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Data Utilities - Contents (continued)


Part_6 - General Purpose Table Summaries
- 'table2' pre-programmed job to accumulate record counts & any 2
  other fields by any desired table argument
- example for sales3 demo file, acum counts,qty,amt by product code
- 'cmtbl2' sample custom written uvcopy job to demo the 'tbl' instruction
- builds 2 tables summarizing this-year & last-year sales
  by city & province

Part_7 - Accumulating numeric data fields
- easiest using the interactive uvhd utility (free download)
- or use pre-programmed batch job to accumulate 1 field & display total
- use copybooks to generate jobs to accumulate ALL numeric fields
- custom written uvcopy jobs to accumulate multiple fields,
  test record types, edit as desired.

Part_8 - Compare 'Variable Length RDW' Data files
- Part 8 here in DataUtilities.htm taken from Part 2 in CMPjobs.doc,
  which is the complete documentation for data file compare jobs.
- CMPjobs.htm covers all file types (Fixed-Length, Micro Focus, etc)
  and alternate jobs to allow for Inserted or Deleted records.
- This Part covering variable-length RDW files was included here in
  DataAnalysis.doc because it was short, but gives you a good idea
  of the data file compare jobs.

Part_9 - 'gentbl1A' & 'gentbl1E' utilities to table summarize all fields
  in a data file (gentbl1A for Asciii & gentbl1E for Ebcdic).
- Driven by copybooks to generate uvcopy jobs that read thru files &
  create tables of values found in each field defined in the copybook.
- Tabling all values would be too voluminous without limiting options.
- Default option 'm30' limits entries retained to 30, then new entries
  added if their value is greater than or less than any existing entry
  & a mid-point entry removed to maintain the number of entries to 30.
- Default option 'v10' limits ~invalid entries retained to 10, after 10
  invalid entries are just counted for the ~Invalid Total line.

Parts 1-8 References vs Part 9 Original

Parts 1-6 are introductions to uvhd,uvhdcob,uvcp,uvsort,uvlist,& uvcopy. The examples & tutorials here are the best place to start learning these utilities, but they are all extensively documented elsewhere in separate manuals.

Part_9 is an original work, this is the only manual that documents gentbl1A/1E. It is particularly valuable if you need to investigate new files from outside sources.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_0. Data Investigation, Manipulation, Analysis,& Validation

Introduction & Overview

This document is intended as a good place to start learning how the Vancouver Utilities can help you investigate, validate, manipulate, sort,& perform summary table analysis on mainframe type files that have been migrated to Unix/Linux systems.

The Vancouver Utilities are designed to provide the mainframe type data file maintenance, manipulation,& sorting capabilities that are not supplied by standard unix/linux system utilities. For example the unix/linux system sort can not sort on packed fields, can not sort files with fixed record lengths (without line-feeds), and can not sort Indexed files.

In addition to the SORT, Vancouver Utilities provide replacements for mainframe utilities such as IDCAMS, IEBGENER, DITTO, FILE-AID,& EASYTRIEVE. Vancouver Utilities provide the powerful data manipulation & data field addressing by column# that is otherwise lacking on unix/linux systems.

test/demo tutorials

Test data files & copybooks are supplied for you to copy to subdirs in your homedir. Step by step operating instructions help you to run the test/demos. Expected reports are listed for comparison to your results. After the test/demos, please try out these utilities on some of your own files.

Vancouver Utility documentation

VU documentation is created & maintained as text files with unix/linux editors, and automatically converted to HTML by a Vancouver Utility job. The text files are found in /home/uvadm/doc/... after the package is installed. They are given suffixes of .doc to indicate documentation (not MS WORD files).

links to related documents on the website

 datautilities.htm - this document
 uvhd.htm      - file investigation vertical hexdump utility
                 - uvhd is a free download from freestuff.htm
 uvhdcob.htm   - displays data field contents with copybook fieldnames
 uvcp.htm      - command line data file reformat utility
 uvsort.htm    - Vancouver sort utility replaces unix sort
                 - inadequate for mainframe type files migrated to unix/linux
 uvlist.htm    - test file list utility for documentation, scripts, programs
                 - ensures page headings with filenames, userid, date/times
 uvcopy1.htm   - powerful data manipulation utility
 uvcopy2.htm   - record/work areas, options, etc
 uvcopy3.htm   - all instructions documented with examples
 cmpjobs.htm   - Data File Compare jobs (for all file types)
 JCLcnv1demo.htm - Best document if you are interested in converting mainframe
                     JCL to Korn shell scripts
                   - part_4 is also best place to start for DATA file conversions
                     translating EBCDIC to ASCII preserving packed fields

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_1. Data Investigation, Manipulation, Analysis,& Validation

Part 1 - Test/Demo Files


1A1. Vancouver Utilities Installation

1A2. Preparation for Vancouver Utilities Self-Training
- setup your profile to use the Vancouver Utilities
- copy demo data files & copybooks to subdirs in your homedir

1A3. Training subdirs/files copied to your homedir

1B1. dat1/custmas1 - demo file for several following tutorials
- customer master Name,Address,& Sales-History file
- sample of fixed-length data file with packed fields
- 256 byte records with 24 x 5 byte packed fields

1C1. dat1/sales1,2,3 - sales detail records
- cust#,slsmn#,date,inv#,product-code,units,price,amount
- sample fixed length file with unpacked numeric fields
- 64 byte records with unpacked Zoned numeric qty/amount
- sales1,2,3 3 versions with different sign conventions

1D1. dat1/vendormas3 - sample RDW file (Record Descriptor Word)
- variable length records
- record size in binary in 4 byte record prefix

1E1. dat1/vendormas3B - sample BDW/RDW file (Block & Record Descriptor Words)

1F1. dat1/vendorpaymas - sample multi-record-type file
- vendor Master & Payment records
- text file & RDW equivalent

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1A1. Data File Investigation,Manipulation,Analysis,& Validation

Vancouver Utilities Installation

We recommend setting up userid 'uvadm' & installing the software in its home directory which would usually be /home/uvadm. We allow for alternate locations' by defining environmental variable '$UV' in the profile as follows:


 export UV=/home/uvadm    #<-- default location of Vancouver Utiltiies
 =====================       - environmental variable in profile

To use the Vancouver utilities, you must setup your profile as described at https://www.uvsoftware.ca/install.htm#A5. If the package is already installed and you are a new user, you can conCATenate the supplied profile onto your existing profile as follows:


 cat /home/uvadm/env/stub_profile >>.profile      <-- for Unix/Korn shell users
 ===========================================
 cat /home/uvadm/env/stub_profile >>.bash_profile <-- for Linux/Bash shell users
 ================================================

subdirs relevant for UV training

Please see all installation directories summarized at install.htm#A2. The following are relevant to the training exercises in this document.

 $UV/bin/...   - contains Vancouver Utility programs (compiled from $UV/src).
                 (uvhd, uvlist, uvcp, uvsort, uvcopy)
               - this directory must be in your PATH (defined in your .profile)
 $UV/dat1/...   - test data files for UV training exercises
                  (custmas1, custmas2, custmas3, sales1, sales2, sales3, etc)
                - demo file for vi mark exercise (uvtrain_contents)
 $UV/mf/cpys    - COBOL copybooks describing record layouts for data files
                  (custmas1.cpy, salesdtl.cpy, etc)
 $UV/mf/maps    - 'cobmap's created from COBOL copybooks
                  (custmas1.map, salesdtl.map, etc)
                - shows field start/end/length/type on right side
 $UV/pf/...     - parameter files (pre-programmed jobs) for the uvcopy utility
                  example: cobmap - create record layouts from COBOL copy-books
                - sub-directoried pf/adm, pf/demo, pf/IBM, pf/util
                  directories defined by env-var PFPATH (see below)
 $UV/sf/...     - script files supplied with the Vancouver Utilities
                  example: uvlp12 - script to execute uvlist & pipe to a printer
                - sub-directoried sf/adm, sf/demo, sf/IBM, sf/util
                - add to your PATH as shown below

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1A2. Preparation for UV training exercises

preparation for Vancouver Utilities Training

We recommend you setup subdirs in your home directory & copy the relevant files to these subdirs as follows:


 #1.  login --> your homedir

 #2a. mkdir dat1      <-- make subdir for demo datafiles
      ==========
 #2b. mkdir cpys      <-- make subdir for copybooks
      ==========
 #2c. mkdir maps      <-- make subdir for copybook layouts (maps)
      ==========
 #2d. mkdir sf        <-- make subdir for scripts (Script Files)
      ==========
 #2e. mkdir pf        <-- make subdir for uvcopy jobs (Parameter Files)
      ==========
 #2f. mkdir tmp       <-- make tmp subdir in your home directory
      =========           to receive output of various exercises

copy training files to your home subdirs


 #3a. cp $UV/dat1/custmas* dat1    <-- customer master files
      =========================      - sample fixed length with packed fields

 #3b. cp $UV/dat1/sales* dat1      <-- sales files
      =======================        - sample fixed length with numeric fields

 #3c. cp $UV/dat1/vendor* dat1     <-- Vendor Master & Payment files
      ========================       - sample RDW variable length files

 #4a. cp $UV/mf/cpys/* cpys   <-- COBOL copybooks
      =====================

 #4b. cp $UV/mf/maps/* maps   <-- cobol copybook MAPS (record layouts)
      =====================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1A3. Preparation for UV training exercises

training subdirs/files in your homedir

 /home/userxx
 :-----dat1                 <-- test/demo data files
 :     :----- custmas0        - EBCDIC file with 24 5 byte packed fields
 :     :----- custmas1        - ASCII file with 24 5 byte packed fields
 :     :----- custmas2        - ASCII file name&address without packed fields
 :     :----- sales0          - EBCDIC sales details with unpacked fields
 :     :----- sales1          - ASCII with EBCDIC sign conventions
 :     :----- sales2          - ASCII with Micro Focus COBOL sign conventions
 :     :----- sales3          - ASCII with separate +/- signs
 :     :----- vendormas3      - Vendor master RDW variable length file
 :     :----- vendorpaymas3   - Vendor payment RDW variable length file
 :-----cpys                 <-- COBOL copybooks
 :     :-----custmas1.cpy     - customer master Name&Adrs + thisyr/lastyr sales
 :     :-----sales.cpy        - detail sales with zoned signs
 :     :-----sales3.cpy       - detail sales with separate +/- signs
 :     :-----vendormas.cpy    - vendor master variable length RDW sample file
 :-----maps                 <-- record layouts (maps) from copybooks
 :     :----- custmas1
 :     :----- sales
 :     :----- sales3
 :     :----- vendormas
 :-----tmp                  <-- for your output files

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1B1. Demo files for Vancouver Utility Tutorials

custmas0 - customer master EBCDIC test file

Mainframe files can not be displayed with most unix/linux utilities because they are usually fixed-length EBCDIC & do not have the LineFeeds required by most unix/linux utilities. 'uvhd' is ideal for this task.


 uvhd dat1/custmas1 r256as2   - display testfile, option 'r256'=record-size,
 ==========================     'a'=translate to ASCII, 's2'=double space
 now=20110128:1615 uvhd /home/uvadm/dat1/custmas0 a
 version=20110114 records=32 rsize=256 fsize=8192
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    EVERGREEN MOTORS LTD.    1815 BOWEN ROAD          NANA
             FFFFFF4444CECDCDCCD4DDEDDE4DEC44444FFFF4CDECD4DDCC4444444444DCDC
             130140000055597955504636920334B000018150266550961400000000005151
          64 IMO          BC V9S1H1    250-754-5531 LARRY WRENCH     ...V@...
             CDD4444444444CC4EFEFCF4444FFF6FFF6FFFF4DCDDE4EDCDCC4444401357000
             946000000000023059218100002500754055310319980695538000000246C000
         128 .........W.....-.........X'..f.....m*..................V'...f...
             0000000005300016000000002570063100095000000000000000013570016000
             0C0000C0270D0540C0000C0098D0263C0444C0000C0000C0000C0246D0056C00
         192 .E@...V'........c..............f..............f.C 19950531
             047000570000000088000000001800068000130000000069C4FFFFFFFF444444
             35C0046D0000C0023C0000C0083C0056D0012C0000C0016D3019950531000000
 rec#=1 rsize=256 fptr=0 fsize=8192 rcount=32
 null=next,r#=rec,s=search,u=update,x=undo,p=print,i=iprint,w=write,e=count
 ,R#=Recsize,h1=char,h2=hex,g=genseq#,c=chkseq#,t=translate(ta,te,tu,tl,tc,tp)
 ,q=quit,k0=helpoff,k1=helpon,?=HelpScreens -->

As you see above there are 32 records in the custmas0 file and it would take a lot of space it we showed them all with uvhd.

We provide other versions of the customer master file with the Name & Address paortion extracted to a separate file. See 'custmas2' listed on the next page.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1B2. Demo files for Vancouver Utility Tutorials

dat1/custmas0,1,2 demo files

          001-006 - customer#
          009-009 - delete flag (D)
          011-035 - customer name
          036-060 - address
          061-076 - city
          078-079 - province
          081-090 - postal code
          091-102 - phone#
          103-120 - contact name
          121-180 - this year sales 12 x 5 byte packed fields
          181-240 - last year sales 12 x 5 byte packed fields
          241-244 - credit codes
          245-250 - date of last change
          251-256 - unused (no LineFeed at end record)

We need 'uvhd' to investigate this file. We cannot use the unix/linux utilities (such as vi, cat, more, lp) becuase of the packed fields & no LineFeeds. But, here is 'dat1/custmas3' which has 1st 80 bytes extracted to a separate file.

 130140    EVERGREEN MOTORS LTD.    1815 BOWEN ROAD          NANAIMO          BC
 132588    GEECOE GENERATOR SERVICESUNIT 170 - 2851 SIMPSON  RICHMOND         BC
 139923    JOHNSTONE BOILER & TANKS 1250 EAST PENDER STREET  VANCOUVER        BC
 142175    LILLY ELECTRIC (1973) LTD16809 - 24TH AVENUE      HOPE             BC
 145264  D MAGRATH SUPPLIES LTD.    1939 KIRSCHNER ROAD      KELOWNA          BC
 147615  X O'CONNER R.V. CENTRE     44430 YALE ROAD WEST     CHILLIWACK       BC
 149304    POINT GREY GOLF & COUNTRY3350 S.W. MARINE DRIVE   VANCOUVER        BC
 150825    RIGGERS INDUSTRIAL       960 - 6TH AVENUE         HOPE             BC
 152355    SHAW, JOHN               477 CARIBOO CRES.        COQUITLAM        BC
 154689    FRANKLIN RESOUCES        805 - 1199 HOWE STREET   VANCOUVER        BC
 157352  D WHYTE, W.                1150 LODGE ROAD          NORTH VANCOUVER  BC
 173320  X ZENITH TRANSPORT         2381 ROGERS AVE          COQUITLAM        BC
 201120    ALLTYPE RENTAL LTD.      BOX 1819                 GRANDE PRAIRIE   AL
 204700    CASE POWER EQUIPMENT     12611-100 ST             GRANDE PRAIRIE   AL
 208060    E&L TRUCKING             LAND & CATTLE LTD        THREE HILLS      AB
 211140  D FORD NEW HOLLAND         BOX 1, 440 ELLIS ROAD    FORT MCMURRAY    AB
 223240  X NICHOLSON CHEVROLET LTD. 7215 ARGYLL ROAD         EDMONTON         AL
 224700    OTTO MOBILES WESTERN LTD.8441 CORONET ROAD        EDMONTON         AL
 231550    SPARROW ELECTRIC CO. LTD.BOX 88                   THREE HILLS      AL
 239300    TERRITORIAL REWIND       BOX 1648                 WHITEHORSE       YK
 237286    WEBER, TOM               BOX 5503                 FORT MCMURRAY    AL
 301120  D ALBERTA GAS CHEMICALS LTD3RD. FLOOR, 11456 JASPER EDMONTON         AL
 306959  X 356582 ALBERTA LTD.      DEVITT NURSERY           CALGARY          AL
 308685    FOOTHILLS ELECTRIC       3932 - 3A ST. N.W.       CALGARY          AL
 313720    MONITREX ENGINEERING LTD 7-2280-39 AVE, N.E.      WHITEHORSE       YK
 315512    PARTS PLUS               BOX 510 MAIN ST          THREE HILLS      AB
 318833    TOP NOTCH CONSTRUCTION   BOX 308, STN J           CALGARY          AL
 400002    ACKLANDS LTD             945 -2ND AVE             PRINCE GEORGE    BC
 401210    COAST RANGE CONSTRUCTION 1103-207 W. HASTINGS ST  VANCOUVER        BC
 402875    HULL, DON & SONS LTD.    BOX 1297                 PRINCE GEORGE    BC
 403887  D MILNER, LARRY            BOX 28 RAU ROAD R.R.8    DAWSON CITY      YK
 406082    PRECAM RENTALS LTD:      10116-94TH AVE           CHILLIWACK       BC

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1C1. Demo files for Vancouver Utility Tutorials

dat1/sales0,1,2,3 - demo files with Zoned Signs

These are Fixed length 64 byte records.

sales0
  • EBCDIC (mainframe file)
sales1
  • transferred to unix/linux & translated to ASCII
  • but zoned signs reflect the mainframe sign conventions
  • this file shown below (after LineFeeds inserted in last byte)
sales2
  • zoned signs corrected for Micro Focus COBOL conventions
  • this file listed on the next page
sales3
  • separate +/- signs
        01-06 - CUST#
        11-12 - SLSMN#
        14-19 - DATE
        21-28 - INVOICE#
        31-36 - PRODUCT#
        39-44 - QTY
        46-52 - PRICE
        54-62 - AMOUNT
        64-64 - linefeed terminated

sales1 - ASCII with mainframe signs

          1         2         3         4         5         6
 1234567890123456789012345678901234567890123456789012345678901234
 CUST#    SLM  DATE   INV#     PROD#     QTY   PRICE    AMOUNT
 130140    21 940802 IN111001  HAM001  00002{ 0000001 00000002{
 130140    21 940802 IN111001  SCR012  00002A 0001001 00002102A
 139923    35 950802 IN111002  CHR001  00002B 0002001 00004402B
 139923    35 950802 IN111002  TAB013  00002C 0003001 00006902C
 139923    35 950807 CR5421    TAB013  00002D 0004001 00009602D
 150825    44 960804 IN1122    HAM001  00002E 0005001 00012502E
 150825    44 960804 IN1122    PLN012  00002F 0006001 00015602F
 201120    44 970807 CR5234    WHIP75  00002G 0007001 00018902G
 223240    65 980816 CR955     HAM001  00002H 0008001 00022402H
 223240    65 980816 IN441     BIT001  00002I 0009001 00026102I
 308685    21 990812 IN6605    SAW001  00001} 0000001 00000001}
 308685    21 990812 IN6605    WID115  00001J 0001001 00001101J
 308685    21 990812 CR8835    TAB013  00001K 0002001 00002401K
 315512    44 000805 IN2251    HAM001  00001L 0003001 00003901L
 315512    44 000805 IN2251    SAW051  00001M 0004001 00005601M
 315512    44 000805 IN2255    WID115  00001N 0005001 00007501N
 400002    85 010812 CR245     HAX129  00001O 0006001 00009601O
 406082    35 020815 IN33001   BBQ001  00001P 0007001 00011901P
 406089    35 020815 IN33001   TAB013  00001Q 0008001 00014401Q
 406082    65 020816 IN441     HAM001  00001R 0009001 00017101R

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1C2. Demo files for Vancouver Utility Tutorials

sales2 - zoned signs for Micro Focus COBOL

          1         2         3         4         5         6
 1234567890123456789012345678901234567890123456789012345678901234
 CUST#    SLM  DATE   INV#     PROD#     QTY   PRICE    AMOUNT
 130140    21 940802 IN111001  HAM001  000020 0000001 000000020
 130140    21 940802 IN111001  SCR012  000021 0001001 000021021
 139923    35 950802 IN111002  HAM001  000022 0002001 000044022
 139923    35 950802 IN111002  TAB013  000023 0003001 000069023
 139923    35 950807 CR5421    TAB013  000024 0004001 000096024
 150825    44 960804 IN1122    HAM001  000025 0005001 000125025
 150825    44 960804 IN1122    PLN012  000026 0006001 000156026
 201120    44 970807 CR5234    WHIP75  000027 0007001 000189027
 223240    65 980816 CR955     HAM001  000028 0008001 000224028
 223240    65 980816 IN441     BIT001  000029 0009001 000261029
 308685    75 990812 IN6605    SAW001  00001p 0000001 00000001p
 308685    75 990812 IN6605    WID115  00001q 0001001 00001101q
 308685    75 990812 CR8835    TAB013  00001r 0002001 00002401r
 315512    33 000805 IN2251    HAM001  00001s 0003001 00003901s
 315512    33 000805 IN2251    SAW051  00001t 0004001 00005601t
 315512    33 000805 IN2255    WID115  00001u 0005001 00007501u
 400002    85 010812 CR245     HAX129  00001v 0006001 00009601v
 406082    55 020815 IN33001   BBQ001  00001w 0007001 00011901w
 406082    55 020815 IN33001   TAB013  00001x 0008001 00014401x
 406082    65 020816 IN441     HAM001  00001y 0009001 00017101y

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1C3. Demo files for Vancouver Utility Tutorials

sales3 - separate +/- signs

          1         2         3         4         5         6
 1234567890123456789012345678901234567890123456789012345678901234
 CUST#    SLM  DATE   INV#     PROD#     QTY   PRICE    AMOUNT
 130140    21 940802 IN111001  HAM001  00020+ 0000001 00000020+
 130140    21 940803 IN111001  SCR012  00021+ 0001001 00021021+
 130140    21 940804 IN111001  CHR001  00022+ 0002001 00044022+
 139923    35 950802 IN111002  TAB013  00023+ 0003001 00069023+
 139923    35 950807 IN111002  TAB013  00024+ 0004001 00096024+
 150825    44 960804 IN1122    HAM001  00025+ 0005001 00125025+
 150825    44 960805 IN1122    HAX129  00026+ 0006001 00156026+
 201120    44 970807 CR5234    WHIP75  00027+ 0007001 00189027+
 223240    65 980816 CR955     HAM001  00028+ 0008001 00224028+
 223240    65 980817 IN441     BBQ001  00029+ 0009001 00261029+
 308685    21 990812 IN6605    SAW051  00010- 0000001 00000010-
 308685    21 990813 IN6605    WHIP75  00011- 0001001 00011011-
 308685    21 990814 CR8835    TAB013  00012- 0002001 00024012-
 315512    44 000805 IN2251    HAM001  00013- 0003001 00039013-
 315512    44 000806 IN2251    SAW051  00014- 0004001 00056014-
 315512    44 000807 IN2255    WHIP75  00015- 0005001 00075015-
 400002    85 010812 CR245     HAX129  00016- 0006001 00096016-
 406082    35 020815 IN33001   BBQ001  00017- 0007001 00119017-
 406082    35 020816 IN33001   TAB013  00018- 0008001 00144018-
 406082    65 020816 IN441     HAM001  00019- 0009001 00171019-

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1D1. using uvhd with Variable-Length files

vendormas3 - sample RDW file


 uvhd dat1/vendormas3 z4  <-- option 'z4' identifies RDW files
 =======================      (record size in 4 byte binary prefix)
 now=20110521:0920 uvhd /home/uvadm/dat1/vendormas3 z4
 version=20110519 lastmod=2011031421 print=p99
 records=8 rsize=23 fsize=249
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ....DELL10 M  Dell Inc.
             01004444332422466624662
             070045CC100D0045CC09E3E
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
          23 ....HP0010 M  Hewlett Packard
             010045333324224676677256666762
             0E008000100D00857C5440013B1240
                      10        20        30        40        50        60
 r#        3 0123456789012345678901234567890123456789012345678901234567890123
          53 .-..IBM010 M  International Business Machines
             020044433324224676766766666247766677246666667
             0D0092D0100D009E452E149FE1C02539E5330D1389E53
                      10        20        30        40        50        60
 r#        4 0123456789012345678901234567890123456789012345678901234567890123
          98 ....MFC010 M  Micro Focus COBOL
             0100444333242246676246677244444
             0F00D630100D00D932F06F35303F2FC
                      10        20        30        40        50        60
 r#        5 0123456789012345678901234567890123456789012345678901234567890123
         129 ....MS0010 M  Microsoft Corp.
             01004533332422466767667246772
             0D00D300100D00D932F3F6403F20E
                      10        20        30        40        50        60
 r#        6 0123456789012345678901234567890123456789012345678901234567890123
         158 ....REDH10 M  Red Hat Linux
             010054443324225662467246677
             0B002548100D0025408140C9E58
                      10        20        30        40        50        60
 r#        7 0123456789012345678901234567890123456789012345678901234567890123
         185 ."..SUN010 M  Sun Microsystems Ltd
             0200554333242257624667677776672476
             020035E0100D0035E0D932F39345D30C44
                      10        20        30        40        50        60
 r#        8 0123456789012345678901234567890123456789012345678901234567890123
         219 ....UVSI10 M  UV Software Inc.
             010055543324225525667767624662
             0E005639100D005603F64712509E3E

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1D2. using uvhd with Variable-Length files


 uvhd dat1/vendormas3 z4  <-- option 'z4' identifies RDW files
 =======================

notes re RDW files

  1. We will use 'uvhd' to display RDW files, because the 'vi' editor cannot display the 4 byte binary prefix. The records size is Big-End binary in the 1st 2 bytes (3rd & 4th bytes are nulls).

  2. For the file listed above, the record sizes in hexadecimal & decimal are: #1=x'17'=23, #2=x'1E'=30, #3=x'2D'=45, #4=x'1F'=31, #5=x'1D'=29, #6=x'1B'=27, #7=x'22'=34, #8=x'1E'=30.

  3. For variable lth files, the record count would be wrong, without building an Index on startup, which uvhd now does since November 2010. Prior to the Index the record# was calculated at each record by dividing the file-size by the record-size.

  4. Building the Index on startup speeds up subsequent finds by record#. Startup reads to EOF storing file-pointers & record#s at 1% filesize intervals. This also locks in the correct record count on all future screens headings. Use 'f#' to find records by their actual record# in Variable-Length files (vs r# for Fixed-length files).

  5. When files are FTP'd from a mainframe, any variable length files should be FTP'd with the RDW option.

  6. By default uvhd displays records on the screen. There are 2 ways to create hard-copy printouts - using command 'p' (print) or 'i' (immediate print)

  7. The 'p' command writes a file of formatted (vertical hexadecimal) records to the tmp/... subdir, appending a date/time stamp, for example:


    uvhd dat1/vendormas3 z4  <-- start uvhd on desired file
    =======================    - displays 1st record & prompts for command
    --> p8                   <-- print(write) 8 records to tmp/...

    uvlp12 tmp/vendormas3_110521_181300P  <-- list Print command output
    ====================================
  1. The 'i' command would write tmp/vendormas2_yymmdd_hhmmssI and print immediately using the 'uvlp12' script, BUT you must have setup your profile to export variable UVLPDEST to identify the printer, for example:


    export UVLPDEST=-dlp0
    =====================

'-d' is the destination option & 'lp0' is the printer (no space allowed)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1E1. using uvhd with Variable-Length files

vendormas3B - BDW/RDW sample file

'vendormas3B' is a BDW/RDW file, similar to 'vendormas3' RDW file shown above. In addition to the 3 byte binary RDW prefix, the records are Blocked & each block is preceded with a BDW prefix.

 Here is the 1st block of the file, shown with uvhd using option 'r256'
 - not option z4 (RDW) or option z8 (BDW). 'r256' displays 256 byte fixed-length
 records, which in this case happens to be the BDW Block size.

The 1st BDW in 1st 2 bytes is x'0100' which is 256, the RDW of 1st record is in bytes 4&5 x'0018' which is 24 (1*16+8=24). Do you see that there are 8 vendor master records in this block ?


 uvhd /home/uvadm/dat1/vendormas3B r256
 ======================================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ........DELL10 - Dell Inc.  ....HP0010 - Hewlett Packard.,..IBM0
             0000010044443322246662466222010045333322246766772566667602004443
             1000080045CC100D045CC09E3E000C008000100D0857C5440013B1240C0092D0
          64 10 - International Business Machines. ..MFC010 - Micro Focus COB
             3322246767667666662477666772466666670200444333222466762466772444
             100D09E452E149FE1C02539E5330D1389E530000D630100D0D932F06F35303F2
         128 OL  ....MS0010 - Microsoft Corp.....REDH10 - Red Hat Linux  .$..
             4422010045333322246676766724677201005444332225662467246677220200
             FC000C00D300100D0D932F3F6403F20E0C002548100D025408140C9E58000400
         192 SUN010 - Sun Microsystems Ltd   . ..UVSI10 - UV Software Inc.
             5543332225762466767777667247622202005554332225525667767624662222
             35E0100D035E0D932F39345D30C4400000005639100D05603F64712509E3E000

option 'z8' for BDW/RDW files

Use option 'z8' to display individual records of a BDW/RDW file. Note that uvhd does not show the 'BDW's, so the RDW of the 1st record below starts at displacement '4'. Here are 1st 2 records (same as RDW on page '1D1').


 uvhd /home/uvadm/dat1/vendormas3B z8
 ====================================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           4 ....DELL10 - Dell Inc.
             010044443322246662466222
             080045CC100D045CC09E3E00
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
          28 ....HP0010 - Hewlett Packard
             0100453333222467667725666676
             0C008000100D0857C5440013B124

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

1F1. Demo files for Vancouver Utility Tutorials

dat1/vendorpaymas0,1,2,3 - demo file with 2 Record Types

'vendorpaymas' is a demo/sample file with 2 record types (Master & Payment). The file is supplied in 4 different file types:

vendorpaymas0
  • EBCDIC Fixed-length 64 byte records
vendorpaymas1
  • ASCII Fixed-length 64 byte records
vendorpaymas2
  • ASCII variable-length Text file (linefeed after last nonblank)
vendorpaymas3
  • ASCII variable-length RDW file (recsize in binary prefix)

dat1/vendorpaymas2 - Text file


 cat dat1/vendorpaymas   <-- display text file version (only 16 records)
 =====================
 DELL10 M  Dell Inc.
 DELL10 P  20100131  INV00010  000195000 Dell server 2900
 HP0010 M  Hewlett Packard
 HP0010 P  20100215  INV00020  000042500 HP LCD Monitor
 HP0010 P  20100215  INV00020  000350000 HP XW9400 Workstation
 IBM010 M  International Business Machines
 IBM010 P  20100321  INV00030  000750000 AIX server
 IBM010 P  20100322  INV00030  000750000 DB2 database software
 MFC010 M  Micro Focus COBOL
 MFC010 P  20100420  INV00040  000320000 Micro Focus Developer
 MS0010 M  Microsoft Corp.
 MS0010 P  20100501  INV00050  000011600 Microsoft Windows 7
 REDH10 M  Red Hat Linux
 REDH10 P  20100620  INV00060  000034900 Red Hat Enterprise 5.4
 SUN010 M  Sun Microsystems Ltd
 UVSI10 M  UV Software Inc.

dat1/vendorpaymas2 - RDW file


 uvhd /home/uvadm/dat1/vendorpaymas3 z4 <-- display RDW file (1st 2 records)
 ======================================   - uvhd shows binary rec-size prefixes
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ....DELL10 M  Dell Inc.
             01004444332422466624662
             070045CC100D0045CC09E3E
 rec#=1 rsize=23 fptr=0 fsize=745 rcount=16
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
          23 .<..DELL10 P  20100131  INV00010  000195000 Dell server 2900
             030044443325223333333322445333332233333333324666276776723333
             0C0045CC10000020100131009E60001000000195000045CC035265202900
 rec#=2 rsize=60 fptr=23 fsize=745 rcount=16

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_2. Data Investigation, Manipulation, Analysis,& Validation

Part 2 - uvhd


2A1. uvhd Program Description
- data file investigation utility
- ideal for mainframe EBCDIC files with packed/binary
- browse,search,select,update,print,translate,etc

2A2. uvhd Display format illustrated
- displays any file in vertical hexadecimal
- 64 byte segments on 3 lines (characters, zones,& digits)

2B1. Investigating EBCDIC files
- demo file custmas0 EBCDIC vs custmas1 ASCII

2B2. Display EBCDIC file with option 'a'
- to translate character line to ASCII so you can read on unix/linux

2C1. Packed Decimal fields
- sample display in vertical hexadecimal

2D1. Example of using uvhd to translate EBCDIC to ASCII
- character fields only, packed fields must be preserved unchanged
- could use uvhd (free download) for simple record layouts
- the full package has comprehensive EBCDIC to ASCII tools, driven by
  the COBOL copybooks to translate high volumes of complex record layouts

uvhd introduction vs detailed documentation

This manual is a good place to start learning uvhd. When you need more details please see complete documentation at https://uvsoftware.ca/uvhd.htm.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2A1. uvhd - file investigation hexdump utility

uvhd - Program Description

'uvhd' is a file investigation utility. It displays any file in vertical hexadecimal and prompts for commands to browse, search, select, update, scan/replace, print, translate, etc. uvhd is an interactive utility with a command line interface and 23 help screens. uvhd is the favorite utility of most UV Software customers.

uvhd is a free download from: https://www.uvsoftware.ca/freestuff.htm

uvhd & Vancouver Utilities

'uvhd' is offered as a Free sample of the Vancouver Utilties which are often used to convert mainframe JCL, COBOL,& DATA to Unix/Linux/Windows Mainframe JCL is converted to Korn shell scripts, and Mainframe COBOL is converted to Micro Focus COBOL to run under Unix/Linux.

uvhd is especially valuable to investigate mainframe type files with packed fields and no linefeeds (required by most unix/linux utilities). uvhd can investigate all file types: sequential, fixed, variable, indexed, text, binary, C-ISAM, D-ISAM, Micro Focus COBOL IDXFORMAT1,3,8)

uvhd displays any file in vertical hexadecimal, 64 byte segments, in 3 line groups (characters on 1st line, zones on 2nd line,& digits on 3rd line).

When you find a data problem, use the print function to capture the evidence and send it to your associates via email or hardcopy printout. Hardcopy or softcopy printouts of binary/packed files with byte position scales are not possible with most other unix utilities.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2A2. uvhd - file investigation hexdump utility

uvhd Display format illustrated

Records are displayed 256 bytes at a time by default (if rcsz not spcfd) in 4 groups of 3 vertical hex lines as follows:

         ...........record in character format (. for non-dspl chars)....
         ...........hex representation of record zones...................
         ...........hex representation of record digits..................

sample display - 1st record from dat1/custmas1 file


 uvhd dat1/custmas1 r256s2   - command to display the testfile
 =========================     options: r256=record-size, s2=double space
 now=20110128:0657 uvhd /home/uvadm/dat1/custmas1 r256s2
 version=20110114 records=32 rsize=256 fsize=8192
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    EVERGREEN MOTORS LTD.    1815 BOWEN ROAD          NANA
             3333332222454545444244545524542222233332445442544422222222224444
             130140000056527255E0DF4F230C44E0000181502F75E02F140000000000E1E1
          64 IMO          BC V9S1H1    250-754-5531 LARRY WRENCH     ..4V|...
             4442222222222442535343222233323332333324455525544442222201357000
             9DF00000000002306931810000250D754D55310C12290725E38000000246C000
         128 .........W0....`........)X}..f3.....\.................4V}...f...
             0000000005300016000000002570063100095000000000000000013570016000
             0C0000C0270D0540C0000C0098D0263C0444C0000C0000C0000C0246D0056C00
         192 .E|...V}.......................f.....<........f.C 19950531
             0470005700000000880000000018000680001300000000694233333333222222
             35C0046D0000C0023C0000C0083C0056D0012C0000C0016D3019950531000000
 rec#=1 rsize=256 fptr=0 fsize=8192 rcount=32
 null=next,r#=rec,s=search,u=update,x=undo,p=print,i=iprint,w=write,e=count
 ,R#=Recsize,h1=char,h2=hex,g=genseq#,c=chkseq#,t=translate(ta,te,tu,tl,tc,tp)
 ,q=quit,k0=helpoff,k1=helpon,?=HelpScreens -->

In this example we entered 'q' to quit after the 1st 256 bytes of the file was displayed, but we could have entered a record# or byte# to explore other areas of the file. Note that the current record# & byte# are displayed at the left of the two top lines (the 0-64 byte scale).

The uvhd display is especially useful for displaying data files that do not have line feeds & therefor are not easily displayed by an editor & for files with packed decimal fields.

Any non displayable characters is represented by periods on the character line (1st line of 3 vertical hex lines). The above sample has a lot due to the multiple packed fields.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2B1. uvhd - file investigation hexdump utility

Investigating EBCDIC files

Both ASCII & EBCDIC versions of a few demo files are provided. Our convention is to use suffix '0' to indicate EBCDIC & '1' for ASCII.

custmas0
  • EBCDIC version of customer master demo file
custmas1
  • ASCII version of customer master demo file
sales0
  • EBCDIC version of sales detail demo file
sales1
  • ASCII version of sales detail demo file

uvhd provides option 'a' to view EBCDIC files. First we will show you the display if you run uvhd on the custmas0 EBCDIC file with NO options. The default record size is 256 which happens to be the record size of custmas0.


 uvhd dat1/custmas0    <-- investigate EBCDIC file with NO options
 ==================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ......@@@@.........@......@...K@@@@....@.....@....@@@@@@@@@@....
             FFFFFF4444CECDCDCCD4DDEDDE4DEC44444FFFF4CDECD4DDCC4444444444DCDC
             130140000055597955504636920334B000018150266550961400000000005151
          64 ...@@@@@@@@@@..@......@@@@...`...`....@@@@@@@@@@@@@@@@@@........
             CDD4444444444CC4EFEFCF4444FFF6FFF6FFFF44444444444444444400000000
             946000000000023059218100006040754055310000000000000000000000C000
         128 .........W0....`........)X|..f3.....\.......................f...
             0000000005300016000000002570063100095000000000000000000000016000
             0C0000C0270C0540C0000C0098C0263C0444C0000C0000C0000C0000C0056C00
         192 .E|...V}.......................f.....<........f..@@@@@@@@@@@@@@@
             047000570000000088000000001800068000130000000069C444444444444444
             35C0046D0000C0023C0000C0083C0056D0012C0000C0016D3000000000000000
 rec#=1 rsize=256 fptr=0 fsize=8192 rcount=32 -->

option 'a' translate character line to ASCII

Of course we can NOT read the data on unix/linux ASCII systems. 'uvhd' provides option 'a' to perform EBCDIC to ASCII translation on the character line of the 64 byte 3 line groups (characters, zones, digits). You can then 'read' the EBCDIC data. Note that the zones & digit lines will show the EBCDIC values. Here is just the 1st 64 byte segment as shown with option 'a'.

 Chars-->  0 130140    EVERGREEN MOTORS LTD.    1815 BOWEN ROAD          NANA
 Zones-->    FFFFFF4444CECDCDCCD4DDEDDE4DEC44444FFFF4CDECD4DDCC4444444444DCDC
 Digits->    130140000055597955504636920334B000018150266550961400000000005151

See the complete 1st record of custmas0 shown below with option 'a'.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2B2. uvhd - file investigation hexdump utility

display EBCDIC file with option 'a'


 uvhd dat1/custmas0 r256as2  <-- redisplay EBCDIC demo file, with option 'a'
 ==========================    - option 'r256' is the default & could be omitted
                               - option 's2' inserts space between 3 line groups
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    EVERGREEN MOTORS LTD.    1815 BOWEN ROAD          NANA
             FFFFFF4444CECDCDCCD4DDEDDE4DEC44444FFFF4CDECD4DDCC4444444444DCDC
             130140000055597955504636920334B000018150266550961400000000005151
          64 IMO          BC V9S1H1    604-754-5531                  ........
             CDD4444444444CC4EFEFCF4444FFF6FFF6FFFF44444444444444444400000000
             946000000000023059218100006040754055310000000000000000000000C000
         128 .........W.....-.........X@..f.....m*.......................f...
             0000000005300016000000002570063100095000000000000000000000016000
             0C0000C0270C0540C0000C0098C0263C0444C0000C0000C0000C0000C0056C00
         192 .E@...V'........c..............f..............f.C
             047000570000000088000000001800068000130000000069C444444444444444
             35C0046D0000C0023C0000C0083C0056D0012C0000C0016D3000000000000000
 rec#=1 rsize=256 fptr=0 fsize=8192 rcount=32 -->

compare EBCDIC & ASCII demo files

Please compare the EBCDIC file display above with the ASCII file shown 2 pages back at '2B2'.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2C1. uvhd - file investigation hexdump utility

Packed Decimal fields

Note that there are 24 * 5 byte packed fields in 120-240. You can identify the end of each packed field by the sign, x'_C' or x'_D'in the last byte. For example here is the 1st 8 bytes of last segment:

         192 .E@...V'   <-- character line
             04700057   <-- zones line
             35C0046D   <-- digits line
               ^    ^   <-- '^' marks sign/end of each packed field

Relevancy of option 'a'

Option 'a' is relevant to sites that are converting from an IBM mainframe to Unix, Linux, or Windows/SFU. The data files are transferred to the new machine where they are converted from EBCDIC to ASCII (see DATAcnv1.htm).

The conversion jobs are generated automatically from the COBOL copybooks. All character type & unpacked numeric fields are translated from EBCDIC to ASCII, while preserving any packed or binary fields unchanged (since they would be destroyed by the translation).

'uvhd' with option 'a' is a vital asset to verify the data conversions, you can spot check the conversion by examining both the input EBCDIC data & the output ASCII file.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

2D1. uvhd - file investigation hexdump utility

uvhd Translate Example

Vancouver Utilities has very comprehensive tools to translate EBCDIC files to ASCII based on the COBOL copybook record layouts - so only character fields are translated & packed fields are preserved unchanged (since translating packed fields would destroy them). Please see DATAcnv1.htm#Part_3.

But, here is an example of using uvhd to translate simple record layouts. You might use this if you have only the free 'uvhd' & not the full Vancouver Utilities package required for the comprehensive translation based on the COBOL copybooks.

This exercise assumes you have setup the directories as shown on page '1A2'. If you are using the free uvhd download, we also assume you have downloaded the demo files from https://www.uvsoftware.ca/VUdemofiles.zip and copied the relevant file (custmas0) to subdir dat1/...


 #1. mkdir tmp1       <-- make subdir for translated file
     ==========         - to preserve input demo file

 #2. cp dat1/custmas0 tmp1  <-- copy EBCDIC demo file to tmp1/custmas0
     =====================    - to preserve input demo file

 #3. uvhd tmp1/custmas0 r256ua  <-- initiate uvhd
     =========================    - option 'u' required to modify records
                                  - option 'a' to show EBCDIC chars in ASCII

 #3a. t99e 0(120)     <-- translate 1st 120 bytes of all records to Ebcdic
      ===========
 #3b. 1               <-- return to 1st record in file
      ===
 #3c. t99e 240(16)    <-- translate last 16 bytes of all records to Ebcdic
      ============
 #3d. q               <-- quit uvhd, so we can rerun without option 'a'
      ===

 #4. uvhd tmp1/custmas0 r256  <-- rerun uvhd
     =======================    - omitting option 'a' (or 'u')

The records in tmp1/custmas0 will now appear as shown on page '2A2' for the ASCII file dat1/custmas1. Note that the packed fields have been preserved.

The 't' command will translate to Ascii,Ebcdic,Upper,Lower,Characters,or Periods (depending on options a,e,u,l,c,p). Some other translate examples:

ta
  • translate entire current record to Ascii
t99e
  • translate ALL records (from current to EOF) to Ebcdic
tl 0(120)
  • translate 1st 120 bytes of current record to Lower case
tc1
  • translate unprintable Characters to blanks EXCEPT LineFeeds x'0A'
    (c2=except CR x'0D', c4=except FF x'0C', c8=except tab x'09')
tc15
  • translate to Chars, except LFs,CRs,FFs,Tabs (1+2+4+8=15)

See full doc for uvhd at https://uvsoftware.ca/uvhd.htm

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_3. Data Investigation, Manipulation, Analysis,& Validation

Part 3 - uvhdcob


3A1. uvhdcob overview
- utility to display data field contents with COBOL copybook field names
- sample display using customer master demo file & copybook

3B1. Preparation to use uvhdcob.
- must convert copybook to 'cobmap' record layout
- using utility provided 'uvcopy cobmap1'

3C1. Operating Instructions for 'uvcopy cobmap1'
- sample 'cobmap' for the customer master demo file

3D1. Verify data command (v)
- creating BAD data for demo

3D2. test/demo verify command
- sample display with BAD data fields indicated
Bad-Numeric, Bad-Packed, Bad-Character

3D3. Notes re verify test/demo

3E1. Options for Verify command
options a-b,c-d,e-f to inhibit Verify on up to 3 areas
- identified by start & end byte displacements
- allows you to search/Verify for other fields in record
 options x1,x2,x4 to inhibit stop-on-err for numeric/packed/character fields
- continues to EOF & displays error counts & record#s of 1st & last error
 options y1,y2,y4 to specify signs: zoned ASCII, zoned EBCDIC, separate +/-
 options z1,z2,z4 to convert all blank numeric/packed fields to zeros
- or convert unprintables in character fields to '.' periods
- option 'p' to print formatted err records to tmp/filename_yymmdd_hhmmssVP
- option 'w' to write original err records to tmp/filename_yymmdd_hhmmssVW
- may follow with lvp/lvw to close files or omit to combine outputs

3F1. Create verify report for entire file --> v99p99x7
- option p99 creates separate tmp/report with error records only
- option x7 inhibits normal stop/display
- quit uvhdcob & use vi/more/lp/uvlp12 on tmp/report
 See sample report for dat1/custmas1x with errs created on 3F1

uvhdcob introduction vs detailed documentation

This manual is a good place to start learning uvhdcob. When you need more details please see complete documentation at https://uvsoftware.ca/uvhdcob.htm.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3A1. Data File Investigation,Manipulation,Analysis,& Validation

uvhdcob

uvhdcob - sample display

 uvhdcob datafile=dat1/custmas1
 20090222:0844 copybook=maps/custmas1 options=
 record#=1 totalrecs=32 recsize=256 filesize=8192 fileptr=0
 rec#       1 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 cm-num                       0   5 n  130140
 002 cm-delete                    6   9
 003 cm-name                     10  34    EVERGREEN MOTORS LTD.
 004 cm-adrs                     35  59    1815 BOWEN ROAD
 005 cm-city                     60  75    NANAIMO
 006 filler001                   76  76
 007 cm-prov                     77  78    BC
 008 filler002                   79  79
 009 cm-postal                   80  89    V9S1H1
 010 cm-phone                    90 101    250-754-5531
 011 cm-contact                 102 119     LARRY WRENCH
 012 cm-thisyr-sales       012  120 124pns 001234567C
 013 cm-lastyr-sales       012  180 184pns 001234567D
 014 filler003                  240 255    C 19950531
 rec#=1 rcount=32 rsize=256 fsize=8192 dat1/custmas1
 null=next,r#=rec,s=search,u=update,x=undo,p=print,i=iprint,w=write,t=tally
 ,v=verify,c=chkseq#,e=exit to uvhd,q=quit,?=help -->
  1. we show only the 1st field of occurs, such as cm-thisyr-sales - 12 x 5 byte packed fields

  2. packed fields shown in horizontal hex x'001234567C' - sign x'_C' in last byte positive, x'_D' negative

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3B1. Data File Investigation,Manipulation,Analysis,& Validation

preparation to use uvhdcob

You can test/demo uvhdcob by copying supplied test files to subdirs in your homedir.


 #1. Login --> your homedir /home/userxx/
     =====

 #2a. mkdir dat1    <-- setup subdir for data files
      ==========
 #2b. mkdir cpys    <-- setup subdir for copybooks
      ==========
 #2c. mkdir maps    <-- subdir for cobmaps(record layouts created from copybooks)
      ==========
 #2d. mkdir tmp     <-- subdir for uvhdcob outputs (prints, selects, verifies)
      =========

copy training files to your home subdirs


 #3a. cp $UV/dat1/custmas* dat1    <-- customer master files
      =========================      - sample fixed length with packed fields
 #3b. cp $UV/dat1/sales* dat1      <-- sales files
      =======================        - sample fixed length with numeric fields
 #3c. cp $UV/dat1/vendor* dat1     <-- Vendor Master & Payment files
      ========================       - sample RDW variable length files

 #4a. cp $UV/mf/cpys/* cpys   <-- COBOL copybooks
      =====================
 #4b. cp $UV/mf/maps/* maps   <-- cobol copybook MAPS (record layouts)
      =====================

convert copybooks to record layouts


 #5. uvcopyx cobmap1 cpys maps uop=q0i7    <-- convert copybooks to cobmaps
     ==================================      see illustration next page -->

uvhdcob demos on following pages


 #6a. uvhdcob dat1/custmas1 maps/custmas   <-- test uvhdcob for customer master
      ==================================     see illustration 2 pages ahead -->

 #6b. uvhdcob dat1/custmas0 maps/custmas a <-- retest with EBCDIC file
      ====================================   - option 'a' translate to ASCII

 #6c. uvhdcob dat1/sales1 maps/sales       <-- test uvhdcob on sales detail
      ==============================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3B2. preparation to use uvhdcob

create record layouts from copybooks

uvhdcob requires prior generation of 'cobmaps' from the COBOL copybooks. You could regenerate the cobmaps as follows:


 #1. Login --> your homedir
     =====

 #2.  uvcopyx cobmap1 cpys maps uop=q0i7
      ==================================
      - generate cobmaps for all files in directory
      - easier than for 1 at a time (as shown below)

 #2a. uvcopy cobmap1,fili1=cpys/custmas1,filo1=maps/custmas1
      ======================================================
      - generate cobmap for 1 copybook at a time

sample cobmap report

 cobmap1  start-end bytes for cobol record fields    199905101503  pg# 0001
 cpys/custmas1                                     RCSZ=0256  bgn-end lth typ
 * custmas1 - cobol copy book for custmas1 file (see uvtrain.
      10 cm-num           pic  9(6).                          000-0005 006 n  06
      10 cm-delete        pic  x(4).                          006-0009 004
      10 cm-name          pic  x(25).                         010-0034 025
      10 cm-adrs          pic  x(25).                         035-0059 025
      10 cm-city          pic  x(16).                         060-0075 016
      10 filler001        pic  x.                             076-0076 001
      10 cm-prov          pic  x(2).                          077-0078 002
      10 filler002        pic  x.                             079-0079 001
      10 cm-postal        pic  x(10).                         080-0089 010
      10 cm-phone         pic  x(12).                         090-0101 012
      10 cm-contact       pic  x(18).                         102-0119 018
      10 cm-thisyr-sales  pic  s9(7)v99 comp-3 occurs 12.     120-0124 005pns 09
      10 cm-lastyr-sales  pic  s9(7)v99 comp-3 occurs 12.     180-0184 005pns 09
      10 cm-thisyr-xft    pic  s9(7)v99 comp-3.               240-0244 005pns 09
      10 cm-lastyr-xft    pic  s9(7)v99 comp-3.               245-0249 005pns 09
      10 filler003        pic  x(6).                          250-0255 006
 *RCSZ=0256                                                       0256

Note that this sample cobmap was used for the uvhdcob sample illustrated on the previous page (please relate the fieldnames & locations).

Also note that the 'cobmaps' are very useful to COBOL shops for other reasons (when you need to know field locations for sorting, verifying data, etc).

You may find it convenient to print cobmaps for all copybooks, and mount in a 3 ring binder for easy reference. See documentation for the 'cobmap' utility in COBaids.doc .

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3C1. uvhdcob - display data files with COBOL field names

demo uvhdcob on customer master


 #1. Login --> your homedir

 #2. ls -lr    <-- list subdirs & files
     ======        (showing only customer master & sales files)
 /home/userxx
 :-----dat1               <-- subdir for test/demo DATA
 :     :----- custmas0        - EBCDIC file with 24 5 byte packed fields
 :     :----- custmas1        - ASCII file with 24 5 byte packed fields
 :     :----- custmas2        - ASCII file name&address without packed fields
 :     :----- sales0          - EBCDIC sales details with unpacked fields
 :     :----- sales1          - ASCII with EBCDIC sign conventions
 :     :----- sales2          - ASCII with Micro Focus COBOL sign conventions
 :     :----- sales3          - ASCII with separate +/- signs
 :     :----- vendormas3      - Vendor master RDW variable length file
 :     :----- vendorpaymas3   - Vendor payment RDW variable length file
 :-----cpys                 <-- COBOL copybooks
 :     :-----custmas1.cpy     - customer master Name&Adrs + thisyr/lastyr sales
 :     :-----sales.cpy        - detail sales with zoned signs
 :     :-----sales3.cpy       - detail sales with separate +/- signs
 :     :-----vendormas.cpy    - vendor master variable length RDW sample file
 :-----maps                 <-- record layouts (maps) from copybooks
 :     :----- custmas1
 :     :----- sales
 :     :----- sales3
 :     :----- vendormas
 :-----tmp                  <-- for your output files

 #2. uvhdcob dat1/custmas1 maps/custmas <-- run uvhdcob specifying
     ==================================     data-file & copybook
 now=20110129:0723 uvhdcob dat1/custmas1
 version=20110114 copybook=maps/custmas
 rec#=1 rcount=32 rsize=256 fsize=8192 fptr=0
 rec#       1           occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 cm-num                       0    5 n  130140
 002 cm-delete                    6    9
 003 cm-name                     10   34    EVERGREEN MOTORS LTD.
 004 cm-adrs                     35   59    1815 BOWEN ROAD
 005 cm-city                     60   75    NANAIMO
 006 filler001                   76   76
 007 cm-prov                     77   78    BC
 008 filler002                   79   79
 009 cm-postal                   80   89    V9S1H1
 010 cm-phone                    90  101    250-754-5531
 011 cm-contact                 102  119     LARRY WRENCH
 012 cm-thisyr-sales       012  120  124pns 001234567C
 013 cm-lastyr-sales       012  180  184pns 001234567D
 014 filler003                  240  255    C 19950531
 rec#=1 rcount=32 rsize=256 fsize=8192 fptr=0
 k1=HelpPrompts,?=HelpScreens -->

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3D1. uvhdcob - display data files with COBOL field names

'v' verify data command

The 'v' command will verify character, numeric,& packed data fields. Verify searches the file for unprintable characters in pic x fields & invalid digits or signs in unpacked or packed numeric fields.

We have provided a doctored version of dat1/custmas1 (dat1/custmas1x) to test/demo uvhdcob. dat1/custmas1x has been updated (using uvhdcob) to create errors in record#s 10 & 20.

creating BAD data

Here are the 'update' commands we used to create the BAD data for the 'verify' command test/demo. We will omit the displays & list only the commands.


 #1. Login --> your homedir (with subdirs shown on previous page)
     =====

 #2. cp dat1/custmas1 dat1/custmas1x  <-- copy to alt file to make bad data
     ===============================

 #3. uvhdcob dat1/custmas1x maps/custmas1 u <-- option 'u' for update
     ======================================

 #4. --> 10              <-- goto record# 10

 #5a. --> u 5(1),'*'     <-- create non-numeric data in numeric field (cust#)
 #5b. --> u 78(1),x'EE'  <-- create unprintable data in character field (prov)
 #5c. --> u 120(1),x'EE' <-- create invalid digits in packed field (thisyr sales)

 #6. --> 20                      <-- goto record# 20

 #7a. --> u 5(2),'XX'            <-- create non-num data in numeric field (cust#)
 #7b. --> u 83(3),x'EEEEEE'      <-- create unprintables in char field (province)
 #7c. --> u 180(5),x'1E2E3E4E5E' <-- create BAD data in packed field (lastyr sls)

test/demo verify command


 #8. uvhdcob dat1/custmas1x maps/custmas1  <-- run on demo file
     ====================================
     - uvhdcob displays 1st record & prompts for command
     - we are omitting 1st record display, see verify displays next page
     --> v99  <-- search/verify until bad record found or EOF reached
                - will display next bad record found & prompt for command
                - if desired, could print(formatted) or write(raw) to tmp/...
     --> vv   <-- may then enter 'vv' to continue search/verify

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3D2. uvhdcob 'verify' test/demo

sample display with BAD data fields

The 'v' verify command searches from the current record until it finds a record with at least 1 bad data field & displays that record. For our demo file (dat1/custmas1x) record# 10 is the 1st record with bad data fields.

--> v99 <-- Verify

 rec#      10           occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 cm-num                       0    5 n  15468*      31353436382A      <-BadN
 002 cm-delete                    6    9
 003 cm-name                     10   34    FRANKLIN RESOUCES
 004 cm-adrs                     35   59    805 - 1199 HOWE STREET
 005 cm-city                     60   75    VANCOUVER
 006 filler001                   76   76
 007 cm-prov                     77   78    B.          42EE              <-BadC
 008 filler002                   79   79
 009 cm-postal                   80   89    V6C2T6
 010 cm-phone                    90  101    604-666-5489
 011 cm-contact                 102  119    RON BENNETT
 012 cm-thisyr-sales       012  120  124pns EE0123456C                    <-BadP
 013 cm-lastyr-sales       012  180  184pns 000000000C
 014 filler003                  240  255    C 19980331
      v99 --> 10 recs verified, 4 ERRs found in rec# 10, 1st at byte# 0
      2 packed & 1 numeric bad signs/digits, 1 unprintables in char fields

--> vv <-- continue Verify

 rec#      20 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 cm-num                       0   5 n  2343XX      323334335858      <-BadN
 002 cm-delete                    6   9
 003 cm-name                     10  34    TERRITORIAL REWIND
 004 cm-adrs                     35  59    BOX 1648
 005 cm-city                     60  75    WHITEHORSE
 006 filler001                   76  76
 007 cm-prov                     77  78    YK
 008 filler002                   79  79
 009 cm-postal                   80  89    Y1A...      593141EEEEEE      <-BadC
 010 cm-phone                    90 101    403-512-3948
 011 cm-contact                 102 119
 012 cm-thisyr-sales       012  120 124pns 000000000C
 013 cm-lastyr-sales       012  180 184pns 1E2E3E4E5E                     <-BadP
 014 filler003                  240 255      20021130
      vv --> 10 recs verified, 10 ERRs found in rec# 20, 1st at byte# 0
      5 packed & 2 numeric bad signs/digits, 3 unprintables in char fields

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3D3. uvhdcob 'verify' test/demo

Notes re uvhdcob Verify demo above

  1. We created intentional errors in records #10 & #20 of dat1/custmas1x.

  2. Note flags on right hand side '<-BadN', '<-BadC', '<-BadP'

  3. cm-num '15468*' flagged because '*' is non-numeric. Hex representation x'313534363858' is shown in case of unprintables.

  4. cm-prov 'B.' is shown in hex as x'42EE' (the '.' is actually x'EE').

  5. cm-thisyr-sales is shown as x'EE0123456C' (1st 2 digits x'EE' invalid).

  6. packed fields are always shown in hex (valid or invalid).

  7. uvhdcob shows only the 1st field of occurs. You could verify all occurs fields using 'genverify1' (see DATAcnv1.htm#8D1) which generates a uvcoy job to verify all numeric & packed fields using the 'vnf' instruction. See the 'vnf' instrucion at uvcopy3.htm#vnf.

  8. As of Dec 2010, uvhdcob now has option 'o2' to display all occurrences

  9. In Jan 2011, we added options 'p' & 'x' to the verify command, which allow you to create a verify report for entire file.


    --> v99p99x7 <-- create a verify report for the entire file
        =======      for separate investigation & printing.
  1. If desired, you may specify option 'z7' to convert any all blank numeric or packed fields to zeros, and convert any unprintable bytes in character fields to '.' periods. See options z1,z2,z4=z7 on the next page.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3E1. uvhdcob 'verify' test/demo

options for Verify

    abcdef - options to inhibit verify in 3 areas ID by offset from/to
 v99a120b239 - inhibit verify in all fields from dsplcmnt 120 to 239
             - to bypass known bad fields (to search for unknown bad fields)
 v99a100b120c200d220e300f320 - inhibit verify in 3 record areas max
 ...x  - option 'x' no-stop-on-error, count for display at EOF (or max# spcfd)
 v99x1 - nostop on errors in numeric fields
 v99x2 - nostop on errors in packed fields
 v99x4 - nostop on errors in character (pic x) fields
 v99x3 - nostop on errors in numeric + packed fields
 v99x7 - nostop on errors in numeric + packed + character fields
 ...y  - option 'y' specify valid signs for numeric fields
 v99y1 - (default) Micro Focus COBOL -0123456789=pqrstuvwxy x'70' zones
 v99y2 - EBCDIC signs: +0123456789={ABCDEFGHI, -0123456789=}JKLMNOPQR
       - defaults to y2 when input file EBCDIC (option 'a' on command line)
 v99y4 - separate +/- signs (leading or trailing)
 ...z - option 'z' correct blank numeric/packed,& pic x with unprintable chars
 v99z1 - convert any all blank numeric fields to numeric zeros
 v99z2 - convert any all blank packed fields to packed zeros with x'_C' sign
 v99z4 - convert any any unprintable characters in pic x fields to '.' periods
 v99z7 - all of above
v99x7p99
  • verify all records (v99) with option x7 for nostop (count only)
  • option 'p99' prints all error to tmp/filename_yymmdd_hhmmssVP
  • option 'w99' writes all error to tmp/filename_yymmdd_hhmmssVW
  • may follow with lvp/lvw to close files or omit to combine outputs

lw/lp/lvw/lvp separate or combined write/print files

 lw  <-- may follow write command with 'lw' command to close file for separate
         output files or omit for combined outputs from multi write commands
 lp  <-- similar command close print files for separate or omit for combined
 lvw <-- similar command close verify writes for separate or omit for combined
 lvp <-- similar command close verify prints for separate or omit for combined

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

3F1. uvhdcob 'verify' test/demo

create verify report for entire file

Use command --> v99p99x7 <-- to create a verify report for the entire file in the tmp/... subdir for separate investigation & printing. Option p99 writes only the error records into tmp/filename_yymmdd_hhmmssVP. Option x7 inhibits the default stop/display on each error record. You can then quit uvhd & use vi/more/lp/uvlp12/... on the tmp/report. Here is a demo using the dat1/custmas file with the 2 bad records created on page '3D1' & displayed on page '3D2'.


 #1.  uvhdcob dat1/custmas1x maps/custmas1  <-- run uvhdcob with demo files
      ====================================
      --> ........  <-- displays 1st record (display omitted to save space here)
 #1a. --> v99p99x7  <-- enter command 'v99' with options 'p99' & 'x7
      ============
 #1b. --> q         <-- quit uvhdcob

 #2.  more tmp/custmas1x_110101_084059VP   <-- display verify error records
      ==================================
 rec#      10           occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 cm-num                       0    5 n  15468*      31353436382A      <-BadN
 002 cm-delete                    6    9
 003 cm-name                     10   34    FRANKLIN RESOUCES
 004 cm-adrs                     35   59    805 - 1199 HOWE STREET
 005 cm-city                     60   75    VANCOUVER
 006 filler001                   76   76
 007 cm-prov                     77   78    B.          42EE              <-BadC
 008 filler002                   79   79
 009 cm-postal                   80   89    V6C2T6
 010 cm-phone                    90  101    604-666-5489
 011 cm-contact                 102  119    RON BENNETT
 012 cm-thisyr-sales       012  120  124pns EE0123456C                    <-BadP
 013 cm-lastyr-sales       012  180  184pns 000000000C
 014 filler003                  240  255    C 19980331
 rec#      20           occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 cm-num                       0    5 n  2343XX      323334335858      <-BadN
 002 cm-delete                    6    9
 003 cm-name                     10   34    TERRITORIAL REWIND
 004 cm-adrs                     35   59    BOX 1648
 005 cm-city                     60   75    WHITEHORSE
 006 filler001                   76   76
 007 cm-prov                     77   78    YK
 008 filler002                   79   79
 009 cm-postal                   80   89    Y1A...      593141EEEEEE20202 <-BadC
 010 cm-phone                    90  101    403-512-3948
 011 cm-contact                 102  119
 012 cm-thisyr-sales       012  120  124pns 000000000C
 013 cm-lastyr-sales       012  180  184pns 1E2E3E4E5E                    <-BadP
 014 filler003                  240  255      20021130

v99p99x7 --> 32 recs verified, 2 with errs, 1st at rec# 10, last at rec# 20 ERRs: 12 packed, 3 numeric signs/digits, 4 unprintable characters --> v99p99x7 closing: tmp/cm1_110102_093123VP

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_4. Data Investigation, Manipulation, Analysis,& Validation

Part 4 - uvcp,uvsort,uvlist - Contents


4A1. uvcp demo#1 - reformat records, change file type & record-size
4A2. uvcp #2 - extract telephone list from Customer Name&Address file
4A3. uvcp #3 - record selection (keep or drop)
4A4. uvcp #4 - convert EBCDIC RDW variable length records to ASCII Text file
4A5. uvcp #5 - convert ASCII text file to EBCDIC RDW variable length records
4A6. uvcp #6 - create 'BDW/RDW' variable length records

4B1. uvsort demo#1 - sort sales3 demo file by product#
4B2. uvsort demo#2 - sort with Unique-Key option
- keep only the 1st record of duplicate key set
4B3. uvsort demo#3 - 'sum' feature, keep only 1st record of product# group
- with accumulated total $amount

4C1. uvlist - Text file listing utility for scripts, programs, parameter files
- page headings with filename, when printed, userID, line#, Page#
- over 50 convenient scripts for various cpi,lpi,Simplex,Duplex,etc

4C2. sample script 'uvlp12' calling uvlist
- add environmental variable 'UVLPDEST' to your profile
- to direct listings to a network printer near you

a few examples to get you started

Part 4 presents a few brief examples of uvcp, uvsort,& uvlist. These should give you an idea of how useful these utilties may be to you - even better if you can take the time to execute these demos & compare your results to the results listed here.

The complete documentation for these utilties can be found at:

uvcp
uvsort
uvlist

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4A1. uvcp - command line Data File Utility

uvcp demo#1 - extract Name&Address text

'uvcp' is handy for reformatting records & extracting desired text info from large records with packed fields. For example, suppose we wish to extract the Name & Address from the customer master & sales history records. You can not do this with the 'vi' editor because there are no LineFeeds and even if there were the packed fields would distort the screen.


 vi dat1/custmas1          <-- attempt to display with 'vi'
 ================            - won't work, get "line too long"

 uvhd dat1/custmas1 r256   <-- display 1st record with uvhd OK
 =======================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    EVERGREEN MOTORS LTD.    1815 BOWEN ROAD          NANA
             3333332222454545444244545524542222233332445442544422222222224444
             130140000056527255E0DF4F230C44E0000181502F75E02F140000000000E1E1
          64 IMO          BC V9S1H1    250-754-5531 LARRY WRENCH     ..4V|...
             4442222222222442535343222233323332333324455525544442222201357000
             9DF00000000002306931810000250D754D55310C12290725E38000000246C000
         128 .........W0....`........)X}..f3.....\.................4V}...f...
             0000000005300016000000002570063100095000000000000000013570016000
             0C0000C0270D0540C0000C0098D0263C0444C0000C0000C0000C0246D0056C00
         192 .E|...V}.......................f.....<........f.C 19950531
             0470005700000000880000000018000680001300000000694233333333222222
             35C0046D0000C0023C0000C0083C0056D0012C0000C0016D3019950531000000

Name & Address extracted from custmas1

Here are the 1st 3 N&A extracted from the 1st 80 bytes of the customer master and sales history records. See the complete file listed on page '1B2'.

 130140    EVERGREEN MOTORS LTD.    1815 BOWEN ROAD          NANAIMO          BC
 132588    GEECOE GENERATOR SERVICESUNIT 170 - 2851 SIMPSON  RICHMOND         BC
 139923    JOHNSTONE BOILER & TANKS 1250 EAST PENDER STREET  VANCOUVER        BC

uvcp demo#1 - solution


 uvcp "fili1=dat1/custmas1,rcs=256,typ=RSF,filo1=tmp/cm1,rcs=80,typ=RST"
 =======================================================================

 vi tmp/cm1          <-- examine extracted file with 'vi' (no problem)
 ==========
note
  • 'rcs=256' after fili1=... defines input record-size
  • 'rcs=80' after filo1=... defines output rec-size (drops packed fields)
  • 'typ=RST' inserts a LineFeed in last byte (col 80) allowing use of vi
  • LineFeeds are not shown by vi, but could see with uvhd as follows:

 uvhd tmp/cm1 t  <-- option 't' for Text file
 ==============

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4A2. uvcp - command line Data File Utility

uvcp demo#2 - reformat, create telephone list

Suppose we wished to create a telephone list from the customer master file. Here are 1st 3 records in the desired format.

 130140    EVERGREEN MOTORS LTD.       250-754-5531  LARRY WRENCH
 132588    GEECOE GENERATOR SERVICES   604-278-4488  HARRY LIGHT
 139923    JOHNSTONE BOILER & TANKS    604-320-1845  GEORGE BROWN

uvcp demo#2 - telephone list solution

 uvcp "fili1=dat1/custmas1,rcs=256,typ=RSF,filo1=tmp/telelist,rcs=80,typ=RST\
 ,clr=35(45),mvc=38(12)90,mvc=52(20)102"
 ============================================================================

notes

  1. From the custmas1 layout on page '1B2', you can see that the telephone# is in columns 91-102 & that the contact name follows in 102-120.

  2. You can enter long commands on 2 lines as shown above, if you end the 1st line with a backslash '\' & continue on 2nd line leaving no spaces.

  3. 'clr=35(45)' clears the area where we intend to put the tel# & contact name.

  4. 'mvc=38(12)90' moves the tel# & 'mvc=52(20)102' moves the contact name.

  5. Note that uvcp addressing is zero relative, so the byte#s are 1 less than the column#s shown in the layout on page '1B2'.

    4A3. uvcp demo#3 - record selection

Suppose we wanted our telephone list only for the province of 'BC'. We could modify the above command to the following:

 uvcp "fili1=dat1/custmas1,rcs=256,typ=RSF,filo1=tmp/telelistBC,rcs=80,typ=RST\
 ,clr=35(45),mvc=38(12)90,mvc=52(20)102,sel=77(2):BC"
 ==============================================================================
  1. 'sel=77(2):BC' selects records with 'BC' in bytes 77-78 (zero relative).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4A4. uvcp - command line Data File Utility

convert EBCDIC RDW file to a Text file

You can use uvcp to translate EBCDIC to ASCII & convert RDW/BDW variable-length files to any desired format (fixed-length, variable-length, text, Indexed). For example, we will convert EBCDIC RDW file vendormas3E to ASCII text records. 'vendormas3E' is the EBCDIC version of ASCII file 'vendormas3' listed on '1D1'. Here is the 1st record shown with uvhd (note the EBCDIC codes).


 uvhd dat1/vendormas3E z4a  <-- option 'z4' identifies RDW files
 =========================    - option 'a' to translate character line to ASCII
                              - zones & digits lines show the EBCDIC codes
 records=8 rsize=23 fsize=249
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ....DELL10 M  Dell Inc.
             0100CCDDFF4D44C8994C984
             0700453310040045330953B

convert vendormas3E to ASCII Text records


 uvcp "fili1=dat1/vendormas3E,rcs=64,typ=RDWz4,filo1=tmp/vm3A,typ=LSTt,tra=0(64)"
 ================================================================================

 more tmp/vm3A      <-- display output file (all 8 records)
 =============

DELL10 M Dell Inc. HP0010 M Hewlett Packard IBM010 M International Business Machines MFC010 M Micro Focus COBOL MS0010 M Microsoft Corp. REDH10 M Red Hat Linux SUN010 M Sun Microsystems Ltd UVSI10 M UV Software Inc.

'tra=0(64)' translates to ASCII

'typ=LSTt' creates a Text file (inserts a Line-Feed after last non-blank). We can use uvhd to see the Line-Feeds as follows:


 uvhd tmp/vm3A t   <-- option 't' designates a Text file
 ===============
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 DELL10 M  Dell Inc..
             44443324224666246620
             45CC100D0045CC09E3EA
Note
  • the last byte is a LineFeed x'0A' in horizontal-hex
  • shown as a '.' period on character line (unprintables are shown as '.'s)
  • the 2nd last byte is a 'real' period x'2E'

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4A5. uvcp - command line Data File Utility

convert an ASCII Text file to RDW record file

We could also do the opposite of the example above - create RDW EBCDIC records from text records.


 uvcp "fili1=tmp/vm3A,rcs=64,typ=LST,filo1=tmp/vm3E,typ=RDWz4,tre=0(64)"
 =======================================================================
      - convert ASCII text file to EBCDIC RDW file

 uvhd tmp/vm3E z4a     <-- display output file (only 1st record shown below)
 =================
 records=8 rsize=23 fsize=248
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ....DELL10 M  Dell Inc.
             0100CCDDFF4D44C8994C984
             0700453310040045330953B
note
  • 'tre=0(64)' translates output from ASCII to EBCDIC
  • 'D' of Dell10 is x'C4' (EBCDIC) vs x'44' (ASCII, on prior page)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4A6. uvcp - command line Data File Utility

create 'BDW/RDW' variable length records

This is an academic exercise, but should help you understand BDW/RDW files. When uvhd reads BDW/RDW files, it bypasses any BDWs & shows only the RDWs. uvhd cannot normally write BDW files, but for this exercise we will create a file with 1 Block by prefixing our sample 8 record RDW file with the block size. We will use uvcp to create the 4 byte BDW as a separate file & then concatenate the 2 files.


 #1. uvcp "fili1=tmp/vm3A,rcs=64,typ=LST,filo1=tmp/vm3E,typ=RDWz4,tre=0(64)"
     =======================================================================
     - create the RDW file (same as on prior page)

 #2. ls -l tmp        <-- get size of RDW file (8 records)
     =========          - 248 on my tests (+4 for BDW)

 #3. echo xx >tmp/xx  <-- create dummy file for input on uvcp below
     ===============

 #4. uvcp "fili1=tmp/xx,rcs=4,typ=RSF,filo1=tmp/vm3B,mvc=0(4):0x00FC0000"
     ====================================================================
     - create 4 byte BDW (to be prefixed to tmp/vm3E from above)
     - x'00FC' = (15*16 + 12*1) = 252

 #5. cat tmp/xx tmp/vm3E >>tmp/vm3EB
     ===============================
     - concat 4 byte BDW + 248 byte RDW file = 252 byte BDW/RDW file

 #6. uvhd /home/uvadm/tmp/vm3EB z8a  <-- display output file
     ==============================      using option 'z8' for BDW/RDW files
     rec#=1 rsize=23 fptr=4 fsize=252 rcount=8
                      10        20        30        40        50        60
     r#    1 0123456789012345678901234567890123456789012345678901234567890123
           4 ....DELL10 M  Dell Inc.
             0100CCDDFF4D44C8994C984
             0700453310040045330953B
Note
  • uvhd bypasses the BDW, which is why fileptr=4 on 1st record
  • BUT, we can see it if we use just option 'r64' to display 1st 64 bytes

 #6. uvhd tmp/vm3EB r64a  <-- display 1st 64, note BDW x'00FC' = 252
     ===================
                      10        20        30        40        50        60
     r#    1 0123456789012345678901234567890123456789012345678901234567890123
           0 ........DELL10 M  Dell Inc.....HP0010 M  Hewlett Packard....IBM0
             0F000100CCDDFF4D44C8994C9840100CDFFFF4D44C8A98AA4D8898980200CCDF
             0C000700453310040045330953B0D0087001004008563533071321940D009240
note
  • 1st 4 bytes is BDW, 2nd 4 bytes is RDW of 1st record
  • 1st 64 bytes contains 1st 2 recors & part of the 3rd

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4B1. 'uvsort' - Vancouver Sort Utility

The UNIX system sort is inadequate for sorting mainframe type files because:

  1. The UNIX sort cannot sort files without LineFeeds (cannot sort fixed-record-lengths, RDW variable length, Indexed files)

  2. The UNIX sort cannot sort on packed, binary, or unpacked zone signed keys

  3. The UNIX sort lacks mainframe capabilities such as: - record selection/deletion based on specified values in specified bytes - selecting records with unique or duplicate keys

    uvsort demo#1 - sort sales3 by product#

See the input file listed on page '1C3'.


 uvsort "fili1=dat1/sales3,rcs=64,typ=RST,filo1=tmp/s3,key1=30(6)"
 =================================================================

 cat tmp/s3   <-- display the output file
 ==========     - note sorted by product#
 223240    65 980817 IN441     BBQ001  00029+ 0009001 00261029+
 406082    35 020815 IN33001   BBQ001  00017- 0007001 00119017-
 130140    21 940804 IN111001  CHR001  00022+ 0002001 00044022+
 130140    21 940802 IN111001  HAM001  00020+ 0000001 00000020+
 150825    44 960804 IN1122    HAM001  00025+ 0005001 00125025+
 223240    65 980816 CR955     HAM001  00028+ 0008001 00224028+
 315512    44 000805 IN2251    HAM001  00013- 0003001 00039013-
 406082    65 020816 IN441     HAM001  00019- 0009001 00171019-
 150825    44 960805 IN1122    HAX129  00026+ 0006001 00156026+
 400002    85 010812 CR245     HAX129  00016- 0006001 00096016-
 308685    21 990812 IN6605    SAW051  00010- 0000001 00000010-
 315512    44 000806 IN2251    SAW051  00014- 0004001 00056014-
 130140    21 940803 IN111001  SCR012  00021+ 0001001 00021021+
 139923    35 950802 IN111002  TAB013  00023+ 0003001 00069023+
 139923    35 950807 IN111002  TAB013  00024+ 0004001 00096024+
 308685    21 990814 CR8835    TAB013  00012- 0002001 00024012-
 406082    35 020816 IN33001   TAB013  00018- 0008001 00144018-
 201120    44 970807 CR5234    WHIP75  00027+ 0007001 00189027+
 308685    21 990813 IN6605    WHIP75  00011- 0001001 00011011-
 315512    44 000807 IN2255    WHIP75  00015- 0005001 00075015-

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4B2. uvsort - Vancouver Sort Utility

uvsort demo#2 - Unique key option 'u'

Demo#2
  • Sort dat1/sales3 by product# (cols 31-36) with data reduction
  • Data Reduction means outputting only 1 record for each
    unique sort key (product# in this case).
  • option key_u1 selects FIRST record of duplicate set
  • option key_u2 selects LAST record of duplicate set
  • option key_u4 selects NO records of duplicate set
    (retaining only records with no duplicate sort keys)

 uvsort "fili1=dat1/sales3,rcs=64,typ=RST,filo1=tmp/s3,key1u1=30(6)"
 ===================================================================

 cat tmp/s3    <-- display output
 ==========
 223240    65 980817 IN441     BBQ001  00029+ 0009001 00261029+
 130140    21 940804 IN111001  CHR001  00022+ 0002001 00044022+
 130140    21 940802 IN111001  HAM001  00020+ 0000001 00000020+
 150825    44 960805 IN1122    HAX129  00026+ 0006001 00156026+
 308685    21 990812 IN6605    SAW051  00010- 0000001 00000010-
 130140    21 940803 IN111001  SCR012  00021+ 0001001 00021021+
 139923    35 950802 IN111002  TAB013  00023+ 0003001 00069023+
 201120    44 970807 CR5234    WHIP75  00027+ 0007001 00189027+

Notes re Unique Key option

  1. option 'u1' of 'key1u1=30(6)' retains only the 1st record of product# groups

  2. compare this output to the prior page sort without "data reduction"

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4B3. uvsort - Vancouver Sort Utility

uvsort demo#3 - 'sum' feature

Demo#3
  • Sort dat1/sales3 by product# with 'sum' of amount cols 54-62

 uvsort "fili1=dat1/sales3,rcs=64,typ=RST,filo1=tmp/s3,key1=30(6),sum=53(9+)"
 ===========================================================================

 cat tmp/s3    <-- display output, note:
 ==========      - output only 1 record per key (product#)
                 - with $amount total summed in cols 54-62
 223240    65 980817 IN441     BBQ001  00029+ 0009001 00142012+
 130140    21 940804 IN111001  CHR001  00022+ 0002001 00044022+
 130140    21 940802 IN111001  HAM001  00020+ 0000001 00139041+
 150825    44 960805 IN1122    HAX129  00026+ 0006001 00060010+
 308685    21 990812 IN6605    SAW051  00010- 0000001 00056024-
 130140    21 940803 IN111001  SCR012  00021+ 0001001 00021021+
 139923    35 950802 IN111002  TAB013  00023+ 0003001 00002983-
 201120    44 970807 CR5234    WHIP75  00027+ 0007001 00103001+

Notes re 'sum' feature

  1. 'sum=53(9+)' sums the $amount field for each group of 'key1=30(6)' (product#) records and outputs 1 record with the total for the product#

  2. The same data reduction as with the prior option 'u1' of 'key1u1=30(6)".

  3. The '+' of 'sum=53(9+)' is necessary to get the desired +/- sign editing on the output records. The default is to generated zoned signs compatible with Micro Focus COBOL (negatives -0123456789 = pqrstuvwxy).

  4. We can have multiple 'sum' fields by coding as sum1,sum2,etc Try the following:

 uvsort "fili1=dat1/sales3,rcs=64,typ=RST,filo1=tmp/s3,key1=30(6)\
 ,sum1=38(6+),sum2=53(9+)"
 =================================================================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4C1. uvlist - Text file listing utility

'uvlist' will list ascii text files (on stdout by default) but with the main intention of piping to a printer. Several scripts are provided (uvlp12,etc) which include the pipe to the printer, so you can list your text files with a minimum of keystrokes.

uvlist was originally developed to list the documentation files of the Vancouver Utility package. These files use the '*eject' pattern (anywhere in 1st 15 cols) to signal a new page. uvlist has proven to be of substantial assistance to customers as a general purpose text file list utility for programs, scripts, documentation, parameter files, etc.

uvlist ensures that all pages of listings have page headings that identify the file-name, date-printed, userid, line# of 1st line on page,& page#. uvlist saves you the frustration that can occur when you pick up a listing without page headings to identify where it came from & when it was created.

Many options are provided for laser printers, for example you can list 132 column reports on 8 1/2 width paper using the 'uvlp18' script which includes the option to print at 18 characters per inch.

uvlp__ scripts - a few of 50+ provided

uvlp12
  • print at 12 cpi (90 chars on 8 1/2" wide)
uvlp12v6
  • print at 12 cpi & 8 lpi (90 chars across & 80 lines down)
    note v6 means 6/48" which is 8 lpi
uvlp14
  • print at 14 cpi (108 chars on 8 1/2" wide)
uvlp12D
  • print DUPLEX at 12 cpi (90 chars across & 60 lines down)
uvlp14L
  • print LANDSCAPE at 14 cpi (150 chars on 11" wide)
uvlp18
  • print at 18 cpi (134 chars on 8 1/2" wide)
uvlp13LD
  • Landscape Duplex, 13 cpi, 8 lpi
  • 132 cols wide & 66 lines down on 8 1/2 x 11
uvlpd12
  • print all files in a directory at 12 cpi

uvlp__ - command examples


 uvlp12 doc/datautilities.doc  <-- list this documentation
 ============================    - defaults to Simplex

 uvlp12D doc/datautilities.doc  <-- list this documentation Duplex
 =============================    - Duplex (double sided)

 uvlp12D doc/datautilities.doc b51e56 <-- list with options Begin/End page#s
 ====================================   - begin at page 51, End at page 56

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

4C2. Scripts provided for uvlist

sample script for uvlist

 #!/bin/ksh
 # uvlp12 - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # uvlp12 - print a file at 12 cpi (90 chars on 8 1/2 x 11)
 #        - pg hdngs with: filename, mod-date, today-date, page#s
 #        - for HP laserjet 4 printers & compatibles
 #        - by Owen Townsend - UV Software Inc - Dec 1994
 #
 #usage: uvlp12 filename [group1] [group2]  <-- may override default options
 #       =================================
 #
 # - 1 of several: uvlp12,uvlp14,uvlp16,uvlp12L,uvlp14L,uvlp12D,uvlp14LD,etc
 # - these scripts invokes uvlist & pipe to the spooler
 # - scriptnames reflect commonly used uvlist options
 #   group1:  p60  = 60 lines per page
 #   group2:  c12  = 12 cpi = 90 chars across on 8 1/2 " paper
 #            m280 = default margin offset by 280/720 inch (for 3 hole punches)
 # - see uvlist.doc for the many other HP PCL5 compatible print options
 #
 # .profile should specify environmental variables for 'lp', for example:
 # export UVLPDEST="-dlp0"       #<-- destination 'lp0' ("-dLPT1" for SFU)
 # export UVLPOPTN="-onobanner"  #<-- 'nobanner' option for lp
 #
 if [ -f "$1" ]; then :
    else echo "ERROR - $1 is not a file"; exit 1; fi
 #
 uvlist $1 p60$2 t1c12$3 | lp $UVLPOPTN $UVLPDEST
 #===============================================
 #note - option 't1' for tray 1 (main paper source on my Lexmark t642)
 #     - option 't4' for alternate tray (t2 & t3 are manual & envelope)
 #     - OR use option 'a1' for letter size (& remove tray code)
 #       and relate letter size to tray# via printer control panel
 uvln=$(basename $0)
 linesbf=$(wc -l $1); linesb=${linesbf% *}; lines=${linesb##* };
 echo "$uvln printing $1 on $UVLPDEST, lines=$lines"
 exit 0

set UVLPDEST to a printer near you

The common_profile defines a default printer that may not be convenient for you. You can override the default by adding a line to your .profile (or .bash_profile) as follows:


 export UVLPDEST="-dlaserjet2"  <-- define printer near you
 =============================
Note
  • must be no space between the "-d" option & the printer-name "laserjet2"
    (or whatever your printer is called)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_5. FTP mainframe DATA files to unix/linux

Part 5 Contents


5A1. problems FTP'ing mainframe data files to unix
- FTP ASCII mode - corrections required

5A2. which solution to CR/LF ?
- solutions for zoned signs

5B1. FTP data from Mainframe to Unix/Linux

5B2. FTP ASCII with demo files

5C1. investigate files FTP'd from mainframe
- sales0 - EBCDIC recsize=64

5C2. sales1a - ASCII with CR/LF appended

5D1. correcting signs after FTP ASCII mode
- ftp/sales1a - BEFORE sign correction
- ftp/sales1a - AFTER sign correction

5E1. correcting Record-Length after FTP ASCII mode
- ftp/sales1a - BEFORE record-length correction
- data/sales1a - AFTER record-length correction

5E2. Best of Both Worlds
- data/sales1a - AFTER recsize correct & LF insert

5F1. uvcp file 'typ' codes

5G1. using uvhd TRANSLATE sign command (vs uvcp) to correct zoned signs
- uvhd is a FREE download from https://www.uvsoftware.ca/freestuff.htm

5G2. using uvhd WRITE command (vs uvcp) to change record-length drop CR/LF
 "Best of Both Worlds" - retaining original rec-size, so no program changes
- inserting LF in last byte (if unused) of original rec-size,
  allows us to use the various unix tools (vi,lp,cat,more,etc).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part 5 Contents (continued)


5H1. Detecting corrupted files after FTP
- ftp/sales1x - corrupted file

5H2. Use 'uvhd' to detect corrupted file

5H3. Using uvhd with Text file option 't'

5I1. finding wrong-size records in large files
varstat3 - table summarize record sizes

5J1. correcting wrong-size records
- dropping/separating wrong-size records
- console log from splitfile2

5K1. splitfile2 - listing of uvcopy job to select desired rec-size
5K2. varstat3 - listing of uvcopy job to table summarize record-sizes

5L1. EBCDIC to ASCII Data Conversion allowing packed/binary
- these more complex data conversions not covered in detail here
- references given to other documents with the operating instructions

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5A1. FTP mainframe DATA files to unix/linux

problems FTP'ing mainframe data files to unix

Part 5 will illustrate problems that can occur when FTP'ing mainframe files to unix/linux. And we will offer solutions to these problems using Vancouver Utilities (uvhd & uvcp).

You can FTP files in either ASCII/text mode or in BINARY mode. If you are have files with packed decimal or binary fields, you must FTP them in BINARY mode - because the ASCII/text mode EBCDIC to ASCII translation destroys packed/binary fields. Ascii mode also appends a Carriage-Return & Line-Feed making each record 2 bytes longer than on the mainframe.

Migrating mainframe applications to unix/linux will be easier if you can avoid changing record sizes - because you will not have to change the prgrams. If you FTP'd the files in ASCII mode you would have to change the COBOL programs to match - increasing the record length by 2 bytes (for the added CR/LF) & changing file type from the default 'ORGANIZATION SEQUENTIAL' to 'organization LINE sequential' (for Micro Focus COBOL).

But it is not that simple if you have files with packed/binary fields. These files must be FTP'd in BINARY mode & then selectively translate only the character fields, preserving the packed/binary fields which must be the same for both mainframe & Micro Focus COBOL. Vancouver Utilities has tools which use the COBOL copybooks to automatically generate jobs to translate the pic x fields & preserve (copy as is) the comp-3/comp fields.

Please see 'Part_6' if you need to convert files with packed &/or binary fields.

FTP ASCII mode - corrections required

This Part 5 will focus on FTP ASCII mode & compensating for the file changes caused by FTP ASCII. There are 2 changes that will require correction if you are inputting the files to the same application (COBOL?) as on the mainframe that expects the same record layout.

  1. The records are now 2 bytes longer due to the ASCII mode addition of a Carriage-Return & a Line-Feed on each record. So you will either have to remove the CR/LF or change the programs to allow the CR/LF.

  2. Records that have zoned signs in unpacked numeric fields will need correction from the mainframe zoned sign conventions to the Micro Focus COBOL zoned signed conventions (or convert to separate signs leading or trailing).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5A2. FTP mainframe DATA files to unix/linux

which solution to CR/LF ?

  1. Drop the CR/LF to make the file exactly the same length as on the mainframe & that is expected by the (COBOL?) application.

  2. Modify the COBOL programs, increasing the record length by 2 bytes OR change the file type from 'ORGANIZATION SEQUENTIAL' to 'organzation LINE sequential'.

Leaving the CR/LF on the records & modifying the programs to expect LINE Sequential has advantages. You can then use the various unix tools (vi,lp,etc) to view & print the file. Without the LineFeeds you could use 'uvhd' to inspect the files.

solutions for zoned signs

  1. For files with zoned signs in unpacked numeric fields, you need to convert the mainframe zoned signs to the Micro Focus COBOL conventions.

  1. OR you might convert them to separate signs leading or trailing - If your numeric fields can afford to lose 1 byte of length to avoid having to change record lengths & therefore the application programs.

The following pages will illustrate the FTP problems & the various solutions.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5B1. FTP mainframe data files to unix

FTP data from Mainframe to Unix/Linux


 #1. Login --> your homedir
     =====

 #2. cdd   --> $RUNDATA  (RUNDATA=/home/cnvdata1 or cnvdata2, etc)
     ===

 #3. cd data    <-- change into the data subdir
     =======

 #4. ftp xxx.xxx.xxx.xxx   <-- FTP to mainframe IP#
     ===================
 #4a. userid--> .....      <-- enter userid
 #4b. passwd--> .....      <-- enter password
 #4c. ascii                <-- translate EBCDIC to ASCII & insert CR/LF
      ---OR---                 (ascii is usually the default)
 #4c. binary               <-- ensure binary transfer
 #4d. site RDW             <-- required if any variable length files
 #4e. cd ...               <-- change to data files directory ??
 #4f. get 'XXX.XXX.XXX'    <-- get desired data files
 #4g. get '...etc....'       - 'single quotes' may be required

 #5. cdd    <-- change back up to working directory (above data/...)
     ===

On some systems the single quotes are retained & the filename may be UPPER case or lower case depending how we entered the filenames. ON unix we certainly need to remove the single quotes & we may need to change the case. We will show you 2 ways to do this - the 'hard way' & the 'easy' way'.


 #5. mv  \'XXX.XXX.XXX\'  xxx.xxx.xxx   <-- the hard way
     ================================     - see #6. the easy way
     - imagine how awkward this would be if hundreds of files

 #6a. renameQQ data  <-- easy way to remove Quotes from All files in subdir
      =============

 #6b. renameL data   <-- easy way to convert to Lower case (All files in subdir)
      ============

 #6b. renameU data   <-- easy way to convert to UPPER case (All files in subdir)
      ============
Note
  • see Part_6 to convert files with packed/binary fields (must FTP binary)
  • this Part 5 corrects files with no packed/binary (can FTP in ASCII mode)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5B2. FTP mainframe data files to unix

FTP ASCII with demo files

We can illustrate FTP ASCII problems by FTP'ing the supplied EBCDIC demo file dat1/sales1 within our unix/linux machine - using IP# 127.0.0.1 (localhost).

 /home/uvadm
 :-----dat1          <-- supplied test data files
 :     :-----sales0  <-- EBCDIC fixed length 64 byte records (no LineFeeds)
 :     :               - with zoned signs in unpacked numeric fields
 :     :-----sales1a <-- ASCII fixed length 66 byte records (CR/LF appended)
 :     :               - as sales0 would be if FTP'd from mainframe in ASCII mode
 :     :-----sales1x <-- corrupted file (not all same-size records)
                       - to demo varstat3 detect & splitfile2 to separate
 /home/mvstest1        <-- mvstest1 for user self-training
 /home/userxx          <-- OR setup testdata in userhomedir
 :-----testdata
 :     :-----ftp         - receive files FTP'd from $UV/dat1/...
 :     :-----data        - copy with corrections to $HOME/testdata/data
 :     :     :-----sales0,sales1a,sales1x <-- files copied from $UV/dat1/...

 #1. Login --> your homedir
 #2. cdd   --> $RUNDATA  (RUNDATA=/home/cnvdata1 or cnvdata2, etc)

 #3. mkdir ftp        <-- make subdir to receive FTP'd files
                        - see existing subdirs on page '3B2'
 #4. cd ftp           <-- change into subdir to receive FTP'd files

 #5. ftp localhost    <-- FTP within our unix/linux machine
       ---OR---
     ftp 127.0.0.1      - IP# for localhost
     =============
 #5a. userid--> uvadm      <-- to get demo files supplied with VU pkg
 #5b. passwd--> .....      <-- enter password
 #5c. cd dat1              <-- change to demo data files directory
 #5d. ascii                <-- translate EBCDIC to ASCII & insert CR/LF
      binary                 - ascii does not work within a unix system
 #5e. get sales0           <-- get EBCDIC demo file
 #5f. get sales1a          <-- get ASCII version with CR/LF
 #5g. bye

 #6. cdd    <-- change back up to working directory (above data/...)
Note
  • FTP ascii mode is ignored & binary mode forced within unix/linux system
  • so we provide 2 demo files
sales0
  • EBCDIC fixed length 64 byte records (no LineFeeds)
sales1a
  • ASCII fixed length 66 byte records (CR/LF appended)
  • as sales0 would be if FTP'd from mainframe in ASCII mode

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5C1. FTP mainframe data files to unix

investigate files FTP'd from mainframe

After FTP'ing use 'uvhd' to investigate the files & confirm as expected. First determine the record size (default 256 if not sepcified on uvhd command). If omitted uvhd suggests the next lower & next higher recsize multiple that divides evenly into the filesize. Or look for repeating patterns in the data. Or since we FTP'd in ascii mode look for the 1st Linefeed x'0A', quit uvhd,& re-enter the command with correct record-size.

sales0 - EBCDIC recsize=64


 uvhd ftp/sales0 r64a    <-- option r64 for record-size
 ====================      - option 'a' to display char line to ASCII
                             so we can read (zones/digits show EBCDIC)
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    21 940802 IN111001  HAM001  00002{ 0000001 00000002{
             FFFFFF4444FF4FFFFFF4CDFFFFFF44CCDFFF44FFFFFC4FFFFFFF4FFFFFFFFC44
             1301400000210940802095111001008140010000002000000001000000002000
 rec#=1 rsize=64 fptr=0 fsize=1280 rcount=20
             cust#   slm#  date  invoice# product   qty unit-price ext$amount
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
          64 130140    21 940802 IN111001  SCR012  00002A 0001001 00002102A
             FFFFFF4444FF4FFFFFF4CDFFFFFF44ECDFFF44FFFFFC4FFFFFFF4FFFFFFFFC44
             1301400000210940802095111001002390120000002100001001000002102100
 rec#=2 rsize=64 fptr=64 fsize=1280 rcount=20

Notes

  1. We show only 1st 2 records (of 20 in demo file).

  2. You can see the data is EBCDIC by coding of zones & digits lines. For example 1st byte '1' is x'F1' (numeric zone x'F_' unique to EBCDIC).

  3. Note that bytes 43 & 61 (units of qty & $amt) have zoned signs. 1st record qty/amt units is '{' x'C0' which is +0 in EBCDIC. 2nd record qty/amt units is 'A' x'C1' which is +1 in EBCDIC. Records 01-10 units are {ABCDEFGHI for +0123456789 Records 11-20 units are {JKLMNOPQR for -0123456789

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5C2. FTP mainframe data files to unix

sales1a - ASCII with CR/LF appended


 uvhd ftp/sales1a r66  <-- option r66 for record-size
 ====================    - omit option 'a' since FTP translated
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    21 940802 IN111001  HAM001  00002{ 0000001 00000002{
             3333332222332333333244333333224443332233333723333333233333333722
             130140000021094080209E1110010081D0010000002B00000001000000002B00
          64 ..
             00
             DA
 rec#=1 rsize=66 fptr=0 fsize=1320 rcount=20
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
          66 130140    21 940802 IN111001  SCR012  00002A 0001001 00002102A
             3333332222332333333244333333225453332233333423333333233333333422
             130140000021094080209E111001003320120000002100001001000002102100
          64 ..
             00
             DA
 rec#=2 rsize=66 fptr=66 fsize=1320 rcount=20

Notes

  1. Records are now 66 bytes, default segment size is 64, so we see 2 bytes on the 2nd segment of the vertical hexadecimal display. These 2 bytes are the Cariage-Return x'0D' & Line-Feed x'0A'

  2. You can see the data is ASCII by coding of zones & digits lines. For example 1st byte '1' is x'31' (numeric zone x'3_' unique to ASCII). Also note that the blanks (x'40' in EBCDIC) are now x'20' in ASCII.

  3. Note that the sign bytes 43 & 61 (units of qty & $amt) have been translated to the equivalent characters in ASCII - which are INCORRECT as signs for unix/linux Micro Focus COBOL, where positives have no zones & negatives -01234567890 are pqrstuvwxy x'70' zones.

  4. See next page where we will show you how to correct the signs with uvcp - using instruction 'tras' (translate EBCDIC signs to Ascii signs).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5D1. FTP mainframe data files to unix

correcting signs after FTP ASCII mode


 uvcp "fili1=ftp/sales1a,rcs=66,typ=RST,filo1=data/sales1a,tras=43(1),tras=61(1)"
 ================================================================================

ftp/sales1a - BEFORE sign correction

 130140    21 940802 IN111001  HAM001  00002{ 0000001 00000002{
 130140    21 940802 IN111001  SCR012  00002A 0001001 00002102A
 139923    35 950802 IN111002  CHR001  00002B 0002001 00004402B
 139923    35 950802 IN111002  TAB013  00002C 0003001 00006902C
 139923    35 950807 CR5421    TAB013  00002D 0004001 00009602D
 150825    44 960804 IN1122    HAM001  00002E 0005001 00012502E
 150825    44 960804 IN1122    PLN012  00002F 0006001 00015602F
 201120    44 970807 CR5234    WHIP75  00002G 0007001 00018902G
 223240    65 980816 CR955     HAM001  00002H 0008001 00022402H
 223240    65 980816 IN441     BIT001  00002I 0009001 00026102I
 308685    21 990812 IN6605    SAW001  00001} 0000001 00000001}
 308685    21 990812 IN6605    WID115  00001J 0001001 00001101J
 308685    21 990812 CR8835    TAB013  00001K 0002001 00002401K
 315512    44 000805 IN2251    HAM001  00001L 0003001 00003901L
 315512    44 000805 IN2251    SAW051  00001M 0004001 00005601M
 315512    44 000805 IN2255    WID115  00001N 0005001 00007501N
 400002    85 010812 CR245     HAX129  00001O 0006001 00009601O
 406082    35 020815 IN33001   BBQ001  00001P 0007001 00011901P
 406089    35 020815 IN33001   TAB013  00001Q 0008001 00014401Q
 406082    65 020816 IN441     HAM001  00001R 0009001 00017101R

ftp/sales1a - AFTER sign correction

 130140    21 940802 IN111001  HAM001  000020 0000001 000000020
 130140    21 940802 IN111001  SCR012  000021 0001001 000021021
 139923    35 950802 IN111002  CHR001  000022 0002001 000044022
 139923    35 950802 IN111002  TAB013  000023 0003001 000069023
 139923    35 950807 CR5421    TAB013  000024 0004001 000096024
 150825    44 960804 IN1122    HAM001  000025 0005001 000125025
 150825    44 960804 IN1122    PLN012  000026 0006001 000156026
 201120    44 970807 CR5234    WHIP75  000027 0007001 000189027
 223240    65 980816 CR955     HAM001  000028 0008001 000224028
 223240    65 980816 IN441     BIT001  000029 0009001 000261029
 308685    21 990812 IN6605    SAW001  00001p 0000001 00000001p
 308685    21 990812 IN6605    WID115  00001q 0001001 00001101q
 308685    21 990812 CR8835    TAB013  00001r 0002001 00002401r
 315512    44 000805 IN2251    HAM001  00001s 0003001 00003901s
 315512    44 000805 IN2251    SAW051  00001t 0004001 00005601t
 315512    44 000805 IN2255    WID115  00001u 0005001 00007501u
 400002    85 010812 CR245     HAX129  00001v 0006001 00009601v
 406082    35 020815 IN33001   BBQ001  00001w 0007001 00011901w
 406089    35 020815 IN33001   TAB013  00001x 0008001 00014401x
 406082    65 020816 IN441     HAM001  00001y 0009001 00017101y

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5E1. FTP mainframe data files to unix

correcting Record-Length from FTP ASCII mode

If you decide to retain the same record-size (so you do not have to change the programs), you can drop the appended CR/LF using 'uvcp' as shown below using our demo file sales1a which expanded from 64 to 66 by ASCII FTP.


 uvcp "fili1=ftp/sales1a,rcs=66,typ=RSF,filo1=data/sales1a,rcs=64,typ=RSF"
 =========================================================================

ftp/sales1a - BEFORE record-length correction


 uvhd ftp/sales1a r66 <-- option r66 rec-size from ASCII FTP & BEFORE drop CR/LF
 ====================   - will show only 1st record of 20 in our demo file
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    21 940802 IN111001  HAM001  00002{ 0000001 00000002{
             3333332222332333333244333333224443332233333723333333233333333722
             130140000021094080209E1110010081D0010000002B00000001000000002B00
          64 ..
             00
             DA
 rec#=1 rsize=66 fptr=0 fsize=1320 rcount=20

data/sales1a - AFTER record-length correction


 uvhd data/sales1a r64  <-- option r64 for rec-size AFTER dropping CR/LF
 =====================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    21 940802 IN111001  HAM001  00002{ 0000001 00000002{
             3333332222332333333244333333224443332233333723333333233333333722
             130140000021094080209E1110010081D0010000002B00000001000000002B00
 rec#=1 rsize=64 fptr=0 fsize=1280 rcount=20

Notes

  1. Restoring the original record-lengths means you do not have to change the programs migrated from the mainframe to unix/linux.

  2. To focus on Record-Length correction, we have omitted the sign-corrections illustrated on the previous page 'tras=43(1),tras=61(1)'. You can combine sign corrections & record-length corrections as required for your files.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5E2. FTP mainframe data files to unix

Best of Both Worlds

Dropping the CR/LF has a disadvantage - you can not use the various unix tools (vi,lp,cat,more,etc) to view & print the records unless the LF is present (the CR is not required for unix/linux systems).

We can keep the LineFeed & the same Record-Length in this case - because there just happens to be 2 unused blanks at the end of our sales1a demo file. The following uvcp specifies output 'typ=RST' (Record Sequential Terminated) which forces a LineFeed into the last byte of 'rcs=64'.


 uvcp "fili1=ftp/sales1a,rcs=66,typ=RSF,filo1=data/sales1a,rcs=64,typ=RST"
 =========================================================================

data/sales1a - AFTER recsize correct & LF insert


 uvhd data/sales1a r64  <-- option r64 for rec-size after dropping CR/LF
 =====================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    21 940802 IN111001  HAM001  00002{ 0000001 00000002{
             3333332222332333333244333333224443332233333723333333233333333720
             130140000021094080209E1110010081D0010000002B00000001000000002B0A
 rec#=1 rsize=64 fptr=0 fsize=1280 rcount=20

Notes

  1. The only difference from the prior page is that output file type is changed from 'typ=RSF' to 'typ=RST', which forces LF into last byte of rcs=64.

  2. Since the original record-lengths have been maintained you do not have to change the programs migrated from the mainframe to unix/linux.

  3. If desired, you can change COBOL program 'ORGANIZATION RECORD SEQUENTIAL' (default) to 'organization LINE sequential' which will write output files with a LineFeed after the last non-blank to allow you to use the various unix tools (vi,lp,cat,more,etc) to view & print your files.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5F1. FTP mainframe data files to unix

uvcp file 'typ' codes

See all file type codes at uvcp.htm#F1. Here are the more relevant codes:

RSF
  • Record Sequential Fixed (no CR/LF appended)
RST
  • Record Sequential Terminated (LF forced in last byte of recsize stated)
LST
  • Line Sequential Terminated
  • for input files the LF is blanked
  • for output files the LF is appended after the stated record-size
LSTt
  • Line Sequential Terminated with option 't' LF after last non-blank
  • for input files the LF is blanked (same as LST)
  • for output files the LF is appended after the last non-blank
    (record scanned backward to find last non-blank)
ISF
  • Indexed Sequential Fixed
RDWz4

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5G1. FTP mainframe files to unix/linux

using 'uvhd' (vs uvcp) to correct zoned signs

Page '5D1' used 'uvcp' to correct zoned signs from mainframe EBCDIC conventions to unix Micro Focus COBOL conventions (positives {ABCDEFGHI to 0123456789 & negatives }JKLMNOPQR to pqrstuvy).


 uvcp "fili1=ftp/sales1a,rcs=66,typ=RST,filo1=data/sales1a,tras=43(1),tras=61(1)"
 ================================================================================
 - 'uvcp' solution from page '5D1', see 'uvhd' Alternative solution below

If you have not purchased the Vancouver Utilities package, you can do the sign corrections using the 'FREE uvhd' from https://www.uvsoftware.ca/freestuff.htm. Need to enter a separate command for each signed field vs uvcp 1 pass.


 #1. Login mvstest1 or yourself --> your homedir

 #2. cdd --> $HOME/testdata (with subdirs ftp,data at least)

 #3. cp ftp/sales1a data/
     ====================
     - copy FTP'd file to the data/ subdir, since uvhd updates in place
     - vs uvcp which copies the file (as seen above)

 #4. uvhd data/sales1a r66u    <-- option 'r66' for Record-size 66
     ======================      - option 'u' required for Update

 #4a. --> t99s 43(1)    <-- translate mainframe signs to unix Micro Focus COBOL
          ==========      - 't99' Translates ALL records (99 assumes 999999999)
                          - option 's' for Sign correction translate table
                          - '43(1)' identifies the start byte & (length)
                          - the units position of the quantity field

 #4b. --> 1             <-- return to record #1 (in preparation for 2nd field)
          ===

 #4c. --> t99s 61(1)    <-- translate the zoned sign in the units of $amount
          ==========

 #5. head data/sales1a  <-- inspect results
     =================    - see all 20 records Before & After on page '5D1'
                          - but here are 1st 2 & last record Before & After
           1         2         3         4         5         6
 012345678901234567890123456789012345678901234567890123456789012345
 CUST#    SLM  DATE   INV#     PROD#     QTY   PRICE    AMOUNT
 130140    21 940802 IN111001  HAM001  00002{ 0000001 00000002{
 130140    21 940802 IN111001  SCR012  00002A 0001001 00002102A
 406082    65 020816 IN441     HAM001  00001R 0009001 00017101R
    - - - 1st 2 & last records BEFORE above, AFTER below - - -
 130140    21 940802 IN111001  HAM001  000020 0000001 000000020
 130140    21 940802 IN111001  SCR012  000021 0001001 000021021
 406082    65 020816 IN441     HAM001  00001y 0009001 00017101y

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5G2. FTP mainframe files to unix/linux

using 'uvhd' (vs uvcp) to drop CR/LF

Page '5E1' used 'uvcp' to change the record-length - dropping the CR/LF so we would not have to change the programs when migrating from mainframe to unix. You can also change record lengths using the 'FREE uvhd'.

We will use the 'w' (Write) command of uvhd with option 'r' for the desired Record-size. Note that the Write command writes the file out to the tmp/... subdir with a date_time stamp (tmp/sales1a_yymmdd_hhmmssW). You can inspect & if OK - copy back changing the name back to the original name.


 #1. Login mvstest1 or yourself --> your homedir

 #2. cdd --> $HOME/testdata (with subdirs ftp,data at least)

 #3. uvhd data/sales1a r66    <-- option 'r66' for Record-size 66
     =====================

 #3a. --> w99r64    <-- Write all records to tmp/sales1a_yymmdd_hhmmssW
          ======      - 'r64' specifies the new Record-size (dropping CR/LF)

 #4. uvhd tmp/sales1a_yymmdd_hhmmssW r64   <-- inpsect output file
     ===================================

 #5. cp tmp/sales1a_yymmdd_hhmmssW data/sales1a
     ==========================================
     - copy back renaming back to the original name

alternative - inserting LF in last byte original recsize

Here is the "Best of Both Worlds" solution - retaining original record-size, so we don't have to chaneg programs AND inserting a LineFeed in the last byte (if unused) of original rec-size, so we can use unix tools (vi,lp,cat,more,etc).


 #3. uvhd data/sales1a r66    <-- option 'r66' for Record-size 66
     =====================

 #3b. --> w99r63t2   <-- Write all records to tmp/sales1a_yymmdd_hhmmssW
          ========     - option 'r63' specifies new Record-size (dropping CR/LF)
                       - option 't2' appends a LineFeed on rec-size (63+1=64)

 #4. uvhd tmp/sales1a_yymmdd_hhmmssW r64   <-- inpsect output file
     ===================================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    21 940802 IN111001  HAM001  00002{ 0000001 00000002{
             3333332222332333333244333333224443332233333723333333233333333720
             130140000021094080209E1110010081D0010000002B00000001000000002B0A
 rec#=1 rsize=64 fptr=0 fsize=1280 rcount=20
Note
  • the LineFeed x'0A' is now in byte 63 (0 relative) 64 (1 relative)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5H1. FTP mainframe files to unix/linux

Detecting corrupted files from FTP

As mentioned earlier, the 1st thing you should do on receiving files via FTP is to investigate with uvhd to confirm as expected - file type & record size. If fixed-length, all records must be the same size & uvhd can check this.

To illustrate, we have provided 'sales1x' which is similar to the previously listed sales1a, but has 5 short records. Here it is:

ftp/sales1x - corrupted file

 130140    21 940802 IN111001  HAM001  00002{ 0000001 00000002{
 130140    21 940802 IN111001  SCR012  00002A 0001001 00002102A
 139923    35 950802 IN111002  CHR001  00002B 0002001 00004402B
 139923    35 950802 IN111002  TAB013  00002C 0003001 00006902C
 139923    35 950807 CR5421    TAB013  00002D 0004001 00009602D
 150825    44 960804 IN1122    HAM001  00002E 0005001 00012502E
 150825    44 960804 IN1122    PLN012  00002F 0006001 00015602F
 201120    44 970807 CR5234    WHIP75  00002G 0007001 00018902G
 223240    65 980816 CR955     HAM001  00002H 0008001
 223240    65 980816 IN441     BIT001  00002I 0009001 00026102I
 308685    21 990812 IN6605    SAW001  00001} 0000001 00000001}
 308685    21 990812 IN6605    WID115  00001J 0001001 00001101J
 308685    21 990812 CR8835    TAB013  00001K
 315512    44 000805 IN2251    HAM001  00001L 0003001 00003901L
 315512    44 000805 IN2251    SAW051  00001M 0004001
 315512    44 000805 IN2255    WID115  00001N 0005001 00007501N
 400002    85 010812 CR245     HAX129  00001O
 406082    35 020815 IN33001   BBQ001  00001P 0007001 00011901P
 406089    35 020815 IN33001   TAB013  00001Q
 406082    65 020816 IN441     HAM001  00001R 0009001 00017101R

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5H2. FTP mainframe files to unix/linux

use uvhd to detect corrupted file

Here are the results of using uvhd on corrupted file ftp/sales1x, which has 5 records shorter than the expected 66 (see listing on previous page).


 uvhd ftp/sales1x r66
 ====================
 records=18 rsize=66 fsize=1236 fsize/rsize(rmndr)=48
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    21 940802 IN111001  HAM001  00002{ 0000001 00000002{
             3333332222332333333244333333224443332233333723333333233333333722
             130140000021094080209E1110010081D0010000002B00000001000000002B00
          64 ..
             00
             DA
 filesize NOT multiple of recsize, 48 bytes remain
 rec#=1 rsize=66 fptr=0 fsize=1236 rcount=18
 If Text file (with LineFeeds),  quit/restart with option 't'
 If Variable (IDXFORMAT3/8,RDW), quit/restart with optn v,or z
 If Fixed length records, change recsize now via command 'R' (R66 current)
 next evenly divisible record sizes lower/higher are --> R12/R103
 OR you may continue (ignoring these warnings) -->
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
          66 130140    21 940802 IN111001  SCR012  00002A 0001001 00002102A
             3333332222332333333244333333225453332233333423333333233333333422
             130140000021094080209E111001003320120000002100001001000002102100
          64 ..
             00
             DA
 filesize NOT multiple of recsize, 48 bytes remain
 rec#=2 rsize=66 fptr=66 fsize=1236 rcount=18

Notes

  1. uvhd displays error message "filesize NOT multiple of recsize" & sugegsts the next lower & higher sizes evenly divisible into the file size. For non-corrupted files (all records same size), you can restart with alternate record sizes until you get the right 1 (no error messages).

  2. BUT, that will not help us here when the file has corrupted records (not all same size as expected for fixed-length files).

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5H3. FTP mainframe files to unix/linux

using uvhd with Text file option 't'

If you are getting "filesize not multiple of recsize" & you can see LineFeeds x'0A' in the data, then quit & change your fixed size option 'r' to option 't' for Text files (record length detrmined by LineFeed at end of each record).


 uvhd ftp/sales1x t   <-- prior page options 'r66' changed to 't'
 ==================     - option 't' determines recor-size by LineFeeds x'0A'
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    21 940802 IN111001  HAM001  00002{ 0000001 00000002{
             3333332222332333333244333333224443332233333723333333233333333722
             130140000021094080209E1110010081D0010000002B00000001000000002B00
          64 ..
             00
             DA
 rec#=1 rsize=66 fptr=0 fsize=1236 rcount=20
                      10        20        30        40        50        60
 r#        2 0123456789012345678901234567890123456789012345678901234567890123
          66 130140    21 940802 IN111001  SCR012  00002A 0001001 00002102A
             3333332222332333333244333333225453332233333423333333233333333422
             130140000021094080209E111001003320120000002100001001000002102100
          64 ..
             00
             DA
 rec#=2 rsize=66 fptr=66 fsize=1236 rcount=20

Notes

  1. Changing fixed size option 'r66' to option 't' avoids the error messages.

  2. It also avoids getting "off-track" when you browse down the file & encounter the different sized records.

  3. You can still jump to any record# in the file, because option t builds an indexed to know where the start of every record is.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5I1. FTP mainframe files to unix/linux

finding wrong-size records in large files

You might run statistics job 'varstat3' on corrupted files to see how many different record-sizes you have & where the 1st record is of each different size.


 #1. Login mvstest1 or yourself --> your homedir

 #2. cdd --> $HOME/testdata (with subdirs ftp,data at least)

 #3. mkdir stats     <-- make subdir to receive stats report

 #4. uvcopy varstat3,fili1=ftp/sales1x,filo1=stats/sales1x
     =====================================================
     - create record-sizes report in stats/sales1x

 #4a. uvcopy varstat3,fili1=ftp/sales1x  <-- Alternative shorter command
      =================================    - output filename defaults as above

 #5. vi stats/sales1x    <-- inspect report
     ================
 varstat3  2011/11/09_17:01:11  record-sizes in dat1/sales1x
 tbl#0001  tblt1f1 c0(5)
 line#     count  %    1strec#  record-sizes
     1         3  15        13  00045
     2         2  10         9  00053
     3        15  75         1  00065
              20*100             *TOTAL*
  1. You can see there are 3 different record-sizes (45, 53,& 65). Ending LineFeeds not counted in these sizes else they would be 46,54,66.

  2. The '1strec#' of 45 byte records occurs at record# 13 in the file, the 1strec# of 53 byte records is at record# 9 & 1strec of 65 byte records is at record #1.

  3. The report tells you that 3 of the 20 records in the file are 45 bytes long, 2 are 53 bytes,& 15 are 65 bytes (correct).

  4. If you are interested, you can inspect 'varstat3' as shown below: It is 1 of the 800+ pre-programmed jobs supplied with Vancouver Utilities.


      vi $UV/pf/IBM/varstat3
      ======================
  1. uvcopy instruction 'tbl' builds the table you see above, capable of tabling thousands to arguments encountered in any seqeunce & dumping the table in sorted sequence at End Of File.

  2. See listing of uvcopy job varstat3 on page '5K2'

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5J1. FTP mainframe files to unix/linux

correcting wrong-size records

You might want to correct the wrong size records, if the application programs must have all same sized records, vs changing program files to 'ORGANIZATION LINE SEQUENTIAL", which would read the file correctly.

We can use 'uvcp' with 'rcs=66', input 'typ=LST' & output 'typ=RST' to force all output recors to 66 bytes with the LF in the last byte.


 uvcp "fili1=ftp/sales1x,rcs=66,typ=LST,filo1=data/sales1x,typ=RST"
 ==================================================================

dropping/separating wrong-size records

If you wanted to drop the wrong-size records, you could run the 'splitfile2' uvcopy job, which reads the file & outputs 2 files - 1st with specified record-size, 2nd with all other-sized records (in case you want to inspect).


 uvcopy splitfile2,fili1=ftp/sales1x,uop=r65
 ===========================================

console log from splitfile2

 uop=q1r220  - option defaults
       r220  - select data recsize 220 (+1 LF total 221)
       r2048 - specify any desired record-size, max 2048
 111109:175930:splitfile2: uvcopy ver=20111103 pf=/home/uvadm/pf/demo/splitfile2
 uvcopy LNX L64  license=111103_99V_930630 site=UV_Software
 User OPtion (uop) defaults  = q1r220r65
 -->null to accept or enter/override -->
 111109:175931:splitfile2: EOF fili01 rds=20 size=1236: ftp/sales1x
 111109:175931:splitfile2: EOF filo01 wrts=15 size=990: data/sales1x
 111109:175931:splitfile2: EOF filo02 wrts=5 size=241: tmp/sales1xx

Notes

  1. You can see that the record counts for the 2 files agree with the varstat3 report (shown on the previous page)

  2. In case you are interested, the uvcopy job is listed on the next page. It is 1 of the 800+ pre-programmed jobs supplied with Vancouver Utilities. The uvcopy interpreter finds its jobs (parameter files) via $PFPATH which is defined in the supplied profiles.


 export PFPATH=$PFPATH,$UV/pf/adm,$UV/pf/demo,$UV/pf/util,$UV/pf/IBM
 ===================================================================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5K1. FTP mainframe files to unix/linux

splitfile2 - uvcopy job to select desired rec-size

 # splitfile2 - uvcopy job stored at $UV/pf/demo/splitfile2 (in $PFPATH)
 # splitfile2 - split file of text records that should all be 1 record-size
 #            - write 2 output files, 1st file all specified recsize
 #              2nd file for all other recsizes
 #            - a more flexible general purpose job than 'splitfile1'
 #              which hard-coded 1 particular record size
 #
 #              ** sample Op. Instrns. for splitfile2 **
 #
 # uvcopy splitfile2,fili1=d1/filex,filo1=d2/filexok,filo2=d2/filexbad,uop=r220
 # ============================================================================
 # - note 'uop=r220' option to select data recsize 220 (+1 for LF total 221)
 # - may specify I/O filenames on comamnd line (as above)
 #
 # uvcopy splitfile2
 # =================
 # - may omit fili1=...,filo1=...,filo2=... filenames & uop=...
 # - uvcopy will show defaults coded & prompt to enter newnames/new rec-size
 #
 opr='uop=q1r220  - option defaults'
 opr='      r220  - select data recsize 220 (+1 LF total 221)'
 opr='      r2048 - specify any desired record-size, max 2048'
 uop=q1r220   # option defaults (actual vs above opr msgs)
 #            # note option 'r' coded as $uopbr on cmn instruction below
 fili1=?dat1/filex,typ=LST,rcs=2048      # recsize by LF max 2048
 filo1=?dat2/filexok,typ=RST,rcs=2048    # fixed recsize option r, LF last byte
 filo2=?dat2/filexbad,typ=LSTt,rcs=2048  #option t of LSTt drops trailing blanks
 @run
        opn    all
 # begin loop to get/put records until EOF
 # note - get stores recsize in register 'v' ($rv)
 loop   get    fili1,a0       fili1 typ=LST rcs=2048 allows recsize 2048 max
        skp>   eof
        cmn    $rv,$uopbr     recsize = option 'r' ?
        skp!   put2
        add    $rv,1              +1 for LineFeed
 put1   put    filo1,a0($rv2048)  filo1 typ=RST inserts LF last byte option size
        skp    loop
 put2   put    filo2,a0       filo2 typ=LST last nonblank determines recsize
        skp    loop
 #
 # EOF - close files & end job
 eof    cls    all
        eoj

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5K2. FTP mainframe files to unix/linux

varstat3 - uvcopy job to table summarize record-sizes

 # varstat3 - create summary table of record-sizes used in text file
 #          - by Owen Townsend, Oct 19/2011
 #
 # uvcopy varstat3,fili1=data/filename,filo2=stats/filename
 # ==========================================================
 #
 #                  ** sample report **
 #
 # RDWstat1  2006/12/17_18:15:17  record-sizes in d0ebc/E2121656
 # tbl#001 pg#001     -argument-
 # line#  count    %  record-sizes
 #     1  10,552  16  00046
 #     2   4,451   7  00065
 #     3  23,347  37  00066
 #     4     367   0  00068
 #     5  21,010  33  00083
 #       - - - etc - - -
 #    18       3   0  00218
 #    19     441   0  00233
 #    20     813   1  00239
 #        62,115*100   *TOTAL*
 #
 opr='$jobname - create summary table of record-sizes used in text files'
 rop=r1        # option to prompt for report disposition
 fili1=?data/input,rcs=2048,typ=LST
 filo1=?stats/$fili1,typ=LSTt,rcs=128
 @run
        opn     all
 #
 # begin loop to get records & build summary table of recsizes
 # - to be dumped at end of file
 man20  get     fili1,a0                 get next RDW record data
        skp>    eof                      (cc set > at EOF)
        mvn     c0(5),$rv                cnvrt binary recsize to numeric
        tblt1f1 c0(5),'record-sizes'     build table in memory
        skp     man20                    repeat get loop til EOF
 #
 # EOF - close files & end job
 eof    mvfv1   f0(80),'record-sizes in $fili1'
        tbpt1s1 filo1,f0(50)             dump table from memory to file
        cls     all
        eoj

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5L1. DATA conversions for VU JCL demos

EBCDIC to ASCII Data Conversion allowing packed/binary

See DATAcnv1.htm#Part_3 for the easiest way to convert data files with packed/binary fields. Script 'gencnvD2' generates conversion jobs, automatically from the COBOL copybooks. The following uvcopy jobs & SQL scripts are generated:

  1. EBCDIC to ASCII conversion, retaining the same record layout, so COBOL programs can execute unchanged on the unix/linux systems. Conversion jobs are generated from the COBOL copybooks so pic x fields can be translated from EBCDIC to ASCII, but any packed/binary fields are preserved (would be destroyed if entire record were translated). Also note that any unpacked numeric fields with zoned signs are transalted from the mainframe EBCDIC conventions to Micro Focus ASCII conventions (positives {ABCDEFGHI to 0123456789 & negatives }JKLMNOPQR to pqrstuvy).

  2. 'uvcopy jobs' are also generated to convert data files to '|' pipe delimited all text files for loading database tables. Packed/binary fields are unpacked & edited with separate signs & decimal points suitable for loading database tables.

  3. SQL scripts are also generated to CREATE the database tables.

  4. SQL scripts are also generated to LOAD the database tables.

Script gencnvD2 generates all 4 types of conversion jobs & scripts. You might initially need only the 1st (EBCDIC to ASCII retaining same layout) if you wish to migrate quickly to unix/linux with the fewest changes required. Later on you might use the jobs/scripts 2,3,4 to convert flat files to database tables.

The data file conversions documented in DATAcnv1.htm#Part_3 are OK for most mainframe files, but do not allow for more complex files used at some mainframe sites. UV Software does provide solutions for those complex files, which are documented in other manuals.

Below is a comparison of datafile conversion capabilities covered in various UV Software documents:

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

5L2. DATA conversions for VU JCL demos

Conversions Documented in DATAcnv1.htm

  1. Simplest & easiest, but adequate for most mainframe data files, EBCDIC to ASCII preserving packed/binary fields.

  2. Complex conversions (such as Multi Record Types & variable length records) are documented in DATAcnv1.htm#Part_5 & DATAcnv1.htm#Part_8.

  3. Simple conversion procedures are made possible by using the copybook name for all generated jobs & by changing the names of the data files to match the copybook names. We can do this since separate subdirs are provided for the various generated jobs & data file types (separate subdirs for EBCDIC,ACII,pipe-delimited).

  4. Original EBCDIC data files are renamed to match the copybook when they are copied into the designated conversion subdir $CNVDATA/d1ebc. After conversion, they are renamed as desired when copied from $CNVDATA/d2asc to $RUNDATA/data for use by the JCL/scripts.

  5. Conversions documented in DATAcnv1.htm#Part_3 are for 1 file at a time. See below for other methods to handle hundreds or thousands of data files.

    more complex EBCDIC to ASCII conversions

DATAcnv1.htm & JCLcnv2real.htm#Part_4 cover the more complex data file conversions (such as multi Record Types & variable length records).

You can retain original data file names (without manual renaming) by creating a control file to relate the copybook-name to the various data-file names with that layout. The conversion jobs are also named from the datafile names (vs copybook names, because there could be multiple datafiles per copybook).

All data files may be converted with 1 command, which is important for the several reconversions requried during testing & the final 'go live' day.

converting data files to database tables

See SQLconvert.htm for more complete documentation on converting data files to pipe|delimited all text files suitable for loading database tables. Packed/binary fields will be unpacked & edited with decimal points & separate signs suitable for loading database tables. We also generate scripts to create & load database tables. Utility 'sqlcreate1' generates scripts for Oracle. Utility 'sqlcreate2' generates scripts for Microsoft SQL Server.

unpacking all packed/binary fields

Or you might want to unpack all packed/binary fields & retain your COBOL batch programs. See DATAcnv1.htm#4I1 thru 4I4 which documents this procedure. You make up a 2nd copybook for each file removing all 'comp-3's & 'comp's. Then run script 'genrfmA' to generate the data conversion jobs automatically. This conversion could be easy if you have used copybooks for all your data files, but obviously more difficult if you have embedded record layouts in your COBOL programs.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_6. Data Investigation, Manipulation, Analysis,& Validation

Part 6 - general purpose Table Summaries - contents


6A1. table2 pre-programmed uvcopy job for table summaries
- table2 demo#1, summarize sales qty & amount by product

6A2. table2 demo - command, prompts, & responses

6B1. table2 demo#2 - customer counts by province

6C1. custom written uvcopy table summaries
cmtbl2 - sample uvcopy job to demo table summaries
- table summarize this-year & last-year sales by city & province

6C2. cmtbl2 - listing of uvcopy instructions

6D1. Concatenating fields for table arguments
cmtbl2a - alternate uvcopy job to demo tabling by province+cityname

6D2. cmtbl2a - listing of uvcopy instructions

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6A1. General Purpose Table Summaries

table2 - pre-programmed uvcopy job for table summaries

'table2' is a pre-programmed uvcopy job to count records & accumulate 1 or 2 numeric fields by any desired argument field. We will demonstrate using the 'sales3' file listed on page '1C3'.

table2 sample report

 table2  2011/05/22_18:19:39  summary of qty & amount by product#
 tbl#0001  tblo1t1 n0($rn32)                   -acum#1-   %        -acum#2-    %
 line#     count  %    1strec#  product#           qty               amount
     1         2  10        10  BBQ001              12   12        1,420.12   31
     2         1   5         3  CHR001              22   22          440.22    9
     3         5  25         1  HAM001              41   41        1,390.41   30
     4         2  10         7  HAX129              10   10          600.10   13
     5         2  10        11  SAW051              24-  24-         560.24-  12-
     6         1   5         2  SCR012              21   21          210.21    4
     7         4  20         4  TAB013              17   17           29.83-   0
     8         3  15         8  WHIP75               1    1        1,030.01   22
              20*100             *TOTAL*           100 *100        4,501.00 *100

Here is the command to call the table2 'uvcopy job' directly, BUT DO NOT use this, RATHER use the table2 'script' which is shorter & easier.


 uvcopy table2,fili1=dat1/sales3,filo1=tmp/sales3_tbl,uop=a30b6c38d6e53f9g3r64
 =============================================================================
 table2 dat1/sales3 RSF a30b6c38d6e53f9g3r64r64  <-- use the table2 'script'
 ==============================================      (shorter than 'uvcopy job')

option prompts

 uop=a0b0c0d0e0f0g0n0p0r0 - option defaults
 uop=a0                 - argument displacement (0 relative)
 uop=  b0               - argument length
 uop=    c0             - acum#1 dsplcmnt (omit c,d,e,f for counts only)
 uop=      d0           - acum#1 length
 uop=        e0         - acum#2 displacement (0 relative)
 uop=          f0       - acum#2 length
 uop=            g1-12  - table formats g1-g12 see uvcopy.doc tbf f1-f12
 uop=            g3     - edit 1st acum qty, acums 2-6 dollars&cents
 uop=        p0         - acum-fields p0=unpacked, p1=packed, p2=binary
 uop=          r8192    - input record size (max 8192)
 uop=a30b6c38d6e53f9g3r64  <-- ENTER this for demo with dat1/sales3 page '1C3'
a30b6
  • product# starts at byte 30, 6 bytes long
c38d6
  • quantity starts at byte 38, 6 bytes long
e53f9
  • amount starts at byte 53, 9 bytes long
g3
  • table format - 1st acum qty, 2nd acum dollars & cents
r64
  • record size
          1         2         3         4         5         6
 1234567890123456789012345678901234567890123456789012345678901234
 CUST#    SLM  DATE   INV#     PROD#     QTY   PRICE    AMOUNT
 130140    21 940802 IN111001  HAM001  00020+ 0000001 00000020+

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6A2. General Purpose Table Summaries

table2 demo - prompts & responses

The table2 demo on the previous page omitted some of the prompts & responses. Here is a rerun, empahsizing the various prompts & responses. Note that the resonses are ==== underlined.


 table2 dat1/sales3 RSF a30b6c38d6e53f9g3r64r64  <-- table2 script
 ==============================================
 - file types: RSF(Fixed,default), LST(Text), RDWz4,RDWz2,RDWz8, ISF,IDXf8
 - output report (filo1) will be: tmp/sales3_tbl_product
 uop=a0b0c0d0e0f0g0n0p0r0 - option defaults
 uop=a0                 - argument displacement (0 relative)
 uop=  b0               - argument length
 uop=    c0             - acum#1 dsplcmnt (omit c,d,e,f for counts only)
 uop=      d0           - acum#1 length
 uop=        e0         - acum#2 displacement (0 relative)
 uop=          f0       - acum#2 length
 uop=            g1     - edit all acums as qty (commas, no decimal)
 uop=            g2     - edit all acums as $ (commas & decimal point)
 uop=            g__    - table formats g1-g12 see uvcopy.doc tbf f1-f12
 uop=        n0         - argument n0=unpacked, n1=packed, n2=binary
 uop=        p0         - acum-fields p0=unpacked, p1=packed, p2=binary
 uop=          r8192    - input record size (max 8192)
 uop=              note - if file type LST specify r > largest possible
 uop=a30b6c38d6e53f9g3r64  <-- ENTER this for demo with dat1/sales3 file
 product#;quantity;amount  <-- ENTER this for demo Field Headings
 Summary of Qty & Amt by Product#  <-- ENTER this for demo Report Heading
 User OPtion (uop) defaults  = q1a0b0c0d0e0f0g0n0p0r0a30b6c38d6e53f9g3r64
 -->null accept or override -->    <-- null response (entered on command line)
 enter table header fieldnames separated by semicolons
 argument;acum1;acum2   <-- enter fieldnames
 --> product;qty;amount     <-- entry for sales3 demo
     ================================================
 enter table heading (optional, prints above fieldnames)
 --> Qty & Amount by Product#  <-- entry for sales3 demo
     ========================
 110523:101907:table2: EOF fili01 rds=20 size=1280: dat1/sales3
 110523:101907:table2: EOF filo01 wrts=8 size=1040: tmp/sales3_tbl_product
 EOJ, Output File written to: tmp/sales3_tbl_product
 default command = more, OR enter: more,print,uvlpr12,edit,null -->
 --> more   <-- entry for sales3 demo (display report on screen)
     ===========================================================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6B1. General Purpose Table Summaries

table2 demo#2 - customer counts by province

Here is another table2 demo using the customer master file, simply counting records (customers) by province. See customer master file listed on page '1B2' & note province is in bytes 77-78 (zero relative) and the record size is 256.


 table2 dat1/custmas1 RSF a77b2r256
 ==================================
 uop=a0b0c0d0e0f0g0n0p0r0 - option defaults
 uop=a0                 - argument displacement (0 relative)
 uop=  b0               - argument length
 uop=    c0             - acum#1 dsplcmnt (omit c,d,e,f for counts only)
 uop=      d0           - acum#1 length
 uop=        e0         - acum#2 displacement (0 relative)
 uop=          f0       - acum#2 length
 uop=            g1     - edit all acums as qty (commas, no decimal)
 uop=            g2     - edit all acums as $ (commas & decimal point)
 uop=            g__    - table formats g1-g12 see uvcopy.doc tbf f1-f12
 uop=        n0         - argument n0=unpacked, n1=packed, n2=binary
 uop=        p0         - acum-fields p0=unpacked, p1=packed, p2=binary
 uop=          r8192    - input record size (max 8192)
 uop=              note - if file type LST specify r > largest possible
 User OPtion (uop) defaults  = q1a0b0c0d0e0f0g0n0p0r0a77b2r256
 -->null accept or override -->    <-- null response (entered on command line)
 enter table header fieldnames separated by semicolons
 argument;acum1;acum2    <-- enter fieldnames
 --> province            <-- entry for custmas1 demo
     ================================================
 enter table heading (optional, prints above fieldnames)
 --> customer counts by province  <-- entry for custmas1 demo
     ===========================
 EOJ, Output File written to: tmp/custmas1_tbl_province
 default command = more, OR enter: more,print,uvlpr12,edit,null -->
 --> more   <-- entry for custmas1 demo (display report on screen)
     =============================================================

table2 demo#2 - report output

 table2  2011/05/23_11:00:58  customer counts by province
 tbl#0001  tblo1t1 n0($rn32)                  -acum#1-    %        -acum#2-    %
 line#     count  %    1strec#  province
     1         3   9        15  AB
     2        10  31        13  AL
     3        16  49         1  BC
     4         3   9        20  YK
              32*100             *TOTAL*                  *                    *

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6C1. General Purpose Table Summaries

custom written uvcopy table summaries

The previous pages used 'table2' a pre-programmed uvcopy job to create 1 table summary report with 1 or 2 accumulators by any 1 argument field. Writing your own uvcopy job would allow you to use up to 6 accumulators in each table and have up to 255 tables in 1 job (1 pass thru the file).

'cmtbl2' is an example of a customer written uvcopy job that reads the customer master file (listed on page '1B2') and builds 2 tables to summarize this-year & last-year sales by city & province. Note that this-year & last-year sales must first be cross-footed (instruction 'xft') from the 12 monthly sales fields before being tabled (instruction 'tbl').

Here is the 'uvcopy' command to call the 'cmtbl2' instructions. uvcopy finds its jobs via PFPATH which includes /home/uvadm/pf/demo/... where cmtbl2 is stored. See the 'cmtbl2' job listed after the sample report below.


 uvcopy cmtbl2    <-- execute 'uvcopy' to interpret instructions in 'cmtbl2'
 =============

cmtbl2 sample report

 cmtbl2  2011/05/23_12:44:07  sales by city name (this yr & last yr)
 tbl#0001  tblt1f2 a60(16)                    -acum#1-    %        -acum#2-    %
 line#   count  %  1strec#  city name     thisyr sales         lastyr sales
     1       3   9      23  CALGARY             894.56    0          847.27    1
     2       2   6       6  CHILLIWACK        4,530.46    4        9,218.36   20
     3       2   6       9  COQUITLAM         1,844.48    1
     4       1   3      31  DAWSON CITY          66.01    0        4,432.47    9
     5       3   9      17  EDMONTON         20,108.35   20       27,919.97   61
     6       2   6      16  FORT MCMURRAY     1,000.00-   1-         849.88    1
     7       2   6      13  GRANDE PRAIRIE    1,546.42    1        2,192.64    4
     8       2   6       4  HOPE             10,837.28   10          173.25
     9       1   3       5  KELOWNA                .00    0
    10       1   3       1  NANAIMO          21,335.56   21       12,152.55-  26-
    11       1   3      11  NORTH VANCOUVER        .00    0
    12       2   6      28  PRINCE GEORGE     1,781.75    1        4,395.21    9
    13       1   3       2  RICHMOND            434.98    0          640.04    1
    14       3   9      15  THREE HILLS       6,048.13    6        3,276.65    7
    15       4  12       3  VANCOUVER        24,112.72   24        1,220.16    2
    16       2   6      20  WHITEHORSE        7,157.23    7        2,427.49    5
            32*100           *TOTAL*         99,697.93 *100       45,440.84 *100
 cmtbl2  2011/05/23_12:44:07  sales by province (this yr & last yr)
 tbl#0002  tblt2f2 a77(2)                     -acum#1-    %        -acum#2-    %
 line#   count  %  1strec#  province      thisyr sales         lastyr sales
     1       3   9      15  AB                  323.13    0        1,534.06    3
     2      10  31      13  AL               27,274.33   27       33,552.35   73
     3      16  50       1  BC               64,877.23   65        3,494.47    7
     4       3   9      20  YK                7,223.24    7        6,859.96   15
            32*100           *TOTAL*         99,697.93 *100       45,440.84 *100

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6C2. General Purpose Table Summaries

cmtbl2 - sample uvcopy job to demo table summaries

 # cmtbl2 - table analysis of customer master sales history file
 #
 # uvcopy cmtbl2  <-- execute uvcopy to interpret this cmtbl2 parameter file
 # =============    - files default as coded below on fili1=... & filo1=...
 #
 opr='$jobname - summarize sales (thisyr&lastyr) by city name & province'
 rop=r1   # prompt oprtr for report disposition at EOJ
 fili1=?dat1/custmas1,rcs=256,typ=RSF
 filo1=?tmp/$fili1.tbl,rcs=80,typ=LSTt
 @run
        opn    all
 # begin loop to read all customer master sales history records
 # - crossfooting & accumulating (tabling) thisyr & lastyr totals
 loop   get    fili1,a0(256)
        skp>   eof
        xft    $ca1,a120(5p),12
        xft    $ca2,a180(5p),12
        tblt1f2 a60(16),'city name;thisyr sales;lastyr sales',$ca1,$ca2
        tblt2f2 a77(2),'province;thisyr sales;lastyr sales',$ca1,$ca2
        skp    loop
 # end of file - print/edit the tables to a file for: lp, cat,or vi
 eof    tbpt1s2 filo1,'sales by city name (this yr & last yr)'
        tbpt2s1 filo1,'sales by province (this yr & last yr)'
        cls    all
        eoj

notes re uvcopy instructions

See all uvcopy instructions documented at uvcopy3.htm. uvcopy instructions are all 3 characters, anything after that are options. Here are a few notes on the more vital instructions in the cmtbl2 job.

get
  • gets the next record from fili1 into area 'a'
  • starting at 1st byte 'a0' for length of (256)
xft
  • cross-foot instruction
xftt1
  • option 't1' for table#1
xftt1f2
  • option 'f2' for table format#2 (qty edit acum1, $edit acums 2-6)
  • op1 a60(16) identifies the table argument (city)
  • op2 'city;thisyr sales...' defines field headings
  • op3,4,5,6,7,8 define up to 6 accumulators
tbp
  • table print (writes memory table to output file at EOF)
tbpt1s2
  • options for table# & space between tables
    op1 filo1 declares output file (defined prior to instructions)
    op2 'sales by city ...' defines an optional extra table heading

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6D1. General Purpose Table Summaries

concatenating fields for table arguments

You might want to concatenate record fields for table arguments. For example you might want to concatenate province + city-name for 1 table argument, so the table would print in sequence by city within province.


 uvcopy cmtbl2a   <-- alternate version for table arg concat province+cityname
 ==============
 cmtbl2a  2011/05/28_15:24:39  sales by province+cityname (this yr & last yr)
 tbl#0001  tblt1f5 c0(20)
 line#     count  %    1strec#  province+cityname         thisyr sales    %
     1         1   3        16  AB FORT MCMURRAY                   .00    0
     2         2   6        15  AB THREE HILLS                  323.13    0
     3         3   9        23  AL CALGARY                      894.56    0
     4         3   9        17  AL EDMONTON                  20,108.35   20
     5         1   3        21  AL FORT MCMURRAY              1,000.00-   1-
     6         2   6        13  AL GRANDE PRAIRIE             1,546.42    1
     7         1   3        19  AL THREE HILLS                5,725.00    5
     8         2   6         6  BC CHILLIWACK                 4,530.46    4
     9         2   6         9  BC COQUITLAM                  1,844.48    1
    10         2   6         4  BC HOPE                      10,837.28   10
    11         1   3         5  BC KELOWNA                         .00    0
    12         1   3         1  BC NANAIMO                   21,335.56   21
    13         1   3        11  BC NORTH VANCOUVER                 .00    0
    14         2   6        28  BC PRINCE GEORGE              1,781.75    1
    15         1   3         2  BC RICHMOND                     434.98    0
    16         4  12         3  BC VANCOUVER                 24,112.72   24
    17         1   3        31  YK DAWSON CITY                   66.01    0
    18         2   6        20  YK WHITEHORSE                 7,157.23    7
              32*100             *TOTAL*                     99,697.93 *100

notes re concat fields for tbl args

  1. Compare this cmtbl2a report to the report shown previously on page '6C1'. We dropped the separate province table.

  2. The table argument is now "province+cityname"

  3. To fit table lines within 80 columns we have dropped 'last year' acums.

  4. We changed table format from 'f2' (16 columns max) to 'f5' (32 columns max) - because our table argument is now 2+1+16=19 (prov + space + city). Note change from 'tblt1f2' to 'tblt1f5' in uvcopy job listing below.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

6D2. General Purpose Table Summaries

cmtbl2a - uvcopy job demo concat fields for tbl args

 # cmtbl2a - table analysis of customer master sales history file
 #         - Alt version to demo concatenating fields for table argument
 #         - use table format 'f5' allow 32 byte arg vs 16 for f2 in cmtbl2
 # uvcopy cmtbl2a  <-- execute uvcopy to interpret this cmtbl2a parameter file
 # ==============
 opr='$jobname - summarize sales (thisyr&lastyr) by province + city-name'
 rop=r1   # prompt oprtr for report disposition at EOJ
 fili1=?dat1/custmas1,rcs=256,typ=RSF
 filo1=?tmp/$fili1.tbl,rcs=80,typ=LSTt
 @run
        opn    all
 # begin loop to read all customer master sales history records
 # - crossfooting & accumulating (tabling) thisyr & lastyr totals
 loop   get    fili1,a0(256)
        skp>   eof
        xft    $ca1,a120(5p),12
        mvc    c0(2),a77         store province to concat city-name
        mvc    c3(16),a60        concat city-name for argument province+cityname
        tblt1f5 c0(20),'province+cityname;thisyr sales;lastyr sales',$ca1
        skp    loop
 # end of file - print/edit the tables to a file for: lp, cat,or vi
 eof    tbpt1s2 filo1,'sales by province+cityname (this yr & last yr)'
        cls    all
        eoj

compare cmtbl2a to previous cmtbl2

  1. Added 'mvc c0(2),a77' & 'mvc c3(16),a60' to concat prov+city for 'tblt1f5 c0(20),...' (table argument 20 bytes vs 16)

  2. Changed tbl format 'f2' to 'f5' because f2 allows only 16 byte args and f5 allows 32 bytes (we need 19 for province + space + city-name).

  3. Dropped 'tblt2' (separate province table), dropped last-year acum.

  4. See complete details of 'tbl' instruction at https://uvsoftware.ca/uvcopy3.htm#tbl.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_7. Data Investigation, Manipulation, Analysis,& Validation

Accumulating numeric fields


7A1. using 'uvhd' or 'uvhdcob' to accumulate any 1 field (interactively)
- may qualify records to be accumulated (via record type, etc)
- see examples using sales detail demo file

7B1. using 'pre-programmed uvcopy job' to accumulate any 1 field
- allows for various file types (fixed,variable,RDW/BDW,IDXFORMAT,text)
- options for field type (numeric, packed, binary)

7C1. writing 'custom uvcopy job' to accumulate 1 or more fields
- complete flexibility to whatever desired
- test record types & accumulate different fields

7D1. using COBOL copybooks to automatically generate a uvcopy job
to accumulate all numeric fields defined in the copybook

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7A1. Data File Investigation,Manipulation,Analysis,& Validation

using uvhd to accumulate any 1 field

Our 1st example will accumulate the $amount field in the sales3 detail file. See all 20 records listed on page '1C3'. hen we run uvhd, it will display the first record & wait for our command.


 uvhd dat1/sales3 r64    <-- run uvhd on the sales3 demo file
 ====================      - option r64 for fixed-length 64 byte records
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    21 940802 IN111001  HAM001  00020+ 0000001 00000020+ .
 rec#=1 rsize=64 fptr=0 fsize=1280 rcount=20
 ,k1=HelpPrompts,?=HelpScreens --> a 53(9) <-- enter acum command
                                   =======

 a 53(9) --> acum total=450100 from 20 recs of 20 read  <-- result
 =====================================================

EOF, enter -1 to see last record, or 1 return to Begin File rec#=21 rsize=0 fptr=1280 fsize=1280 rcount=20 ,k1=HelpPrompts,?=HelpScreens --> q <-- quit request - program ended

example#2 - acum sales for salesman #21


 uvhd dat1/sales3 r64
 ====================
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 130140    21 940802 IN111001  HAM001  00020+ 0000001 00000020+ .
 rec#=1 rsize=64 fptr=0 fsize=1280 rcount=20
 ,k1=HelpPrompts,?=HelpScreens --> a 53(9),,10(2),'21' <-- acum qualified
                                   ===================     for slsmn #21
                      10        20        30        40        50        60
 r#       13 0123456789012345678901234567890123456789012345678901234567890123
         768 308685    21 990814 CR8835    TAB013  00012- 0002001 00024012- .

 a 53(9),,10(2),'21' --> acum total=30030 from 6 recs of 20 read  <-- result
 ===============================================================

searched to EOF & reset to last record accumulated rec#=13 rsize=64 fptr=768 fsize=1280 rcount=20 ,k1=HelpPrompts,?=HelpScreens --> q <-- quit request - program ended

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7A2. using uvhd to accumulate any 1 field

notes re uvhd acum

  1. You must enter 2 commas after the acum field & prior to the qualifier field. IE the ',,' in 53(9),,10(2),'21' are required.

  2. When qualifiers are used, uvhd displays the last matching record, but does continue searching to End Of File.

  3. You can use 2 qualifiers, for example: --> a 53(9),,10(2),'21',30(3),'TAB' <-- acum if slsmn 21 & product 'TAB...'

  4. You can use greater-than, less-than, or not-equal conditions on qualifiers --> a 53(9),,10(2),>'50' <-- acum if slsmn# > '50'

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7B1. Data File Investigation,Manipulation,Analysis,& Validation

acum1 - acumulate any 1 field

'acum1' is a pre-programmed uvcopy job to accumulate any 1 field in any file. There are 2 ways to use it - directly with uvcopy or via script 'acum1'. Here are examples of both ways, but the script is easier to use (fewer keystrokes).


 #1. uvcopy acum1,fili1=dat1/sales,typ=RSF,uop=a53b9e2r64 <-- direct call
     ====================================================

 #2. acum1 dat1/sales3 RSF a53b9e2r64   <-- script 'acum1' calls uvcopy 'acum1'
     ================================

Either call prompts for options in case you want to add more options or override any specified on the command line.

 file types: RSF(Fixed,default), LST(Text), RDWz4,RDWz2,RDWz8, ISF,IDXf8
 uop=a0b0c1d0e0f0r0  - default options
     a0              - field dsplcmnt (zero relative)
       b0            - field length (must specify)
         c1          - cross-foot number of fields
         c12         - example for 12 months sales
            d0       - field type numeric unpacked
            d1       - field type binary
            d2       - field type packed
              e0     - edit 0 decimals
              e2     - edit 2 decimals
                f0   - ASCII file
                f1   - EBCDIC file
                  r0 - recsize if typ=RSF
 User OPtion (uop) defaults  = q1a0b0c1d0e0f0r0a53b9e2r64
 -->null to accept or enter/override -->    <-- null to accept cmd line as is
 110524:162621:acum1: EOF fili01 rds=20 size=1280: dat1/sales3
 110524:162621:acum1: EOF filo01 wrts=2 size=120: tmp/dat1_sales3_acum1

EOJ, Output File written to: tmp/dat1_sales3_acum1 report command: more,print,uvlpr12,edit,null --> more <-- display reprot

 acum1: dsplcmnt 53, length 9, occurs 1, recsize 64: dat1/sales3
 field total-->           4,501.00  records=          20
 =======================================================

acum1 example#2 for custmas1

Run the following command on the custmas1 file & see if you get the result shown below. Option 'c12' crossfoots 12 5 byte packed fields starting at 120.


 acum1 dat1/custmas1 RSF a120b5c12d2e2r256
 =========================================
 acum1: dsplcmnt 120, length 5, occurs 12, recsize 256: dat1/custmas1
 field total-->          99,697.93  records=          32
 =======================================================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7C1. Data File Investigation,Manipulation,Analysis,& Validation

writing custom uvcopy jobs to accumulate fields

'acumvpm1' is an example of a custom written 'uvcopy job' to accumulate record fields for final totals. See the job (Parameter/Instruction file) listed on the next page. Note the many #comments documenting the job.

The input file is lsited on page '1F1' but also listed here for convenience. Note 2 record types 'M' or 'P' in byte 7 & Payment amount in bytes 30-38. The uvcopy job must test the Record Type & accumulate the Payment amount.

The file listed below is dat1/vendorpaymas2 which is the 'TEXT' file equivalent of the actual input file dat1/vendorpaymas3 which is an 'RDW' file with binary record-length prefixes (must be listed with uvhd as on page '1F1').

          10        20        30        40        50        60
 0123456789012345678901234567890123456789012345678901234567890123
 ================================================================
 DELL10 M  Dell Inc.
 DELL10 P  20100131  INV00010  000195000 Dell server 2900
 HP0010 M  Hewlett Packard
 HP0010 P  20100215  INV00020  000042500 HP LCD Monitor
 HP0010 P  20100215  INV00020  000350000 HP XW9400 Workstation
 IBM010 M  International Business Machines
 IBM010 P  20100321  INV00030  000750000 AIX server
 IBM010 P  20100322  INV00030  000750000 DB2 database software
 MFC010 M  Micro Focus COBOL
 MFC010 P  20100420  INV00040  000320000 Micro Focus Developer
 MS0010 M  Microsoft Corp.
 MS0010 P  20100501  INV00050  000011600 Microsoft Windows 7
 REDH10 M  Red Hat Linux
 REDH10 P  20100620  INV00060  000034900 Red Hat Enterprise 5.4
 SUN010 M  Sun Microsystems Ltd
 UVSI10 M  UV Software Inc.

Execute job & observe results


 uvcopy acumvpm1,fili1=dat1/vendorpaymas3,filo1=tmp/vpmtotal
 ===========================================================

 uvcopy acumvpm1   <-- shorter command (files default as above)
 ===============

110525:100809:acumvpm1: uvcopy ver=20110511 pf=/home/uvadm/pf/demo/acumvpm1 uvcopy DISAM ext=.dat LNX L64 license=110511_99V_930630 site=UV_Software dat1/vendorpaymas3 = default fili01 - null accept or reenter ? tmp/vpmtotal = default filo01 - null accept or reenter ? 110525:100810:acumvpm1: EOF fili01 rds=16 size=745: dat1/vendorpaymas3 110525:100810:acumvpm1: EOF filo01 wrts=1 size=56: tmp/vpmtotal EOJ, Output File written to: tmp/vpmtotal report dispostion, enter: more,print,uvlpr12,edit,null --> more <--


 Payment Total = 2454000, input-file: dat1/vendorpaymas3
 =======================================================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7C2. writing custom uvcopy jobs to accumulate fields

acumvpm1 - uvcopy job to demo accumulation

Note
  • this is a listing of the file stored at /home/uvadm/pf/demo/acumpvpm1
  • this Parameter/Instruction File is interpreted by the uvcopy interpreter
  • 1st 24 lines are #comments (documentation & operating instructions)
 # acumvpm1 - accumulate payments in VendorPayMas3
 #          - uvcopy job to demo accumulating numeric fields
 #          - by Owen Townsend, UV Software, April 24/2011
 #          - documented at www.uvsoftware.ca/datautilities.htm#7C1
 #
 # Execute uvcopy to interpret this instruction file as follows:
 # - default input is dat1/vendorpaymas3 (an RDW demo file)
 # - default output is tmp/vpmtotal (option rop=r1 prompts for more,lp,etc)
 #
 # uvcopy acumvpm1,fili1=dat1/vendorpaymas3,filo1=tmp/vpmtotal
 # ===========================================================
 #
 # uvcopy acumvpm1   <-- shorter command (files default as above)
 # ===============
 #
 # Default input is dat1/vendorpaymas3 (RDW demo file with 2 Record Types)
 # - here are 1st 2 records, see all at www.uvsoftware.ca/datautilities.htm#7C1
 # - R/T M/P is in byte 7 & payment amount is in bytes 30-39 (0 relative)
 #
 # DELL10 M  Dell Inc.
 # DELL10 P  20100131  INV00010  000195000 Dell server 2900
 #          10        20        30        40        50        60
 # 0123456789012345678901234567890123456789012345678901234567890123
 #
 rop=r1   # Run OPtions to prompt for report disposition at EOJ (more,lp,etc)
 fili1=?dat1/vendorpaymas3,typ=RDWz4,rcs=64
 filo1=?tmp/vpmtotal,typ=LSTt,rcs=80
 @run
        opn    all                  open files
 # begin loop to get records, test R/T, accumulate payments until EOF
 man20  get    fili1,a0(64)         get next record
        skp>   man90                (cc set > at EOF)
        cmc    a7(1),'P'            Payment record (vs Master record) ?
        skp!   man20                no - return to get next record
        add    $ca1,a30(9)          acum Payment $amount
        skp    man20                return to get next record
 #
 # EOF - write total to outfile & end job (option rop=r1 prompts for more,lp,etc)
 man90  mvfv1  b0(80),'Payment Total = $ca1, input-file: $fili1'
        put    filo1,b0(80)         write to output file
        cls    all                  close files
        eoj                         End Of Job

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7C3. writing custom uvcopy jobs to accumulate fields

acumvpm2 - example#2 custom written uvcopy jobs

'acumvpm2' is an extension of acumvpm1 above. In addition to accumulating the payment amounts, this job lists the payment records with the payment amount edited on the right hand side (see report at bottom of this page). See the acumvpm2 Instruction file listed on the following page.

Execute job & observe results


 uvcopy acumvpm2,fili1=dat1/vendorpaymas3,filo1=tmp/vpmlist
 ===========================================================

 uvcopy acumvpm2   <-- shorter command (files default as above)
 ===============

110525:100818:acumvpm2: uvcopy ver=20110511 pf=/home/uvadm/pf/demo/acumvpm2 uvcopy DISAM ext=.dat LNX L64 license=110511_99V_930630 site=UV_Software dat1/vendorpaymas3 = default fili01 - null accept or reenter ? tmp/vpmlist = default filo01 - null accept or reenter ? 110525:100819:acumvpm2: EOF fili01 rds=16 size=745: dat1/vendorpaymas3 110525:100819:acumvpm2: EOF filo01 wrts=9 size=684: tmp/vpmlist EOJ, Output File written to: tmp/vpmlist default command = null, OR enter: more,print,uvlpr12,edit,null --> more <--

 DELL10 P  20100131  INV00010  000195000 Dell server 2900           1,950.00
 HP0010 P  20100215  INV00020  000042500 HP LCD Monitor               425.00
 HP0010 P  20100215  INV00020  000350000 HP XW9400 Workstation      3,500.00
 IBM010 P  20100321  INV00030  000750000 AIX server                 7,500.00
 IBM010 P  20100322  INV00030  000750000 DB2 database software      7,500.00
 MFC010 P  20100420  INV00040  000320000 Micro Focus Developer      3,200.00
 MS0010 P  20100501  INV00050  000011600 Microsoft Windows 7          116.00
 REDH10 P  20100620  INV00060  000034900 Red Hat Enterprise 5.4       349.00
                                                   Payment Total   24,540.00

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7C2. writing custom uvcopy jobs to accumulate fields

acumvpm2 - edit,list,& accumulate payments

 # acumvpm2 - edit, list, & accumulate payments in VendorPayMas3
 #          - sample/demo of custom written uvcopy jobs
 #          - by Owen Townsend, UV Software, April 24/2011
 #          - documented at www.uvsoftware.ca/datautilities.htm#7C2
 #
 # Execute uvcopy to interpret this instruction file as follows:
 # - default input is dat1/vendorpaymas3 (an RDW demo file)
 # - default output is tmp/vpmlist (option rop=r1 prompts for report more,lp,etc)
 #
 # uvcopy acumvpm2,fili1=dat1/vendorpaymas3,filo1=tmp/vpmlist
 # ==========================================================
 #
 # uvcopy acumvpm2   <-- shorter command (files default as above)
 # ===============
 #
 # Default input is dat1/vendorpaymas3 (RDW demo file with 2 Record Types)
 # - here are 1st 2 records, see all at www.uvsoftware.ca/datautilities.htm#7C1
 # - R/T M/P is in byte 7 & payment amount is in bytes 30-39 (0 relative)
 #
 # DELL10 M  Dell Inc.
 # DELL10 P  20100131  INV00010  000195000 Dell server 2900
 #          10        20        30        40        50        60
 # 0123456789012345678901234567890123456789012345678901234567890123
 #
 rop=r1   # Run OPtions to prompt for report disposition at EOJ
 fili1=?dat1/vendorpaymas3,typ=RDWz4,rcs=64
 filo1=?tmp/vpmlist,typ=LSTt,rcs=80
 @run
        opn    all                  open files
 # begin loop to get records, test R/T, accumulate payments until EOF
 man20  get    fili1,a0(256)        get next record (RDW variable lth)
        skp>   man90                (cc set > at EOF)
        cmc    a7(1),'P'            Payment record (vs Master record) ?
        skp!   man20                no - return to get next record
        mvc    b0(80),a0            move pmt record to output (print) area
        add    $ca1,a30(9)          acum Payment $amount
        edt    b64(12),a30(9),'zzzz,zzz.99-'  edit $amt to outarea
 man30  put    filo1,b0(80)         write (print) to outfile
        skp    man20                return to get next record
 #
 # EOF - write total to outfile & end job (option rop=r1 prompt for more,lp,etc)
 man90  clr    b0(80),' '
        mvf    b50(20),'Payment Total'
        edt    b64(12),$ca1,'zzzz,zzz.99-'  edit $total to outarea
        put    filo1,b0(80)         write to output file
        cls    all                  close files
        eoj                         End Of Job

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7D1. Verifying Conversions by Accumulating All Numeric fields

genacum2 - generate jobs to accum ALL numeric fields

'genacum2' is a Vancouver Utility to generate uvcopy jobs to accumulate all numeric fields (packed/zoned/binary) in 1 or all data files. The jobs are generated automatically from the 'cobmap' record layout (which is generated from the COBOL copybook).

You might use these reports to verify data file conversions, OR to get a 2nd opinion on suspicious looking totals on your COBOL reports.

We will demonstrate using the 'custmas1' copybook & datafile, which has occurs fields for 12 months sales this year & last. Here are the illustrated procedures to generate & execute the 'genacum2' job for custmas1.


 #1. Login --> your homedir

 #2. ls -l   <-- list subdirs created previously
               - should be: dat1, cpys, maps, tmp

 #3a. mkdir pfa1 pfa2   <-- make subdirs for genacum2 jobs
      ===============

 #3b. mkdir rpts        <-- make subdir for output reports
      ==========

Directories Required

 /home/userxx
 :-----cpys/          <-- copybooks
 :     :----custmas1    - copybook 'custmas1' for datafile 'custmas1'
 :-----dat1/          <-- data files (previously converted to ASCII)
 :     :-----custmas1   - datafile 'custmas1'
 :-----maps/          <-- record layouts (cobmaps) created from copybooks
 :     :-----custmas1   - cobmap 'custmas1'
 :-----pfa1/          <-- uvcopy jobs generated from cobmaps by 'genacum2'
 :     :-----custmas1   - uvcopy job 'custmas1' to add all fields in datafile
 :-----pfa2/          <-- uvcopy jobs copied here before execution
 :     :                - modified if if multiple record types
 :     :                - &/or modified to insert actual datafile names
 :-----rpts/          <-- reports written here
 :     :-----custmas1   - reports named same as copybook & datafile
                          (run uvdata52 to change to datafilenames)
  1. We will create subdirs pfa1,pfa2,pfa3 for the generated 'acum' jobs using the same filename for copybook,datafile,cobmap,& uvcopy acum job which simplifies the generation scripts & operating instructions

  2. We can run the generated job from pfa1 (still named same as copybook) The pfa1/... job expects input datafile in dat1/ & writes report to rpts/

  3. If modifications are required copy jobs to pf

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7D2. Verifying Conversions by Accumulating All Numeric fields

convert copybook to cobmap (prelim to genacum2)


 #4. uvcopy cobmap1,fili1=cpys/custmas1,filo1=maps/custmas1
     ======================================================
     - convert copybook to 'cobmap' (record layout)

cobmap (record layout) for custmas1

 cobmap1  start-end bytes for cobol record fields    200902200736  pg# 0001
 cpys/custmas1.cpy                                 RCSZ=00256  bgn  end  lth typ
 * custmas1 - cobol copy book for custmas1 file (see uvtrain.
      10 cm-num           pic  9(6).                          0000 0005  006 n  6
      10 cm-delete        pic  x(4).                          0006 0009  004
      10 cm-name          pic  x(25).                         0010 0034  025
      10 cm-adrs          pic  x(25).                         0035 0059  025
      10 cm-city          pic  x(16).                         0060 0075  016
      10 filler001        pic  x.                             0076 0076  001
      10 cm-prov          pic  x(2).                          0077 0078  002
      10 filler002        pic  x.                             0079 0079  001
      10 cm-postal        pic  x(10).                         0080 0089  010
      10 cm-phone         pic  x(12).                         0090 0101  012
      10 cm-contact       pic  x(18).                         0102 0119  018
      10 cm-thisyr-sales  pic  s9(7)v99 comp-3 occurs 12.     0120 0124  005pns 9
      10 cm-lastyr-sales  pic  s9(7)v99 comp-3 occurs 12.     0180 0184  005pns 9
      10 filler003        pic  x(16).                         0240 0255  016
 *RCSZ=00256

On the next page, we will run genacum2 to generate a job to add all fields in test file dat1/custmas1 (based on the copybook converted to a cobmap as shown on the previous page).

But first (to understand the datafile), you might run 'uvhdcob' which displays the data field contents beside the COBOL copybook fieldnames.


 #5. uvhdcob dat1/cutsmas1 maps/custmas1  <-- execute uvhdcob for custmas1
     ===================================

uvhdcob display - 1st record of dat1/custmas1

 rec#       1 fieldname occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 cm-num                       0   5 n  130140
 002 cm-delete                    6   9
 003 cm-name                     10  34    EVERGREEN MOTORS LTD.
 004 cm-adrs                     35  59    1815 BOWEN ROAD
 005 cm-city                     60  75    NANAIMO
 006 filler001                   76  76
 007 cm-prov                     77  78    BC
 008 filler002                   79  79
 009 cm-postal                   80  89    V9S1H1
 010 cm-phone                    90 101    250-754-5531
 011 cm-contact                 102 119     LARRY WRENCH
 012 cm-thisyr-sales       012  120 124pns 001234567C
 013 cm-lastyr-sales       012  180 184pns 001234567D
 014 filler003                  240 255    C 19950531

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7D3. Verifying Conversions by Accumulating All Numeric fields

Generate 1 job to accum ALL num-fields in 1 file


 #6. uvcopy genacum2,fili1=maps/custmas1,filo1=pfa1/custmas1,uop=f0
     ==============================================================
     - convert cobmap to uvcopy job to accumulate All numeric fields

console prompts for options

 uop=f2m1 - option defaults (for generated jobs)
     f0   - Input file default typ=RSF fixed lth (in generated job)
     f2   - Input typ=RDWz2 (2 byte BIG-end binary recsize hdrs)
     f4   - Input file typ=RDWz4 (4 byte hdrs, bytes 3&4 nulls)
       m1 - read copybook record layouts from maps/... in $CNVDATA
       m2 - read copybook record layouts from maps/... in $RUNLIBS
 User OPtion (uop) defaults  = q1f2m1
 -->null to accept or enter/override --> f0m0 <-- f0 typ=RSF, m0 ./maps
Note
  • specify option 'f0' to generate job with 'typ=RSF' (on fili1=... below)
  • default f2 would generate typ=RDW2
  • specify option 'm0' if maps/ is in current dir vs $CNVDATA

generated job pfa1/custmas1

 # uvcopy job to accumulate all numeric fields in a datafile
 opr='jobname=JOBNAME   - pfa2 name=datafilename'
 opr='copybook=custmas1 - maps/pfa1 name=copybookname'
 uop=q0i3,rop=r1  # minimize startup msgs, prompt for report disposition
 was=a33000c200000t100000  # 33K maxrec, 200K cpybkmap, 100K acum tbl
 fili1=?dat1/custmas1,rcs=00256,typ=RSF
 filo1=?rpts/custmas1,rcs=128,typ=LSTt
 fili2=?maps/custmas1,rcs=128,typ=LST
 @run
        opn    all
        rtb    fili2,c0(100),c0(100)  read cobmap into area c table
        clr    t0(100000),x'00'       clear area t for accumulator table
 loop   get    fili1,a0               get next record into area 'a'
        skp>   eof
        add    $cb3,1                 count all record types
 #      ---    ---      <-- insert any Record Type tests here
        mvn    $rt,0                  init register t for current record
        mvn    $rs,0                  init register s for current record
        add    ts9800(8b),1           count 1st (or only) R/T in copybook
        bal    acum,'0,6,z,1,cm-num'
        bal    acum,'120,5,p,12,cm-thisyr-sales'
        bal    acum,'180,5,p,12,cm-lastyr-sales'
        skp    loop
 eof    bal    dump                   dump tables of fieldnames & acummulators
        cls    all
        eoj
 @pf2=acum2.sub

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

7D4. Verifying Conversions by Accumulating All Numeric fields

Execute job to acum all numeric fields in custmas1


 #7. uvcopy pfa1/custmas1     <-- execute job to add all numeric fields custmas1
     ====================

 #8. uvlp12 rpts/custmas1     <-- print report
     ====================

sample report

 genacum2  RCSZ=00256 DataFile=dat1/custmas1
 2009/02/26_09:22:03  copybook=cpys/custmas1.cpy
 line# Field-Name                 occurs start lth type         field-total
    1  cm-num                             0005 006  n            7,259,353
    2  cm-thisyr-sales                12  0124 005 pns           99,697.93
    3  cm-lastyr-sales                12  0184 005 pns           45,440.85
          Record count        32 **      Hash-Total             21,773,231 **

generate jobs for ALL files

The above procedures were for 1 file at a time. For major projects we would want to generate jobs for all copybooks & all datafiles. See the procedures to generate jobs for all copybooks & all datafiles documented at https://www.uvsoftware.ca/datacnv1.htm#8G5.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_8 CMPjobs - Comparing data files

Part 8 - comparing Variable Length RDW data files


8A1. Comparing Variable Length RDW files - Overview
'uvcmp2' to compare Variable Length RDW files
vs 'uvcmp1' for FIXED Length files (see CMPjobs.htm#Part_1)
'uvcmpRDW1' script easier way to run uvcmp2 (shorter commands)
Variable Length file types (RDW, BDW/RDW, BDW/RDW unblocked, Micro Focus)

8B1. Setup Directories to demo uvcmp2
- may setup 'cmpdata' directory in your homedir
- make subdirs & copy test/demo files from /home/uvadm/dat1/...

8C1. testRDWe & testRDWea - test files supplied for VarLth comparison demo
- display with 'uvhd', since binary RDW conflicts with 'vi'

8D1. Op. Instrns to compare RDW files using uvcopy job 'uvcmp2' directly
8D2. console log
8D3. sample report

8E1. Op. Instrns to compare RDW files using script 'uvcmpRDW1'
8E2. console log
8E3. sample report

notes re compare documentation

This Part 8 in this document DataUtilities.htm is taken from Part 2 in CMPjobs.htm, which is the complete documentation for file compare jobs. CMPjobs.htm covers all file types (Fixed-Length, Micro Focus Indexed, etc) and alternate jobs to allow for Inserted or Deleted records.

This Part covering variable-length RDW files was included here in DataAnalysis.htm because it was short, but gives you a good idea of the data file compare jobs.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8A1. Comparing Variable Length RDW Data Files

comparing variable length RDW files - Overview

We will document 2 ways to compare variable length RDW files

  1. using uvcopy job 'uvcmp2' directly

  2. using script 'uvcmpRDW1' which calls uvcmp2 (easiest method).

    types of Variable Length files

  3. RDWz4 - records have 4 byte a 'Record Descriptor Word' (prefix) - record size in 1st 2 bytes, next 2 bytes x'0000' unused - the binary value is the total slot (-4 for data-size) - record size is in binary 'Big-End' (mainframe vs Intel Little-End) - we will illustrate RDWz4 file compares here.

  4. RDWz2 - records have 2 byte a 'Record Descriptor Word' (prefix) (the 2 null bytes are omitted) - the binary value is the data-size (+2 for total slot size)

  5. BDW/RDW - Blocks have 4 byte 'BDW's containing records with 4 byte 'RDW's - this is the traditional IBM variable length tape format

  6. BDW+RDW - unblocked records where every record has both BDW & RDW the same.

  7. Micro Focus variable length Sequential or Indexed (IDXFORMAT3/IDXFORMAT8) - file header 128 or 1024 bytes - each record has 2 or 4 byte prefix (not same as RDW) - see format described in uvhd.htm#X1 thru X5. - could convert to fixed length using uvhd with option 'v' (or uxcp, uxsort, uxcopy) - Micro Focus varlth compares are illustrated in CMPjobs.htm#Part_3

    sample variable length RDW data files

Before documenting the comparison procedures, we will illustrate the RDW variable length file format on the next few pages.


8B1. testRDWe - sample VARIABLE length data file in RDW format.

8B2. testRDWea - Alternate test file for comparison
- with some intentional mismatches

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8B1. Comparing Variable Length RDW Data Files

subdirs used to compare varlth files

 /home/userxx                <-- your home directory
 :-----...
 :-----cmpdata               <-- working directory for datafile compares
 :     :
 :     :-----dat1            <-- subdir to store test files
 :     :     :-----testRDWe      <-- sample RDW file for comparison
 :     :     :-----testRDWea     <-- 'a'lternate file with some mismatches
 :     :
 :     :-----rptcmp          <-- subdir to receive comparison reports
 :     :     :-----testRDWe      <-- comparison report
 :     :                           - name same as file#1
 :     :
 :     :-----tmp             <-- subdir required by uvhd for any Printing
 :     :     :-----testRDWe_090217_124800P  <-- Print file created by uvhd

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8B2. Comparing Variable Length RDW Data Files

setup subdirs & copy test/demo files


 #1. login to your homedir & mkdir cmpdata (if not already existing)

 #2. cd cmpdata                 <-- change into comparison directory
     ==========

 #3. mkdir dat1 rptcmp tmp      <-- make subdirs required
     =====================

 #4a. cp /home/uvadm/dat1/testRDWe dat1
      =================================
      - copy 1st EBCDIC varlth file to be compared to dat1/...

 #4b. cp /home/uvadm/dat1/testRDWea dat1
      ==================================
      - copy 2nd EBCDIC varlth file to be compared to dat1/...

 #5. uvhd dat1/testRDWe z4a
     ======================
     Investigate 1st variable lth file with options:
     z4 - use binary recsize in 1st 2 bytes of record
     a  - translate uvhd character line from EBCDIC to ASCII

 #5a. --> p8     <-- Print 8 records (all)
                   - see printout below
Note
  • see the demo file listed on the next page (via uvhd)
  • can NOT list with text utilities (lp,vi,etc) due to binary RDW

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8C1. Comparing Variable Length RDW Data Files

testRDWe - sample VARIABLE length RDW data file


 uvhd dat1/testRDWe z4a1  <-- option z4 = variable lth binary 1st 2 bytes
 =======================    - option a1 = translate char line to Ascii
                            - zones & digits lines show EBCDIC codes
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ....DELL10 - Dell Inc.
             0100CCDDFF464C8994C98444
             080045331000045330953B00
 r#        2
          24 ....HP0010 - Hewlett Packard
             0100CDFFFF464C8A98AA4D889898
             0C00870010000856353307132194
 r#        3
          52 ....IBM010 - International Business Machines
             0200CCDFFF464C9A8998A899894CAA898AA4D888898A
             0C009240100009535951396513024295522041389552
 r#        4
          96 ....MFC010 - Micro Focus COBOL
             0200DCCFFF464D88994C98AA4CDCDD44
             00004630100004939606634203626300
 r#        5
         128 ....MS0010 - Microsoft Corp.
             0100DEFFFF464D8899A98A4C9994
             0C0042001000049396266303697B
 r#        6
         156 ....REDH10 - Red Hat Linux
             0100DCCCFF464D884C8A4D89AA44
             0C00954810000954081303954700
 r#        7
         184 ....SUN010 - Sun Microsystems Ltd
             0200EEDFFF464EA94D8899AAAA89A4DA8444
             040024501000024504939628235420334000
 r#        8
         220 ....UVSI10 - UV Software Inc.
             0200EEECFF464EE4E98AA8984C984444
             0000452910000450266361950953B000
  1. RDW format stores RecLth in 1st 2 bytes binary (next 2 bytes null/unused)

  2. RecLth prefix for 1st rec is x'0018' = ((1*16)+(8*1)) = 24 bytes

  3. RecLth is Big-End binary, right adjusted & zero left filled (like mainframe).

  4. Many Unix/Linux systems are Intel architecture, left adjusted & 0 right fill. To perform calculations on Intel, binary 24 would be x'1800' (not x'0018'). uvcopy provides option 's' (switch), specify as (2bs) vs (2b).

  5. See mode doc for RDW & BDW files on pages X1-X5 of uvhd.htm#X1.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8C2. Comparing Variable Length RDW Data Files

testRDWea - Alternate file for compare (with mismatches)


 uvhd dat1/testRDWea z4a1  <-- option z4 = variable lth binary 1st 2 bytes
 ========================    - option a1 = translate char line to Ascii
                             - zones & digits lines show EBCDIC codes
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ....DELL10 - Dell Inc.
             0100CCDDFF464C8994C98444
             080045331000045330953B00
 r#        2
          24 ....HP0010 - Hewlett Packard
             0100CDFFFF464C8A98AA4D889898
             0C00870010000856353307132194
 r#        3
          52 ....IBM010 - International Business Mothers
             0200CCDFFF464C9A8998A899894CAA898AA4D9A889A4
             0C009240100009535951396513024295522046385920
 r#        4
          96 ....MFC010 - Micro Focus COBOL
             0200DCCFFF464D88994C98AA4CDCDD44
             00004630100004939606634203626300
 r#        5
         128 ....MS0010 - Microsoft Corp.
             0100DEFFFF464D8899A98A4C9994
             0C0042001000049396266303697B
 r#        6
         156 ....REDH10 - Red Hat Linux
             0100DCCCFF464D884C8A4D89AA44
             0C00954810000954081303954700
 r#        7
         184 ....SUN010 - Sun Microsystems Corp
             0200EEDFFF464EA94D8899AAAA89A4C99944
             040024501000024504939628235420369700
 r#        8
         220 ....UVSI10 - UV Software Inc.
             0200EEECFF464EE4E98AA8984C984444
             0000452910000450266361950953B000
  1. Can you see the differences between this file 'testRDWea' & 'testRDWe' listed on the previous page.

  2. If not, it is a good reason to use these comparison jobs, especially when your data files could be thousands of times bigger.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8D1. Comparing Variable Length RDW Data Files

compare RDW files method #1 - use uvcmp2 directly

Assuming the 2 EBCDIC RDW files have already been copied to subdir dat1/... in your homedir (as shown on page '8B1').


 #1. uvcopy uvcmp2,fili1=dat2/testRDWe,fili2=dat2/testRDWea,uop=y1z4
     ===============================================================
     - run comparison utility
       option y1 - translate char line to ASCII (zone & digit lines EBCDIC)
       option z4 - RDW file with 4 byte recsize prefixes (vs 2 bytes)
     - see the console log displayed on the next page --->

 #2. uvlp12 rptcmp/testRDWe
     ======================
     - print output report

alternative 'script uvcmpRDW1' vs 'uvcopy uvcmp2'


 #1a. uvcmpRDW1 dat1/testRDWe dat1/testRDWea z4y1  <-- execute alternative
      ===========================================      (shorter command)
      - option 'z4' (4 byte prefixes) is default & may be omitted
      - option 'y1' translates character line to ASCII (for EBCDIC files)
Note
  • see script 'uvcmpRDW1' Op. Instrns. on page '8E1'
  • see script 'uvcmpRDW1' listing at CMPjobs.htm#2X1

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8D2. Comparing Variable Length RDW Data Files

console log from uvcmp2 2 FixLth files written by uvhd


 uvcopy uvcmp2,fili1=dat2/testRDWe,fili2=dat2/testRDWea,uop=y1z4
 ===============================================================
 uvcmp2 - compare files & print diffs in hex with "*" markers (Nov28/07)
 uop=q1a0b0c0d0e0f0g0h0p30r8192s8t500000u3x2y0  - option defaults
       a#              - dsplcmnt to exclude field #1
       b#              - length   to exclude field #1
         c#            - dsplcmnt to exclude field #2
         d#            - length   to exclude field #2
           e#          - dsplcmnt to exclude field #3
           f#          - length   to exclude field #3
             g#        - dsplcmnt to exclude field #4
             h#        - length   to exclude field #4
               n0      - print column scale only at begin report
               n1      - print column scale for each record mismatch pair
                p30    - line# to test overflow at begin record only
                  r8192- record size (largest RDW recsize, default 8192)
                s8     - StopPrint count, after s# mismatched records
              t500000  - StopRead count, after s# mismats, else read to EOF
            u1         - stop print at last segment with mismatch
            u2         - inhibit printing all matching segments after 1st
          x#           - hex or char print (x1 char, x2 hex, x0 auto)
        y0             - do NOT translate, file already ASCII, default(y0)
        y1             - translate char lines to ASCII (for EBCDIC files)
      z2               - RDWz2 2 byte RDW recsize prefixes (vs RDWz4 dflt)
 090217:131431:uvcmp2: uvcopy ver=20090212 pf=/home/uvadm/pf/util/uvcmp2
 uvcopy DISAM ext=dat LNX L64  license=20090212V site=UV_Software
 User OPtion (uop) defaults  = q1p30r8192s8t500000u3x2y0y1
 -->null to accept or enter/override -->
 ==================== EOF or StopPrint/StopRead count reached  ==============
 File1Total=0, File2Total=0, StopPrint=8, StopRead=500000
 File1Reads=8, TotalMisMatches=2, MisMatchesPrinted=2
 090217:131431:uvcmp2: EOF fili01 rds=8 size=252: dat1/testRDWe
 090217:131431:uvcmp2: EOF fili02 rds=8 size=252: dat1/testRDWea
 090217:131431:uvcmp2: EOF fild03 size=4096: rptcmp
 090217:131431:uvcmp2: EOF filo03 wrts=32 size=1496: rptcmp/testRDWe hits=2
 EOJ, Output File written to: rptcmp/testRDWe
 default command = null, OR enter: more,print,uvlpr12,edit,null -->

See differences report listed on the next page --->

Note
  • we will not demo 'exclude field' options here
  • see exclude field options at CMPjobs.htm#1E1.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8D3. Comparing Variable Length RDW Data Files

testRDWe - sample comparison report

 ${JOBID2} uvcmp2 - compare RDW files, print mismatched records, '*' flag diffs
 2009/02/17_13:14:31 uop=q1p30r8192s8t500000u3x2y0y1
 recsize  reccount   file-size  typ Report=rptcmp/testRDWe
 1: 8192         0*          0  RDW  File1=dat1/testRDWe
 2: 8192         0*          0  RDW  File2=dat1/testRDWea
                       1         2         3         4         5         6
 f#rec#byte# 0123456789012345678901234567890123456789012345678901234567890123
 ============================================================================
 1    3    0 IBM010 - International Business Machines
     40      CCDFFF464C9A8998A899894CAA898AA4D888898A
             9240100009535951396513024295522041389552
                                              ** ****
 2    3      IBM010 - International Business Mothers
             CCDFFF464C9A8998A899894CAA898AA4D9A889A4
             9240100009535951396513024295522046385920
                                              ** ****
 1    7    0 SUN010 - Sun Microsystems Ltd
     32      EEDFFF464EA94D8899AAAA89A4DA8444
             24501000024504939628235420334000
                                       ****
 2    7      SUN010 - Sun Microsystems Corp
             EEDFFF464EA94D8899AAAA89A4C99944
             24501000024504939628235420369700
                                       ****
 ==================== EOF or StopPrint/StopRead count reached  ==============
 File1Total=0, File2Total=0, StopPrint=8, StopRead=500000
 File1Reads=8, TotalMisMatches=2, MisMatchesPrinted=2

Notes

  1. As you can see '*'s are used to indicate misscompares.

  2. Note that f#rec#byte# shows the file (1 or 2), record#,& byte# of segment where mismatches occur.

  3. The number under 'f#rec#' (for file#1) is the record-size of the pair. If file#2 different you would see another error message.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

8E1. Comparing Variable Length RDW Data Files

compare varlth files - script uvcmpRDW1

Script 'uvcmpRDW1' makes it easier to compare variable length RDW files because:

  1. The command line is shorter.

  2. The script assumes you are in a working directory where you have permissions & groupID match to allow you to make subdirs & write files.

  3. The script will make subdir rptcmp if not already present in current working directory.

    compare varlth files - script 'uvcmpRDW1' Op. Instrns.


 #1. uvcmpRDW1 dat1/testRDWe dat1/testRDWea z4y1    <-- execute script
     ===========================================
      - option 'z4' (4 byte prefixes) is default & may be omitted
      - option 'y1' translates character line to ASCII (for EBCDIC files)

 #2a. vi rptcmp/testRDWe        <-- view report
      ==================

 #2b. uvlp12 rptcmp/testRDWe    <-- print report
      ======================
Note

uvcmpRDWAI2 compare RDW files allowing inserts/deletes

Also see 'uvcmpRDWAI2/uvcmpRDWEI2' to compare variable length RDW files allowing for inserts & deletes at CMPjobs.htm#4H1.


 uvcmpRDWAI2 dat1/vendormas3 dat1/vendormas32 r64k0l6
 ====================================================
 - compare 2 RDW files, report created in rptcmp/vendormas3

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Part_9. Data Summary Table Analysis & Validation

Part_9 - gentbl1A/gentbl1E


9A1. Data table analysis Overview

9B1. Sample reports

9C1. Subdirs for Generating & Executing table summary jobs

9C2. Setting up subdirs in your homedir

9D1. Generate Table summary jobs - for 1 file at a time

9E1. Generate table summary job - Illustrated
 copybook maps/custmas0x will generate pft1/custmas0x table summary job
- copybook maps/custmas0x input listed

9E2. Execute 'gentbl1E' to Generate table summary job 'custmas0x'
- console display, prompts for option changes
- option 'm15' maximum table entries to be maintained
- option 'v4' maximum Invalid table entries to be maintained
- option 'd3' (default) converts invalid numerics to zeros
- option 'd4' clears invalid numeric fields (shows hex on right side)

9E3. Listing of gnerated table summary job pft1/custmas0x
- 14 fields in copybook generate code for 14 summary tables
- thisyear/lastyear sales occurs 12 handled by index register & code loop

9E4. copy/rename/modify table summary job for easier testing/check/rerun
- copy pft1/custmas0x to pft1/cm03v4 & modify
- reduce no of fields from 13 to 3
- remove crossfoot of 12 months sales to just Jan sales

9E5. Execute modified table summary job pft1/cm03v4
- output report listed, 3 tables (cust#, city, Jan sales)

9F1. further modify table summary job to test accumulation for total Jan sales
- copy pft1/cm03v4 to pft1/cm03v4a & modify
- reduce to just 1 table for January sales
- add accumulator for Jan sales total
- could accumulate up to 6 fields on right side of report

9F2. Execute modified table summary job
- output report listed (showing Jan sales total)

9G1. use 'uvhdcob' to investigate problems reported by table summary jobs

9H1. operating instructions - All files in directory

9I1. inserting data-file-names in table analysis jobs

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9A1. Data Summary Table Analysis & Validation

Data Table Analysis - Overview

'gentbl1A' & 'gentbl1E' are Vancouver Utilities to table summarize all fields in a data file (gentbl1A for Asciii & gentbl1E for Ebcdic).

gentbl1A/gentbl1E are driven by the COBOL copybooks to generate uvcopy jobs that will read thru data files & create a summary table of values found in each & all fields defined in the COBOL copybook.

Obviously tabling all values would be too voluminous without limiting options. Default option 'm30' limits the no of entries retained to 30. After 30, new entries are added if their value is greater than or less than any existing entry, & a mid-point entry is removed to maintain the number of entries to 30.

Default option 'v10' limits the no of ~invalid entries retained to 10. After 10, invalid entries are just counted for the ~Invalid Total line.

You can generate the data analysis/validation jobs for 1 file at a time, or you can generate the jobs for all COBOL copybooks in the directory.

The generated jobs are named the same as the copybooks, and are written to a separate subdir 'pft1'. The generated jobs assume the data files have the same name as the copybook and are stored in subdir 'dat1'.

This means that no changes are required to the generated jobs, if you can copy/rename your data files into subdir 'dat1', but it us very simple to edit the generated jobs with the actual directory/filename where your data files currently exist.

If you have dozens or hundreds of copybooks/datafiles, there is an optional utility 'uvdata52', that uses a control file to relate copybooknames to datafilenames as it copies the jobs from 'pft1' to pft2'. Of course you have to 1st edit/create the control-file.

changing options m30 & v10

There are 2 times you can change the 'm' & 'v' options. #1 when you generate the jobs for 1 or all copybooks. #2 you may edit the generated jobs to change the defaults. After executing & investigating the 1st run reports, you may well want to rerun with expanded options for problem fields.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9B1. Data Summary Table Analysis & Validation

sample reports

 cm03v4  2011/01/30_12:25:42  Table Analysis all fields in dat1/custmas0x
 tbl#0005  tblt005f7d3s0m015v04 a60(16ce)
 line#     count  %    1strec#  10 CM-CITY pic x(16).
     1         2   6        23   CALGARY
     2         6  18        13   EDMONTON
     3         1   3        21   FORT MCMURRAY
     4         1   3         1   NANAIMO
     5         1   3        11   NORTH VANCOUVER
     6         1   3         9   PORT COQUITLAM
     7         1   3        30   PRINCE GEORGE
     8         1   3        26   THREE HILLS
     9         3   9         3   VANCOUVER
    10         1   3        16   WARBURG
    11         2   6        20   WHITEHORSE
    12         1   3         6  ~CHIL.IWACK          C3C8C9D3EEC9E6C1C3D24040404
    13         1   3         8  ~HOPE.               C8D6D7C5EE40404040404040404
    14         1   3         5  ~K.LOWNA             D2EED3D6E6D5C14040404040404
    15         2   6        10  ~VA.COUVER           E5C1EEC3D6E4E5C5D9404040404
    16         6  18            ~Invalid Total
              32*100             *TOTAL*
 cm03v4  2011/01/31_16:15:59  Table Analysis all fields in dat1/custmas0x
 tbl#0012  tblt012f7d3s2m020v04 a120(5pe)
 line#     count  %    1strec#  10 CM-THISYR-SALES pic s9(7)v99 comp-3 occurs 12.
     1        16  50         4                    0+
     2         1   3        17                 1443+
     3         1   3         3                 1528+
     4         1   3        31                 2487+
     5         1   3         2                 4680+
     6         1   3        32                 5246+
     7         1   3        14                21248-
     8         1   3        28                27483+
     9         1   3        27                38355+
    10         2   6         1              1234567+
    11         1   3         8  ~               123+ 000000123E
    12         1   3        15  ~           1200567+ 0012FF567C
    13         1   3        10  ~           1234007+ 001234AA7C
    14         2   6         5  ~           1234560+ 00123456EE
    15         6  18            ~Invalid Total
              32*100             *TOTAL*
Note
  • the following pages will illustrate how the data table analysis/validate
    jobs are generated & executed.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9C1. Data Summary Table Analysis & Validation

subdirs for generating & executing table summary jobs

 /home/userxx
 :-----tablesum
 :     :-----cpys               <-- subdir for COBOL COPYBOOKs
 :     :     :-----custmas0x.cpy  <-- copybook for customer master
 :     :     :-----sales0.cpy     <-- copybook for sales detail
 :     :     :
 :     :-----dat1               <-- subdir for test/demo DATA
 :     :     :-----custmas0x      <-- customer master data file
 :     :     :-----sales0         <-- sales detail data file
 :     :     :
 :     :-----maps               <-- subdir for RECORD LAYOUTs (cobmaps)
 :     :     :-----custmas0x      <-- cobmap for customer master
 :     :     :-----sales0         <-- cobmap for sales detail
 :     :     :
 :     :-----pft1               <-- subdir for table analysis JOBS
 :     :     :-----custmas0x      <-- table analysis job for customer master
 :     :     :-----sales0         <-- table analysis job for sales detail
 :     :     :
 :     :-----tbls               <-- subdir for data table analysis REPORTs
 :     :     :-----custmas0x      <-- table analysis report customer master
 :     :     :-----sales0         <-- table analysis report for sales detail
 :     :     :
 :     :-----tmp                <-- subdir for uvhd/uvhdcob print/verify files
Note
  • file-names here are the same in all subdirs
  • this makes the generation procedures easier
  • see further below for instructions to insert actual data-file-names
    into the generated jobs (using a control-file & utility 'uvdata52')

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9C2. Data Summary Table Analysis & Validation

setup subdirs in your homedir

We suggest you setup these subdirs within a table summary analyze/verify super directory within your homedir.


 #1. Login ---> your home dir
     =====

 #2. mkdir tablesum     <-- make superdir for subdirs
     ==============

 #3. mkdir cpys dat1 maps pft1 tbls tmp <-- make subdirs for generation/execution
     ==================================

 #4a. cp /home/uvadm/dat1/custmas0x dat1         <-- copy data-files
      ==================================
 #4b. cp /home/uvadm/dat1/sales0 dat1
      ===============================

 #5a. cp /home/uvadm/mf/cpys/custmas0x dat1   <-- copy COBOL copy-books
      =====================================
 #5b. cp /home/uvadm/mf/cpys/sales0 dat1
      ==================================

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9D1. Data Summary Table Analysis & Validation

Generate Table summary jobs - for 1 file at a time

Note
  • we will demo using 'custmas0x' & 'sales0' which are EBCDIC data files
  • corresponding copybooks custmas0x & sales0 makes generation easier
  • 'gentbl1E' is used to generate table analysis jobs for EBCDIC files
  • 'gentbl1A' would be used for ASCII files (custmas1 & sales1 are ASCII)

 #1. uvcopy cobmap1,fili1=cpys/custmas0x,filo1=maps/custmas0x
     ========================================================
     - convert copybook to record layout (cobmap)

 #2. uvcopy gentbl1E,fili1=maps/custmas0x,filo1=pft1/custmas0x,uop=m15v6
     ===================================================================
     - generate verify job from record layout
     - default options changed from 'uop=m30v10' as desired ('uop=m15v6' here)
     - m15 = max total lines, v6 = max invalid lines

 #3. vi pft1/custmas0x        <-- inspect output, modify for multi R/T's
     =================            (no manual changes required for custmas0x)

 #4. uvcopy pft1/custmas0x,[fili1=dat1/custmas0x,filo1=tbls/custmas0x]
     =================================================================
     - execute table analysis/verify job for customer master file
     - I/O subdirs/files will default as shown above in [...]

 #4a. uvcopy pft1/custmas0x   <-- execute (easier) [files default as above]
      =====================

 #5.  vi tbls/custmas0x       <-- inspect table analysis/verify report
      =================
 #5a. uvlp12 tbls/custmas0x  <-- print table analysis/verify report
      =====================
Note
  • the following pages will repeat the above instructions, BUT with
    illustrations of the generated jobs & reports

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9E1. Data Summary Table Analysis & Validation

generate table summary job - Illustrated


 #1. uvcopy cobmap1,fili1=cpys/custmas0x,filo1=maps/custmas0x
     ========================================================
     - convert copybook to record layout (cobmap)

cobmap to generate table summary job

 cobmap1  start-end bytes for cobol record fields    201012221438  pg# 0001
 cpys/custmas0x.cpy                                 RCSZ=00256  bgn  end  lth typ
 * custmas0x - cobol copy book for custmas0x file (see uvtrain).
      10 cm-num           pic  9(6).                          0000 0005  006 n  6
      10 cm-delete        pic  x(4).                          0006 0009  004
      10 cm-name          pic  x(25).                         0010 0034  025
      10 cm-adrs          pic  x(25).                         0035 0059  025
      10 cm-city          pic  x(16).                         0060 0075  016
      10 filler001        pic  x.                             0076 0076  001
      10 cm-prov          pic  x(2).                          0077 0078  002
      10 filler002        pic  x.                             0079 0079  001
      10 cm-postal        pic  x(10).                         0080 0089  010
      10 cm-phone         pic  x(12).                         0090 0101  012
      10 cm-contact       pic  x(18).                         0102 0119  018
      10 cm-thisyr-sales  pic  s9(7)v99 comp-3 occurs 12.     0120 0124  005pns 9
      10 cm-lastyr-sales  pic  s9(7)v99 comp-3 occurs 12.     0180 0184  005pns 9
      10 filler003        pic  x(16).                         0240 0255  016
 *RCSZ=00256

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9E2. demo generate/execute table summary for custmas0x

generate table summary job


 #2. uvcopy gentbl1E,fili1=maps/custmas0x,filo1=pft1/custmas0x,uop=d4m15v4
     =====================================================================
     - run gentbl1E to generate table summary from copybook 'custmas0x'
     - displays option defaults & prompts for option changes
     - see 'change option defaults *NOTE*' below
 gentbl1E - generate uvcopy jobs to table analyze all fields in copybook
 uop=f0d3m030v10 - option defaults
     f0          - Input file typ=RSF fixed lth (in generated job)
     f2          - Input typ=RDWz2 (2 byte BIG-end binary recsize hdrs)
     f4          - Input file typ=RDWz4 (4 byte hdrs, bytes 3&4 nulls)
     f8          - Input file typ=RDWz8 (BDW + RDW)
       d3        - convert invalid digits in numeric fields to zeros
       d4        - clear invalid numeric fields, show hex on rightside
         m030    - max tbl entries, lower/higher added mid-point removed
             v10 - max Invalid entries, excess added for ~Invalid Total
 110313:150859:gentbl1E: uvcopy ver=20110313 pf=/home/uvadm/pf/IBM/gentbl1E
 uvcopy DISAM ext=.dat LNX L64  license=110313_00V_930601 site=UV_Software
 User OPtion (uop) defaults  = q1f0d3m030v10
 -->null to accept or enter/override --> d4m15v4 <------- change options *NOTE*
 110127:163805:gentbl1E: EOF fili01 rds=19 size=1309: maps/custmas0x
 110127:163805:gentbl1E: EOF filo01 wrts=47 size=1719: pft1/custmas0x

Note re Options

  1. Note default options (listed above) may be changed by entering overrides on the command line (see #2. uvcopy ...,uop=d4m15v4 above), OR entering the override options at the prompt (see -->null accept...--> d4m15v4 <--). We entered override options both places for illustration.

  2. See generated job listed on the next page & note effects of user options. uop=f0 (default) codes fili1=... as 'typ=RSF'. Enter option 'f4' to code input file type as 'typ=RDW'.

  3. Note effect of override options 'uop=d4m15v4' on generated tbl instructions. Here is the 1st 'tbl' for default options & for the override options


      tblt001f7d3s1m030v10 a0(6ze),'10 CM-NUM pic 9(6).' <-- default options
      =========**==*******==============================
      tblt001f7d4s1m015v04 a0(6ze),'10 CM-NUM pic 9(6).' <-- sample overrides
      =========**==*******==============================
d3
  • default converts any invalid digits in numeric fields to zeros
d4
  • clears any numeric fields with invalid digits/signs (shows hex on right)
m030
  • default max valid entries in table, after which higher/lower entries
    added, but mid-point entry removed to maintain max 30
m015
  • changes max valid entries to 15
v10
  • default max invalid entries, after which additional invalid entries
    are counted for the ~Invalid Total entry
v04
  • changes max invalid entries to 4

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9E3. demo generate/execute table summary for custmas0x

generated table summary job

 # uvcopy job to generate jobs to table analyze all fields in copybook
 opr='jobname=JOBNAME   - pft2 name=datafilename'
 opr='copybook=custmas0x - maps/pft1 name=copybookname'
 arg1=JOBNAME,arg2=custmas0x
 rop=u400v500000
 was=a33000b33000
 fili1=?dat1/custmas0x,rcs=00256,typ=RSF
 filo1=?tbls/custmas0x,rcs=128,typ=LSTt
 @run
        opn     all
 loop   get     fili1,a0
        skp>    eof
        mvc     b0(00256),a0
        tra     b0(00256)           translate to Ascii
        tblt001f7d3s1m015v04 a0(6ze),'10 CM-NUM pic 9(6).'
        tblt002f7d3s0m015v04 a6(4ce),'10 CM-DELETE pic x(4).'
        tblt003f7d3s0m015v04 a10(25ce),'10 CM-NAME pic x(25).'
        tblt004f7d3s0m015v04 a35(25ce),'10 CM-ADRS pic x(25).'
        tblt005f7d3s0m015v04 a60(16ce),'10 CM-CITY pic x(16).'
        tblt006f7d3s0m015v04 a76(1ce),'10 FILLER001 pic x.'
        tblt007f7d3s0m015v04 a77(2ce),'10 CM-PROV pic x(2).'
        tblt008f7d3s0m015v04 a79(1ce),'10 FILLER002 pic x.'
        tblt009f7d3s0m015v04 a80(10ce),'10 CM-POSTAL pic x(10).'
        tblt010f7d3s0m015v04 a90(12ce),'10 CM-PHONE pic x(12).'
        tblt011f7d3s0m015v04 a102(18ce),'10 CM-CONTACT pic x(18).'
 #
        mvn     $ra,0
        mvn     $rj,0
 oci010 tblt012f7d3s2m015v04 aa120(5pe),'10 CM-THISYR-SALES pic s9(7)v99 comp-3 occurs 12.'
        add     $ra,00005
        add     $rj,00001
        cmn     $rj,00012
        skp<    oci010
 #
        mvn     $ra,0
        mvn     $rj,0
 oci020 tblt013f7d3s2m015v04 aa180(5pe),'10 CM-LASTYR-SALES pic s9(7)v99 comp-3 occurs 12.'
        add     $ra,00005
        add     $rj,00001
        cmn     $rj,00012
        skp<    oci020
 #
        tblt014f7d3s0m015v04 a240(16ce),'10 FILLER003 pic x(16).'
        skp     loop
 #
 eof    tbp     filo1,'Table Analysis all fields in $fili1'
        cls     all
        eoj
Note
  • thisyear/lastyear sales occurs 12 handled by index register & code loop

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9E4. demo generate/execute table summary for custmas0x

modify table summary job before execution

Before illustrating the execution of the table summary job, we will reduce the number of fields/tables from 14 to 3 (manageable for this documentation).


 #3a. cp pft1/custmas0x pft1/cm03v4  <-- copy/rename to preserve original
      =============================

 #3b. vi pft1/cm03v4   <-- edit generated job, reduce to 3 fields
      ==============     - edited result below (original above)
 # uvcopy job to generate jobs to table analyze all fields in copybook
 # cm03v4 - custmas0x stripped down to just 3 fields & drop crossfoot 12 mths
 #        - also see cm03v4a, just 1 field (sales) but also accumulated
 #        - output name changed to $jobname to preserve each jobs output
 rop=u400v500000
 was=a33000b33000
 fili1=?dat1/custmas0x,rcs=00256,typ=RSF
 filo1=?tbls/$jobname,rcs=128,typ=LSTt
 @run
        opn     all
 loop   get     fili1,a0
        skp>    eof
        tblt001f7d3s1m015v04 a0(6ze),'10 CM-NUM pic 9(6).'
        tblt005f7d3s0m015v04 a60(16ce),'10 CM-CITY pic x(16).'
 oci010 tblt012f7d3s2m015v04 a120(5pe),'10 CM-THISYR-SALES pic s9(7)v99 comp-3 occurs 12.'
        skp     loop
 #
 eof    tbp     filo1,'Table Analysis all fields in $fili1'
        cls     all
        eoj

Notes

  1. We renamed as cm03v4 to indicate 3 fields & option v10 reduced to v4.

  2. We eliminated the code loop (at tag 'oci010') to simplify the job. Our simplified job summarizes only the 1st of the 12 monthly sales fields. 'CM-THISYR-SALES' is now actually just 'January sales'.

  3. Sales field definition 'aa120(pe)' changed to 'a120(5pe)', removed register 'a' (2nd 'a' of aa120(pe)), but would not matter (would be 0 for 1st field).

  4. We eliminated some other non-essential code. 'mvc' & 'tra' generated in case of a multi-record-type file & you want to display bad records in ASCII.

  5. We changed the name of the output file, see line 8 before/after as follows:


 filo1=?tbls/custmas0x     <-- as generated in original job 'custmas0x'
 =====================
 filo1=?tbls/$jobname     <-- changed to '$jobname' to not overwrite custmas0x
 ====================       - output report will same as jobname ('cm03v4' here)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9E5. demo generate/execute table summary for custmas0x

Execute table summary job


 #4. uvcopy pft1/cm03v4,[fili1=dat1/custmas0x,filo1=tbls/custmas0x]
     ==============================================================
     - execute table analysis/verify job for customer master file
     - I/O subdirs/files will default as shown above in [...]

 #4a. uvcopy pft1/custmas0x   <-- execute (easier) [files default as above]
      =====================

table summary report

 cm03v4  2011/01/31_16:15:59  Table Analysis all fields in dat1/custmas0x
 tbl#0001  tblt001f7d3s1m015v04 a0(6ze)
 line#     count  %    1strec#  10 CM-NUM pic 9(6).
     1         1   3         1               120140
     2         1   3         2               132588
     3         1   3         3               139923
     4         1   3         4               142175
     5         1   3         6               147615
     6         1   3         7               149304
     7         1   3        13               201120
     8         1   3        14               204700
     9         1   3        16               211140
    10         1   3        26               315512
    11         1   3        31               403887
    12         1   3         5  ~            105264  F1EEF5F2F6F4
    13         1   3        10  ~            105688  F1EEF5F6F8F8
    14         1   3        15  ~            205060  F2F0F5EEF6F0
    15         1   3        20  ~            234090  F2F3F4EEF9F0
    16         6  18            ~Invalid Total
              32*100             *TOTAL*
 cm03v4  2011/01/31_16:15:59  Table Analysis all fields in dat1/custmas0x
 tbl#0005  tblt005f7d3s0m015v04 a60(16ce)
 line#     count  %    1strec#  10 CM-CITY pic x(16).
     1         2   6        23   CALGARY
     2         6  18        13   EDMONTON
     3         1   3        21   FORT MCMURRAY
     4         1   3         1   NANAIMO
     5         1   3        11   NORTH VANCOUVER
     6         1   3         9   PORT COQUITLAM
     7         1   3        30   PRINCE GEORGE
     8         1   3        26   THREE HILLS
     9         3   9         3   VANCOUVER
    10         1   3        16   WARBURG
    11         2   6        20   WHITEHORSE
    12         1   3         6  ~CHIL.IWACK          C3C8C9D3EEC9E6C1C3D24040404
    13         1   3         8  ~HOPE.               C8D6D7C5EE40404040404040404
    14         1   3         5  ~K.LOWNA             D2EED3D6E6D5C14040404040404
    15         2   6        10  ~VA.COUVER           E5C1EEC3D6E4E5C5D9404040404
    16         6  18            ~Invalid Total
              32*100             *TOTAL*

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9E6. demo generate/execute table summary for custmas0x

 cm03v4  2011/01/31_16:15:59  Table Analysis all fields in dat1/custmas0x
 tbl#0012  tblt012f7d3s2m020v04 a120(5pe)
 line#     count  %    1strec#  10 CM-THISYR-SALES pic s9(7)v99 comp-3 occurs 12.
     1        16  50         4                    0+
     2         1   3        17                 1443+
     3         1   3         3                 1528+
     4         1   3        31                 2487+
     5         1   3         2                 4680+
     6         1   3        32                 5246+
     7         1   3        14                21248-
     8         1   3        28                27483+
     9         1   3        27                38355+
    10         2   6         1              1234567+
    11         1   3         8  ~               123+ 000000123E
    12         1   3        15  ~           1200567+ 0012FF567C
    13         1   3        10  ~           1234007+ 001234AA7C
    14         2   6         5  ~           1234560+ 00123456EE
    15         6  18            ~Invalid Total
              32*100             *TOTAL*

Notes

  1. Invalid digits in packed/numeric fields are converted to '0' zeros.

  2. Invalid digits 'AA' in hex display on right are converted to '00' in '1234007+' left side of table argument.

  3. Note the options on the 'tbl' instruction shown in the table headings:


      tbl#0012  tblt012f7d3s2m015v04 aa120(5pe)
      =========================================

See all tbl options at https://uvsoftware.ca/datautilities.htm#tbl. Here is a description of the options used in this case:

t012
  • the Table#
f7
  • table Format, 'f7' allows 48 bytes for the argument field
d3
  • default converts any invalid digits in numeric fields to zeros
d4
  • clears any numeric fields with invalid digits/signs (shows hex on right)
s2
  • edits signed fields with '+' or '-' (s1 would show '-' only)
m015
  • max entries, after 15, higher/lower entries added and a mid-point entry
    removed to maintain entries at 15.
v04
  • max Invalid entries, after 4, invalid entries are just counted for the
    '~Invalid Total' line count.
  1. The entry count field will not necessarily add up due to options 'm' & 'v' There were 32 records in the file & option 'm15' limited entries shown in tables #1 & #5 on previous page.

  2. Option 'm15' did not limit entries in table #12 above, but option 'v4' did. You see the true number of ~Invalid entries on the ~Invalid Total line.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9E7. demo generate/execute table summary for custmas0x

option d4 clear invalid numeric fields

If we had specified option 'd4' (vs default 'd3') the table on the prior page would then appear as shown below - with invalid numeric fields cleared, but still showing the hexadecimal value on the right.

 cm03v4  2011/01/31_16:15:59  Table Analysis all fields in dat1/custmas0x
 tbl#0012  tblt012f7d3s2m020v04 a120(5pe)
 line#     count  %    1strec#  10 CM-THISYR-SALES pic s9(7)v99 comp-3 occurs 12.
     1        16  50         4                    0+
     2         1   3        17                 1443+
     3         1   3         3                 1528+
     4         1   3        31                 2487+
     5         1   3         2                 4680+
     6         1   3        32                 5246+
     7         1   3        14                21248-
     8         1   3        28                27483+
     9         1   3        27                38355+
    10         2   6         1              1234567+
    11         1   3         8  ~                    000000123E
    12         1   3        15  ~                    0012FF567C
    13         1   3        10  ~                    001234AA7C
    14         2   6         5  ~                    00123456EE
    15         6  18            ~Invalid Total
              32*100             *TOTAL*
  1. You could modify option 'd3' to 'd4' on the 'tbl' instructions in the generated table summary job pft1/custmas0x (in this example).


 #1a. vi pft1/custmas0x   <-- edit to modify options
      =================
      tbl#0012  tblt012f7d3s2m015v04 aa120(5pe)  <-- d3 default
      ===================**====================
      tbl#0012  tblt012f7d4s2m015v04 aa120(5pe)  <-- d4 modified
      ===================**====================

 #1b. uvcopy pft1/custmas0x   <-- rerun the table summary job
      =====================
  1. OR, you could regenerate the table summary jobs, specifying override option 'd4'


 #2a. uvcopy gentbl1E,fili1=maps/custmas0x,filo1=pft1/custmas0x,uop=d4m15v4
      =====================================================================
      - regenerate table summary job with desired options
  1. Regenerating is best if you are creating table summary jobs for all copybooks


 #3a. uvcopyx gentbl1E maps pft1 uop=q0i7d4m15v4
      ==========================================
      - generate table sumary jobs for ALL copybooks (cobmaps)

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9F1. Data Summary Table Analysis & Validation

further modify table summary job & rerun

If desired you can add up to 6 accumulator fields on the right side of the table argument. To illustrate we will reduce out table job to just the 1 table for January sales & define the Jan sales field as an accumulator on the right.


 #3a. cp pft1/cm03v4 pft1/cm03v4a  <-- copy/rename to preserve original
      ===========================

 #3b. vi pft1/cm03v4a  <-- edit generated job, reduce to 1 field (Jan sales)
      ===============    - edited result below (before editing on page '9E4').
 # uvcopy job to generate jobs to table analyze all fields in copybook
 # cm03v4a - custmas0x stripped down to just 1 fields & drop crossfoot 12 mths
 #         - table analyze just Jan sales, but also accumulate
 #         - output name changed to $jobname to preserve each jobs output
 rop=u400v500000
 was=a33000b33000
 fili1=?dat1/custmas0x,rcs=00256,typ=RSF
 filo1=?tbls/$jobname,rcs=128,typ=LSTt
 @run
        opn     all
 loop   get     fili1,a0
        skp>    eof
 oci010 tblt012f7d3s2m015v04 a120(5pe),'Jan-Sales pic s9(7)v99 comp-3;Jan-sales',a120(5pe)
        skp     loop
 #
 eof    tbp     filo1,'Table Analysis all fields in $fili1'
        cls     all
        eoj

Notes

  1. Compare this to the 'before change' job on page '9E4'.

  2. We defined accumulator field 'a120(5pe)' on the right side of the 'tbl' instruction.

  3. We also inserted a field heading ';Jan-acum' in the field description.

  4. ';'s are used as field separators in the 'heading line'. When executed uvcopy uses them to aligh the field-names over the argument & accumulators

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9F2. demo generate/execute table summary for custmas0x

Execute table summary job


 #4. uvcopy pft1/cm03v4a,[fili1=dat1/custmas0x,filo1=tbls/custmas0x]
     ===============================================================
     - execute table analysis/verify job for customer master file
     - I/O subdirs/files will default as shown above in [...]

 #4a. uvcopy pft1/custmas0x   <-- execute (easier) [files default as above]
      =====================
 cm03v4a  2011/01/31_16:33:30  Table Analysis all fields in dat1/custmas0x
 tbl#0012  tblt012f7d3s2m015v04 a120(5pe)                           -acum#1-    %
 line#     count  %    1strec#  Jan-Sales pic s9(7)v99 comp-3      Jan-sales
     1        16  50         4                    0+                       0    0
     2         1   3        17                 1443+                   1,443    0
     3         1   3         3                 1528+                   1,528    0
     4         1   3        31                 2487+                   2,487    0
     5         1   3         2                 4680+                   4,680    0
     6         1   3        32                 5246+                   5,246    0
     7         1   3        14                21248-                  21,248-   0
     8         1   3        28                27483+                  27,483    0
     9         1   3        27                38355+                  38,355    0
    10         2   6         1              1234567+               2,469,134   33
    11         1   3         8  ~               123+ 000000123E          123    0
    12         1   3        15  ~           1200567+ 0012FF567C    1,200,567   16
    13         1   3        10  ~           1234007+ 001234AA7C    1,234,007   16
    14         2   6         5  ~           1234560+ 00123456EE    2,469,120   33
    15         6  18            ~Invalid Total
              32*100             *TOTAL*                           7,467,492 *100

Notes

  1. To fit this documentation, we edited out 18 spaces between the table argument & the accumulator.

  2. Removing 18 spaces was possible here because the table argument allows up to 18 digits for packed/numeric & 28 bytes for the hex display.

  3. We could define up to 6 accumulators on the right of the table argument. The max line length would then be 203 bytes.


    4. See https://www.uvsoftware.ca/uvcopy3.htm#tbl for complete details.

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9G1. Data Summary Table Analysis & Validation

using 'uvhdcob' to investigate problems

'uvhdcob' is the best tool to investigate problems reported by the table summary jobs (generated by 'gentbl1E' from COBOL copybooks).

After examining the 'custmas0x' table summaries on the last few pages '9E5' thru '9F2', you might want to see the record#s reported to have invalid fields. Note that the table summaries report the 1st record# with each invalid data field reported & record #5 is the 1st on those reports.


 uvhdcob dat1/custmas0x maps/custmas0x a   <-- run uvhdcob on datafile & copybook
 =====================================
 --> displays 1st record & prompts for command
     - 1st record display omitted here
 --> 5     <-- enter 5 to see 1st record reported to have errors
 --> v99   <-- OR use verify command to search for records with invalid fields
 rec#       5           occurs  bgn end typ<------ data (hex if typ=p/b) ----->
 001 cm-num                       0    5 n  1.5264      F1EEF5F2F6F4      <-BadN
 002 cm-delete                    6    9      D
 003 cm-name                     10   34    MAGRATH SUPPLIES LTD.
 004 cm-adrs                     35   59    1939 KIRSCHNER ROAD
 005 cm-city                     60   75    K.LOWNA     D2EED3D6E6D5C1404 <-BadC
 006 filler001                   76   76
 007 cm-prov                     77   78    BC
 008 filler002                   79   79
 009 cm-postal                   80   89    V1Y4N7
 010 cm-phone                    90  101    250-843-2987
 011 cm-contact                 102  119
 012 cm-thisyr-sales       012  120  124pns 00123456EE                    <-BadP
 013 cm-lastyr-sales       012  180  184pns 000000000C
 014 filler003                  240  255    C 19970131
 rec#=5 rcount=32 rsize=256 fsize=8192 fptr=1024
 k1=HelpPrompts,?=HelpScreens -->

notes

  1. Note flags on right hand side '<-BadN', '<-BadC', '<-BadP'

  2. cm-num '1.5264' non-numeric, the '.' is the x'EE' in F1EEF5F2F6F4 <-BadN

  3. cm-city 'K.LOWNA' unprintable character, x'EE' in D2EED3D6E6D5C1404 <-BadC

  4. cm-thisyr-sales packed field invalid digit/sign x'EE' 00123456EE <-BadP

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9H1. Data Summary Table Analysis & Validation

operating instructions - All files in directory


 #1. uvcopyx cobmap1 cpys maps uop=q0i7p0
     ====================================
     - convert all COBOL copybooks to layouts

 #2. uvcopyx gentbl1E maps pft1 uop=q0i7f0m15v6
     ==========================================
     - convert all cobmaps to table analysis/verify jobs
     - options 'uop=q0i7f0m15v6'
       q0i7 - inhibits prompts (desired when generating multiple jobs)
       f0   - Fixed Length files (as specified in copybook/map
       m15 = max total lines, v6 = max invalid lines

 #3. vi pft1/*          <-- inspect all generated jobs, modify for multi R/T's
     =========              (no changes required for custmas0x or sales0)

 #4. uvcopyxx 'pft1/*'  <-- execute all jobs (I/O files coded in jobs)
     =================    - input files in dat1/... output reports in tbls/...

 #5. vi tbls/*         <-- inspect all table analysis/verify reports
     =========

 #5a. uvlpd12 tbls     <-- print all table analysis/verify reports
      ============       - 'uvlpd12' prints all files in subdir
                         - vs 'uvlp12' (used earlier) to print 1 file
Note
  • generated table analysis jobs are named the same as the copybook/maps
    & the I/O filenames within the jobs are the same as the copybook/maps
  • this makes the generation procedures easier
  • if you can rename your data-files the same as the copybooks/maps/jobs
    & store in subdir dat1/..., you can run the jobs with no manual changes
  • see below for instructions to insert actual data-file-names
    into the generated jobs (using a control-file & utility 'uvdata52')

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

9I1. Data Summary Table Analysis & Validation

inserting data-file-names in table analysis jobs


 #0a. mkdir ctl

 #0b. mkdir pft2

 #0. vi ctl/ctlfile54    <-- create control file for uvdata52
     ================      - to relate copybook-names to data-file-names
                           - see sample below
     ar.customer.master     cpy=custmas    typ=RSF   rcs=00256
     ar.sales.detail        cpy=sales      typ=RSF   rcs=00064
     ap.vendor.master       cpy=vendormas  typ=RDWz4 rcs=08192
Note
  • will repeat Instructions to generate all jobs with control fiel
  • similar to page above with insert of 'uvdata52'

 #1. uvcopyx cobmap1 cpys maps uop=q0i7p0
     ====================================
     - convert all COBOL copybooks to layouts

 #2. uvcopyx gentbl1E maps pft1 uop=q0i7f0m15v6
     ==========================================
     - convert all cobmaps to table analysis/verify jobs

 #3. uvcopy uvdata52,fili1=ctl/ctlfile54,fild2=pft1,fild3=pft2
     =========================================================
     - insert datafilenames (vs copybooknames) while copying pft1 to pft2/...
     - also renames output jobs same as datafilenames (not copybooknames)

 #4. vi pft2/*          <-- inspect all generated jobs, modify for multi R/T's
     =========              (no changes required for custmas0x or sales0)

 #5. uvcopyxx 'pft2/*'  <-- execute all jobs (I/O datafilenames coded in jobs)
     =================    - input files in dat1/... output reports in tbls/...

 #6. vi tbls/*         <-- inspect all table analysis/verify reports
     =========

 #6a. uvlpd12 tbls     <-- print all table analysis/verify reports
      ============       - 'uvlpd12' prints all files in subdir
                         - vs 'uvlp12' (used earlier) to print 1 file

Goto:   Begin this doc End this doc Index this doc Contents this library UVSI Home-Page

Visitor Counters for ThisYear and LastYear