ADMjobs - Contents

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

ADMjobs.doc - Applications Administration for Unix/Linux

ADMjobs - Contents (continued)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_0. ADMjobs.doc - Applications Administration for Unix/Linux

Introduction & Overview

ADMjobs.doc discusses several subjects that are vital to any successful unix or linux installation (filesystems, backup/restore, cron, console logging,etc).

ADMjobs.doc is intended to be used with the following items, which document conversion of IBM mainframes to unix & linux.

MvsJclPerl.htm - MVS JCL Conversion to Perl script

We will emphasize the importance of the automatic backup system. This should be setup very soon after installation so that you have protection from inadvertently wiping out your programs & jobs (before you become a Unix/Linux expert).

It is a great comfort to know that every night your libraries & data are automatically backed up to the DAT tape & also saved to alternate disc directories for convenient & immediate recovery.

Part_1 will document setting up user profiles for uvadm, appsadm, programmers, & operators. This repeats some of the install.htm instructions, but this is vital and is related to this document (backup scripts, console logging, etc).

Copyright 1993-2009, UV Software Inc.

Owen Townsend, UV Software, 4667 Hoskins Rd., North Vancouver BC, V7K2R3

 Tel: 604-980-5434 Fax: 604-980-5404

mailto:owen@uvsoftware.ca

http://www.uvsoftware.ca

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_0. ADMjobs.doc - Applications Administration for Unix/Linux

References & Books for Unix/Linux Administration

1. Red Hat Enterprise Installation guide. I recommend you download the 350 page .pdf document & print Duplex on 3 hole punched paper for 3 ring binder.
   
   
   

   www.redhat.com -> support -> documentation -> Red Hat Enterprise Linux

2. Red Hat Enterprise Deployment guide. I recommend you download the 900 page .pdf document & print Duplex on 3 hole punched paper for 3 ring binder.
   
   
3. Red Hat Enterprise Linux & Fedora 4th Edition, O'Reilly, Bill McCarty
   
   
4. Red Hat Linux Networking & System Adminitration 3rd Edition, Wiley, Terry Collings & Kurt Wall
   
   
5. The New Kornshell, Prentice Hall, Morris Bolsky & David Korn
   
   
6. The C Programming Language, Prentice Hall, Brian Kernighan & Dennis Ritchie
   
   

   useful Downloads

Here are a few downloads relevant to mainframe conversions to Unix/Linux.

Look for the binary matching your Unix/linux architecture

See more info & configuration on pages '8F1' - 8F3.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_1. ADMjobs.doc - Applications Administration - setup profiles

Setting up profiles - Contents

 1C4. utilities_profile - simplified profile to access programs & scripts only
     - alternative to above 3 part profiles (G1,G2,G3), which are recommended
       for sites converting from a mainframe

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1A1. ADMjobs.doc - Applications Administration for Unix/Linux

uvadm - Vancouver Utility home directory

 /home/uvadm
 :-----batDOS            - BAT files for VU on native Windows
 :-----binDOS            - binaries for VU on native Windows (cc by lcc-win32)
 :-----bin             <-- binaries (uvcopy,uvsort,etc) distros are RedHat Linux
 :-----binSFU            - binaries for SFU (Services For Unix) on Windows
 :-----ctl               - control files for various purposes
 :-----dat1              - test data files
 :-----doc               - Vancouver Utilities documentation (text)
 :-----dochtml           - documentation in HTML (same as on www.uvsoftware.ca)
 :-----env         <------ profiles for Unix/Linux, SFU, Cygwin,& Uwin
 :-----hdr               - hdr files for C compiles
 :-----htmlcode          - merged into dochtml when text converted to HTML
 :-----lib               - libraries for C compiles (subfunctions,DISAM,etc)
 :-----mvstest         <-- test/demos for MVS JCL/COBOL mainframe conversions
 :     :-----...         - many subdirs omitted, see MVSJCL.htm
 :-----perlm           <-- Perl Modules (support JCL conversions to Perl scripts)
 :-----perls           <-- Perl Scripts (few, most scripts are ksh in sf/.../...)
 :-----pf              <-- Parameter Files for uvcopy & uvqrpg
 :     :-----adm         - administrative jobs
 :     :-----demo        - demo jobs
 :     :-----IBM         - IBM mainframe conversion jobs
 :     :-----util        - utility jobs
 :-----sf              <-- Script Files
 :     :-----adm         - administrative scripts
 :     :-----demo        - demo scripts
 :     :-----IBM         - IBM mainframe conversion scripts
 :     :-----util        - utility scripts
 :-----sfun              - ksh functions (used in converted JCL/scripts)
 :-----src             <-- Vancouver Utilities C source code
 :-----srcf              - C source for various sub-functions
 :-----tf                - test files for various examples in doc
 :-----tmp               - tmp subdir (test/demo outputs)
 :-----vsetest         <-- test/demos for VSE JCL/COBOL mainframe conversions
 :     :-----...         - many subdirs omitted, see VSEJCL.htm

The profiles (listed on the following pages) are intended to be used with uvadm, appsadm, mvstest,& vsetest. You will need only minor changes to use for your programmers & operators.

The /home/uvadm sub-directories are illustrated here to clarify the procedures required should you find reasons to modify any of the Vancouver Utility scripts or uvcopy jobs or programs at your site.

Note that the uvadm subdirs for 'sf' (script files) & 'pf' (uvcopy parameter files) (or uvcopy jobs) are further sub-directoried as shown above, but there is no need for you to subdirectory the appsadm/sf or appsadm/pf.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1B1. ADMjobs.doc - profiles for Vancouver Utilities

'profiles' provided in /home/uvadm/env

 /home/uvadm/env      <--- profiles provided here
 :-----common_defines    - defines TEST/PROD LIBS/DATA so stub_profile
 :                         can use $symbols to define RUNLIBS/RUNDATA
 :-----common_profile    - common profile (called by stub_profile)
 :                         defines PATH's etc using $RUNLIBS/$RUNDATA
 :-----stub_profile      - will be renamed .profile (ksh) or .bash_profile (bash)
 :                       - defines RUNLIBS/RUNDATA for programmers & operators
 :-----root_profile      - profile for root, copy to /root/.bash_profile (RedHat)
 :                         to access Vancouver Utility scripts & uvcopy jobs
 :
 /home/appsadm/env    <--- setup user 'appsadm' & copy from /home/uvadm/env/*
 :-----common_defines    - customize profiles here in /home/appsadm/env/...
 :-----common_profile    - common profile (called by stub_profile)
 :-----stub_profile      - customize & copy to homedirs .profile or .bash_profile

Mainframe conversion sites should setup an application administrator userid 'appsadm', copy /home/uvadm/env/* to /home/appsadm/env,& customize profiles there depending on the locations of their libraries & data.

Do NOT customize profiles in /home/uvadm/env/... because they would be overwritten when a new version of Vancouver Utilities is installed.

We recommend the concept of 'stub' & 'common' profiles. The shell profile in each user's homedir is a 'stub' that calls 'common_defines' & 'common_profile' which are stored in /home/appsadm/env/...

1. 'common_defines' defines locations of PRODLIBS,PRODDATA,TESTLIBS,TESTDATA,& software package HOME dirs (uvadm, COBDIR, ORACLE_BASE, ORACLE_HOME, etc)
   
   
2. The supplied 'stub_profile' is copied to homedirs, renamed as '.bash_profile' for bash shell (Linux) & '.profile' for Korn shell (Unix), and then modified depending on whether the user is a programmer or operator.
   

• 'programmer' stub_profiles define RUNLIBS=$TESTLIBS & RUNDATA=$TESTDATA.
• 'operator' stub_profiles define RUNLIBS=$PRODLIBS & RUNDATA=$PRODDATA.

3. 'common_profile' then defines the 'PATH's using $RUNLIBS,$COBDIR,$UV,etc For example: export PATH=$PATH:$RUNLIBS/jcls (converted JCL/scripts).
   
   
4. '$RUNDATA' determines data-file locations indirectly as follows: $RUNDATA defines the superdir housing all data-files. All JCL/scripts call a common function 'jobset51' which changes directory to $RUNDATA (cd $RUNDATA). The JCL converter inserts jobset51 at the begining of all converted JCL/scripts and then addresses all data files relative to $RUNDATA.
   

Note that stub profiles must call 'common_defines' & 'common_profile' using '.' (dot execution), which means the 'export's made in the common_defines & common_profile will still be effective on return to the users profile.

This system is a big advantage for any site with multiple users, it means the sysadmin can update common_defines & common_profile once in 1 place (actually 2 places) & those changes are effective for all users.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1B2. ADMjobs: profiles for Vancouver Utilities on Unix/Linux

Profiles are vital for Unix/Linux

Profiles are vital to the success of any Unix/Linux site and especially for sites converting from a mainframe. You can not find much practical advice on 'how to setup user profiles' in books or on the internet, so here are my methods which have been proven successful at over 50 conversion sites.

UV Software supplies the recommended profiles with the Vancouver Utilities, or you can save them from http://www.uvsoftware.ca/admjobs.htm#1C1 thru 1C4. BUT, before you try to use them, it is important to understand the concepts. I assume the reader has some basic understanding of profile functions.

The most important profile function is to define search 'PATH's to scripts, programs,& (indirectly) to data-files. Some other functions are to define aliases, terminal types,& to capture console logs.

Without direction, an inexperienced unix programmer would probably define everything (PATHs,aliases,etc) in the profile in his home directory. Then the 1st programmer's profile might be copied to the homedirs of other programmers working on the same system.

You can see a big problem developing - when they need to change search PATHs, etc, they would have to update the multiple profiles in the homedirs of all programmers & operators.

Here is a better system to overcome the problem described above, and to provide many other benefits described further below. The solution is to split the profile in 3 parts (common_defines, stub_profile,& common_profile).

common_defines

• distributed in /home/uvadm/env/common_defines
• should be copied to & maintained in /home/appsadm/env/common_defines
• maintained by 'appsadm' (the applications administrator), in his homedir.

• defines (exports) locations of TESTLIBS,TESTDATA,PRODLIBS,PRODDATA
  (superdirs with multiple subdirs to store scripts,programs,& data-files)
• TESTLIBS,TESTDATA,PRODLIBS,PRODDATA are environmental variables defined
  in the common_defines for use by the stub_profile & common_profile.

• common_defines also defines the BASE/HOME 'super-directories' for software
  packages such as Vancouver Utilities (UV), Micro Focus COBOL (COBDIR),
  Oracle (ORACLE_BASE & ORACLE_HOME), etc.
• these definitions (exported $SYMBOLs) will be used in the common_profile
  to define the search PATHs to scripts, programs, data-files.
• On exceptional occasions the common_defines definition might be overridden
  in the stub_profile of sys admins to test new versions of software packages.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

stub_profile

• distributed in /home/uvadm/env/stub_profile
• should be copied to & maintained in /home/appsadm/env/stub_profile
• 'stub_profile' is the name assigned by UV Software
• copied to user homedirs & renamed as .profile for the Korn shell (Unix)
  or .bash_profile for the Bash shell (Linux).
• modified depending on whether the user is a programmer or operator.

• 'programmer' stub_profiles define RUNLIBS=$TESTLIBS & RUNDATA=$TESTDATA.
• 'operator' stub_profiles define RUNLIBS=$PRODLIBS & RUNDATA=$PRODDATA.

• the stub_profile 1st calls 'common_defines', then defines RUNLIBS/RUNDATA,
  before calling the 'common_profile', which uses the $RUNLIBS/$RUNDATA
  definitions to setup PATH's to search for scripts, programs,& data-files.

• note that the stub_profile must '.' (dot-execute) the common_defines &
  common_profile to retain their definitions on return to the stub_profile

common_profile

• distributed in /home/uvadm/env/common_profile
• should be copied to & maintained in /home/appsadm/env/common_profile
• maintained by the applications administrator (in his homedir)

• for Vancouver Utility mainframe conversion sites, the common_profile defines
  PATH's to JCL/scripts, programs,& data-files, by using $RUNLIBS/$RUNDATA
  environmental variables, which were defined in the stub_profile, using
  $TESTLIBS/$TESTDATA or $PRODLIBS/$PRODDATA, which were defined in the
  common_profile.

• for software packages (Vancouver Utilities, Micro Focus COBOL, Oracle, etc)
• the common_profile defines PATH's to scripts, programs,& data-files,
  by using environmental variables $UV, $COBDIR, etc, which were defined
  in the common_profile.

• On exceptional occasions the common_defines definition might be overridden
  in the stub_profile of sys admins to test new versions of software packages.
  You could redefine UV for Vancouver Utilities or COBDIR for Micro Focus COBOL.

RUNLIBS & RUNDATA determine PATHs to programs/scripts & data

The PATH to JCL/scripts depends on $RUNLIBS ($TESTLIBS or $PRODLIBS), example:

 export PATH=$PATH:$RUNLIBS/jcls    <-- PATH to JCL/scripts (in common_profile)
 ===============================

'$RUNDATA' determines data-file locations indirectly as follows:

• $RUNDATA defines the superdir housing all data-files. All JCL/scripts call a
  common function 'jobset51' which changes directory to $RUNDATA (cd $RUNDATA).

• The JCL converter inserts jobset51 at the begining of all converted
  JCL/scripts and then addresses all data files relative to $RUNDATA.

• See a sample JCL conversion at http://www.uvsoftware.ca/mvsjcl.htm#1C1

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1B3. ADMjobs: profiles for Vancouver Utilities on Unix/Linux

Advantages of common_defines, stub_profile,& common_profile

The benefits of this system are HUGE:

1. You never need to change JCL/scripts between TESTING & PRODUCTION.
   
   
2. Programmers execute JCL/scripts from $TESTLIBS/jcls that access data from $TESTDATA/... because their profiles set RUNLIBS/RUNDATA as TESTLIBS/TESTDATA
   
   
3. Operators execute JCL/scripts from $PRODLIBS/jcls that access data from $PRODDATA/... because their profiles set RUNLIBS/RUNDATA as PRODLIBS/PRODDATA
   
   
4. This system could allow testing & production on the same machine. They are usually on separate machines, and this system provides great flexibility for both these situations.
   
   
5. Different programmers can have their own set of TESTDATA simply by changing the definition of RUNDATA in their profile.
   
   
6. On production machines, different operators could operate on different sets of data for different companies on the same machine, using the same JCL/scripts & COBOL programs with no changes required.
   
   
7. The system administrator (appsadm) can make changes in 1 place to affect all users vs having to update profiles in all user homedirs. Actually I should have said 2 places (common_defines & common_profile).
   
   
8. The sysadmin would control any updates to common_defines & common_profile. On test systems the programmers would control updates to their stub_profile.
   
   
9. Programmers have the flexibility required. They can select different sets of testdata using $symbols defined in common_defines (by the sysadmin).
   
   
10. This system makes it easy to test new versions of software packages (Vancouver Utilities, COBOL, Oracle, etc). You can modify the stub_profile (in homedirs of selected users) to define the BASE/HOME dir of the new package, overriding the existing definitions in common_defines. The stub_profile then calls the commmon_profile that defines the PATHs to the bin & script subdirs within the defined BASE/HOME dirs.
    

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1C0. ADMjobs: profiles for Vancouver Utilities on Unix/Linux

Profile Listings

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1C1. ADMjobs.doc - profiles for Vancouver Utilities

common_defines - critical env-vars for profiles

 # common_defines - critical environmental variables for Vancouver Utilities
 #                - this file distributed in /home/uvadm/env/common_defines
 # - copy to /home/appsadm/env/... before modifying for your site
 #   (won't lose your changes when new version of Vancouver Utilities installed)
 # profiles in 3 parts: common_defines, stub_profile, common_profile
 # - stub_profile copied to homedirs, renamed .profile or .bash_profile
 # - stub_profile calls common_defines & common_profile (from /home/appsadm/env)
 # - see more explanations at www.uvsoftware.ca/admjobs.htm#Part_1
 #
 # Define critical site dependent PATH variables here in common_defines
 # - define superdir PATHs for various software packages
 # - Vancouver Utilities, appsadm, COBOL, RPG, Oracle, SyncSort, etc
 # - search PATHs in common_profile depend on these base values
 #
 export UV=/home/uvadm                # <-- UV homedir symbol used below
 export APPSADM=/home/appsadm         # <-- site applications administrator
 export LOGDIR=$APPSADM               # <-- cons log subdirs usually in APPSADM
 export COBDIR=/opt/microfocus/cobol  #<-- default for Server Express
 export LPSDIR=/home/ezadm/om/server  #<-- for EasySpooler
 export RPGADM=/home/rpgadm           # Morada RPG compiler
 export ORACLE_BASE=/h22/oracle
 export ORACLE_HOME=$ORACLE_BASE/product/11.1.0/db_2
 export OH=$ORACLE_HOME               # shortform
 export ORACLE_OWNER=oracle
 export ORACLE_SID=demo1
 export DB2DIR=/h23/db2/db2xc         # DB2DIR for RHEL at UVSI
 #export DB2DIR=/h23/db2/v95          # - should have been this
 export DB2INSTANCE=db2inst1          # DB2INSTANCE = homedir DB2 install
 export MYSQL=/var/lib/mysql          # MYSQL default on Linux
 export ODBCINI=/usr/local/etc/odbc.ini
 export ODBCSYSINI=/usr/local/etc
 #
 # export SYNCSORT=/home/ssadm        #<-- SYNCSORT at UVSI
 # export SYNCSORT=/export/home/syncsort/sunos/mvssortconv/  #<-- at EDS
 # export PATH=$PATH:/opt/sfw/bin:/usr/ccs/bin:/opt/SUNWspro/bin # SUN stuff ?
 #
 #Nov2008 - env-vars added for uycopy-API to SQL DataBase Tables
 #        - see www.uvsoftware.ca/sqldemo.htm#Part_2
 export DBhost1=localhost  # DBhost1,DBuser1,DBpass1 connect for uycopy input
 export DBuser1=$LOGNAME
 export DBpass1=${LOGNAME}00
 export DBhost2=localhost  # DBhost2,DBuser2,DBpass2 connect for uycopy output
 export DBuser2=$LOGNAME
 export DBpass2=${LOGNAME}00
 # defaults: DBhost=localhost,DBuser=$LOGNAME,DBpass=(prompt if undefined)
 #---------------------------------------------------------------------------
 #
 #             ** common_defines required by Mainframe conversion sites **
 #
 # critical environmental variables for TESTING & PRODUCTION
 # common_defines allow you to define the locations of your TESTLIBS/TESTDATA
 # & PRODLIBS/PRODDATA in 1 place for easy changes (vs multiple profiles)
 # see suggested profiles, disc partitioning & directory designs in ADMjobs.doc
 # - at least 4 disc partitions p1,p2,p3,p4 recommended for production sites
 # - p1=testlibs+testdata,p2=prodlibs+proddata,p3=backup+restore,p4=cnvdata
 #
 export TESTLIBS=/p1/testlibs  #<-- examples for user sites
 export TESTDATA=/p1/testdata  #  - see overrides for 'mvstest' below
 export PRODLIBS=/p2/prodlibs
 export PRODDATA=/p2/proddata
 export BACKUP=/p3/backup
 export RESTORE=/p3/restore
 export CNVDATA=/p4/cnvdata    # convert data files
 export CMPDATA=/p4/cmpdata    # compare data files
 export HOMEDIRS=/home         # HOMEDIRS=/export/home for SUN solaris
 export TAPERWD=/dev/st0       # rewind tape device for Linux SCSI
 export TAPENRW=/dev/nst0      # NO rewind tape device for Linux SCSI
 #
 # override above examples with 'mvstest' definitions for testing at UV Software
 # user implementation would delete following & modify above
 # - depending on user site disc partitioning & file design
 export TESTLIBS=/home/mvstest/testlibs
 export TESTDATA=/home/mvstest/testdata
 export CNVDATA=/home/mvstest/cnvdata
 export CMPDATA=/home/mvstest/cmpdata
 export BACKUP=/h33/backup    #<-- for testing at UV Software
 export RESTORE=/h33/restore
 #
 #                  ** stub_profile - Execution Sequence **
 #
 # 'stub_profile' must be copied to homedirs & renamed as '.profile' for the
 #  Korn shell or 'bash_profile' for bash shell. One of major reasons for
 #  splitting the profiles into 3 parts is to provide flexibility for the
 #  definition of RUNLIBS/RUNDATA as TESTLIBS/TESTDATA or PRODLIBS/PRODDATA
 #  depending on the user (programmer or operator).
 #
 # 1. stub_profile First executes common_defines (this file) via '.' execution
 #    to retain the definitions made in common_defines. See the 'export's for
 #    TESTLIBS/TESTDATA & PRODLIBS/PRODDATA above (here in common_defines).
 #
 # 2. stub profiles define RUNLIBS,RUNDATA,etc using $symbol definitions
 #    made in common_defines, for example:
 #    export RUNLIBS=$TESTLIBS  #<-- for programmers ($PRODLIBS for operators)
 #    export RUNDATA=$TESTDATA  #<-- for programmers ($PRODDATA for operators)
 #    export CNVDATA=$CNVDATA   #<-- for programmers (n/a to operators)
 #    export CMPDATA=$CMPDATA   #<-- for programmers (n/a to operators)
 #
 # 3. stub profiles then '.' dot execute the common_profile
 #    which uses $RUNLIBS/$RUNDATA as part of PATH to JCL/scripts, etc
 #    $RUNLIBS & $RUNDATA also used in 'jobset51' called by ksh JCL/scripts
 #    or in 'jobset61' called by Perl scripts.
 #
 #Note - the 'stub_profile' distributed in /home/uvadm/env/... has some extra
 #       insertions prior to the #2 definitions above, as follows:
 # export TESTLIBS=$HOME/testlibs   #<-- temporary for UV testing
 # export TESTDATA=$HOME/testdata   #<-- see notes below
 # export CNVDATA=$HOME/cnvdata
 # export CMPDATA=$HOME/cmpdata
 # - above exports (in stub_profile) are temporary for UV test/demos
 #   or for programmers who wish to have Libs & Data in their home directory
 # - must be removed for production operators, see more in the stub_profile
 #-----------------------------------------------------------------------------
 #
 #                  ** site dependent optional variables **
 #
 export LOGMSGACK=y     #<-- operator must acknowledge logmsg1/2 prompts
 #=================
 # - JCL converter inserts logmsg1/2 functions in converted Korn shell scripts
 # - 'LOGMSGACK=y' forces operator to reply 'y' to continue
 # - change to 'LOGMSGACK=n' to inhibit the operator prompt
 # - default here in common_defines could be overridden in user profiles
 #Note - running JCL/scripts by cron automatically disables operator prompts
 #
 export RUNTYPE=TXXX    #<-- define RUNTYPE T/P (Test/Production)
 #==================
 # profiles could export RUNTYPE=TXXX for programmers, RUNTYPE=PXXX for operators
 # - scripts can then test RUNTYPE to see if Test or Production, for example:
 #   if [[ "$RUNTYPE" == P* ]]; then llp ...; fi #<--print if Prod (not if Test)
 # - the 'XXX' portion allows you to add other codes as you find convenient
 # - testing for 'P*' or 'T*' allows for these other codes
 # - for example if you added code 'Y' for Year end, you would test as follows:
 #   if [[ "$RUNTYPE" == *Y* ]]; then ...; fi   #<--test for Year-end code
 #------------------------- end of common_defines ---------------------------

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1C2. ADMjobs.doc - profiles for Vancouver Utilities

'stub' profile in homedirs calling 'common' profile

 # stub_profile - this file distributed in /home/uvadm/env/stub_profile
 #              - to be copied to /home/appsadm/env/... for user customization
 #              - then copied to user homedirs & renamed as '.profile' for ksh
 #                OR '.bash_profile' for bash/linux
 #              - see more at www.uvsoftware.ca/admjobs.htm#Part_1
 #
 # Also see 'stub_profile_appsadm' alternate version of this 'stub_profile'
 # - to be copied to /home/appsadm/env & customized for programmers/operators
 #
 # profiles in 3 parts: #1=common_defines, #2=stub_profile, #3=common_profile
 # - stub_profile calls common_defines & common_profile
 # - profiles distributed in /home/uvadm/env/...
 # - should copy to /home/appsadm/env/... before modifying for your site
 #   (won't lose your changes when new version of Vancouver Utilities installed)
 #
 #                    ** profile execution sequence **
 #
 # 1. common_defines is '.' dot executed by stub_profiles (which have been
 #    copied to homedirs,& renamed .profile for ksh or .bash_profile for bash).
 #
 #*2. stub profile (copied/renamed .profile/.bash_profile in all user homedirs)
 #    defines RUNLIBS,RUNDATA,etc using $symbols in common_defines, for example:
 #
 #    export RUNLIBS=$TESTLIBS    #<-- for programmers ($PRODLIBS for operators)
 #    export RUNDATA=$TESTDATA    #<-- for programmers ($PRODDATA for operators)
 #    export CNVDATA=$CNVDATA
 #    export CMPDATA=$CMPDATA
 #
 # 2x. Avoid hard-coded exports in stub_profile
 #     (by using $SYMBOLS defined in common_defines)
 #     BUT, on exception, could override some $SYMBOLs defined in common_defines
 #     For example - testing new versions of software packages (by sys admins)
 #     - could override defs for $UV, $COBDIR, etc
 #
 # 3. stub profiles then '.' dot execute the 'common_profile' (see further below)
 #    which uses $RUNLIBS/$RUNDATA as part of PATH to JCL/scripts, etc
 #    $RUNLIBS & $RUNDATA also used in 'jobset51' function called by ksh scripts
 #    or in 'jobset61' subrtn called by Perl scripts
 #
 #              ** RUNDATA/RUNLIBS - critical for conversion sites **
 #
 # - variables RUNDATA & RUNLIBS point to TEST or PRODuction files (see below)
 #   depending on whether user is a programmer or operator
 # - this stub_profile should be used to run the test/demos documented
 #   in Part_1 of MVSJCL.doc, where you are instructed to setup a separate
 #   userid login 'mvstest' (or 'vsetest') & copy the supplied subdirs/files
 #   from /home/uvadm/mvstest/* to /home/mvstest/...
 # - 'stub' contains user preferences: TERM, stty erase char, aliases,logging,etc
 #---------------------------------------------------------------------------
 #
 # #1 - call 'common_defines'
 #    - allows admins to make changes in 1 place to affect all users
 #      (vs having to change profiles in all user homedirs)
 #    - called via '.' so $variables will persist in current environment
 #
 . /home/uvadm/env/common_defines     #<-- call common_defines from 'uvadm'
 #===============================     #  - distributed in /home/uvadm/env
 # . /home/appsadm/env/common_defines #<-- copy to 'appsadm' for your conversions
 # ==================================    - customize for your conversion
 #Note - see stub_profile_appsadm with uvadm #cmtd & appsadm uncmtd
 #
 # common_defines - defines critical site dependent $variables for:
 #                - Vancouver Utilities,appsadm,COBOL,RPG,Oracle,SyncSort,etc
 # - search PATHs in common_profile depend on superdir defs in common_defines
 # - possibly overridden by redefs here in stub_profiles
 # $variables defined in common_defines are:
 # - $UV, $APPSADM, $COBDIR,
 # - $TESTDATA/$TESTLIBS, $PRODDATA/$PRODLIBS, $RUNDATA/$RUNLIBS
 # - see more explanations as #comments in common_defines
 #   and at www.uvsoftware.ca/admjobs.htm#Part_1
 #
 #----------------------------------------------------------------------------
 #
 # #2x. Exception coding in stub_profile to override defs in common_defines:
 #
 export TESTLIBS=$HOME/testlibs #<-- uncommented (active) in stub_profile
 export TESTDATA=$HOME/testdata #  - #commented out in stub_profile_appsadm
 export CNVDATA=$HOME/cnvdata   #  - to preserve the TESTLIBS/TESTDATA
 export CMPDATA=$HOME/cmpdata   #    definitions in prior common_defines
 #
 #Note - above exports are temporary for UV test/demos or for programmers
 #        who wish to have their own set of Libs & Data in their home directory
 # Must be removed for production operators
 # - when stub_profile_appsadm copied to .bash_profile in operator homedirs
 # Programmers might use for initial testing & self education
 # - remove TESTLIBS=$HOME/testlibs, etc when conversion team given the go ahead
 #   since all programmers should work on the common set of libraries
 # - might use TESTDATA=$HOME/testdata in homedirs to prevent conflicts
 #   (programmers could #comment or uncomment as required)
 #-------------------------------------------------------------------------
 #
 # #2. define critical VARIABLES using $VARIABLES defined in common_defines
 #
 export RUNLIBS=$TESTLIBS   #<-- for programmers ($PRODLIBS for operators)
 export RUNDATA=$TESTDATA   #<-- for programmers ($PRODLIBS for operators)
 export CNVDATA=$CNVDATA
 export CMPDATA=$CMPDATA
 #
 #Note - above exports are intended for programmer profiles
 #     - RUNLIBS/RUNDATA must be changed to PRODLIBS/PRODDATA for production
 #       when this stub_profile copied to homedirs of operators
 #       (& renamed as .bash_profile for bash/linux or .profile for ksh/unix)
 #-------------------------------------------------------------------------
 #
 # #3. call common_profile (supplied in /home/uvadm/env/...)
 #          - you should copy to & customize in /home/appsadm/env/...
 #          - see listings in install.doc, ADMjobs.doc,or MVSJCL.doc
 #          - contains the bulk of profile setups (PATH defs, etc)
 #          - so you can update these in 1 place for all users
 #          - call via '.' so 'export's persist on return to stub
 #
 . /home/uvadm/env/common_profile    #<-- call common_profile from 'uvadm'
 #===============================    #  - distributed in /home/uvadm/env
 #. /home/appsadm/env/common_profile #<-- copy to 'appsadm' for your conversions
 #===================================   - customize for your conversion
 #Note - see stub_profile_appsadm with uvadm #cmtd & appsadm uncmtd
 #-----------------------------------------------------------------------------
 #
 #                       ** user preferences **
 #
 # - code user specific profile requirements here in this 'stub' profile
 # - to override/modify definitions in the common profile called above
 #
 export TERM=linux         # TERM - modify depending on your terminal
 #                         # (vt100,xterm,at386,ansi,etc)
 stty erase '^?'           # erase char - modify depending on your terminal
 #                         #  '^?' for linux/at386, '^H' for vt100,ansi,xterm
 export UVLPDEST="-dlp0"   # define printer for 'uvlp' scripts
 alias l='ls -l'           # code your desired aliases here
 #--------------------------------------------------------------------------
 #
 #                        ** Console Logging **
 #
 # - uncomment 7 '##' lines below to activate console logging
 # - must setup subdirs matching $LOGNAME in $APPSADM/log1/...,log2/...,log3/...
 # - subdirs log1,log2,log3 hold logfiles for: current file, month, lastmonth
 # - see details at www.uvsoftware.ca/admjobs.htm#Part_6
  ## login1 || exit 2          # exit here if 2nd login
  ## logfixA $LOGNAME          # process log1 file to log2 (to allow read/print)
  ## echo "console logging requires aliases to be stored in .bashrc or .kshrc"
  ## echo "--> . aliases <-- 'dot' execute aliases NOW, if not in .bashrc/.kshrc"
  ## echo "--> logview   <-- execute logview script to see prior console logs"
  ## echo "script $APPSADM/log1/$LOGNAME/$(date +%y%m%d_%H%M%S)"
  ## exec script $APPSADM/log1/$LOGNAME/$(date +%y%m%d_%H%M%S)
 # - 'exec script' must be the last non-comment line in the profile
 # - 'script' disables 'aliases', put them in '.bashrc' to make them effective
 #--------------------------- end of stub_profile ---------------------------

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1C3. ADMjobs.doc - profiles for Vancouver Utilities

'common' profile called by 'stub' profile in homedirs

 # common_profile - common to all users of Vancouver Utilities,COBOL,etc
 #                - defines search PATHs based on supdir defs in common_defines
 #                - defines common aliases, TERM type, prompts, etc
 #
 # profiles in 3 parts: common_defines, stub_profile, common_profile
 # - home dirs contain only the 'stub_profile'
 # - stub_profiles must be renamed '.bash_profile' for bash & '.profile' for ksh
 # - 'stub_profiles' call the 'common_profile'
 # - any user customizations must be made in the stub profiles
 #
 # - sample common_profile distributed in /home/uvadm/env/...
 # - you should setup userid appsadm & copy profiles to /home/appsadm/env/...
 #   so installing new versions of uvadm will not overwrite your common profiles
 #
 # - stub profiles in various homedirs should call common profile in appsadm
 # - user 'stub' profiles must '.' (dot) execute the common_profile
 #   to retain the exports & aliases when common_profile returns to the stub
 #
 # - There is no need for test & prod versions of common_profile, since
 #   RUNLIBS/RUNDDATA are defined in the stub before calling the common
 # - 'test' & 'prod' versions of stub_profile_.... might be convenient
 #   so administrators could copy stub_profile_test to programmer homedirs
 #   & stub_profile_prod to homedirs of operators (rename .profile/.bash_profile)
 # - RUNLIBS/RUNDATA would point to either test or prod superdirs
 #
 #-----------------------------------------------------------------------------
 # common_defines - 1st executed (of common_defines,stub_profile,common_profile)
 #                - defined critical site dependent PATH variables
 #                - superdir PATHs for various software packages
 #                - Vancouver Utilities,appsadm,COBOL,RPG,Oracle,SyncSort,etc
 #
 # NOTE - search PATHs defined here in common_profile depend on superdir defs
 #        in common_defines (possibly overridden in stub_profiles)
 # NOTE - following superdir defs are #commented out here in common_profile
 #        (see actual definitions in common_defines)
 #
 # export UV=/home/uvadm                # <-- UV homedir symbol used below
 # export APPSADM=/home/appsadm         # <-- site applications administrator
 # export LOGDIR=$APPSADM               # <-- cons log subdirs usually in APPSADM
 # export COBDIR=/opt/microfocus/cobol  #<-- default for SX 4.0
 # export LPSDIR=/home/ezadm/om/server  #<-- for EasySpooler
 # export RPGADM=/home/rpgadm           # Morada RPG compiler
 # export ORACLE_BASE=/h22/oracle
 # export ORACLE_HOME=$ORACLE_BASE/product/11.1.0/db_1
 # export ORACLE_OWNER=oracle
 # export ORACLE_SID=orcl
 # export SYNCSORT=/home/ssadm        #<-- SYNCSORT at UVSI
 # export SYNCSORT=/export/home/syncsort/sunos/mvssortconv/  #<-- at EDS
 # export PATH=$PATH:/opt/sfw/bin:/usr/ccs/bin:/opt/SUNWspro/bin # SUN stuff ?
 #
 #Nov2008 - env-vars added for uycopy-API to SQL DataBase Tables
 #        - see www.uvsoftware.ca/sqldemo.htm#Part_2
 # export DBhost1=localhost  # DBhost1,DBuser1,DBpass1 connect for uycopy input
 # export DBuser1=$LOGNAME
 # export DBpass1=${LOGNAME}00
 # export DBhost2=localhost  # DBhost2,DBuser2,DBpass2 connect for uycopy output
 # export DBuser2=$LOGNAME
 # export DBpass2=${LOGNAME}00
 #----------------------------------------------------------------------------
 # - critical environmental variables for TESTING & PRODUCTION
 # - RUNLIBS/RUNDATA defines as $TESTLIBS/$TESTDATA or $PRODLIBS/$PRODDATA
 # - controls your JCL conversions to Korn or Perl scripts & their executions
 # - copy to /home/appsadm/env/common_defines before modifying for your site
 # - allows administrators to make changes in 1 place to affect all users
 # - also see suggested disc partitioning & directory designs in ADMjobs.doc
 # - here are some examples from common_defines, but #commented out here
 #
 # export TESTLIBS=/p1/testlibs
 # export TESTDATA=/p1/testdata
 # export PRODLIBS=/p2/prodlibs
 # export PRODDATA=/p2/proddata
 # export BACKUP=/p3/backup
 # export RESTORE=/p3/restore
 # export CNVDATA=/p4/cnvdata
 # export CMPDATA=/p4/cmpdata
 #
 #----------------------------------------------------------------------------
 # Note - the sequence of events must be as follows:
 #
 # 1. common_defines is '.' dot executed by stub_profiles (which have been
 #    copied to homedirs,& renamed .profile for ksh or .bash_profile for bash).
 #
 # 2. stub profiles then define RUNLIBS,RUNDATA,etc using $symbols
 #    from common_defines, for example:
 #
 #    export RUNLIBS=$TESTLIBS    #<-- for programmers ($PRODLIBS for operators)
 #    export RUNDATA=$TESTDATA    #<-- for programmers ($PRODDATA for operators)
 #    export CNVDATA=$CNVDATA
 #    export CMPDATA=$CMPDATA
 #
 # 3. stub profiles then '.' dot execute the common_profile
 #    which uses $RUNLIBS/$RUNDATA as part of PATH to JCL/scripts, etc
 #    $RUNLIBS & $RUNDATA also used in 'jobset51' function  called by ksh scripts
 #    or in 'jobset61' subrtn called by Perl scripts
 #
 #----------------------------------------------------------------------------
 # Verify that critical environmental variables have been defined
 # (by common_defines or stub_profile before this common_profile called)
 if [[ "$UV" = "" || "$APPSADM" = "" || "$COBDIR" = "" ]]; then
    echo "UV=$UV, APPSADM=$APPSADM,or COBDIR=$COBDIR not defined"
    echo "- enter to exit"; read $reply; exit 99; fi
 if [[ "$RUNLIBS" = "" || "$RUNDATA" = "" ]]; then
    echo "RUNLIBS=$RUNLIBS or RUNDATA=$RUNDATA not defined"
    echo "- enter to exit"; read $reply; exit 99; fi
 #
 #----------------------------------------------------------------------------
 # setup PATH for Vancouver Utilities programs & scripts (uvadm & appsadm)
 # - append onto system PATH, using symbols defined above ($UV, $APPSADM, etc)
 #
 export PATH=$PATH:/usr/sbin
 export PATH=$PATH:$HOME/bin:$HOME/sf:$APPSADM/bin:$APPSADM/sf:$RUNLIBS/sf
 export PATH=$PATH:$UV/bin:$UV/sf/adm:$UV/sf/demo:$UV/sf/util:$UV/sf/IBM
 #
 #Note - APPSADSM appears before UV so user modified scripts/jobs in APPSADM
 #       can be stored in $APPSADM & be found prior to original versions in $UV
 #     - $UV/sf subdirectoried to adm,demo,util,IBM (April2003+)
 #
 #----------------------------------------------------------------------------
 # setup 'PFPATH' for uvcopy & uvqrpg interpreter to find Parameter Files (jobs)
 export PFPATH=$RUNLIBS/pf,$RUNLIBS/pfx3,$RUNLIBS/ezts,$HOME/pf,$APPSADM/pf
 export PFPATH=$PFPATH,$UV/pf/adm,$UV/pf/demo,$UV/pf/util,$UV/pf/IBM
 # - use symbol $UV (defined above) to shorten PFPATH definition
 # - UV/pf/... follows RUNLIBS,APPSADM,HOME to allow user duplicate names
 # - uvcopy accepts ',' delimiters as well as ':' in case of SFU on Windows
 #
 #----------------------------------------------------------------------------
 # setup PATH & FPATH for JCL/scripts converted from mainframe Vancouver Utils
 # - see www.uvsoftware.ca/mvsjcl.htm or www.uvsoftware.ca/vsejcl.htm
 export PATH=$PATH:$RUNLIBS/jcls  # RUNLIBS defined in common_profile
 export FPATH=$RUNLIBS/sfun       # ksh functions used by VU JCL/scripts
 #Note - RUNLIBS defined in common_profile (maybe TESTLIBS or PRODLIBS)
 # - possibly modified in users stub_profile before used here in common_profile
 #
 #----------------------------------------------------------------------------
 # env-vars for perl scripts (added June 27/2006)
 # - alternative conversion of JCL to perl scripts (vs Korn shell scripts)
 # - PERL5LIB defines directories to be searched for perl modules
 export PERL5LIB=$UV/perlm:$PERL5LIB  # perl modules 'use'd by perl scripts
 #
 #----------------------------------------------------------------------------
 # misc setups for Vancouver Utilities
 export DISAMEXT="dat"   # .dat/.idx Indexed files for uvsort,uvcopy,uvcp,etc
 # DISAMEXT="dat" causes uvsort,etc to expect .dat on data partition of ISAM file
 # MicroFocus COBOL equivalent is 'IDXNAMETYPE=2' in $EXTFH/extfh.cfg (see below)
 #
 # printer destinations for VU laser printing scripts
 # - modify UVLPDEST to the network printer closest to you
 export UVLPDEST="-dlp0"       # default dest for uvlp(uvlist) scripts
 export UVLPOPTN="-onobanner"  # for unix/linux (SFU does not allow)
 export UVHDPRINT=uvlp12       # script for uvhd 'i' immediate print command
 export UVHDPWIDE=uvlp12L      # script for uvhd 'iprint' Landscape 100 chs/line
 #
 #-------------------------------------------------------------------------
 # Easy Spooler setups
 export LPDEST="lp0"           # default dest if lp -d unspcfd
 export LPOPTION="-dlpt1 -onob -odata=a"  # options for EasySpooler
 #
 #-------------------------------------------------------------------------
 # For Micro Focus COBOL Server Express
 # - see compile scripts mfcbl1 & mfcblA in /home/uvadm/sf/IBM
 # - Also required to compile 'ux' programs (uxcp,uxsort,uxcopy,uxqrpg) with
 # - cobccuv script to support Indexed Seqntl Variable length (IDXFORMAT3 files)
 # - COBDIR defined in common_defines with other site dependent path variables
 #
 export PATH=$PATH:$COBDIR/bin
 export LD_LIBRARY_PATH=$COBDIR/lib:$LD_LIBRARY_PATH
 export LANG=en_US                   # fix animator display carets vs data
 #note - for COBOPT & EXTFH, might change $UV to $RUNLIBS ??
 export COBOPT=$UV/ctl/cobdirectives # compiler options & Directives
 export COBCPY=$RUNLIBS/cpys         # copybook search, compile script overrides
 export EXTFH=$UV/ctl/extfh.cfg      # file handler configuration
 # - [XFH-DEFAULT] IDXNAMETYPE=2 for .dat ext on data part (.idx on index part)
 export ASLMFRETRY="5,15"  # limited licenses, retry 5 times, wait 15 secs btwn
 export COBMAPDIR=$RUNLIBS/maps # for uvhdcob (display data with fieldnames)
 # export COBMODE=64              # generate 64 bit programs (activate Jan14/09)
 #Sep2008 COBMODE=64 Err-> CE0001 odbcecm depmod n/f, OK if #-C SQL cobdirectives
 # - but uxcopy,etc will not link with uvlib.a in 64 bit mode ???
 # - ld: lib/uvlib64.a(errmsg.o) relocation R_X86_64_32 cant make shared object
 # - w/o COBMODE=64, can compile w H64, but execute gives err 114 signal 11
 # - must compile uvlib32.a with -m32 & uxcopy with H32
 # define TERM & TERMINFO for MicroFocus color coding in animator
 # #commented out here?, could define in stub profile depending on Terminal
 # export TERMINFO=$COBDIR/terminfo # provides color coding in MFC animator
 # export TERM=ansi                     # for animator (& vim multicolor)
 #
 #----------------------------------------------------------------
 # setup for DB2 - see export DB2DIR in common_defines for: DB2DIR
 # now define PATHs based on common_defines
 export PATH=$PATH:$DB2DIR/bin:$DB2DIR/adm
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$DB2DIR/lib32:$DB2DIR/lib64
 export DB2PATH=$DB2DIR/bin:$DB2DIR/adm
 #
 #-------------------------------------------------------------------------
 # setup for MySQL & ODBC on RHEL at UVSI, see exports in common_defines
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib32:/usr/local/lib:/usr/lib:/usr/lib64
 #
 #-------------------------------------------------------------------------
 # setup for Oracle - see exports in common_defines for:
 # ORACLE_SID, ORACLE_OWNER, ORACLE_BASE, ORACLE_HOME
 # now define PATHs based on common_defines
 PATH=$PATH:$ORACLE_HOME/bin    # :$ORACLE_HOME/precomp/demo

 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib:$ORACLE_HOME/lib32
 #
 #-------------------------------------------------------------------------
 # SYNCSORT requirements
 # SYNCSORT homedir defined in common_defines with other site dependent variables
 # export PATH=$PATH:$SYNCSORT/bin
 # export MANPATH=$MANPATH:$SYNCSORT/man
 # export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$SYNCSORT/lib
 #
 #-------------------------------------------------------------------------
 # Morada RPG user requirements
 export PATH=$PATH:$RPGADM/bin      # append RPG bin to PATH
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$RPGADM/lib # for Morada RPG
 export SIGN_OVERRIDES=UN-7         # x'70' neg zone signs for RPG programs
 export RPGCDIR=$RPGADM             # alternate def for compile scripts ?
 # - see more about Morada RPG at www.uvsoftware.ca/vserpg.htm
 # - see Part 8 for additional env-vars to generate RPG distribution
 #
 #-------------------------------------------------------------------------
 # define TERM & dependent 'erase' char
 export TERM=linux         # TERM - modify depending on your terminal
 #                         # (vt100,xterm,at386,ansi,etc)
 stty erase '^?'           # erase char - modify depending on your terminal
 #                         #  '^?' for linux/at386, '^H' for vt100,ansi,xterm
 # stty intr '^C'          # interrupt ^C, (probably already default ?)
 #-------------------------------------------------------------------------
 # Miscellaneous, Recommended items
 umask 002                     # permissions 775 dirs, 664 files
 set -o ignoreeof              # disallow logoff via ctl D (use exit)
 HOST1=$(uname -n)             # add to PS1 prompt if desired
 trm=$(tty)                    # capture terminal device for PS1
 trmv=${trm#/dev/}             # remove prefix /dev/
 export PS1='<@$trmv:$LOGNAME:$PWD> '
 export EDITOR=vi              # for Korn shell history
 export VISUAL=vi              # for Korn shell history
 export HISTSIZE=1000;         # Korn shell history file size
 #
 #-------------------------------------------------------------------------
 # alias commands to prompt for overwrite (highly recommended)
 # - use option '-f' when you have many files  (rm -f tmp/*, etc)
 alias rm='rm -i'              # confirm removes
 alias mv='mv -i'              # confirm renames
 alias cp='cp -i'              # confirm copy overwrites
 alias rmf='rm -f'             # force removes (no prompts)
 alias mvf='mv -f'             # force renames (no prompts)
 alias cpf='cp -f'             # force copies (no prompts)
 alias l='ls -l'               # save keystrokes
 alias more='less'             # less is way better than more
 alias vi='vim'                # use vim
 # alias vi='\vi'                # use vi (Red Hat Linux defaults to vim)
 alias grep='grep -Hn'         # ensure filename & line# on matching lines
 alias df='df -H'              # ensure -H on df (Disc Free Human readable)
 alias du='du -s -h -c '       # -s summary -h Human-read, -c grand total
 alias uname='uname -a'        # ensure -a on uname (All info)
 alias cduv='cd $UV'           # Vancouver Utilities
 alias cdl='cd $RUNLIBS'       # quick access to LIBS superdir
 alias cdd='cd $RUNDATA'       # DATA superdir
 alias cdc='cd $CNVDATA'       # data CONVERSION superdir
 alias cdk='cd $CMPDATA'       # data COMPARISON superdir
 alias cdb='cd $BACKUP'        # BACKUP superdir
 alias cdr='cd $RESTORE'       # RESTORE superdir
 alias cdoh='cd $ORACLE_HOME'  # Oracle homedir
 alias cdmy='cd $MYSQL'        # MySQL install default
 alias cddb2='cd $DB2DIR'      # DB2 install dir
 # aliases - ineffective if console logging activated (in user stub profile)
 #         - ifso, place aliases in .bashrc (or .kshrc, etc)
 #--------------------------------------------------------------------------
 #NOTE - stub profiles (in user home dirs) can activate console logging here
 #     - does not apply here in common_profile
 #     - see coding at end of stub_profile
 #     - see documentation at www.uvsoftware.ca/admjobs.htm
 #------------------------- end of common_profile --------------------------

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1C4. ADMjobs.doc - profiles for Vancouver Utilities

profile for utilities ONLY

 # profile - for users of Vancouver Utilities only
 #         - simplified profile to access VU programs in /home/uvadm/bin/...
 # programs - uvsort, uvhd, uvcopy, uvcp, uvlist, uvqrpg, etc
 # scripts  - over 500 Korn shell scripts in /home/uvadm/sf/...
 # uvcopy jobs - over 500 uvcopy parameter files in /home/uvadm/pf/...
 # note - this profile NOT for Vancouver Utility mainframe conversion sites
 #      - conversion site profiles are more complex in 3 parts
 #        (common_defines, stub_profile, common_profile)
 #      - see www.uvsoftware.ca/admjobs.htm#Part_1
 #
 export UV=/home/uvadm   # <-- UV homedir symbol used below
 # setup PATH for Vancouver Utilities programs & scripts (uvadm & appsadm)
 # - append onto system PATH, using symbol $UV defined above
 export PATH=$PATH:$UV/bin:$UV/sf/adm:$UV/sf/demo:$UV/sf/util:$UV/sf/IBM
 #
 # setup 'PFPATH' for uvcopy & uvqrpg interpreter to find Parameter Files (jobs)
 export PFPATH=$HOME/pf:$UV/pf/adm:$UV/pf/demo:$UV/pf/util:$UV/pf/IBM
 #
 #----------------------------------------------------------------------------
 # misc setups for Vancouver Utilities
 export DISAMEXT="dat"   # .dat/.idx Indexed files for uvsort,uvcopy,uvcp,etc
 # DISAMEXT="dat" causes uvsort,etc to expect .dat on data partition of ISAM file
 # MicroFocus COBOL equivalent is 'IDXNAMETYPE=2' in extfh.cfg
 #
 # printer destinations for VU laser printing scripts
 # - modify UVLPDEST to the network printer closest to you
 export UVLPDEST="-dlp0"       # default dest for uvlp(uvlist) scripts
 export UVLPOPTN="-onobanner"  # for unix/linux (SFU does not allow)
 export UVHDPRINT=uvlp12       # script for uvhd 'i' immediate print command
 export UVHDPWIDE=uvlp12L      # script for uvhd 'iprint' Landscape 100 chs/line
 #-------------------------------------------------------------------------
 #Note - down to here is mandatory for Vancouver Utilities
 #     - following lines may be deleted/modified for established sites
 #-------------------------------------------------------------------------
 # Miscellaneous, Recommended items
 umask 002                     # permissions 775 dirs, 664 files
 set -o ignoreeof              # disallow logoff via ctl D (use exit)
 HOST1=$(uname -n)             # add to PS1 prompt if desired
 trm=$(tty)                    # capture terminal device for PS1
 trmv=${trm#/dev/}             # remove prefix /dev/
 export PS1='<@$trmv:$LOGNAME:$PWD> '
 export EDITOR=vi              # for Korn shell history
 export VISUAL=vi              # for Korn shell history
 export HISTSIZE=1000;         # Korn shell history file size
 #-------------------------------------------------------------------------
 # define TERM & dependent 'erase' char
 export TERM=linux         # TERM - modify depending on your terminal
 #                         # (vt100,xterm,at386,ansi,etc)
 stty erase '^?'           # erase char - modify depending on your terminal
 #                         #  '^?' for linux/at386, '^H' for vt100,ansi,xterm
 # stty intr '^C'          # interrupt ^C, (probably already the default ?)
 #------------------ end of Vancouver Utility profile stuff -----------------

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1C5. ADMjobs.doc - profiles for Vancouver Utilities

'.bashrc' - required for console logging

 # bashrc - initialization file for the bash shell (vs kshrc for Korn shell)
 #        - file stored at /home/uvadm/env/...
 #        - '.' of '.bashrc' omitted for visibility
 #NOTE - copy to your homedir & rename .bashrc(bash/linux) or .kshrc(ksh/unix)
 #
 # - aliases coded here, as well as in .bash_profile(linux) or .profile(unix)
 # - useful if console logging via 'script' command (see ADMjobs.doc)
 # - aliases & umask in profile get lost by the 'script' console logging command
 # - this saves having to remember '. aliases' after login when logging
 # - could also code functions here
 #
 # alias commands to prompt for overwrite (highly recommended)
 # - use option '-f' when you have many files  (rm -f tmp/*, etc)
 alias rm='rm -i'              # confirm removes
 alias mv='mv -i'              # confirm renames
 alias cp='cp -i'              # confirm copy overwrites
 alias rmf='rm -f'             # force removes (no prompts)
 alias mvf='mv -f'             # force renames (no prompts)
 alias cpf='cp -f'             # force copies (no prompts)
 #
 # aliases for quick 'cd's to commonly accessed directories
 # - requires env-vars RUNLIBS, RUNDATA, CNVDATA in your profile
 alias cdl='cd $RUNLIBS'       # quick access to libs superdir
 alias cdd='cd $RUNDATA'       # quick access to data superdir
 alias cdc='cd $CNVDATA'       # quick access to data conversion superdir
 alias cdb='cd $BACKUP'        # quick access to backup superdir
 alias cdr='cd $RESTORE'       # quick access to restore superdir
 #
 # misc aliases
 alias l='ls -l'               # save keystrokes
 alias vi='vim'                # use vim (vs vi)
 alias grep='grep -n'          # ensure -n option used on grep
 #
 # set umask, which also gets lost when console logging
 umask 002                     # ensure dirs 775 & files 664
 #--------------------------- end of bashrc ---------------------------------

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1C6. ADMjobs.doc - profiles for Vancouver Utilities

root_profile - root access to Vancouver Utilities

 # root_profile - for Red Hat Enterprise Linux & Vancouver Utilities
 # - supplied with Vancouver Utilities in /home/uvadm/env/root_profile
 # - to be copied/renamed to /root/.bash_profile
 # - to allow root to access Vancouver Utility scripts & uvcopy jobs
 # - includes handy aliases such as alias l='ls -l'
 # - includes prompt to identify current directory
 # - could also use for user's who need access to Vancouver Utilities
 #   but do not need the power & flexibility of stub & common profiles
 #
 # Get the aliases and functions
 if [ -f ~/.bashrc ]; then . ~/.bashrc; fi
 # above is only significant code in default RHEL root profile
 # following code added by Owen Townsend, UV Software
 #--------------------------------------------------------------------------
 # setup PATH for Vancouver Utilities programs & scripts (uvadm & appsadm)
 # - append onto system PATH, using symbols defined above ($UV, $APPSADM)
 export UV=/home/uvadm                # <-- UV homedir symbol used below
 export APPSADM=/home/appsadm         # <-- site applications administrator
 export PATH=$PATH:$HOME/bin:$HOME/sf:$APPSADM/bin:$APPSADM/sf:$RUNLIBS/sf
 export PATH=$PATH:$UV/bin:$UV/sf/adm:$UV/sf/demo:$UV/sf/util:$UV/sf/IBM
 # - $UV/sf subdirectoried to adm,demo,util,IBM (April2003+)
 # - APPSADSM appears before UV so user modified scripts/jobs in APPSADM
 #   can be stored in $APPSADM & be found prior to original versions in $UV
 #
 # setup 'PFPATH' for uvcopy interpreter to find Parameter Files (jobs)
 export PFPATH=$RUNLIBS/pf:$RUNLIBS/pfx3:$HOME/pf:$APPSADM/pf
 export PFPATH=$PFPATH:$UV/pf/adm:$UV/pf/demo:$UV/pf/util:$UV/pf/IBM
 #--------------------------------------------------------------------------
 # define prompt for root
 export PS1='$LOGNAME:$PWD# '
 #--------------------------------------------------------------------------
 # alias commands to prompt for overwrite (highly recommended)
 # - use option '-f' when you have many files  (rm -f tmp/*, etc)
 alias rm='rm -i'              # confirm removes
 alias mv='mv -i'              # confirm renames
 alias cp='cp -i'              # confirm copy overwrites
 alias l='ls -l'               # save keystrokes
 alias more='less'             # less is way better than more
 # alias vi='vim'                # use vim
 # alias vi='\vi'                # use vi (Red Hat Linux defaults to vim)
 alias grep='grep -n'          # ensure -n option used on grep
 #--------------------------------------------------------------------------
 # define COBOL env-vars for root on Red Hat Enterprise Linux
 # - needed to install Micro Focus COBOL & apply patches
 export COBDIR=/opt/microfocus/cobol   # default COBDIR for RHEL
 export PATH=$PATH:$COBDIR/bin
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$COBDIR/lib
 export LANG=en_US  # fix MFC SX 4.0 animator displaying carets vs data
 #--------------------------------------------------------------------------

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1C7. ADMjobs: profiles for Vancouver Utilities on Unix/Linux

permissions: directories 775 & files 664

The profiles set umask 002, which means the permissions will be 775 for directories & 664 for files, created by users of these profiles with umask 002.

Also be sure to assign a common group-ID (we suggest 'apps') to the programmers & operators who are working on a common set of directories & files (JCL, COBOL,& DATAfiles).

Unix normally defaults umask 022, which means subdirs would be 755 & files 644, which would not allow users to write in directories created by other members of the team.

Making umask 002 (dirs 775 files 664) & ensuring all team members in a common group allows team members to write into a common set of directories for JCL, COBOL,& DATA. We are in effect extending security to the group.

permissions vital for nightly cron jobs

Nightly batch jobs could fail due to files with bad permisions or group. Nightly batch jobs are scheduled by a crontab owned by 'appsadm' (see crontabs in 'Part_5'). Files with bad permissions migt be FTP to the site or somebody may have used 'root' to copy a file & forgot to fix permissions.

See 'chmod_custom1' '7B8' sample script that could be run before the nightly batch jobs to ensure permissions on all data directories/files 775/664 and group 'apps'. You could also reset owner to 'appsadm' if you want to see who changed what files during the day (or reset owner more infrequently). This sample script has hard-coded directories & permissions for reliability. You would customize for your site.

Note that 'root' should be used only when necessary (fixing permissions, etc). It is too dangerous to run application scripts with root privileges. Of course the chmod_custom1 script must be scheduled by a root crontab, but all batch jobs would be scheduled by 'appsadm' crontabs. And appsadm shares group 'apps' with all operators & programmers who access the data files.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1C8. ADMjobs.doc - profiles for Vancouver Utilities

stub_profile_appsadm

 # stub_profile_appsadm - this file distributed in /home/uvadm/env/...
 #                      - special version of 'stub_profile'
 # - To be copied to /home/appsadm/env/...
 #   and renamed as '.bash_profile' for Linux or '.profile' for Unix
 # - we have dropped many explanatory #cmts here in stub_profile_appsadm
 # - please study original stub_profile cmts to understand how it all works
 #
 # #1 - call 'common_defines'
 # . /home/uvadm/env/common_defines   #<-- call common_defines from 'uvadm'
 #=================================   #  - distributed in /home/uvadm/env/...
 . /home/appsadm/env/common_defines   #<-- copy to 'appsadm' for your conversions
 #=================================      - customize for your conversion
 #Note - this is stub_profile_appsadm, so uvadm is #cmtd & appsadm is uncmntd
 #
 # #2. define critical VARIABLES using $VARIABLES defined in common_defines
 #     - OR overridden above in stub_profile or stub_profile_appsadm
 export RUNLIBS=$TESTLIBS   #<-- for programmers ($PRODLIBS for operators)
 export RUNDATA=$TESTDATA   #<-- for programmers ($PRODLIBS for operators)
 export CNVDATA=$CNVDATA
 export CMPDATA=$CMPDATA
 #Note - above exports are intended for programmer profiles
 #     - RUNLIBS/RUNDATA must be changed to PRODLIBS/PRODDATA for production
 #
 # #3. call common_profile (supplied in /home/uvadm/env/...)
 # . /home/uvadm/env/common_profile  #<-- call common_profile from 'uvadm'
 #=================================  #  - distributed in /home/uvadm/env/...
 . /home/appsadm/env/common_profile  #<-- copy to 'appsadm' for your conversions
 #=================================     - customize for your conversion
 #Note - this is stub_profile_appsadm, so uvadm is #cmtd & appsadm is uncmntd
 #
 #                       ** user preferences **
 # User Preferences - override definitions in the common profile called above
 export TERM=linux         # TERM - modify depending on your terminal
 #                         # (vt100,xterm,at386,ansi,linux,etc)
 stty erase '^?'           # erase char - modify depending on your terminal
 #                         #  '^?' for linux/at386, '^H' for vt100,ansi,xterm
 export UVLPDEST="-dlp0"   # define printer for 'uvlp' scripts
 alias l='ls -l'           # code your desired aliases here
 #
 #                        ** Console Logging **
 # - the normal stub_profile has 7 ##cmt lines here that you uncmt for logging
 # - this stub_profile_appsadm omits those lines (NOT required for appsadm)
 # - see details at www.uvsoftware.ca/admjobs.htm#Part_6
 #------------------------ end of stub_profile_appsadm ------------------------

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1C9. ADMjobs.doc - profiles for Vancouver Utilities

stub_profile_appsadm_cronlogdemo

 # stub_profile_appsadm_cronlogdemo - file distributed in /home/uvadm/env/...
 #                                  - to be copied to /home/appsadm/env/...
 # Special version of profile to demo capturing logs from jobs run by cron
 # - defines RUNLIBS & RUNDATA as /home/mvstest/testlibs & testdata
 #   ==============================================================
 # - see www.uvsoftware.ca/admjobs.htm#5I1 - 5K6
 #
 # This stub_profile_appsadm_cronlogdemo called directly by 'cronscript1'
 # - which is scheduled by 'crontab2' & 'crontabtest2'
 # - since 'cron' environment has NO profile to setup PATHs, etc
 #
 # We have dropped a lot of explanatory #cmts here in cronlogdemo version
 # - see explanatory #cmts in original /home/uvadm/env/stub_profile
 # - especially see '#2X Exception coding' below required for cron/log demo
 #
 # #1 - call 'common_defines'
 # . /home/uvadm/env/common_defines   #<-- call common_defines from 'uvadm'
 #=================================   #  - distributed in /home/uvadm/env/...
 . /home/appsadm/env/common_defines   #<-- copy to 'appsadm' for your conversions
 #=================================      - customize for your conversion
 #Note - this is stub_profile_appsadm_cronlogdemo, uvadm #cmtd, appsadm uncmntd
 #
 # #2X. Exception coding to demo capturing logs from JCL/scripts run by cron
 #      - see www.uvsoftware.ca/admjobs.htm#5I1 - 5K6
 #
 #=====================================
 export TESTLIBS=/home/mvstest/testlibs #<-- test cron mail log capture
 #=====================================    - these exports are VITAL to the demo
 export TESTDATA=/home/mvstest/testdata #  - demo in ADMjobs.doc#5I1 - 5K6
 #=====================================
 #
 # #2. define critical VARIABLES using $VARIABLES defined in common_defines
 #     - OR overridden above in stub_profile or stub_profile_appsadm
 export RUNLIBS=$TESTLIBS
 export RUNDATA=$TESTDATA
 #
 # #3. call common_profile (supplied in /home/uvadm/env/...)
 # . /home/uvadm/env/common_profile  #<-- call common_profile from 'uvadm'
 #=================================  #  - distributed in /home/uvadm/env/...
 . /home/appsadm/env/common_profile  #<-- copy to 'appsadm' for your conversions
 #=================================     - customize for your conversion
 #Note - end stub_profile_appsadm_cronlogdemo, uvadm is #cmtd, appsadm uncmntd

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1D1. ADMjobs.doc - Applications Administration

appsadm - Application Administrator

We recommend you setup a login/userid 'appsadm' to serve as the applications administrator for the unix/linux site. The appsadm home directory would hold various scripts, crontabs, log files, etc used in application administration. Here are some suggested sub-directories:

 /home/appsadm
 :-----bin              - binaries for site developed/modified programs
 :-----cronlog1         - 1 line status messages from logcron1 function
 :-----cronlog2         - console messages from jobs run by cron mail
 :     :-----             captured by cronmailsave1 & date_time stamped
 :-----ctl              - control files (miscellaneous)
 :-----env              - master copies of profiles etc
 :                        modified appropriately for your site
 :-----log1             - console logging files (currently active)
 :     :-----user1        - sub-directoried by user login
 :     :-----user2
 :     :-----.etc.
 :-----log2             - console logging files (for current month)
 :     :-----user1
 :     :-----user2
 :     :-----.etc.
 :-----log3             - console logging files (for last month)
 :     :-----user1
 :     :-----user2
 :     :-----.etc.
 :-----logs             - backup logs (run by cron)
 :
 :-----pf             <-- uvcopy jobs developed/modified by site admin
 :-----sf             <-- shell scripts developed/modified by appsadm
 :-----src              - source for any programs developed/modified by appsadm
 :-----tmp

appsadm vs uvadm

Do not confuse 'appsadm' (applications administrator userid/login) with 'uvadm' (the Vancouver Utilities administrator userid/login). Please see the uvadm subdirs illustrated on page '1A1'.

One important purpose of appsadm is to hold the modified versions of any profiles, scripts & uvcopy jobs that you need to customize at your site. Copy files you need to change from /home/uvadm/.../... to /home/appsadm/.../...

 cp  /home/uvadm/env/...     /home/appsadm/env/...
 cp  /home/uvadm/sf/.../...  /home/appsadm/sf/...
 cp  /home/uvadm/pf/.../...  /home/appsadm/pf/...

This protects you form losing your customized versions when you install a future new version of Vancouver Utilities, which would overwrite /home/uvadm.

Note that the recommended profile (listed previously) searches the PATH & PFPATH of appsadm before uvadm, so any scripts & uvcopy jobs that you modify will be found before any of the original scripts/jobs.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1D2. ADMjobs.doc - Applications Administrator

Op Instrns to setup 'appsadm'

We will use 'useradd' (command line method) here, but you can use the GUI sysadm screen if you prefer. In any case you have to login as 'root' to setup a new user, but be sure to login as 'appsadm' before you setup the profile, make subdirs,& copy any files.

 #1. login as 'root'

 #2. groupadd apps        <-- setup group 'apps', if not already setup
     =============            (when uvadm was setup in install.htm)

 #3.  useradd -m -g apps -s /bin/bash appsadm   <-- setup user 'appsadm'
      =======================================
 #3a. useradd -m -d /export/home/appsadm -g apps -s /bin/bash appsadm
      ===============================================================
      - must specify '-d ...' homedir option for SUN Solaris

 #3a. passwd uvadm        <-- setup password desired
      ============

'-m' is the option to create the home directory (/home/appsadm).

'-g apps' assigns the group. Assign it as you wish, but it is VERY important that you assign the same group as for uvadm, and the programmers, analysts, & other users who are going to use the Vancouver Utilities & share information on your UNIX system. This is also related to the recommended system permissions for file read/write/execute, which extends security to the 'group' level, using 'umask 002'.

'-s /bin/bash' specifies the 'bash' shell (the default on Linux systems). For Unix I recommend 'ksh' if 'bash' not available.

These shells are much superior to 'sh' (Bourne shell, default on some Unix systems). The 'history' feature of bash & ksh is a reason enough to upgrade.

The Korn shell is recommended for all scripts - 1st line is '#!/bin/ksh'. All scripts used in the installation procedures have been verified under ksh. The JCL converters create 'ksh' shells since they use some features that are lacking in the 'bash' shell. But 'bash' is easier to use as the login shell.

While you are still logged in as 'root', you might as well setup other user logins, you will require. For example if you plan on running the test/demos described in MVSJCL.htm or VSEJCL.htm, you will need to setup 'mvstest' or 'vsetest'.

 #4a. useradd -m -g apps -s /bin/bash mvstest   <-- for MVS JCL test/demos
      =======================================
 #4b. useradd -m -g apps -s /bin/bash prgmr1    <-- setup programmers ...
      ======================================
 #4c. useradd -m -g apps -s /bin/bash oper1     <-- setup operators ...
      =====================================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1D3. ADMjobs.doc - Applications Administrator

Op Instrns setup 'appsadm' (continued)

 #5. login as 'appsadm' --> /home/appsadm

 #6a. mkdir bin cronlog1 cronlog2 ctl log1 log2 log3 logs pf sf src tmp
      =================================================================
      - setup subdirs, see '1D1'

 #6b. mkdir log1/oper1 log2/oper1 log3/oper1 log1/oper2 log2/oper2 log3/oper2 etc
      =======================================================================
      - setup subdirs matching logins that will be using console logging

 #7. cp /home/uvadm/env/* env    <-- copy profiles from uvadm to appsadm
     =========================

     bashrc               - bash 'rc' aliases req'd if console logging
     kshrc                - rename as kshrc for Korn shell (vs bash shell)
     common_defines       - defines TEST/PROD LIBS/DATA so stub_profile
                            can use $symbols to define RUNLIBS/RUNDATA
     common_profile       - common profile (called by stub_profile)
                            defines PATH's etc using $RUNLIBS/$RUNDATA
     stub_profile         - stub profile (rename to .profile or .bash_profile)
                          - modify RUNLIBS/RUNDATA for programmers & operators
     stub_profile_appsadm - alt version to copy to /home/appsadm/env & modify
     stub_profile_test    - could make diff versions for prgmrs & oprtrs
     stub_profile_prod    - for copying to homedirs of new users

modify common_defines

 #8. vi env/common_defines             <-- modify for your site
     =====================
     - Modify 'common_defines' for your site
     - modify TESTLIBS,TESTDATA,PRODLIBS,PRODDATA for your site locations

      export TESTLIBS=/p1/testlibs  #<-- examples for user sites (line 51)
      export TESTDATA=/p1/testdata  #  - see overrides for 'mvstest' below
      export PRODLIBS=/p2/prodlibs
      export PRODDATA=/p2/proddata
         --- 7 more exports to be modified depending on site locations ---

      export TESTLIBS=/home/mvstest/testlibs # - override for 'mvstest' (line 66)
      export TESTDATA=/home/mvstest/testdata # - for initial testing
         --- 6 more 'mvstest' overrides ---

      export COBDIR=/opt/microfocus/cobol
      ===================================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1D4. ADMjobs.doc - Applications Administrator

modify stub_profile_appsadm

 #9. vi env/stub_profile_appsadm       <-- modify for your site
     ===========================
     - modify 'stub_profile_appsadm'
     - define RUNLIBS/RUNDATA using common_defines $TEST/$PROD LIBS/DATA

 #10. vi env/stub_profile_appsadm    <-- other changes to stub_profiles
      ===========================
      - Make any other changes you need to the stub profile before you copy it
        to the home dirs of your programmers & operators.

 #10a. Modify TERM & 'stty erase' character depending on user's terminal
       (distribution has TERM=linux & stty erase '^?')

       export TERM=linux    # TERM - modify depending on your terminal
       =================    # (vt100,xterm,at386,ansi,etc)

       stty erase '^?'      # erase char - modify depending on your terminal
       ===============      #  '^?' for linux/at386, '^H' for vt100,ansi,xterm

 #10b. Modify UVLPDEST to a central laser printer at your site.
       (after copying stubs to user homedirs, users can modify to a nearby laser)

       export UVLPDEST="-dlp0"     <-- change 'lp0' to your laser printer
       =======================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1D5. ADMjobs.doc - Applications Administrator

additional optional changes to stub profiles

For most unix/linux OS's I think you can simply copy the supplied stub_profile to user homedirs, overwriting the default .profile or .bash_profile that is created when you setup new users (via useradd or the GUI).

But if desired you could read in the default .profile or .bash_profile, at the begining of the VU supplied stub_profile, before copying to user homedirs.

 #10c. vi env/stub_profile_appsadm    <-- additional optional? change
       ===========================
       - read in the '.profile' from your OS (SUN,HP,AIX,etc)
         at the begining of the supplied stub_profile
       - write & quit

 #10d. Should not define 'COBDIR' in user profiles. COBDIR should be defined
       only in the 'common_profile', since there is usually no need to have
       different COBDIRs for different users. See #14 on '1D7'.

      export COBDIR=/opt/microfocus/cobol
      ===================================

#11. create _test & _prod versions of stub_profile

If you are performing JCL conversions, you will need different versions of the stub_profile for programmers & operators. It is convenient to setup stub_profile_test & stub_profile_prod in /home/appsadm/env/... and then you can simply copy the appropriate version to the homedirs of your programmers & operators.

The main difference is the definition of RUNLIBS & RUNDATA which are intended to point to the 'test' or 'prod' libraries & data appropriate for the user (programmer or operator).

 #11. login appsadm --> /home/appsadm
      =============
      - we should already be in /home/appsadm with stub_profile in 'env'

 #11a. cp env/stub_profile_appsadm env/stub_profile_test
       =================================================

 #11b. cp env/stub_profile_appsadm env/stub_profile_prod
       =================================================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1D6. ADMjobs.doc - Applications Administrator

#12. Modify RUNLIBS & RUNDATA in _test & _prod stubs

Lines 48-51 of the supplied stub_profile are as follows:

 export RUNLIBS=$HOME/testlibs
 #============================
 export RUNDATA=$HOME/testdata
 #============================

These definitions work well for the JCL conversion test/demo jobs documented in MVSJCL.htm#Part_1, but you would modify for your own conversions depending on where you plan to store your own JCLs, COBOLs, & Data files.

For example the file system design described in Part_2. defines the following libraries & data file locations.

      export TESTLIBS=/p1/testlibs   #<-- define LIBS & DATA for your site
      export TESTDATA=/p1/testdata
      export PRODLIBS=/p2/prodlibs
      export PRODDATA=/p2/proddata
      export BACKUP=/p3/backup
      export RESTORE=/p3/restore
      export CNVDATA=/p4/cnvdata

 These SYMBOLS are defined in 'common_defines' distributed in /home/uvadm/env/...
 but already copied from /home/uvadm/env/* to /home/appsadm/env/... (on '1D3').

I suggest you modify these $SYMBOLS in /home/appsadm/env/... as you desire for your site & then use them to define RUNLIBS/RUNDATA in the stub_profile (also now in /home/appsadm/env/... & to be copied over .profiles in user homedirs).

We have already duplicated the original stub_profile to _test & _prod versions (on the previous page), so we would define RUNLIBS/RUNDATA as follows:

 export RUNLIBS=$TESTLIBS       <-- in stub_profile_test for programmers
 export RUNDATA=$TESTDATA

 export RUNLIBS=$PRODLIBS       <-- in stub_profile_test for operators
 export RUNDATA=$PRODDATA

Note that the stubs '.' dot execute the master_defines, so the $TEST/$PROD symbols persist in the current environment & are available for use in the stub_profiles.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1D7. ADMjobs.doc - Applications Administrator

#13. copy 'stub' profiles to homedirs

You can now copy the stub_profiles to the homedir of your programmers & operators, renaming as .bash_profile for bash (or .profile for ksh).

 #13a. cp env/stub_profile_test /home/prgmr1/.bash_profile
       ===================================================
       ... etc for other programmers ...

 #13b. cp env/stub_profile_prod /home/oper1/.bash_profile
       ==================================================
       ... etc for other operators ...

 #13c. cp /home/uvadm/env/stub_profile /home/mvstest/.bash_profile (or vsetest)
       ===========================================================

copy .bashrc to homedirs

 #14. cp env/bashrc /home/user1/.bashrc
      ==================================
      ... etc for other users (programmers & operators) ...

'.bashrc' should be copied to the homedirs of any programmers & operators who might use 'console logging' (see Part_6). Console logging is activated by uncommenting the 'script' command at the end of the profile. 'script' is another level of the shell which causes any 'aliases' & 'umaks' in the profile to be lost.

See bashrc listed on page '1C5'. It contains the same aliases & umask as the common_profile (listed on page '1C3').

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1E1. Preparations for Onsite Training & Conversion Assistance

Preparations for Onsite Visit

UV Software recommends 3 to 6 weeks of onsite training & assistance to quick-start your conversion. In the first 3 or 4 weeks, we can usually do the training & convert all the JCL, COBOL,& DATA and get you started testing. The testing & parallel running could take 6 months or a year for large sites. To get optimum results from the onsite visit, please ensure the following preparations are made.

Hardware

1. Select a server such as (in alpha order): Dell 2900, HP ProLiant ML350 G6, IBM Power 570, Sun Fire x4140. Configuration might include 8 gigs memory, 8 discs of 144 GB SCSI (or 300-500 gigs SATA), using hardware RAID to mirror 2 discs for the OS and 6 discs for application data.
   
   
2. Backup tape drive. You might select an LTO tape drive to backup the entire system. RAID makes system failure very unlikely, so I suggest regular backups only for the application libraries and data using cheaper DAT tapes. See ADMjobs.htm#2G1 for recommended supplier & tape drive models.
   
   
3. 3480/3490 tape drive. You may need a 3480/3490 tape drive to load the data tapes from your mainframe. Some sites may need a 3480/3490 to exchange data with other sites. See ADMjobs.htm#2G1 for recommended supplier and tape drive models.
   
   
4. Laser printer with Duplex capability (HP PCL5 compatible) - on the network & configured in Unix/Linux - 50 ppm simplex (30 ppm duplex) - suggestions: Lexmark T652DTN or HP laserjet P4115DN - we have scripts to fit mainframe reports (132 col x 66 lines) on 8 1/2 x 11 (by printing Landscape at 13cpi & 8lpi, Simplex or Duplex)
   
   

   Software

5. Select & install your Operating System. UV Software recommends Red Hat Linux (current version 5.1)
   
   
6. Select & install a COBOL compiler. UV Software recommends Micro Focus COBOL (current version 5.1).
   
   
7. UV Software will install the 'Vancouver Utilities Mainframe conversion pkg' the first day onsite. We might not send prior to coming onsite, because we are busy customizing the conversion utilities based on the customers sample JCLs.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1E2. Preparations for Onsite Training & Conversion Assistance

select your conversion team

8. Select your conversion team to receive the training & perform the conversion. Conversion teams are usually 3 or 4 (but we will train as many as desired). It is convenient if 1 of these can be the system administrator for your unix/linux system.
   
   
9. Note that somebody on the team must have the root password. We will not normally need root access but we will need it to install the Vancouver Utilities & sometimes transferred files have incorrect permissions that we can correct only with root access.
   
   

   setup users, profiles,& file systems

10. Setup user login accounts for your team. Also setup 'uvadm' & 'appsadm' as recommended by UV Software. See http://www.uvsoftware.ca/install.htm Assign a common groupid to the team users, uvadm,& appsadm (suggest 'apps'). Note that the supplied profiles have umask 002 so files will be 664 & directories 755, which allows our group to work on the common set of libraries & data.
    
    
11. See recommended profiles at: http://www.uvsoftware.ca/admjobs.htm#Part_1 A 'stub_profile' is installed in each users homedir which calls a common_defines & a common_profile stored in the 'appsadm' homedir. These are powerful concepts that make system administration much easier.
    
    
12. Plan the directory structure you would like to use for your libraries & data, see the suggestions at http://www.uvsoftware.ca/admjobs.htm#Part_2.
    

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1E3. Preparations for Onsite Training & Conversion Assistance

transfer Mainframe files to Unix/Linux

13. Transfer Mainframe JCL to Unix/Linux - if your mainframe does not have 'zip', transfer to a PC, also translating to ASCII, then zip & FTP to Unix/Linux - be sure to transfer all referenced PROCs & SYSIN/INCLUDE modules (these should be in separate zip files) - it is a good idea to date stamp your zip files, for example: you might name them as jclYYMMDD.zip & procYYMMDD.zip
    
    
14. Transfer Mainframe COBOL to Unix/Linux - transfer to a PC, translating to ASCII, then zip & FTP to Unix/Linux - be sure to transfer all copybooks in a separate zip file - zip files might be named as cblYYMMDD.zip & cpyYYMMDD.zip
    
    
15. Transfer Mainframe control card libraries (SYSIN, PARM) - zip file might be named as parmYYMMDD.zip
    
    
16. Transfer Mainframe DATA to Unix/Linux - must FTP binary (FTP ASCII will destroy packed/binary fields) - might not transfer all data files before the onsite visit begins, but please transfer enough data to test some JCL/scripts - for initial testing, can you identify some JCLs that require only sequential & indexed (VSAM) files, with no database or online preparation.
    

17. Transfer Mainframe LISTCAT reports that describe the DATA files. - UV Software will extract data file information into control files to improve the conversion of both JCL & DATA
    
    

    JCL/COBOL preview by UV Software

For a faster and more efiicient conversion, please send us all JCL,PROCs,Parms, COBOL,& Copybooks several weeks prior to the scheduled on-site conversion. We will spend up to 1 week (at no extra charge) investigating & optimizing the conversions for your particular coding habits.

No DATA files are required. But if desired, you could send the data files for just 1 small standalone system for us to parrallel & return the reports for you to compare to the mainframe.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1E4. Preparations for Onsite Training & Conversion Assistance

Classroom Setup

18. Classroom or boardroom table - PC's on network to Unix/Linux - terminal emulator (PUTTY is a free download) - projector & screen - if easy, enough PCs for each student & UV Software - Could manage with minimum 1 terminal (students take turns on keyboard) - for only 2 or 3 students could even manage without projector if we have a 21 inch monitor - laser printer in the classroom or nearby (on network & configured to unix)
    
    
19. Classroom hours: - Let UV Software know what hours will work for you & your personnel - We know they probably have other ongoing responsibilities, and we have many other conversion activities to perform when not in class. - Might be 1/2 days 5 days a week - Or might be full days Tues, Wed, Thurs (some personnel may have Mon/Fri off)
    
    
20. Internet access - the terminal on the network to unix/linux is usually a PC with terminal emulator (Putty) & therefore should have access to the Internet - We have found it convenient to access the UV Software Internet web site to display documentation which students can relate to their hardcopy doc - We also need to be able to read our email via webmail to our ISP
    

other preparations

21. Please discuss these preparations with UV Software before we arrive on-site. Please tell us about the backgrounds of the conversion team members, so we can have a better idea of the training required.
    
    
22. Please arrange security cards so UV Software can work at least part of the weekends. If required ask 1 staff member to work with UV Software.
    
    
23. Please study the optional enhancements that can be made during the conversion (6 pages begining on page 'K0'). Please discuss with UV Software.
    
    
24. If you think of any additonal enhancements that you would like to make during the conversion, please discuss them with UV Software. We often make suggested enhancements at no charge when we think such enhancements would be useful to other future customers.
    

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

1F1. ADMjobs: Preparations for Onsite Training & Conversion Assistance

On-site Conversion Training Plan

UV Software recommends a 4 or 5 week on-site visit to train the conversion team in using the Vancouver Utilities to convert mainframe JCL, COBOL,& DATA to Korn shell scripts, Micro Focus COBOL,& ASCII data files for Unix/Linux systems.

1. First we will study sample conversions which list the original JCL followed by the converted Korn shell script equivalents for several test/demo jobs. See the sample conversions at http://www.uvsoftware.ca/mvsjcl.htm#Part_1.
   
   
2. After we understand the JCL conversions, we will perform them & execute the resulting Korn shell scripts using supplied test files. MVSJCL.htm#Part_1 also lists the expected outputs from the demo JCL/scripts & COBOL programs.
   
   
3. For mainframers new to Unix/Linux, there is much to be learned about the Unix/Linux environment required to run jobs on Unix/Linux. It is vital to understand the Vancouver Utility profiles & functions that support the execution of converted scripts. See http://www.uvsoftware.ca/admjobs.htm.
   
   
4. In addition to learning the Vancouver 'Conversion Utilities' it is vital to learn how to use the Vancouver 'Data Utilities' (uvsort,uvcp,uvhd,uvcopy,etc) that replace various mainframe utilities (SORT,IDCAMS,DITTO,Easytrieve,etc). 'uvhd' is the first utility that students need to learn - to investigate & verify conversions of mainframe data files with packed/binary fields.
   
   
5. Rather than teach the Vancouver 'Data Utilities' in isolation, we will teach them when the need arises, during our conversion & execution of the JCL/COBOL test/demo jobs. For a summary of the utilities most useful to conversions, please see http://www.uvsoftware.ca/cnvaids.htm.
   
   
6. After the sample conversions & executions in MVSJCL.htm#Part_1, we will begin conversion of the customer JCL, COBOL,& DATA, following the step by step guide in http://www.uvsoftware.ca/mvsjcl.htm#Part_4.
   
   
7. The mass conversion scripts make it easy to quickly convert all mainframe JCL and COBOL files. The Vancouver Utilties also make it easy to generate the 'DATA' conversion jobs (from COBOL copybooks), but executing them takes much longer than for JCL & COBOL, because the DATA files are usually much larger, and they first have to be transferred from the mainframe.
   
   
8. Our initial objective is to convert enough data files to allow us to begin testing some JCL/scripts & COBOL programs. Then we will assign each team member a JCL/script to be tested & the results compared to the mainframe.
   
   
9. It usually takes 2 or 3 weeks to get to this point (where we can begin testing). The team members can now work independently testing their assigned systems & consulting with the UV Software instructor as required.
   
   
10. For the remaining 2 or 3 weeks that UV Software is onsite, we usually meet in the classroom for only 2 hours a day to discuss common problems & continue training on the utilities that are relevant to the site (file maintenance, Easytrieve conversions, etc).
    

See more details at uvprices.htm#I1 thru I10

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_2 RAID Arrays & File Designs - Basic & Alternate

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2A1. ADMjobs: Administration for Unix/Linux

Installing Red Hat Enterprise Linux

UV Software recommends Red Hat Enterprise Linux (current version 5.2).

As of July 2008, UV Software is using RHEL 5.1 for software development. We ordered the DVD media kit & 1st year basic support for only $349. 'basic support' means no telephone support, you post help requests on the Red Hat support web site, but this works very well.

For production sites, you would probably want telephone support and a 3 or 5 year term.

Install Guide for RHEL

Since the media kit does not include hard-copy documentation, we recommend you download & print the following 2 manuals from the Red Hat website.

 https://www.redhat.com/docs/manuals/enterprise  <-- RHEL manuals

1. Installation Guide - download 'pdf' document (370 pages)
   
   
2. Deployment Guide - download 'pdf' document (900 pages)
   

We printed Duplex on 3 hole punched paper on our 35 ppm laser printer and mounted in 3 ring binders.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2B1. ADMjobs: RAID, Partitioning,& Making File Systems

RAID & Partitioning

Here is a suggested RAID & Partition design that might be used in a typical small server, such as the Dell 2800, using Red Hat Enterprise Linux. Our example will configure 6 * 73 gig SCSI discs into 2 RAID arrays as follows:

1. Array#1 - 2 * 73 gig discs, RAID1, mirrored, effective storage 73 gig.
   
   
2. Array#2 - 4 * 73 gig discs, RAID5, striped, effective storage 220 gig.
   

RAID1 sda - for O/S & home dirs

The Red Hat O/S install procedure provides the 'disc druid' GUI program that makes it easy to setup the desired partitions. I suggest the RAID1 O/S array (sda) would be partitioned as follows:

 / (root) 20 gig
 /boot     1 gig
 /swap     4 gig
 /tmp      2 gig
 /usr     20 gig
 /var      6 gig
 /home    20 gig
     ------------
    total 73 gig

Each user will of course have a /home directory, but the home dirs should not be used for production data or libraries (which should always be maintained on the RAID5 array as documented in the next section below).

You might also use /home dirs for for 3rd party software packages, for example:

 /home/uvadm   - Vancouver Utilities
 /home/appsadm - Applications Administrator
 /home/tipadm  - Inglenet TIP/ix

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2B2. ADMjobs: RAID, Partitioning,& Making File Systems

RAID5 sdb - for application Data & Libraries

I suggest the RAID5 array 'sdb' might be partitioned into 4 file systems of about 55 gig each (assuming the 4 * 73 gig discs provide 220 gig effective storage of the 292 physical disc storage). We will name the partitions as p1,p2,p3,p4 and we intend to use them as follows:

 /p1 - test-libraries & test-data
 /p2 - production-libraries & production-data
 /p3 - backup & restore
 /p4 - conversion data

This partitioning makes our backups & restores easier to handle. Another significant point is that a run away program cannot fill up the whole file system since it would be restrained to the production or test data partition.

Partitioning RAID5 sdb

The vendor (Dell, etc) might install the OS on the RAID1 array, but would not usually partition the RAID5 application array. It is easier to partition sdb during OS install on sda, since there is usually a GUI program to partition discs during the OS install.

On Red Hat Enterprise, the GUI program 'disc druid' cannot be used after the OS install, but we can use the command line tools 'fdisk' & 'mkfs'. You can look up the 'man' pages to see how to run these commands, but briefly:

Note that Intel hardware & OS's such as Linux only allow 4 primary partitions on a disc (or RAID array). They do allow partition#4 to be specified as an 'extended' partition & subdivided into many more partitions. But we will assign the 4 primary partitions as 55 gig each.

Here is a summary of the steps required to partition our disc array. The detailed instructions follow on the next few pages.

 #1. fdisk /dev/sdb                   <-- partition the disc
                                        - see details on following page

 #2. mkfs.ext3 /dev/sdb1,2,3,4        <-- make file systems

 #3. e2label /dev/sdb1 p1,p2,p3,p4    <-- assign Labels to disc partitions

 #4. mkdir /p1 /p2 /p3 /p4            <-- make mount points

 #5. vi /etc/fstab                    <-- setup mount commands for reboots

Note that some of the sample printouts are from my partitioning of 1 147 gig non-raid disc vs the 4 * 73 gig disc RAID array at a customer site.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2B3. ADMjobs: RAID, Partitioning,& Making File Systems

Partition the disk with 'fdisk'

 #1. fdisk /dev/sdb     <-- start interactive session
     ==============       - to partition sdb

 #1a. n          <-- make 'n'ew partition sdb1 (55 gig)
      --> p      <-- 'p'rimary partition
      --> 1      <-- first cylinder
      --> 4400   <-- last cylinder

 #2b. n          <-- make 'n'ew partition sdb2 (55 gig)
      --> p      <-- 'p'rimary partition
      --> 4401   <-- first cylinder
      --> 8800   <-- last cylinder

 #2c. n          <-- make 'n'ew partition sdb3 (55 gig)
      --> p      <-- 'p'rimary partition
      --> 8801   <-- first cylinder
      --> 13200  <-- last cylinder

 #2d. n          <-- make 'n'ew partition sdb3 (55 gig)
      --> p      <-- 'p'rimary partition
      --> 13201  <-- first cylinder
      --> 17849  <-- last cylinder

 #2e. p    <-- print partition table

      Disk /dev/sdb: 146.8 GB, 146815737856 bytes
      255 heads, 63 sectors/track, 17849 cylinders
      Units = cylinders of 16065 * 512 = 8225280 bytes
         Device Boot      Start         End      Blocks   Id  System
      /dev/sdb1               1        4400    35342968+  83  Linux
      /dev/sdb2            4401        8800    35343000   83  Linux
      /dev/sdb3            8801       13200    35343000   83  Linux
      /dev/sdb4           13201       17849    37343092+  83  Linux

 #2f. w    <-- write partition table & exit

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2B4. ADMjobs: RAID, Partitioning,& Making File Systems

mkfs - make file systems

Actually, we will use 'mkfs.ext3', which is the Red Hat version of 'mkfs' that will create a journalling file system (fstype=ext3).

 #2a. mkfs.ext3 /dev/sdb1    <-- make ext3 file system on sdb1
 #2b. mkfs.ext3 /dev/sdb2               ... etc ...
 #2c. mkfs.ext3 /dev/sdb3
 #2d. mkfs.ext3 /dev/sdb4

Label the File Systems

 #3a. e2label /dev/sdb1 p1
 #3b. e2label /dev/sdb2 p2
 #3c. e2label /dev/sdb3 p3
 #3d. e2label /dev/sdb4 p4

make mount points

Next we need to make 'mount points' (empty directories at the / root level) for mounting our newly created file systems.

 #4. mkdir /p1 /p2 /p3 /p4    <-- make mount points for file systems
     =====================

mounting the file systems

 #5a. mount /dev/sdb1 /p1     <-- mount file system /dev/sdb1 on /p1
 #5b. mount /dev/sdb2 /p2                 ... etc ...
 #5c. mount /dev/sdb3 /p3
 #5d. mount /dev/sdb4 /p5

 #5e. mount -l               <-- display all mounted filesystems
      ========

      /dev/sda1 on /boot type ext3 (rw) [/boot1]
      /dev/sda2 on /     type ext3 (rw) [/]
      /dev/sda3 on /home type ext3 (rw) [/home]
      /dev/sda4 on /var  type ext3 (rw) [/var]
      /dev/sdb1 on /p1   type ext3 (rw) [/p1]
      /dev/sdb2 on /p2   type ext3 (rw) [/p2]
      /dev/sdb3 on /p3   type ext3 (rw) [/p3]
      /dev/sdb4 on /p4   type ext3 (rw) [/p4]

The above 'mount' commands are not permanent. We need to edit /etc/fstab to have the mounts performed automatically on subsequent reboots. Please see the next page.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2B5. ADMjobs: RAID, Partitioning,& Making File Systems

add mount commands to /etc/fstab

 #6. vi /etc/fstab
     =============

    label       mount-point  file-type  defaults  dump fsck
 ==========================================================

 LABEL=/             /          ext3    defaults    1   1
 LABEL=/boot1        /boot      ext3    defaults    1   2
 LABEL=/p1           /p1        ext3    defaults    1   2    <-- add
 LABEL=/p2           /p2        ext3    defaults    1   2    <-- add
 LABEL=/p3           /p3        ext3    defaults    1   2    <-- add
 LABEL=/p4           /p4        ext3    defaults    1   2    <-- add
 LABEL=/home         /home      ext3    defaults    1   2
 proc                /proc      proc    defaults    0   0
 LABEL=/var          /var       ext3    defaults    1   2
 LABEL=SWAP-sda3     swap       swap    defaults    0   0

1. /etc/fstab is listed above AFTER 4 lines inserted to mount the new partitions automatically on reboots
   
   
2. The easy way to insert them is to duplicate an existing line (such as the 'boot' or 'home' line) & change the LABEL & mount point.
   
   
3. Alternatively, you could use the GUI disc management tools to mount partitions either temporarily or permanently.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2C0. ADMjobs: File System Design

In the previous section, we setup a RAID array, partitioned it, and made file systems on each partition (p1,p2,p3,p4). After adding directories for our application libraries and data, our file systems will look something like the following:

 /root
 /...              <-- unix/linux O/S directories
 /home             <-- home directories
 :----user1
 :----etc---
 :
 /p1                 <---- /p1 file system mount point
 :-----testlibs        <-- RUNLIBS=$TESTLIBS=/p1/testlibs
 :     :-----cbls        - COBOL programs
 :     :-----jcls        - JCL/scripts
 :     :---etc---        - see other subdirs at ADMjobs.htm#2C1
 :-----testdata      <---- RUNDATA=$TESTDATA=/p1/testdata
 :     :-----mstr        - data files (or use topnodes as subdirs)
 :     :-----jobtmp      - job temporary files
 :     :---etc---        - see other subdirs at ADMjobs.htm#2C2
 /p2                 <---- /p2 file system mount point
 :-----prodlibs        <-- RUNLIBS=$PRODLIBS=/p2/prodlibs
 :     :-----cbls        - COBOL programs (production)
 :     :-----jcls        - JCL/scripts (production)
 :     :---etc---
 :-----proddata
 :     :-----mstr        - data files (or use topnodes as subdirs)
 :     :-----jobtmp      - job temporary files
 /p3                 <---- /p3 file system mount point
 :-----backup            - backup & restore directories
 :-----restore
 /p4                 <---- /p4 file system mount point
 :-----cnvdata           - data conversion directories
 :     :----d1ebc        - EBCDIC data files from mainframe
 :     :----d2asc        - converted to ASCII (preserving packed)

It is important to make separate file systems for our application libraries & data. You should not use /home directories for these because:

1. The /home file system would not usually have sufficient space allocated for your large data files.
   
   
2. You should have separate file systems to protect against runaway COBOL programs filling up the entire file system, rather than just the data file system. Other file systems are also protected against corruption in any 1 file system.
   
   
3. You should have separate file systems designed to facilitate your backup & restore procedures. You should be able to do backup/restores separately for testlibs, testdata, prodlibs,& proddata.
   
   
4. Separate file systems facilitate writing backup tapes with multiple archives. The restore script allows you to restore just the desired archive, rather than the entire tape, saving time & disc space.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2C1. ADMjobs: Directory Design for YOUR Conversion & Testing

Libraries for Your Conversion & Testing

 /p1/testlibs
 :--*--cbl0          - COBOL programs ('*' means files present)
 :-----cbl1          - cleaned up, cols 1-6 & 73-80 cleared, etc
 :-----cbl2          - cnvMF5 converts mainframe COBOL to MicroFocus COBOL
 :-----cbls          - copy here (standard source library) before compiling
 :-----cblx          - compiled COBOL programs (.int's)
 :--*--parm0         - control cards (SORT FIELDS, etc)
 :-----parms         - control cards with 73-80 cleared
 :--*--cpy0          - for COBOL copybooks
 :-----cpy1          - cleaned up, cols 1-6 & 73-80 cleared, etc
 :-----cpy2          - cnvMF5 converts mainframe COBOL to MicroFocus COBOL
 :-----cpys          - copy here (standard copybook library)
 :--*--jcl0          - test/demo JCLs supplied
 :-----jcl1          - intermediate conversion 73-80 cleared
 :-----jcl2          - PROCs expanded from procs
 :-----jcl3          - JCLs converted to Korn shell scripts
 :-----jcls          - copy here manually 1 by 1 during test/debug
 :--*--proc0         - test/demo PROCs supplied
 :-----procs         - will be merged with jcl1, output to jcl2
 :-----prns          - .prn files from MS WORD 'print to a file' for overlays
 :-----ovls          - overlays to print forms+data (see pcloverlay & uvoverlay)
 :-----rpts          - for optional statistics reports
 :-----sf            - for misc scripts you may wish to write
 :-----tmp           - tmp subdir used by various conversions

DATA Directories for Your Testing

 /p1/testdata
 :-----ap          <-- directories created for topnodes of data filenames
 :-----ar              (Accounts Payable, Accounts Receivable, etc)
 :-----gl
 :-----py
 :-----jobctl      <-- working directories shared by all applications
 :-----joblog
 :-----jobtmp
 :-----rpts        <-- reports created by COBOL programs
 :-----rptsovls    <-- some reports can be printed with overlays (see uvoverlay)
 :-----sysout
 :-----tmp
 :-----wrk

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2C2. Directory Design for YOUR Conversion, Testing,& Production

Libraries for Your Production

 /p2/prodlibs
 :-----cbls          - end point for converted COBOL programs
 :-----cblst         - cobol source listings from compiles
 :-----cblx          - compiled COBOL programs (.int's)
 :-----cpys          - converted, ready for compiles
 :-----jcl3          - JCLs converted to Korn shell scripts
 :-----jcls          - copy here manually 1 by 1 during test/debug
 :-----prns          - .prn files from MS WORD 'print to a file' for overlays
 :-----ovls          - overlays to print forms+data (see pcloverlay & uvoverlay)
 :-----pf            - uvcopy jobs to replace utilities (easytrieve,etc)
 :-----sf            - for misc scripts you may wish to write
 :-----sfun          - korn shell functions (jobset41,logmsg,etc)
 :-----tmp           - tmp subdir used by various conversions

Please compare these production libraries to the conversion & testing libraries on the preceding page. Note that many original mainframe & intermediate conversion subdirs have been dropped, retaining only the fully converted subdirs of COBOL programs, copybooks,& JCL(now Korn shell scripts).

DATA Directories for Your Production

 /p2/proddata
 :-----ap          <-- directories created for topnodes of data filenames
 :-----ar              (Accounts Payable, Accounts Receivable, etc)
 :-----gl
 :-----py
 :-----jobctl      <-- working directories shared by all applications
 :-----joblog
 :-----jobtmp
 :-----rpts        <-- reports created by COBOL programs
 :-----rptsovls    <-- some reports can be printed with overlays (see uvoverlay)
 :-----sysout
 :-----tmp
 :-----wrk

Also note that we have added some subdirs, such as 'prns' & 'ovls' in prodlibs and 'rptsovls' in proddata. These are used to form overlays that might be used to print some of your reports created by your COBOL programs.

See uvoverlay.htm which documents these procedures & gives an example of printing letters with a letterhead overlay.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2C3. Directory Design for YOUR Conversion, Testing,& Production

Directories for Backup & Restore

 /p3/backup
 :-----homedirs             <-- $HOMEDIRS backup from last night
 :     :-----appsadm          - showing only 1 user to save lines
 :     :     :-----ctl          & showing only a few subdirs in 1st user
 :     :     :-----env
 :     :     :-----logs
 :     :     :-----...
 :-----homedirs-1           <-- $HOMEDIRS backup from 2 nights ago
 :     :-----...same as above...
 :-----proddata             <-- $PRODDATA backup from last night
 :     :-----ap
 :     :-----ar
 :     :-----gl
 :     :-----rpts
 :     :-----wrk
 :-----proddata-1           <-- $PRODDATA backup from 2 nights ago
 :     :-----...same as above...
 :-----prodlibs             <-- $PRODLIBS backups from last night
 :     :-----cbls
 :     :-----cpys
 :     :-----ctl
 :     :-----jcls
 :     :-----parms
 :-----prodlibs-1           <-- $PRODLIBS backup from 2 nights ago
 :     :-----...same as above...
 :-----zip                  <-- last nights backup (only)
 :     :-----homedirs_070529_0301.zip
 :     :-----proddata_070529_0302.zip  <-- sample for May 29/2007
 :     :-----prodlibs_070529_0303.zip
 :     :-----...
 :-----Day                  <-- Daily backups in .zip files for last 40 days
 :     :-----homedirs_070419_0301.zip
 :     :-----proddata_070419_0302.zip  <-- 40 days ago = April 19/2007
 :     :-----prodlibs_070419_0303.zip
 :     :-----...(39 sets not shown)
 :-----Month                <-- Monthly backups in .zip files for last 15 months
 :     :-----homedirs_060201_0301.zip
 :     :-----proddata_060201_0302.zip  <-- 15 months ago = Feb 1/2006
 :     :-----prodlibs_060201_0303.zip
 :     :-----...(14 sets not shown)
 :-----Year                 <-- Yearly backups in .zip files for last 7 years
 :     :-----homedirs_000501_0301.zip
 :     :-----proddata_000501_0302.zip  <-- 7 years ago = Jan 1/2000
 :     :-----prodlibs_000501_0303.zip
 :     :-----...(6 sets not shown)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2D1. ADMjobs: Directory & File-Design - Principles

Directory & File-Design - Principles

1. A good Directory Design is critical to the success of your conversion and to the future efficiency & flexibility of your unix/linux installation.
   
   
2. The basic principle is to have multiple separate file systems and group your libraries & data into file systems in a manner that facilitates backup & restore. You can group libraries & data under super-directories (prodlibs,proddata,testlibs,testdata) that allow you to backup to multiple separate archives on tape & if necessary restore the desired archive without having to restore the entire tape (saving time & disc space).
   
   
3. Separate file systems limit data corruption & protects against runaway programs. Runaway programs can fill up the Test Data file system without harming production data or libraries. Data directories can be very large & should be assigned to large file systems (library directories of programs & JCL are comparatively small).
   
   
4. Testing & Production will probably be on separate machines, but this design allows them to be on 1 machine. You could also have different test libraries & data for different programmers
   
   
5. This design follows good unix/linux principles. There are at least 3 or 4 levels of directories before files are assigned. This gives us more flexibility & better file protection.
   
   
6. The design might allow for multiple related companies (aaco,bbco,ccco,etc), and/or multiple applications (apay,arcv,ordr,etc).
   
   
7. Backup tapes are written as multiple archives. This makes it easy to recover systems separately. It is easy to have enough disc space to restore just the system in question & then retrieve desired files over to production data or libraries.
   
   
8. On-disc backup is recommended for at least 1 day's data files & 2 days libraries. This is very convenient if you discover problems within 1 or 2 days. Programmers have the assurance that they can quickly recover from mistakes (rm *).
   
   

   RUNLIBS & RUNDATA

RUNLIBS & RUNDATA are the 2 critical environmental variables (defined in the profile of programmers & operators) that point to the appropriate Libraries & Data for Testing & Production (prodlibs/proddata or testlibs/testdata).

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2D2. ADMjobs: Directory & File-Design - Principles

RUNLIBS, RUNDATA,& CNVDATA - Concepts

1. RUNLIBS, RUNDATA,& CNVDATA are environmental variables (in the .profile of programmers & operators) that control access to JCL/scripts/programs & Data files.
   
   
2. When programmers & operators enter JCL/script names to be executed, RUNLIBS (in the .profile) determines which Library directories are searched (Test or Production).
   
   
3. When the JCL/script executes, RUNDATA determines which directories of Data files are accessed (Test data or Production data).
   
   
4. CNVDATA is a convenience during conversion. $CNVDATA is coded in the conversion scripts so you can easily change locations just by changing your .profile.
   

5. RUNLIBS & RUNDATA mean you do not have to modify your JCL/scripts as you move them from testing machines to production machines. RUNLIBS & RUNDATA in the login .profiles supply the top-level portion of the pathnames to the Libraries & Data files.
   
   
6. RUNLIBS & RUNDATA allow you to run various systems separately on the same machine. This is important if you want to maintain separate console logs for the various systems.
   
   
7. RUNLIBS & RUNDATA allow you to control which operators can run which systems. Operators cannot run unauthorized systems, since their .profile PATH & RUNLIBS will not find the unauthorized JCL/scripts. Operators that need to run multiple systems would be provided with the multiple logins required. Terminal emulators make it easy to run the multiple sessions on 1 physical terminal.
   
   
8. Dedicated logins can be used to enforce the tight control required for computer room batch shift operations. You might setup logins that match the system/application IDs. Examples might be apay,arcv,ordr for separate applications, or aaco,bbco,ccco,etc for separate companies.
   
   
9. I suggest the above userids be used for your production operators. Your programmers would use their personal logins, but modify RUNLIBS & RUNDATA in their .profile as required for testing the various systems.
   
   
10. The .profiles also include aliases (cdl,cdd,& cdc) which allow you to easily switch between the superdirs ($RUNLIBS,$RUNDATA,$CNVDATA).
    

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2E1. ADMjobs: Directory & File-Design - Principles

Programmer & Operator Logins

The preceding pages have discussed our file design, which might be summarized as follows:

 /home             <-- home directories
 :----prgmr1
 :----prgmr2
 :----oper1
 :----oper2
 /p1                 <---- /p1 file system mount point
 :-----testlibs        <-- RUNLIBS=$TESTLIBS=/p1/testlibs
 :     :-----cbls        - COBOL programs
 :     :-----jcls        - JCL/scripts
 :     :---etc---        - see other subdirs at ADMjobs.htm#2C1
 :-----testdata      <---- RUNDATA=$TESTDATA=/p1/testdata
 :     :-----mstr        - data files (or use topnodes as subdirs)
 :     :-----jobtmp      - job temporary files
 :     :---etc---        - see other subdirs at ADMjobs.htm#2C2
 /p2                 <---- /p2 file system mount point
 :-----prodlibs        <-- RUNLIBS=$PRODLIBS=/p2/prodlibs
 :     :-----cbls        - COBOL programs (production)
 :     :-----jcls        - JCL/scripts (production)
 :     :---etc---
 :-----proddata
 :     :-----mstr        - data files (or use topnodes as subdirs)
 :     :-----jobtmp      - job temporary files
 /p3                 <---- /p3 file system mount point
 :-----backup            - backup & restore directories
 :-----restore
 /p4                 <---- /p4 file system mount point
 :-----cnvdata           - data conversion directories
 :     :----d1ebc        - EBCDIC data files from mainframe
 :     :----d2asc        - converted to ASCII (preserving packed)

RUNLIBS/RUNDATA = testlibs/testdata OR prodlibs/proddata

RUNLIBS & RUNDATA are assigned in the profiles, which were discussed & listed begining on page '1C1. 'common_defines' should define LIBS & DATA for use by the 'stub_profiles' as shown below:

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2E2. ADMjobs: Directory & File-Design - Principles

LIBS/DATA defs in 'common_defines'

      export TESTLIBS=/p1/testlibs
      export TESTDATA=/p1/testdata
      export PRODLIBS=/p2/prodlibs
      export PRODDATA=/p2/proddata
      export BACKUP=/p3/backup
      export RESTORE=/p3/restore
      export CNVDATA=/p4/cnvdata

RUNLIBS/RUNDATA defs in 'stub_profiles'

      export RUNLIBS=/p1/testlibs       <-- stub_profile_test for programmers
      export RUNDATA=/p1/testdata

      export RUNLIBS=/p2/prodlibs       <-- stub_profile_prod for operators
      export RUNDATA=/p2/proddata

aliases for quick changes

The profiles also define several aliases that make it easy for programmers & operators to get to various frequently used directories. 'cdl' & 'cdd' are especially convenient.

 alias cdl='cd $RUNLIBS'     <-- /p1/testlibs(prgmrs) or /p2/prodlibs(oprtrs)

 alias cdd='cd $RUNDATA'     <-- /p1/testdata(prgmrs) or /p2/proddata(oprtrs)

 alias cdb='cd $BACKUP'      <-- /p3/backup

 alias cdr='cd $RESTORE'     <-- /p3/restore

 alias cdc='cd $CNVDATA'     <-- /p4/cnvdata

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2F0. ADMjobs: File System Design Alternatives

On pages '2B1'-2B3, we setup a RAID array, partitioned it, and made file systems on each partition (p1,p2,p3,p4). On pages '2C1'-2C3, we suggested a basic directory design for application libraries & data (testlibs, testdata, proddata, prodlibs,& backups).

In this section we will present some alternatives in case you need multiple sets of libraries & data.

We will follow the basic design with some alternative designs for organizations with multiple companies &/or multiple separate applications on the same machine.

Basic Design - High Level (big picture)

 /p1               <-- /p1 file system mount point
 :----testlibs       - test-libraries & test-data
 :----testdata
 /p2               <-- /p2 file system mount point
 :----prodlibs       - production-libraries & production-data
 :----proddata
 /p3               <-- /p3 file system mount point
 :----backup         - backup & restore directories
 :----restore
 /p4               <-- /p4 file system mount point
 :----cnvdata        - data conversion directories
 :    :----d1ebc
 :    :----d2asc

From here on, we will illustrate basic & alternative designs only for 'proddata'. The libraries could have a similar design, or they may be more integrated (if the same programs & scripts are used for different companies).

Note that the 'RUNLIBS' & 'RUNDATA' definitions in the profiles will facilitate these alternatives. For production operators using the basic design above, the definitions would be:

 export PRODLIBS=/p2/prodlibs
 ============================

 export PRODDATA=/p2/proddata
 ============================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2F1. ADMjobs: File System Design

Basic Design - proddata only

 /p2/proddata
 :-----ap          <-- directories created for topnodes of filenames
 :-----ar
 :-----gl
 :-----py
 :-----jobctl      <-- standard directories shared by all applications
 :-----joblog
 :-----jobtmp
 :-----rpts
 :-----sysout
 :-----tmp
 :-----wrk

These directory illustrations are created by the 'dtree' script and show only directories (no files). But in the following illustrations, I will show a few data files to ensure your complete understanding.

When we convert mainframe data files, we use the top-node as a sub-directory within 'proddata' (path defined by $RUNDATA). We also convert to lower case. Here are a few examples:

 AR.CUSTOMER.MASTER        <-- Mainframe file naming conventions
 AR.SALES.ITEMS
 GL.ACCOUNT.MASTER
 GL.ACCOUNT.TRANS

/p2/proddata/ar/customer.master /p2/proddata/ar/sales.items /p2/proddata/gl/account.master /p2/proddata/gl/account.trans

Basic Design dtree - with sample data files

 /p2/proddata
 :-----ar
 :     :-----customer.master
 :     :-----sales.items
 :-----gl
 :     :-----account.master
 :     :-----account.trans

The following pages will show some alternatives to this basic design, using only these 2 subdirs & 4 files for illustration purposes.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2F2. ADMjobs: Alternative File System Designs

Multi Separate Companies - format #1

We will describe these alternate designs as 'multi-company' but of course these designs could apply to divisions, departments, or whatever is relevant.

 /p2/aaco                            <-- company 'aa'
 :-----proddata
 :     :-----ar
 :     :     :-----customer.master
 :     :     :-----sales.items
 :     :-----gl
 :     :     :-----account.master
 :     :     :-----account.trans
 /p2/bbco                            <-- company 'bb'
 :-----proddata
 :     :-----ar
 :     :     :-----customer.master
 :     :     :-----sales.items
 :     :-----gl
 :     :     :-----account.master
 :     :     :-----account.trans
 /p2/prodlibs
 :     :-----cbls
 :     :-----jcls
 :     :------etc-

The above design could be used when there is no interaction required between the companies. When processing 'aaco', there is no need to access any files in 'bbco' & vice-versa. RUNDATA in operator profiles would be defined as 1 of the following 2:

 export RUNDATA=/p2/aaco/proddata
 ================================
        - - - OR - - -
 export RUNDATA=/p2/bbco/proddata
 ================================

But RUNLIBS could define the same set of libraries if the same programs & JCL/scripts could be used for both companies:

 export RUNLIBS=/p2/prodlibs
 ===========================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2F3. ADMjobs: Alternative File System Designs

Multi Separate Companies - format #2

In this design, we will code the company directory following 'proddata' rather than preceding. This still does not allow company file sharing.

You might use format#1 (previous page) if you wanted to be able to backup all data for any 1 company into 1 archive.

You might use format#2 (this page) if you wanted to backup all data for all companies into 1 archive.

 /p2/proddata
 :-----aaco                            <-- company 'aa'
 :     :-----ar
 :     :     :-----customer.master
 :     :     :-----sales.items
 :     :-----gl
 :     :     :-----account.master
 :     :     :-----account.trans
 :-----bbco                            <-- company 'bb'
 :     :-----ar
 :     :     :-----customer.master
 :     :     :-----sales.items
 :     :-----gl
 :     :     :-----account.master
 :     :     :-----account.trans
 /p2/prodlibs
 :     :-----cbls
 :     :-----jcls
 :     :------etc-

Like format#1, this design does not allow interaction between the companies if the same set of JCL/scripts is used for both companies. RUNDATA in operator profiles would be defined as 1 of the following 2:

 export RUNDATA=/p2/proddata/aaco
 ================================
        - - - OR - - -
 export RUNDATA=/p2/proddata/bbco
 ================================

RUNLIBS could define the same set of libraries if the same programs & JCL/scripts could be used for both companies:

 export RUNLIBS=/p2/prodlibs
 ===========================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2F4. ADMjobs: Alternative File System Designs

Multi Company - allowing interactions

We will discuss this under 2 scenarios, depending on whether the mainframe file naming allowed for multi-company or whether multi-companies are to be added after the conversion from mainframe to unix/linux.

#1 - Mainframe file names were multi-company

In this scenario some data filenames might be:

 AACO.AR.CUSTOMER.MASTER     <-- Mainframe files for company AACO
 AACO.GL.ACCOUNT.MASTER
 BBCO.AR.CUSTOMER.MASTER     <-- Mainframe files for company BBCO
 BBCO.GL.ACCOUNT.MASTER

 /p2/proddata/aaco/ar.customer.master  <-- JCL converter default names
 /p2/proddata/aaco/gl.account.master     - subsystem (ar,gl) part of filename
 /p2/proddata/bbco/ar.customer.master
 /p2/proddata/bbco/gl.account.master

 /p2/proddata/aaco/ar/customer.master  <-- JCL converter option
 /p2/proddata/aaco/gl/account.master     - make subdirs from top 2 nodes
 /p2/proddata/bbco/ar/customer.master      (vs just the top-node)
 /p2/proddata/bbco/gl/account.master

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2F5. ADMjobs: Alternative File System Designs

#2 - multi-companies added after conversion

To add a 2nd company after conversion (& allow inter company datafile access), you would probably clone a 2nd set of JCL/scripts, renaming & modifying as required.

UV Software has the tools to automate the changes required.

1. Copy/rename a 2nd set of JCL/scripts, inserting company name (abbreviation) into the new JCL/script names.
   
   
2. Insert another directory level in front of all existing file definitions within the JCL/scripts, for example:
   

 export CUSTMAS=ar/customer.master       <-- original JCL conversion
 =================================

 export CUSTMAS=aaco/ar/customer.master  <-- modified file definition
 ======================================

Remember that all JCL/scripts change directory at the begining. Line 9 of all converted JCL/scripts calls common function 'jobset41' which performs 'cd $RUNDATA'. So the effective datafile path name becomes:

 export CUSTMAS=/p2/proddata/aaco/ar/customer.master  <-- effective file def
 ===================================================

Actually the JCL converter uses the 'exportfile' function rather than the native 'export' (to display filenames on console), so this would be

 exportfile CUSTMAS /p2/proddata/aaco/ar/customer.master  <-- effective file def
 =======================================================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2G1. ADMjobs.doc - Applications Administration

Tape Drives for Backup

You might select an LTO tape drive (high speed, high capacity) to backup the entire system. RAID makes system failure very unlikely, so I suggest regular backups only for the application libraries and data using cheaper DAT tapes. I suggest the following supplier for backup tape drives.

I suggest the following DAT tape drives. These drives are priced like used, but are often new (excess stock never sold). The cartridges are only $5 each.

      HP C5683 DDS4 20/40 GB SCSI 68 pin internal $110
      (ask for external version if your server does not provide internal slots)

      Quantum STD2401LW-S 20/40 GB DDS4 SCSI 68 pin internal $110

3480/3490 Tape Drives

You may need a 3480/3490 tape drive to load the data tapes from your mainframe. Some site may need a 3480/3490 to exchange data with other sites. I suggest the following supplier & following tape drives:

      IBM 3490E reconditioned & certified $2,795.00
      - reads both 3480 & 3490, writes only 3490
      - highly recommended if you do NOT need to write 3480

      Fujitsu M2483/5H reconditioned & ceritifed $2,495.00
      - reads both 3480 & 3490, writes both 3480 & 3490
      - recommended if you DO need to write 3480

SCSI controllers

I recommend the following SCSI controller cards for the tape drives. You get them from Coastal Micro or Comco. These are PCI cards for tower PCs & servers.

      Adaptec 2944UW SCSI controller 68 pin only

      Adaptec 29160N SCSI controller 50 & 68 pin

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

2H1. ADMjobs.doc - Applications Administration

Setup Summary for Unix/Linux Hardware/Software

1. Install Hardware
   
   
2. Install O/S (Red Hat Enterprise recommended, RHEL 5.1 in 2008) https://www.redhat.com/docs/Red_Hat_Enterprise_Linux/Installation_Guide/ https://www.redhat.com/docs/Red_Hat_Enterprise_Linux/Deployment_Guide/ Also see RAID, partitioning,& file design at ADMjobs.htm#Part_2
   
   
3. Install Vancouver Utilities install.htm#B1 - setup user 'uvadm' install.htm#B2 - copy VU CD to /home/uvadm & unzip install.htm#B3 - setup profiles (supplied in /home/uvadm/env)
   
   
4. setup other users (appsadm, site programmers & operators)
   
   
5. Install Micro Focus COBOL & any other 3rd party software (RDBMS, EasySpooler, etc)
   
   
6. See 23 point preparations for mainframe conversion - see pages '1E1' thru '1E4'.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_3 ADMjobs - Basic Backup & Restore

Backup & Restore - Contents

Backup/Restore using manual commands

Disc Backup/Restore scripts

Tape Backup/Restore scripts

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3A1. ADMjobs: Backup & Restore

Introduction & Overview

Part 3 is an education on unix/linux basic backup & restore.

1. We will illustrate directory trees for production Data & Libraries and suggest corresponding directories for backup & restore on disc.
   
   
2. We will give examples using manual commands ('cp -r', tar,& cpio) to copy production data & library trees to backups on-disc & on-tape.
   
   
3. Then we present Korn shell scripts to backup/restore to/from disc & tape.
   

The scripts here in Part 3 can be used standalone with no dependencies on site conventions, profiles,or environmental variables.

also see Part 4 Advanced Backup/Restore

After you understand these simple backup commands & scripts, please see Part_4, which presents an advanced system for backups automatically scheduled by 'cron'.

Advantages of Part 4 backup system

1. This system takes advantage of the massive inexpensive disc systems now available to maintain zip file backups on-disc for the last 40 days, the last 15 end of month backups,& the last 7 end of year backups.
   
   
2. Backups automatically scheduled by 'cron'
   
   
3. Two days backup on-disc, unzipped, instantly available for file recovery.
   
   
4. Zipped file of nightly backups maintained on-disc for 40 days, 1st of month zip file maintained on-disc for 15 months, & 1st of year zip file maintained on-disc for 7 years.
   
   
5. Zip file names have embedded date & time to eliminate confusion. For example, the 3 nightly zip files used in the backup examples are: proddata_yymmdd_HHMM.zip, prodlibs_yymmdd_HHMM.zip, homedirs_yymmdd_HHMM.zip.
   
   
6. This backup system is intended for the most frequently changing files (production data, production libraries,& home dirs), that may need quick restore. Most sites will already have backups for the entire system, but those probably do not provide the quick restore possible with these smaller targeted backup files.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3B1. ADMjobs: Backup & Restore

Backup & Restore Directories

Part_2 suggested some file designs that might be used for testing & production. Here is a short summary, with the emphasis on production backup & restore.

 /home
 :----uvadm
 :    :----sf/adm/...   <-- backup/restore scripts provided by UV Software
 :----appsadm
 :    :----sf/...       <-- backup/restore scripts copied to /home/appsadm/sf/
                          - customized as required for your site

 /p2                  <---- /p2 file system mount point
 :----proddata          <-- PRODuction DATA
 :     :-----ap
 :     :-----ar             <-- multiple subdirs in proddata
 :     :-----gl
 :     :-----jobtmp
 :     :-----rpts
 :     :-----sysout
 :     :-----tmp
 :     :-----wrk
 :-----prodlibs          <-- $PRODLIBS backups from last night
 :     :-----cbls
 :     :-----cpys
 :     :-----ctl            <-- multiple subdirs in prodlibs
 :     :-----jcls
 :     :-----parms
 :     :-----sfun
 /p3                  <---- /p3 file system mount point
 :----backup              - backup directories (on-disc)
 :    :----proddata       - backup dir for proddata
 :    :    ...              - multiple subdirs in backup/proddata
 :    :----prodlibs       - backup dir for prodlibs
 :    :    ...              - multiple subdirs in backup/proddata
 :    :----...
 :----restore           <-- restore directories (from tape)
 :    :----proddata       - restore area for proddata
 :    :    ...              - multiple subdirs in restore/proddata
 :    :----prodlibs       - restore area for prodlibs
 :    :    ...              - multiple subdirs in restore/prodlibs

1. Backup directories on-disc allow immediate recovery when required
   
   
2. This shows only 1 days backup on-disc but later in this section we show how to maintain 2 days backup unzipped & 40 days zipped
   
   
3. If you need to restore files, don't attempt to restore directly, but rather restore to the /p3/restore/... areas, for investigation & selection of desired files to copy over to /p2/proddata or /p2/prodlibs
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3C1. ADMjobs: Backup & Restore

Backup/Restore - On-Disc - Manual commands

To illustrate the backup/restore commands, let's assume we wish to backup /p2/proddata directory to /p3/backup/proddata. Please see the dtree diagrams listed previously, but here is a partial view.

 /p2
 :----proddata          <-- production data superdir
 :    :----ap
 :    :----ar           <-- subdirs (from mainframe file top-nodes)
 :    :----gl
 :    :----py

 /p3
 :----backup            <-- backup directories
 :    :----proddata     <-- proddata Backup before Night batch
 :    :    :----ap
 :    :    :----ar
 :    :    :----gl

Before backups such as this, you should remove all old files from the destination. Otherwise old files would remain in the backup directory that are not present in today's files. This could be a serious error in production file systems.

On-Disc Backup using 'cp -r'

 #1. cd /p2/proddata                 <-- change to input file superdir
     ===============

 #2. rm -rf /p3/backup/proddata/*    <-- remove all old files & subdirs
     ============================

 #3. cp -rf * /p3/backup/proddata    <-- copy all proddata files to proddata
     ============================

Option '-r' (recursive) is vital here, to copy all files from all subdirs. Option 'f' (force) inhibits any overwrite prompts in case you are using alias cp='cp -i' in your .profile (recommended). Actually we can omit 'f' here since we 1st removed all files from the destination.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3C2. ADMjobs: Backup & Restore - Manual Commands

On-Disc Backup using 'cpio' (manually)

We will use the same example as illustrated on the previous page.

 #1. cd /p2/proddata                  <-- change to input file superdir
     ===============

 #2. rm -rf /p3/backup/proddata/*     <-- remove all old files & subdirs
     ============================

 #3. find . -print | cpio -pdmv /p3/backup/proddata <-- backup all proddata
     ==============================================

cpio options

      i=input, o=output, p=pass (copy to destination directory)
      c - compatible ASCII headers (allows transfer to other machines)
      d - directory creation as required
      m - modification times will be retained
      v - verbose (displays filenames copied)
      B - block size 5120 (vs 512), use 'Q' for 65K if your system allows
      I - Input device name follows (vs standard Input)
      O - Output device name follows (vs standard Output)

copycpio1/2 scripts recommended

These scripts are safer than manual commands since they verify that you are positioned where you say you are & that the output directory is empty.

Script 'copycpio1' ensures you are positioned in the INPUT directory & that the output directory is empty.

Script 'copycpio2' ensures you are positioned in the OUTPUT directory & that the output directory is empty.

Example copy /p2/proddata/... to /p3/backup/proddata

 #1. cd /p2/proddata                            <-- change to input directory
 #2. ls -l /p3/backup/proddata | more           <-- ensure outdir OK to erase
 #3. rm -rf /pr/backup/proddata/*               <-- clear output directory
 #4. copycpio1 /p2/proddata /p3/backup/proddata <-- do it
     ==========================================

Please see the copycpio1 & copycpio2 scripts listed on pages '3D1' & '3D2'.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3C3. ADMjobs: Backup & Restore - Manual Commands

Tape Backup using 'tar' (manually)

Our Tape backups will use the 1st SCSI Tape device for Linux systems (/dev/st0). You may have to modify depending on your version of Unix/Linux, and which particular tape device you wish to use.

 #1. cd /p2/proddata       <-- change to input superdir to be backed up
     ===============

 #2. tar cvf /dev/st0 .    <-- backup to tape from '.' (current dir)
     ==================

tar options

Tape Restore using 'tar' (manually)

Let's assume we only want to restore a few files from the backup tape. We will restore the entire tape to our designated restore area in the /p3 filesystem. Then we can copy the desired files over to the production filesystem (/p2/proddata/...).

It is possible to specify the desired files on the restore command, but it is awkward & often you discover that you need additional files, that then require additional time consuming passes of the tape. IE - you will wish you had restored everything the 1st time.

 #1. cd /p3/restore/proddata  <-- change to the restore area
     =======================

 #2. rm -rf *                 <-- remove all old files
     ========                   - naked '*' can be dangerous
                                - be sure you are in the right place (via pwd)

 #3. tar xvf /dev/st0         <-- restore all files from tape to current dir
     ================

 #4. cp selected-files /p2/proddata/...   <-- recover desired files
     ==================================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3C4. ADMjobs: Backup & Restore - Manual Commands

Tape Backup using 'cpio' (manually)

 #1. cd /p2/proddata                     <-- change to input file superdir
     ===============

 #2. find . -print | cpio -ocvBO /dev/st0 <-- backup all files & subdirs to tape
     ====================================

Tape Restore using 'cpio'

We will assume the same situation as in the 'tar' example on the previous page. We only want to recover a few files, but we will restore the entire tape to a designated restore work area,& then selectively copy the desired files to the production directories.

 #1. cd /p3/restore/proddata  <-- change to the restore area
     =======================

 #2. rm -rf *                 <-- remove all old files
     ========                   - ensure you are in right place (pwd)

 #3. cpio -icvdmBI/dev/st0    <-- restore all files from tape to current dir
     =====================

 #4. cp selected-files?? /p2/proddata   <-- recover desired files
     ================================

option for Block size on output

Option 'B' of '-ocvBO' above specifies block-size 5120 bytes

OPtion 'C' can specify a desired block-size, but timing is not much different as long as you specify at least 5120 (512 is definitely slower).

Large block sizes might use less tape but I don't know how much less. Here is an example specifying block-size 512,000 bytes

 #2. find . -print | cpio -ocvBO /dev/st0 -C512000
     ============================================
     - backup all files & subdirs to tape writing blocks of 512,000 bytes

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3D0. ADMjobs: Disc Backup & Restore Scripts

Disc Backup/Restore scripts

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3D1. ADMjobs: Disc Backup & Restore Scripts

copycpio1 - copy current dir to 2nd empty dir

 #!/bin/ksh
 # copycpio1 - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # copycpio1 - copy a directory tree to a 2nd directory
 #           - input may have sub-directories to any level
 #           - output will have exactly the same tree structure
 #           - you must be in input directory
 #           - output directory must be empty
 #           - you must specify full pathname of indir & outdir
 #           - fail-safe, paranoid script
 #           - also see copycpio2, similar to this *copycpio1, except:
 #*copycpio1 - requires you to be in input directory
 # copycpio2 - requires you to be in output directory
 #
 # Example - copy /p2/proddata/... to /p3/backup/proddata
 #
 # 1. cd /p2/proddata                  <-- change to input directory
 # 2. ls -l /p3/backup/proddata | more <-- ensure outdir OK to erase
 # 3. rm -rf /pr/backup/proddata/*     <-- clear output directory
 # 4. copycpio1 /p2/proddata /p3/backup/proddata <-- do it
 #    ==========================================
 #
 d1="$1"; d2="$2";
 if [[ -d "$d1" && -d "$d2" ]]; then :
    else echo "usage: copycpio1 indir outdir"
         echo "       ======================"
         echo " - arg1 & arg2 must be directories (input & output)"
         exit 91; fi
 #
 dpath=$(pwd)                      # capture current directory path
 if [[ $dpath != $d1 ]]
    then echo "you must be in the input directory $d1";
          exit 92; fi
 #
 ls $2 >/tmp/copycpio_emptytest
 if [[ -s /tmp/copycpio_emptytest ]]
    then echo "the output directory must be empty $d2";exit 93; fi
 #
 echo "copy $d1 to $d2 OK ? (or kill)"; read reply
 cd $d1                          # <-- change to indir
 #
 find . -print | cpio -pdmv $d2  # <-- copy all files to outdir
 #=============================
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3D2. ADMjobs: Disc Backup & Restore Scripts

copycpio2 - copy directory to current empty dir

 #!/bin/ksh
 # copycpio2 - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # copycpio2 - copy a directory tree to a 2nd directory
 #           - input may have sub-directories to any level
 #           - output will have exactly the same tree structure
 #           - you must be in output directory & it must be empty
 #           - you must specify full pathname of outdir & it must match .
 #           - fail-safe, paranoid script
 #           - also see copycpio1, similar to this *copycpio2, except:
 # copycpio1 - requires you to be in input directory
 #*copycpio2 - requires you to be in output directory
 #
 # Example - copy /p2/proddata/... to /p3/backup/proddata
 #
 # 1. cd /p3/backup/proddata         <-- change to output directory
 # 2. ls -l | more                   <-- ensure outdir OK to erase
 # 3. rm -rf *                       <-- clear output directory
 # 3a. rm -rf /p3/backup/proddata/*  <-- safer (avoid unqualified '*')
 # 4. copycpio2 /p2/proddata /p3/backup/proddata <-- do it
 #    ==========================================
 #
 d1="$1"; d2="$2";
 if [[ -d "$d1" && -d "$d2" ]]; then :
    else echo "usage: copycpio2 indir outdir"
         echo "       ======================"
         echo " - arg1 & arg2 must be directories (input & output)"
         exit 91; fi
 #
 dpath=$(pwd)                      # capture current directory path
 if [[ $dpath != $d2 ]]
    then echo "you must be in output directory (& it must be empty) $d2";
          exit 92; fi
 #
 ls . >/tmp/copycpio_emptytest
 if [[ -s /tmp/copycpio_emptytest ]]
    then echo "the current directory (output) must be empty $d2";exit 93; fi
 #
 echo "copy $d1 to current dir $d2 OK ? (or kill job)"; read reply
 cd $d1                          # <-- change to indir
 #
 find . -print | cpio -pdmv $d2  # <-- copy all files to outdir
 #=============================
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3D3. ADMjobs: Disc Backup & Restore Scripts

sortcpio1 - copy directory, sorting filenames

 #!/bin/ksh
 # sortcpio - Korn shell script from UVSI stored in: /home/uvadm/sf/adm/
 # sortcpio  - sort filenames & copy a directory to a 2nd directory
 #           - so filenames will be in sequence for backups, etc
 #             (ls sorts filenames, but not tar, cpio, du, etc)
 #           - input may have sub-directories to any level
 #           - output will have exactly the same tree structure
 #             except directory names & filenames will be sorted
 #
 if [[ -d $1 && -d $2 ]]; then :
    else echo "usage: sortcpio indir outdir (outdir empty)"
         echo "       ====================="; exit 1; fi
 #
 echo "current directory must be outside of indir & have tmp subdir - OK ?";read
 echo "outdir can be anywhere outside of indir & must be empty - OK ?";read
 #
 cwd=$(pwd)
 cd $1; d1=$(pwd); cd $cwd
 cd $2; d2=$(pwd); cd $cwd
 if [[ ! -d tmp ]];then mkdir tmp; fi
 cd $d1
 #
 find . -print >$cwd/tmp/files1
 #=============================
 sort -o$cwd/tmp/files2 $cwd/tmp/files1
 #=====================================
 cat $cwd/tmp/files2 | cpio -pdmv $d2
 #===================================
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3E0. ADMjobs: Tape Backup & Restore scripts

Tape Backup/Restore scripts

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3E1. ADMjobs: Tape Backup/Restore scripts

backupT1 - backup to DAT tape using find & cpio

 #!/bin/ksh
 # backupT1 - backup any 1 directory tree to DAT tape using find & cpio
 #          - Korn shell script from UVSI stored in: /home/uvadm/sf/adm/
 #
 # usage: backupT1 directory
 #        ==================
 #
 if [ ! -d "$1" ]; then echo "usage: backupT1 directory"; exit 9; fi
 #
 echo "backup $1 to DAT tape - OK ?"; read reply
 #
 cd $1
 find . -print | cpio -ocvBO/dev/st0
 exit 0
 #
 #note - tape device shown is the 1st SCSI device on a Linux system
 #     - you must modify for other devices & other unix systems
 #
 # See more advanced backup/restores at: www.uvsoftware.ca/admjobs.htm#Part_4
 # - defining directories & tape devices in 1 place for easier maintenance
 #   (/home/appsadm/env/common_defines '.' executed by .bash_profile)
 # - easier to install backup scripts at new sites
 # - easier to make changes when directories or tape devices change

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3E2. ADMjobs: Tape Backup/Restore scripts

backupT1NRW - backup to DAT tape No Rewind

 #!/bin/ksh
 # backupT1NRW - backup any 1 directory tree to DAT tape using find & cpio
 #               with no rewind, so you can stack multiple archives on 1 tape
 #             - Korn shell script from UVSI stored in: /home/uvadm/sf/adm/
 #
 # usage: backupT1NRW directory
 #        =====================
 #
 if [ ! -d "$1" ]; then echo "usage: backupT1NRW directory"; exit 9; fi
 #
 echo "backup $1 to DAT tape - OK ?"; read reply
 #
 cd $1
 find . -print | cpio -ocvBO/dev/nst0
 exit 0
 #Note - tape device is NO Rewind for 1st SCSI Tape on Linux
 # - you must modify for other devices & other unix systems
 #
 # See more advanced backup/restores at: www.uvsoftware.ca/admjobs.htm#Part_4
 # - defining directories & tape devices in 1 place for easier maintenance
 #   (/home/appsadm/env/common_defines '.' executed by .bash_profile)
 # - easier to install backup scripts at new sites
 # - easier to make changes when directories or tape devices change

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3E3. ADMjobs: Tape Backup/Restore scripts

backupT2 - backup to Multi-File tape

 #!/bin/ksh
 # backupT2 - backup multiple directory trees to multiple archives on DAT tape
 #          - Korn shell script from UVSI stored in: /home/uvadm/sf/adm/
 #
 # usage: backupT2
 #        ========
 #
 # This script is a sample backup script to multi-archive tape using cpio
 # - also see 'restoreT1' to restore any 1 archive to a work area
 #   (for investigation & extract of desired files)
 #
 # assign symbols for REWIND/NOREWIND devices
 # - for easier changes for various unix/linux systems (also see note at end)
 TAPERWD=/dev/st0           # rewind tape device for Linux SCSI
 TAPENRW=/dev/nst0          # NO rewind tape device for Linux SCSI
 #
 #note - tape device shown is the 1st SCSI device on a Linux system
 # - you must modify for other devices & other unix systems
 # - also see backupNtape showing how tape devices can be defined in 1 place
 #
 echo "backupT2 - backup proddata, prodlibs,& home/dirs to DAT tape OK"
 read reply
 mt -f $TAPERWD rewind       # rewind for Linux
 #
 cd /p2/prodlibs
 echo "backup /p2/prodlibs - archive #0 (1st on tape)"
 find . -print | cpio -ocvBO/$TAPENRW
 #
 cd /p2/proddata
 echo "backup /p2/proddata - archive #1 (2nd on tape)"
 find . -print | cpio -ocvBO/$TAPENRW
 #
 cd /home
 echo "backup all /home/...dirs - archive #2 (3rd on tape)"
 find . -print | cpio -ocvBO/$TAPENRW
 #
 echo "backups complete - rewinding tape"
 mt -f $TAPERWD rewind           # rewind for Linux
 exit 0
 #
 # See more advanced backup/restores at: www.uvsoftware.ca/admjobs.htm#Part_4
 # - defining directories & tape devices in 1 place for easier maintenance
 #   (/home/appsadm/env/common_defines '.' executed by .bash_profile)
 # - easier to install backup scripts at new sites
 # - easier to make changes when directories or tape devices change

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3E4. ADMjobs: Tape Backup/Restore scripts

restoreT1 - restore any 1 archive from tape

 #!/bin/ksh
 # restoreT1 - Korn shell script from UVSI stored in: /home/uvadm/sf/adm/
 # restoreT1 - restore any 1 archive from tape to an empty work space
 #             for investigation & extract of desired files
 #
 #usage: 1. change to directory where files are to be restored
 #       2. remove any old files (directory must be empty)
 #       3. restoreT1 tapefile#(0,1,2,etc) directory
 #          ========================================
 #sample: 1. cd /p2/prodlibs        <-- change to PRODuction LIBrarieS
 #        2. rm  -fr *              <-- remove old files & subdirs
 #        3. restoreT1 0 /dev/nst0  <-- restore from 1st file (archive #0)
 #           =====================    - use 1 for 2nd archive, 2 for 3rd, etc
 #
 # - you must specify full path name of restore directory &
 #   it must match your current directory & it must be empty
 # assign symbols for REWIND/NOREWIND devices
 # - for easier changes for various unix/linux systems (also see note at end)
 TAPERWD=/dev/st0           # rewind tape device for Linux SCSI
 TAPENRW=/dev/nst0          # NO rewind tape device for Linux SCSI
 # capture command arguments & verify
 integer fno="$1"           # capture tape file number (0 relative)
 cdir="$2"
 dpath=$(pwd)               # capture current directory path
 echo "restoreT1: arg1=file#=$fno, arg2=curdirpath=$cdir"
 if ((fno < 1 || fno > 99))
    then echo "USAGE: restoreT1 tapefile#(0,1,2,etc) curdirfullpath";
         echo "  arg1 invalid - tapefile# (0 relative 0-99)"
         exit 91; fi
 if [[ ! -d "$cdir" ]]
    then echo "USAGE: restoreT1 tapefile#(0,1,2,etc) curdirfullpath";
         echo "  arg2 invalid - must be full path to current(restore) directory"
         echo "       - restore directory must be empty & you must be in it"
         exit 92; fi
 if [[ $dpath != $cdir ]]
    then echo "you must be in the directory specified"; exit 92; fi
 ls . >/tmp/restore_emptytest
 if [[ -s /tmp/restore_emptytest ]]
    then echo "the current directory must be empty";exit 93; fi
 #
 echo "restore tapefile# $fno to current directory ($cdir) OK ?"; read reply
 echo "- are you sure tapefile# $fno correct for $cdir ? (kill/rekey if not)"
 read reply
 mt -f $TAPERWD rewind       # ensure tape rewound (for Linux)
 mt -f $TAPENRW fsf $fno     # forward space to desired archive (for Linux)
 cpio -icvdmBI $TAPENRW      # restore spcfd archive to cur dir
 #=====================
 echo "tapefile# $fno restored to $cdir, rewinding tape"
 mt -f $TAPERWD rewind       # rewind tape after (for Linux)
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

3F1. ADMjobs: Tape Backup & Restore scripts

Restoring Backup tapes to a New System

With Raid, you may never crash, but at some point you will want to move to new hardware. Here is the procedure to restore the backup tape archives created by backupT2 (listed on page '3E3').

1. Install new Hardware
   
   
2. Install O/S (Red Hat Enterprise recommended, RHEL 5.1 in 2008) https://www.redhat.com/docs/Red_Hat_Enterprise_Linux/Installation_Guide/ https://www.redhat.com/docs/Red_Hat_Enterprise_Linux/Deployment_Guide/ Also see RAID, partitioning,& file design at ADMjobs.htm#Part_2
   
   
3. Restore homedirs from archive #3 of backupT2 tape
   

 #3a. cd /home
 #3b. mt -f /dev/st0 fsf 2     <-- forward space 2 files (to 3rd archive)
 #3c. cpio -icvdmBI /dev/nst0  <-- restore 3rd archive (homedirs)

4. Setup users on new system. Note that '-m' option (create homedir) is omitted below (since homedirs restored from tape).
   

 #4a. groupadd apps
 #4b. adduser -d/home/uvadm   -gapps uvadm
 #4c. adduser -d/home/appsadm -gapps appsadm
 #4c. adduser -d/home/userxxx -gapps userxxx
 #4_.            - - - - - etc - - - - -

5. Setup the directories to hold your production Libraries & Data - as on the former system or as suggested on ADMjobs.htm#2C0 thru 2F5.
   
   
6. Restore production Libraries & Data. Note that PRODLIBS & PRODDATA are defined in common_defines (executed when appsadm logs on).
   

 #6a. Logon appsadm --> /home/appsadm
 #6b. cdl           --> $PRODLIBS (/p2/prodlibs on page '2C0' example)
 #6c. mt -f /dev/st0 fsf 0     <-- forward space 0 files (1st archive)
 #6d. cpio -icvdmBI /dev/nst0  <-- restore 1st archive (production libraries)
 #6e. cdd           --> $PRODDATA (/p2/proddata on page '2C0' example)
 #6f. mt -f /dev/st0 fsf 1     <-- forward space 1 files (2nd archive)
 #6g. cpio -icvdmBI /dev/nst0  <-- restore 2nd archive (production libraries)

7. Alternatively - use restoreT1 script (listed on page '3E4')
   

 #7a. Logon appsadm --> /home/appsadm
 #7b. cdl           --> $PRODLIBS (/p2/prodlibs on page '2C0' example)
 #7c. restoreT1 0 $PRODLIBS   <-- restore 1st archive (production libraries)
 #7d. cdd           --> $PRODDATA (/p2/proddata on page '2B0' example)
 #7e. restoreT1 1 $PRODDATA   <-- restore 2nd archive (production libraries)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_4 ADMjobs - Advanced Backup & Restore

Advanced Backup & Restore - Contents

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Advanced Backup & Restore

Backup & Restore - Contents (continued)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Advanced Backup & Restore

Backup & Restore - Contents (continued)

 4F10. purgeold2 - purge files from a directory older than a specified no of days
      ----------- based on the dates embedded in the filenames
      uvcopy job called by backupPurge2 script to remove:
      - Daily backup files older than 40 days
      - Monthly backup files older than 450 days
      - Yearly backup files older than 2555 days
      Alternate backupPurge1 uses directory entry dates vs embedded dates
      - which could be wrong if files copied without -p option

4A1. Introduction & Overview

Part_3 was an education on unix/linux basic backup & restore. The scripts were simple & could be used standalone with no dependencies on site conventions, profiles,or environmental variables.

Here in Part 4 ('http://www.uvsoftware.ca/admjobs.htm#Part_4'), we will present an advanced backup/restore system which uses 'cron' to automatically schedule backups to disc & to tape.

The scripts are well commented & listed in Part 4, so new users should be able to learn something about unix/linux Korn shell scripts & crontabs.

BACKUP directory permissions, owner, group

Since the backup scripts are scheduled by crontabs owned by 'appsadm', you must ensure appsadm can write to the backup directories. If you set them up with root, you could fix as follows:

 #1. chmod 775 $BACKUP   <-- set permissions
     =================

 #2. chown appsadm:apps $BACKUP <-- set owner & goup
     ==========================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4A2. ADMjobs: Advanced Backup & Restore

Advantages of Part 4 backup system

1. This system takes advantage of the massive inexpensive disc systems now available to maintain zip file backups on-disc for the last 40 days, the last 15 end of month backups,& the last 7 end of year backups.
   
   
2. Backups automatically scheduled by 'cron'
   
   
3. Two days backup on-disc, unzipped, instantly available for file recovery.
   
   
4. Zipped file of nightly backups maintained on-disc for 40 days, 1st of month zip file maintained on-disc for 15 months, & 1st of year zip file maintained on-disc for 7 years.
   
   
5. Zip file names have embedded date & time to eliminate confusion. For example, the 3 nightly zip files used in the backup examples are: proddata_yymmdd_HHMM.zip, prodlibs_yymmdd_HHMM.zip, homedirs_yymmdd_HHMM.zip.
   
   
6. This backup system is intended for the most frequently changing files (production data, production libraries,& home dirs), that may need quick restore. Most sites will already have backups for the entire system, but those probably do not provide the quick restore possible with these smaller targeted backup files.
   
   
7. Nightly zip files backed up to tape by cron & tapes cycled over 31 days. For smaller sites with single tape units, I suggest you simply label 31 tapes with numbers 1 to 31. Each morning the operator changes tapes & inserts the tape# for today's day of month, with no need to write anything.
   
   
8. The nightly backup tapes can be stored onsite in fireproof vault. The end of month tape should be taken offsite & new tape inserted in rotation, progressing thru the 31 tapes, so tapes are reused maximum 31 times.
   
   
9. Directories & devices defined in 1 place (/home/appsadm/env/common_defines), to make it easier, to install backup scripts at new sites & to make changes, when directories or tape devices change.
   
   
10. This is achieved by having all scripts '.' dot-execute the 'appsadm profile' & 'common_defines' which define the symbols ($PRODDATA, $PRODLIBS, $BACKUP, $TAPERWD, $TAPENRW), which are used thruout the scripts. Please see the profiles listed begining on page http://www.uvsoftware.ca/admjobs.htm#1B1.
    
    
11. Executing the backups under cron means the console logs will be 'mailed' to the crontab owner, which is 'appsadm', so the site administrator can investigate any errors.
    
    
12. These scripts use zip/unzip for Linux, but Unix users may have to change these to gzip/gunzip or to tar+compress/uncompress/untar.
    

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4B1. ADMjobs: Advanced Backup & Restore

Backup & Restore Directories

Part_2 suggested some file designs that might be used for testing & production. Here is a short summary, with the emphasis on production backup & restore.

 /home
 :----uvadm
 :    :----sf/adm/...   <-- backup/restore scripts provided by UV Software
 :----appsadm
 :    :----sf/...       <-- backup/restore scripts copied to /home/appsadm/sf/
                          - customized as required for your site

 /p1                  <---- /p2 file system mount point
 :----testdata          <-- TEST DATA
 :     :-----ap             (subdirs same as proddata below)
 :----testlibs          <-- TEST LIBrarieS
 :     :-----cbls           (subdirs same as prodlibs below)

 /p2                  <---- /p2 file system mount point
 :----proddata          <-- PRODuction DATA
 :     :-----ap
 :     :-----ar             <-- multiple subdirs in proddata
 :     :-----gl
 :-----prodlibs          <-- $PRODLIBS backups from last night
 :     :-----cbls
 :     :-----cpys        <-- multiple subdirs in prodlibs
 :     :-----jcls
 /p3                  <---- /p3 file system mount point
 :----backup              - backup directories (on-disc)
 :     :----proddata       - backup dir for proddata (unzipped, quick restore)
 :     :    ...              - multiple subdirs in backup/proddata
 :     :----prodlibs       - backup dir for prodlibs (unzipped, quick restore)
 :     :    ...              - multiple subdirs in backup/proddata
 :     :-----zip                             <-- last nights backup (only)
 :     :     :-----proddata_070529_0302.zip  <-- sample for May 29/2007
 :     :-----Day                             <-- Daily backup .zips last 40 days
 :     :     :-----proddata_070419_0302.zip  <-- sample for April 19/2007
 :     :-----Month                           <-- Monthly backup .zips 15 mths
 :     :     :-----proddata_060201_0302.zip  <-- sample for Feb 1/2006
 :     :-----Year                            <-- Yearly backup .zips last 7 years
 :     :     :-----proddata_000501_0302.zip  <-- sample for Jan 1/2000

 :-----restore           <-- restore directories (from tape)
 :     :----proddata       - restore area for proddata
 :     :    ...            - multiple subdirs in restore/proddata

1. Backup directories on-disc allow immediate recovery when required
   
   
2. This shows only 1 days backup on-disc but later in this section we show how to maintain 2 days backup unzipped & 40 days zipped
   
   
3. If you need to restore files, don't attempt to restore directly, but rather restore to the /p3/restore/... areas, for investigation & selection of desired files to copy over to /p2/proddata or /p2/prodlibs
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4C1. ADMjobs: Advanced Backup & Restore

Directories for Backup & Restore

 /p3/backup
 :-----homedirs             <-- $HOMEDIRS backup from last night
 :     :-----appsadm          - showing only 1 user to save lines
 :     :     :-----ctl          & showing only a few subdirs in 1st user
 :     :     :-----logs
 :     :     :-----...
 :-----homedirs-1           <-- $HOMEDIRS backup from 2 nights ago
 :     :-----...same as above...
 :-----proddata             <-- $PRODDATA backup from last night
 :     :-----ap
 :     :-----ar
 :     :-----gl
 :-----proddata-1           <-- $PRODDATA backup from 2 nights ago
 :     :-----...same as above...
 :-----prodlibs             <-- $PRODLIBS backups from last night
 :     :-----cbls
 :     :-----cpys
 :     :-----jcls
 :-----prodlibs-1           <-- $PRODLIBS backup from 2 nights ago
 :     :-----...same as above...
 :-----testdata             <-- $TESTDATA backup from last night
 :-----testdata-1           <-- $TESTDATA backup from 2 nights ago
 :-----testlibs             <-- $TESTLIBS backup from last night
 :-----testlibs-1           <-- $TESTLIBS backup from 2 nights ago
 :-----zip                  <-- last nights backup (only)
 :     :-----homedirs_070529_0301.zip
 :     :-----proddata_070529_0302.zip  <-- sample for May 29/2007
 :     :-----prodlibs_070529_0303.zip
 :     :-----...
 :-----Day                  <-- Daily backups in .zip files for last 40 days
 :     :-----homedirs_070419_0301.zip
 :     :-----proddata_070419_0302.zip  <-- 40 days ago = April 19/2007
 :     :-----prodlibs_070419_0303.zip
 :     :-----...(39 sets not shown)
 :-----Month                <-- Monthly backups in .zip files for last 15 months
 :     :-----homedirs_060201_0301.zip
 :     :-----proddata_060201_0302.zip  <-- 15 months ago = Feb 1/2006
 :     :-----prodlibs_060201_0303.zip
 :     :-----...(14 sets not shown)
 :-----Year                 <-- Yearly backups in .zip files for last 7 years
 :     :-----homedirs_000501_0301.zip
 :     :-----proddata_000501_0302.zip  <-- 7 years ago = Jan 1/2000
 :     :-----prodlibs_000501_0303.zip
 :     :-----...(6 sets not shown)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4D1. ADMjobs: Advanced Backup & Restore

 # cronbackupNight - crontab file to run backup scripts
 #                 - Nightly1, Monthly1, Yearly
 #                 - by Owen Townsend, UV Software, May 19/2007
 #                 - see doc at www.uvsoftware.ca/admjobs.htm#Part_4
 #
 # - crontab file supplied with Vancouver Utilities in /home/appsadm/sf/adm/...
 # - I suggest you setup userid 'appsadm' to house your crontabs & cron scripts
 #   then copy supplied file to /home/appsadm/sf/cronbackupNight & modify as reqd
 # - see backup scripts (backupPROD,backupTEST,backupHOME,backupMonth,backupYear)
 # - these should also be copied to /home/appsadm/sf/... & modified there as reqd
 # - this crontab & the backup scripts should run under the 'appsadm' user id
 # - much safer than running under root
 # - works OK if appsadm in same group as users who created files & directories
 #   & if permissions 775 for directories & 664 for files (umask 002 in profiles)
 # - use a separate crontab for root if you need to backup system files
 #
 #                    ** updating & installing crontab files **
 #
 # - we recommend you store this crontab file in /home/appsadm/sf/...
 #   update it (with vi) when required & re-install, using the crontab command
 #   (IE - the master copy is in /home/appsadm/sf/... not the installed version)
 # - log on as 'appsadm' to issue the 'crontab' command
 #   (crontab command installs crontabs only for the logged on user)
 # - the 'crontab' command (#4 below) copies the specified crontab file
 #   to the 'real' crontab file which would be: /var/spool/cron/appsadm
 # - Before 1st use, you must logon as root, &  add the 'appsadm'
 #   userid to: /etc/cron.allow if it exists. If it does not exist
 #   you can use cron unless your userid exists in /etc/cron.deny
 #
 # 1. logon as appsadm --> /home/appsadm
 # 2. vi sf/cronbackupNight      - edit this file as required
 # 3. crontab -r                 - remove old crontabs for user (appsadm)
 #                                 (ok since this file is the master copy)
 # 4. crontab sf/cronbackupNight - activate new crontab for appsadm
 # 5. crontab -l                 - list crontab onfile to confirm installation
 #
 # arguments to crontab are as follows:
 # Minute Hour DayofMth MthofYr DayofWeek <----command---->
 #   00     3     *        *       2-6      #<-- codes used for 1st cmd below
 #
 # 00 3 * * 2-6 /home/appsadm/sf/backupPROD   #Nightly backup proddata & prodtest
 #=======================================
 15 3 * * 2-6 /home/appsadm/sf/backupTEST   #Nightly backup testdata & testtest
 #=======================================
 # 30 3 * * 2-6 /home/appsadm/sf/backupHOME   #Nightly backup homedirs
 #=======================================
 # 00 4 01 * * /home/appsadm/sf/backupMonth   #Monthly on 1st of month at 4 AM
 #=======================================
 # 30 4 01 01 * /home/appsadm/sf/backupYear   #Yearly on Jan 1 at 4:30 AM
 #=======================================
 # 00 5 * * 2-6 /home/appsadm/sf/backupNtape  #Nightly backup to tape
 #========================================
 #Note - Nightly backups for Monday to Friday after 12AM coded as 2-6 (Tues-Sat)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4E1. ADMjobs: Advanced Backup & Restore

mail from cron after backup scripts

When scripts are run by cron, the console messages are automatically 'mailed' to the owner of the crontab file (appsadm in this case).

This is a great convenience & the appsadm administrator should check this mail to ensure no problems occurred in overnight backups & any other cron jobs.

cron mail after backupNight

 From appsadm@uvsoft3.uvsoft.ca  Sun May 20 11:30:05 2007
 Return-Path: <appsadm@uvsoft3.uvsoft.ca>
 Received: from uvsoft3.uvsoft.ca (uvsoft3.uvsoft.ca [127.0.0.1])
 	by uvsoft3.uvsoft.ca (8.12.10/8.12.10) with ESMTP id l4KIU5dc007947
 	for <appsadm@uvsoft3.uvsoft.ca>; Sun, 20 May 2007 11:30:05 -0700
 Received: (from appsadm@localhost)
 	by uvsoft3.uvsoft.ca (8.12.10/8.12.10/Submit) id l4KIU0ix007914
 	for appsadm; Sun, 20 May 2007 11:30:00 -0700
 Date: Sun, 20 May 2007 11:30:00 -0700
 Message-Id: <200705201830.l4KIU0ix007914@uvsoft3.uvsoft.ca>
 From: root@uvsoft3.uvsoft.ca (Cron Daemon)
 To: appsadm@uvsoft3.uvsoft.ca
 Subject: Cron <appsadm@localhost> /home/appsadm/sf/backupNight  #Nightly backup to disc
 X-Cron-Env: <SHELL=/bin/sh>
 X-Cron-Env: <HOME=/home/appsadm>
 X-Cron-Env: <PATH=/usr/bin:/bin>
 X-Cron-Env: <LOGNAME=appsadm>
 Status: RO

 backupNight - backup $PRODDATA, $PRODLIBS,& $HOMEDIRS
  - removes 2 days ago backup proddata-1, prodlibs-1, homedirs-1
  - renames yesterdays proddata,prodlibs,homedirs (append -1)
  - makes new empty output dirs for today's backups
  - changes in turn into each input directory
  - runs the copycpio1 script to copy all subdirs & files to backup dir
 enter to proceed (will not stop here if run under cron)

 copy /home/mvstest/testdata to /home5/backup/proddata OK ? (or kill)
 /home5/backup/proddata/./ap
 /home5/backup/proddata/./ar
 /home5/backup/proddata/./ar/customer.master
 /home5/backup/proddata/./ar/customer.nameadrs.list100
 /home5/backup/proddata/./ar/sales.items
    - - - lines removed for this illustration - - -
 /home5/backup/proddata/./pysave/test5a
 317 blocks

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4E2. ADMjobs: Advanced Backup & Restore

cron mail after backupNight (continued)

 copy /home/mvstest to /home5/backup/prodlibs OK ? (or kill)
 /home5/backup/prodlibs/./cbls
 /home5/backup/prodlibs/./cbls/car100.cbl
 /home5/backup/prodlibs/./cbls/car120.cbl
    - - - lines removed for this illustration - - -
 /home5/backup/prodlibs/./parms/pgl200s1
 /home5/backup/prodlibs/./parms/ppy200s2
 3760 blocks

 copy /home/appsadm to /home5/backup/homedirs OK ? (or kill)
 /home5/backup/homedirs/./env
 /home5/backup/homedirs/./env/common_defines
 /home5/backup/homedirs/./env/common_profile
 /home5/backup/homedirs/./env/stub_profile
    - - - lines removed for this illustration - - -
 /home5/backup/homedirs/./sfun/exportfile
 /home5/backup/homedirs/./sfun/jobset5
 /home5/backup/homedirs/./sfun/logmsg1
 784 blocks

   adding: ap/ (stored 0%)
   adding: ar/ (stored 0%)
   adding: ar/customer.master (deflated 66%)
   adding: ar/customer.nameadrs.list100 (deflated 54%)
   adding: ar/sales.items (deflated 70%)
    - - - lines removed for this illustration - - -
   adding: pysave/test5a (stored 0%)

   adding: cbls/ (stored 0%)
   adding: cbls/car100.cbl (deflated 64%)
   adding: cbls/car120.cbl (deflated 65%)
   adding: cbls/car130.cbl (deflated 64%)
    - - - lines removed for this illustration - - -
   adding: sf/backupPurge2 (deflated 62%)
   adding: sf/backupYear (deflated 57%)

 backupNight completed, files in $BACKUP/zip are:
 total 808
 -rw-rw-r--    1 appsadm  users      143823 May 20 11:30 homedirs_070520_1130.zip
 -rw-rw-r--    1 appsadm  users       66054 May 20 11:30 proddata_070520_1130.zip
 -rw-rw-r--    1 appsadm  users      594983 May 20 11:30 prodlibs_070520_1130.zip
 accumulated files in $BACKUP/Day (/home5/backup/Day) =        3

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4E3. ADMjobs: Advanced Backup & Restore

cron mail after backupMonth

 From appsadm@uvsoft3.uvsoft.ca  Sun May 20 11:35:01 2007
 Return-Path: <appsadm@uvsoft3.uvsoft.ca>
 Received: from uvsoft3.uvsoft.ca (uvsoft3.uvsoft.ca [127.0.0.1])
 	by uvsoft3.uvsoft.ca (8.12.10/8.12.10) with ESMTP id l4KIZ0dc007976
 	for <appsadm@uvsoft3.uvsoft.ca>; Sun, 20 May 2007 11:35:00 -0700
 Received: (from appsadm@localhost)
 	by uvsoft3.uvsoft.ca (8.12.10/8.12.10/Submit) id l4KIZ0H5007967
 	for appsadm; Sun, 20 May 2007 11:35:00 -0700
 Date: Sun, 20 May 2007 11:35:00 -0700
 Message-Id: <200705201835.l4KIZ0H5007967@uvsoft3.uvsoft.ca>
 From: root@uvsoft3.uvsoft.ca (Cron Daemon)
 To: appsadm@uvsoft3.uvsoft.ca
 Subject: Cron <appsadm@localhost> /home/appsadm/sf/backupMonth   #Monthly on 1st of month at 4 AM
 X-Cron-Env: <SHELL=/bin/sh>
 X-Cron-Env: <HOME=/home/appsadm>
 X-Cron-Env: <PATH=/usr/bin:/bin>
 X-Cron-Env: <LOGNAME=appsadm>
 Status: RO

 backupMonth - copy current backup zip files to Month backup directory
 - intended for cron on 1st each month at 4 AM (but could run manually)
 - preceded by backupNight, backed up $PRODDATA $PRODLIBS $HOMEDIRS
   & zipped them to $BACKUP/zip/... with date/time stamps
 - this script simply copies contents of $BACKUP/zip/* to $BACKUP/Month
 - see more documentation at: www.uvsoftware.ca/admjobs.htm#Part_4
 enter to proceed (will not stop here if run under cron)
 stty: standard input: Invalid argument
 stty: standard input: Invalid argument
 backupMonth completed, files in $BACKUP/zip are:
 total 808
 -rw-rw-r--    1 appsadm  users      143823 May 20 11:30 homedirs_070520_1130.zip
 -rw-rw-r--    1 appsadm  users       66054 May 20 11:30 proddata_070520_1130.zip
 -rw-rw-r--    1 appsadm  users      594983 May 20 11:30 prodlibs_070520_1130.zip
 accumulated files in $BACKUP/Month are:
 total 808
 -rw-rw-r--    1 appsadm  users      143823 May 20 11:35 homedirs_070520_1130.zip
 -rw-rw-r--    1 appsadm  users       66054 May 20 11:35 proddata_070520_1130.zip
 -rw-rw-r--    1 appsadm  users      594983 May 20 11:35 prodlibs_070520_1130.zip
 file count in $BACKUP/Month (/home5/backup/Month) =        3

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4E4. ADMjobs: Advanced Backup & Restore

cron mail after backupNtape

 From appsadm@uvsoft3.uvsoft.ca  Sun May 20 11:46:19 2007
 Return-Path: <appsadm@uvsoft3.uvsoft.ca>
 Received: from uvsoft3.uvsoft.ca (uvsoft3.uvsoft.ca [127.0.0.1])
 	by uvsoft3.uvsoft.ca (8.12.10/8.12.10) with ESMTP id l4KIkDdc008030
 	for <appsadm@uvsoft3.uvsoft.ca>; Sun, 20 May 2007 11:46:19 -0700
 Received: (from appsadm@localhost)
 	by uvsoft3.uvsoft.ca (8.12.10/8.12.10/Submit) id l4KIj0p4008021
 	for appsadm; Sun, 20 May 2007 11:45:00 -0700
 Date: Sun, 20 May 2007 11:45:00 -0700
 Message-Id: <200705201845.l4KIj0p4008021@uvsoft3.uvsoft.ca>
 From: root@uvsoft3.uvsoft.ca (Cron Daemon)
 To: appsadm@uvsoft3.uvsoft.ca
 Subject: Cron <appsadm@localhost> /home/appsadm/sf/backupNtape  #Nightly backup to tape
 X-Cron-Env: <SHELL=/bin/sh>
 X-Cron-Env: <HOME=/home/appsadm>
 X-Cron-Env: <PATH=/usr/bin:/bin>
 X-Cron-Env: <LOGNAME=appsadm>
 Status: RO

 stty: standard input: Invalid argument
 stty: standard input: Invalid argument
 backup $BACKUP/zip/* to tape ($BACKUP=/home5/backup)
 - intended for cron at 5 AM (but could be run manually)
 - follows backupNight which created date stamped .zip files
 /home5/backup/zip/homedirs_yymmdd_HHMM.zip
 /home5/backup/zip/prodlibs_yymmdd_HHMM.zip
 /home5/backup/zip/proddata_yymmdd_HHMM.zip
 - enter to backup all files in $BACKUP/zip/* to tape
 .
 ./proddata_070520_1130.zip
 ./prodlibs_070520_1130.zip
 ./homedirs_070520_1130.zip
 158 blocks
 backups complete - rewinding & unloading tape

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4F0. ADMjobs: Advanced Backup & Restore

Backup & Restore scripts

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Advanced Backup & Restore

Backup & Restore - Contents (continued)

 4F10. purgeold2 - purge files from a directory older than a specified no of days
                 - based on the dates embedded in the filenames
      uvcopy job called by backupPurge2 script to remove old backups
      Alternate backupPurge1 uses directory entry dates vs embedded dates
      - which could be wrong if files copied without -p option

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4F1. ADMjobs: Advanced Backup & Restore

 #!/bin/ksh
 # backupPROD - backup $PRODDATA, $PRODLIBS (nigtly backup)
 #            - intended to be run by cron at 3 AM (but could run manually)
 #            - also see backupTEST for $TESTDATA & $TESTLIBS
 #            - by Owen Townsend, UV Software, May 18/2007
 #            - see doc at www.uvsoftware.ca/admjobs.htm#Part_4
 #
 # backupPROD       <-- no arguments required
 # ==========
 #
 #Note - we will minimize necessity to modify backup scripts for new sites
 #     - by using environmental variables $PRODDATA,$PRODLIBS,$BACKUP
 # - variables defined in /home/appsadm/env/common_defines (with profiles)
 #   assuming site admin followed advice in www.uvsoftware.ca/admjobs.htm#Part_1
 # - will '.' dot-execute site admin's .bash_profile to define $SYMBOLs & PATHs
 #
 export APPSADM=/home/appsadm    # define site admin superdir
 #===========================      (only absolute path in these scripts)
 . $APPSADM/.bash_profile        # define $PRODDATA,$PRODLIBS,$BACKUP,etc
 #===========================
 #
 echo "backupPROD - backup \$PRODDATA & \$PRODLIBS"
 echo "PRODDATA=$PRODDATA PRODLIBS=$PRODLIBS"
 echo " - intended to be run by cron at 3 AM (but could run manually)"
 echo " - removes 2 days ago backup proddata-1, prodlibs-1"
 echo " - renames yesterdays proddata,prodlibs (append -1)"
 echo " - makes new empty output dirs for todays backups"
 echo " - changes in turn into each input directory"
 echo " - runs the copycpio1 script to copy all subdirs & files to backup dir"
 echo "enter to proceed (will not stop here if run under cron)"; read reply
 #
 #Note - this script may be run by cron at 3 AM (but could run manually)
 #     - see recommendations at: www.uvsoftware.ca/admjobs.htm#Part_4 & Part 5
 # - crontab file should be invoked by user 'appsadm' (not root)
 # - appsadm must belong to same group as users creating files to be backed up
 # - all directories must have perms 775 & all files 664
 #   (they will if using the profiles recommended in ADMjobs Part 1)
 #
 # verify that env-vars set OK & required subdirs present
 if [[ -d $PRODDATA && -d $PRODLIBS && -d $BACKUP/zip && -d $BACKUP/Day ]];
   then :
   else echo "\$PRODDATA=$PRODDATA"
        echo "\$PRODLIBS=$PRODLIBS"
        echo "\$BACKUP/zip=$BACKUP/zip"
        echo "\$BACKUP/Day=$BACKUP/Day"
        echo "\$APPSADM=$APPSADM"
        echo "1 or more of above directories NOT found:"
        echo "- or \$variables not set by \$APPSADM/.bash_profile"
        exit 99; fi
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 # we will maintain 2 days backup of unzipped data & libraries
 # - for quick recovery if missing files discovered early (else unzip backups)
 rm -rf $BACKUP/proddata-1               # remove 2 days ago backups
 rm -rf $BACKUP/prodlibs-1
 mv $BACKUP/proddata $BACKUP/proddata-1  # change name of yesterdays backups
 mv $BACKUP/prodlibs $BACKUP/prodlibs-1
 mkdir $BACKUP/proddata                  # make new dirs for todays backups
 mkdir $BACKUP/prodlibs
 #
 # now execute copycpio1 script from each superdir to be backed up
 # - must be in directory to be backed up & directory must be empty
 cd $PRODDATA                           # change into input superdir
 copycpio1 $PRODDATA $BACKUP/proddata   # backup all levels of subdirs & files
 #===================================
 cd $PRODLIBS
 copycpio1 $PRODLIBS $BACKUP/prodlibs
 #===================================
 #
 # now zip the backups
 # clear old files from zip subdir & zip today's backups into it
 rm -f $BACKUP/zip/prod*
 #======================
 cd $BACKUP/proddata
 zip -r $BACKUP/zip/proddata_$(date +%y%m%d_%H%M).zip .
 #=====================================================
 cd $BACKUP/prodlibs
 zip -r $BACKUP/zip/prodlibs_$(date +%y%m%d_%H%M).zip .
 #=====================================================
 #
 # copy today's zip files into the Day backup dir
 # - backupPurge script will remove files older than 40 days
 cp $BACKUP/zip/* $BACKUP/Day
 #===========================
 #
 echo "backupPROD completed, files in \$BACKUP/zip are:"
 ls -l $BACKUP/zip
 days=$(ls $BACKUP/Day | wc -l)
 echo "accumulated files in \$BACKUP/Day ($BACKUP/Day) = $days"
 exit 0

4F2. backupTEST

'backupTEST' is the same as backupPROD (listed above), but backs up testdata & testlibs (vs proddata & prodlibs).

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4F3. ADMjobs: Advanced Backup & Restore

 #!/bin/ksh
 # backupHOME - backup $HOMEDIRS (nigtly backup)
 #            - intended to be run by cron at 3 AM (but could run manually)
 #            - by Owen Townsend, UV Software, Sep18/2009
 #            - backupHOME now separate from backupPROD & backupTEST
 #            - see doc at www.uvsoftware.ca/admjobs.htm#Part_4
 #
 # backupHOME       <-- no arguments required
 # ==========
 #
 #Note - we will minimize necessity to modify backup scripts for new sites
 #     - by using environmental variables $HOMEDIRS & $BACKUP
 # - variables defined in /home/appsadm/env/common_defines (with profiles)
 #   assuming site admin followed advice in www.uvsoftware.ca/admjobs.htm#Part_1
 # - will '.' dot-execute site admin's .bash_profile to define $SYMBOLs & PATHs
 #
 export APPSADM=/home/appsadm    # define site admin superdir
 #===========================      (only absolute path in these scripts)
 . $APPSADM/.bash_profile        # define $HOMEDIRS, $BACKUP, etc
 #===========================
 #
 echo "backupHOME - backup \$HOMEDIRS=$HOME to \$BACKUP=$BACKUP"
 echo " - intended to be run by cron at 3 AM (but could run manually)"
 echo " - removes 2 days ago backup & renames yesterdays to homedirs-1"
 echo " - makes new empty output homedirs for todays backups"
 echo " - changes into \$HOMEDIRS=$HOMEDIRS"
 echo " - runs the copycpio1 script to copy all subdirs & files to backup dir"
 echo "enter to proceed (will not stop here if run under cron)"; read reply
 #
 #Note - this script may be run by cron at 3 AM (but could run manually)
 #     - see recommendations at: www.uvsoftware.ca/admjobs.htm#Part_4 & Part 5
 # - crontab file should be invoked by user 'appsadm' (not root)
 # - appsadm must belong to same group as users creating files to be backed up
 # - all directories must have perms 775 & all files 664
 #   (they will if using the profiles recommended in ADMjobs Part 1)
 #
 # verify that env-vars set OK & required subdirs present
 if [[ -d $HOMEDIRS && -d $BACKUP/zip && -d $BACKUP/Day ]]; then :
   else echo "\$HOMEDIRS=$HOMEDIRS"
        echo "\$BACKUP/zip=$BACKUP/zip"
        echo "\$BACKUP/Day=$BACKUP/Day"
        echo "\$APPSADM=$APPSADM"
        echo "1 or more of above directories NOT found:"
        echo "- or \$variables not set by \$APPSADM/.bash_profile"
        exit 99; fi
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 # we will maintain 2 days backup of unzipped homedirs
 # - for quick recovery if missing files discovered early (else unzip backups)
 rm -rf $BACKUP/homedirs-1              # remove 2 days ago backups
 mv $BACKUP/homedirs $BACKUP/homedirs-1 # change name of yesterdays backups
 mkdir $BACKUP/homedirs                 # make new dirs for todays backups
 #
 # now execute copycpio1 script from superdir to be backed up
 # - must be in directory to be backed up & directory must be empty
 cd $HOMEDIRS                           # change into input superdir
 copycpio1 $HOMEDIRS $BACKUP/homedirs   # backup all levels of subdirs & files
 #===================================
 #
 # now zip the backups
 # clear old files from zip subdir & zip today's backups into it
 rm -f $BACKUP/zip/homedirs*
 cd $BACKUP/homedirs
 zip -r $BACKUP/zip/homedirs_$(date +%y%m%d_%H%M).zip .
 #=====================================================
 #
 # copy today's zip files into the Day backup dir
 # - backupPurge script will remove files older than 40 days
 cp $BACKUP/zip/* $BACKUP/Day
 #===========================
 #
 echo "backupHOME completed, files in \$BACKUP/zip are:"
 ls -l $BACKUP/zip
 days=$(ls $BACKUP/Day | wc -l)
 echo "accumulated files in \$BACKUP/Day ($BACKUP/Day) = $days"
 exit 0
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4F4. ADMjobs: Advanced Backup & Restore

 #!/bin/ksh
 # backupMonth - copy current backup zip files to Month backup directory
 #             - intended for cron on 1st each month at 4 AM (could run manually)
 #             - by Owen Townsend, UV Software, May 18/2007
 #             - see doc at www.uvsoftware.ca/admjobs.htm#Part_4
 #
 # backupMonth       <-- no arguments required
 # ===========
 #
 echo "backupMonth - copy current backup zip files to Month backup directory"
 echo "- intended for cron on 1st each month at 4 AM (but could run manually)"
 echo "- preceded by backupNight, backed up \$PRODDATA \$PRODLIBS \$HOMEDIRS"
 echo "  & zipped them to \$BACKUP/zip/... with date/time stamps"
 echo "- this script simply copies contents of \$BACKUP/zip/* to \$BACKUP/Month"
 echo "- see more documentation at: www.uvsoftware.ca/admjobs.htm#Part_4"
 echo "enter to proceed (will not stop here if run under cron)"; read reply
 #
 #Note - we will minimize necessity to modify backup scripts for new sites
 #     - by using environmental variables $PRODDATA,$PRODLIBS,$BACKUP
 # - variables defined in /home/appsadm/env/common_defines (with profiles)
 #   assuming site admin followed advice in www.uvsoftware.ca/admjobs.htm#Part_1
 # - will '.' dot-execute site admin's .bash_profile to define $SYMBOLs & PATHs
 #
 export APPSADM=/home/appsadm    # define site admin superdir
 #===========================      (only absolute path in these scripts)
 . $APPSADM/.bash_profile        # define $PRODDATA,$PRODLIBS,$BACKUP,etc
 #===========================
 #
 # verify that env-vars set OK & required subdirs present
 if [[ -d $BACKUP/zip && -d $BACKUP/Month ]]; then :
   else echo "\$BACKUP/zip=$BACKUP/zip"
        echo "\$BACKUP/Month=$BACKUP/Month"
        echo "\$APPSADM=$APPSADM"
        echo "1 or more of above directories NOT found:"
        echo "- or \$variables not set by \$APPSADM/.bash_profile"
        exit 99; fi
 #
 # copy today's (1st of month) zip files into the Month backup dir
 # - backupPurge script will remove files older than 15 months
 cp -i $BACKUP/zip/* $BACKUP/Month
 #================================
 #
 echo "backupMonth completed, files in \$BACKUP/zip are:"
 ls -l $BACKUP/zip
 echo "accumulated files in \$BACKUP/Month are:"
 ls -l $BACKUP/Month
 months=$(ls $BACKUP/Month | wc -l)
 echo "file count in \$BACKUP/Month ($BACKUP/Month) = $months"
 exit 0
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4F5. ADMjobs: Advanced Backup & Restore

 #!/bin/ksh
 # backupYear - copy current backup zip files to Year backup directory
 #            - for cron on Jan 1 each year at 4:30 AM (could run manually)
 #            - by Owen Townsend, UV Software, May 18/2007
 #            - see doc at www.uvsoftware.ca/admjobs.htm#Part_4
 #
 # backupYear       <-- no arguments required
 # ===========
 #
 echo "backupYear - copy current backup zip files to Year backup directory"
 echo "- intended for cron on Jan 1 each year at 4:30 AM (could run manually)"
 echo "- preceded by backupNight, backed up \$PRODDATA \$PRODLIBS \$HOMEDIRS"
 echo "  & zipped them to \$BACKUP/zip/... with date/time stamps"
 echo "- this script simply copies contents of \$BACKUP/zip/* to \$BACKUP/Year"
 echo "- see more documentation at: www.uvsoftware.ca/admjobs.htm#Part_4"
 echo "enter to proceed (will not stop here if run under cron)"; read reply
 #
 #Note - we will minimize necessity to modify backup scripts for new sites
 #     - by using environmental variables $PRODDATA,$PRODLIBS,$BACKUP,etc
 # - variables defined in /home/appsadm/env/common_defines (with profiles)
 #   assuming site admin followed advice in www.uvsoftware.ca/admjobs.htm#Part_1
 # - will '.' dot-execute site admin's .bash_profile to define $SYMBOLs & PATHs
 #
 export APPSADM=/home/appsadm    # define site admin superdir
 #===========================      (only absolute path in these scripts)
 . $APPSADM/.bash_profile        # define $PRODDATA,$PRODLIBS,$BACKUP,etc
 #===========================
 #
 # verify that env-vars set OK & required subdirs present
 if [[ -d $BACKUP/zip && -d $BACKUP/Year ]]; then :
   else echo "\$BACKUP/zip=$BACKUP/zip"
        echo "\$BACKUP/Year=$BACKUP/Year"
        echo "\$APPSADM=$APPSADM"
        echo "1 or more of above directories NOT found:"
        echo "- or \$variables not set by \$APPSADM/.bash_profile"
        exit 99; fi
 #
 # copy today's (1st of year) zip files into the Year backup dir
 # - backupPurge script will remove files older than 7 years
 cp -i $BACKUP/zip/* $BACKUP/Year
 #===============================
 #
 echo "backupYear completed, files in \$BACKUP/zip are:"
 ls -l $BACKUP/zip
 echo "accumulated files in \$BACKUP/Year are:"
 ls -l $BACKUP/Year
 years=$(ls $BACKUP/Year | wc -l)
 echo "file count in \$BACKUP/Year ($BACKUP/Year) = $years"
 exit 0
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4F6. ADMjobs: Advanced Backup & Restore

 #!/bin/ksh
 # backupTapeA - backup Nightly to tape (DAT or DLT)
 #             - this script stored in: /home/appsadm/sf/backupTapeA
 #             - run by crontab file stored at: /home/appsadm/sf/cronbackupNight
 #
 # usage: backupTapeA   - no args required
 #        ===========
 #
 # - this script run by cron at 5 AM, following Nightly backups to disc
 # - files already zipped by backupNight script as follows:
 #
 # 1. $BACKUP/zip/homedirs_yymmdd_HHMM.zip
 # 2. $BACKUP/zip/prodlibs_yymmdd_HHMM.zip
 # 3. $BACKUP/zip/proddata_yymmdd_HHMM.zip
 #
 # - use 'restore1' to restore files to $RESTORE/zip/...
 # - then copy desired .zip file to appropriate area & unzip
 #   ($RESTORE/homedirs, $RESTORE/prodlibs, or $RESTORE/proddata)
 # - clear out any old files before restore,
 # - after restore, unzip, investigate, & copy desired files back to
 #   /home/..., $PRODDATA/..., or $PRODLIBS/...
 #
 #Note - we will minimize necessity to modify backup scripts for new sites
 #     - by using environmental variables $BACKUP, $TAPERWD, $TAPENRW
 # - variables defined in /home/appsadm/env/common_defines (with profiles)
 #   assuming site admin followed advice in www.uvsoftware.ca/admjobs.htm#Part_1
 # - will '.' dot-execute site admin's .bash_profile to define $SYMBOLs & PATHs
 #
 export APPSADM=/home/appsadm    # define site admin superdir
 #===========================      (only absolute path in these scripts)
 . $APPSADM/.bash_profile        # define $PRODDATA,$PRODLIBS,$BACKUP,etc
 #===========================
 #
 # verify that env-vars set OK & required subdirs present
 if [[ -d $BACKUP/zip && -d $APPSADM && -c $TAPERWD && -c $TAPENRW ]]; then :
   else echo "\$BACKUP/zip=$BACKUP/zip"
        echo "\$APPSADM=$APPSADM"
        echo "\$TAPERWD=$TAPERWD   # rewind tape Linux SCSI"
        echo "\$TAPENRW=$TAPENRW   # NO rewind tape Linux SCSI"
        echo "1 or more of above directories or devices NOT found:"
        echo "- or \$variables not set by \$APPSADM/.bash_profile"
        exit 99; fi
 #
 #Note - tape devices for site are assigned in \$APPSADSM/env/common_defines
 #     - 1st SCSI tape device for Linux would be as follows:
 # TAPERWD=/dev/st0     # rewind tape device for Linux SCSI
 # TAPENRW=/dev/nst0    # NO rewind tape device for Linux SCSI
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 echo "backup \$BACKUP/zip/* to tape (\$BACKUP=$BACKUP)"
 echo "- intended for cron at 5 AM (but could be run manually)"
 echo "- follows backupNight which created date stamped .zip files "
 echo "$BACKUP/zip/homedirs_yymmdd_HHMM.zip"
 echo "$BACKUP/zip/prodlibs_yymmdd_HHMM.zip"
 echo "$BACKUP/zip/proddata_yymmdd_HHMM.zip"
 echo "- enter to backup all files in \$BACKUP/zip/* to tape"
 read reply               # Note - will not wait for reply when run by cron
 #
 mt -f $TAPERWD rewind    # ensure tape rewound
 #
 cd $BACKUP/zip
 #
 find . -print | cpio -ocvBO/$TAPENRW
 #===================================
 #
 # could append other backups on end of $BACKUP/zip archive ??
 #
 echo "backups complete - rewinding & unloading tape"
 mt -f $TAPERWD rewind           # rewind tape
 mt -f $TAPERWD offline          # unload tape
 exit 0
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4F7. ADMjobs: Advanced Backup & Restore

 #!/bin/ksh
 # restoreTapeA - Korn shell script from UVSI stored in: /home/uvadm/sf/adm/
 # restoreTapeA - restore cpio tape to an empty work space
 #              - this job restores all files from the 1st archive on tape
 # restoreTape1 <-- alternate job for multi-archive tapes
 #
 #usage: 1. change to directory where files are to be restored
 #       2. remove any old files (directory must be empty)
 #       3. restoreTapeA directory
 #          ======================
 #
 #sample 1. cd $RESTORE/zip
 #       2. rm  -fr *
 #       3. restoreTapeA $RESTORE/zip
 #          =========================
 #
 # - you must specify full path name of restore directory &
 #   it must match your current directory & it must be empty
 #
 #Note - we minimize necessity to modify backup scripts for new sites
 #     - by using environmental variables $TAPERWD, $TAPENRW
 # - variables defined in /home/appsadm/env/common_defines (with profiles)
 #   assuming site admin followed advice in www.uvsoftware.ca/admjobs.htm#Part_1
 # - will '.' dot-execute site admin's .bash_profile to define $SYMBOLs & PATHs
 #
 export APPSADM=/home/appsadm    # define site admin superdir
 #===========================      (only absolute path in these scripts)
 . $APPSADM/.bash_profile        # define $PRODDATA,$PRODLIBS,$BACKUP,etc
 #===========================
 #
 # verify that env-vars set OK & required subdirs present
 if [[ -d $APPSADM && -c $TAPERWD && -c $TAPENRW ]]; then :
   else echo "\$APPSADM=$APPSADM"
        echo "\$TAPERWD=$TAPERWD   # rewind tape Linux SCSI"
        echo "\$TAPENRW=$TAPENRW   # NO rewind tape Linux SCSI"
        echo "1 or more of above directories or devices NOT found:"
        echo "- or \$variables not set by \$APPSADM/.bash_profile"
        exit 99; fi
 #
 #Note - tape devices for site are assigned in $APPSADSM/env/common_defines
 #     - 1st SCSI tape device for Linux would be as follows:
 # TAPERWD=/dev/st0     # rewind tape device for Linux SCSI
 # TAPENRW=/dev/nst0    # NO rewind tape device for Linux SCSI
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 # capture command arguments & verify
 cdir="$1"                   # capture arg1 directory path
 dpath=$(pwd)                # cpature current directory path
 echo "restoreTapeA: arg1=curdirpath=$cdir"
 #
 if [[ ! -d "$cdir" ]]
    then echo "USAGE: restoreTapeA curdirfullpath";
         echo "  arg2 invalid - must be full path to current(restore) directory"
         echo "       - restore directory must be empty & you must be in it"
         exit 92; fi
 #
 if [[ $dpath != $cdir ]]
    then echo "you must be in the directory specified"; exit 92; fi
 #
 ls . >/tmp/restore_emptytest
 if [[ -s /tmp/restore_emptytest ]]
    then echo "the current directory must be empty";exit 93; fi
 #
 echo "restore tape to current directory ($cdir) OK ?"; read reply
 read reply
 #
 mt -f $TAPERWD rewind       # ensure tape rewound
 cpio -icvdmBI $TAPENRW      # restore spcfd archive to cur dir
 #=====================
 echo "tape restored to $cdir, rewinding tape"
 mt -f $TAPERWD rewind       # rewind tape after
 exit 0
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4F8. ADMjobs: Advanced Backup & Restore

 #!/bin/ksh
 # backupPurge1 - purge backup files older than specified limits
 #              - intended for cron each night at 2:30 AM (could run manually)
 #              - by Owen Townsend, UV Software, May 18/2007
 #              - see doc at www.uvsoftware.ca/admjobs.htm#Part_4
 #
 # backupPurge1       <-- no arguments required
 # ============
 #
 echo "backupPurge1 - purge backup files older than specified limits"
 echo "- intended for cron each night at 2:30 AM (but could run manually)"
 echo "- followed by backupNight, backupMonth(1st of month), backupYear(Jan 1)"
 echo "- removes files from \$BACKUP/Day older than 40 days"
 echo "- removes files from \$BACKUP/Month older than 15 months"
 echo "- removes files from \$BACKUP/Year older than 7 years"
 echo "- see more documentation at: www.uvsoftware.ca/admjobs.htm#Part_4"
 echo "enter to proceed (will not stop here if run under cron)"; read reply
 #
 #Note - we will minimize necessity to modify backup scripts for new sites
 #     - by using environmental variables $PRODDATA,$PRODLIBS,$BACKUP
 # - variables defined in /home/appsadm/env/common_defines (with profiles)
 #   assuming site admin followed advice in www.uvsoftware.ca/admjobs.htm#Part_1
 # - will '.' dot-execute site admin's .bash_profile to define $SYMBOLs & PATHs
 #
 export APPSADM=/home/appsadm    # define site admin superdir
 #===========================      (only absolute path in these scripts)
 . $APPSADM/.bash_profile        # define $PRODDATA,$PRODLIBS,$BACKUP,etc
 #===========================
 #
 # verify that env-vars set OK & required subdirs present
 if [[ -d $BACKUP/Day && -d $BACKUP/Month && -d $BACKUP/Year ]]; then :
   else echo "\$BACKUP/Day=$BACKUP/Day"
        echo "\$BACKUP/Month=$BACKUP/Month"
        echo "\$BACKUP/Year=$BACKUP/Year"
        echo "\$APPSADM=$APPSADM"
        echo "1 or more of above directories NOT found:"
        echo "- or \$variables not set by \$APPSADM/.bash_profile"
        exit 99; fi
 #
 # now remove files older than desired limits
 find $BACKUP/Day   -mtime   +40 -exec rm -fr {} \;
 #=================================================
 find $BACKUP/Month -mtime  +150 -exec rm -fr {} \;
 #=================================================
 find $BACKUP/Year  -mtime +2555 -exec rm -fr {} \;
 #=================================================
 #
 echo "after backupPurge1, files in \$BACKUP/Day are:"; ls -l $BACKUP/Day
 echo "after backupPurge1, files in \$BACKUP/Month are:"; ls -l $BACKUP/Month
 echo "after backupPurge1, files in \$BACKUP/Year are:"; ls -l $BACKUP/Year
 days=$(ls $BACKUP/Day | wc -l); echo "file count in \$BACKUP/Day = $days"
 months=$(ls $BACKUP/Month | wc -l); echo "file count \$BACKUP/Month = $months"
 years=$(ls $BACKUP/Year | wc -l); echo "file count in \$BACKUP/Year = $years"
 exit 0
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4F9. ADMjobs: Advanced Backup & Restore

 #!/bin/ksh
 # backupPurge2 - purge backup files older than specified limits
 #              - intended for cron each night at 2:30 AM (could run manually)
 #              - by Owen Townsend, UV Software, May 18/2007
 #              - see doc at www.uvsoftware.ca/admjobs.htm#Part_4
 #
 # backupPurge2      <-- no arguments required
 # ============
 #
 # backupPurge2 is an alternate to backupPurge1 which uses find & -mtime:
 #
 # find $BACKUP/Day   -mtime   +40 -exec rm -fr {} \;
 #===================================================
 # find & -mtime would not work if files were copied/moved (without -p option)
 # backupPurge2 will calculate age based on _yymmdd_ embedded in filenames.
 #
 echo "backupPurge2 - purge backup files older than specified limits"
 echo "- intended for cron each night at 2:30 AM (but could run manually)"
 echo "- followed by backupNight, backupMonth(1st of month), backupYear(Jan 1)"
 echo "- removes files from \$BACKUP/Day older than 40 days"
 echo "- removes files from \$BACKUP/Month older than 15 months"
 echo "- removes files from \$BACKUP/Year older than 7 years"
 echo "- see more documentation at: www.uvsoftware.ca/admjobs.htm#Part_4"
 echo "enter to proceed (will not stop here if run under cron)"; read reply
 #
 export APPSADM=/home/appsadm    # define site admin superdir
 #===========================      (only absolute path in these scripts)
 . $APPSADM/.bash_profile        # define $PRODDATA,$PRODLIBS,$BACKUP,etc
 #===========================
 # verify that env-vars set OK & required subdirs present
 if [[ -d $BACKUP/Day && -d $BACKUP/Month && -d $BACKUP/Year ]]; then :
   else echo "\$BACKUP/Day=$BACKUP/Day"
        echo "\$BACKUP/Month=$BACKUP/Month"
        echo "\$BACKUP/Year=$BACKUP/Year"
        echo "\$APPSADM=$APPSADM"
        echo "1 or more of above directories NOT found:"
        echo "- or \$variables not set by \$APPSADM/.bash_profile"
        exit 99; fi
 # call uvcopy job 'purgeold2' to remove files, based on age calculations
 # - using _yymmdd_ embedded in filenames & specified no of days old
 # - purging independent of dir entry dates (in case files moved w/o -p option)
 #
 uvcopy purgeold2,fild1=$BACKUP/Day,arg1=40,uop=q0i7f1
 #====================================================
 uvcopy purgeold2,fild1=$BACKUP/Month,arg1=450,uop=q0i7f1
 #=======================================================
 uvcopy purgeold2,fild1=$BACKUP/Year,arg1=2555,uop=q0i7f1
 #=======================================================
 echo "after backupPurge2, files in \$BACKUP/Day are:"; ls -l $BACKUP/Day
 echo "after backupPurge2, files in \$BACKUP/Month are:"; ls -l $BACKUP/Month
 echo "after backupPurge2, files in \$BACKUP/Year are:"; ls -l $BACKUP/Year
 days=$(ls $BACKUP/Day | wc -l); echo "file count in \$BACKUP/Day = $days"
 months=$(ls $BACKUP/Month | wc -l); echo "file count \$BACKUP/Month = $months"
 years=$(ls $BACKUP/Year | wc -l); echo "file count in \$BACKUP/Year = $years"
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

4F10. ADMjobs: Advanced Backup & Restore

 # purgeold2 - purge files from a directory older than a specified no of days
 #           - based on the dates embedded in the filenames
 #           - by Owen Townsend, UV Software, May 19/2007
 #           - see documentation at www.uvsoftware.ca/admjobs.htm#Part_4
 #
 # uvcopy purgeold2,fild1=directory,arg1=days
 # ==========================================
 #
 # This uvcopy job is called by the backupPurge2 script
 # (alternate backupPurge1 uses directory entry dates vs embedded dates)
 # The backuppurge2 script calls this job to remove:
 # - Daily backup files older than 40 days
 # - Monthly backup files older than 450 days
 # - Yearly backup files older than 2555 days
 #
 # uvcopy purgeold2,fild1=$BACKUP/Day,arg1=40
 # uvcopy purgeold2,fild1=$BACKUP/Month,arg1=450
 # uvcopy purgeold2,fild1=$BACKUP/Year,arg1=2555
 #
 #     ** sample filenames in $BACKUP/Day **
 #
 # homedirs_070501_0841.zip    <--
 # proddata_070501_0841.zip    <--dropped if run on 070610
 # prodlibs_070501_0841.zip    <--
 #
 # homedirs_070502_0841.zip
 # proddata_070502_0841.zip
 # prodlibs_070502_0841.zip
 #   - - - - etc - - - -
 # homedirs_070610_0841.zip
 # proddata_070610_0841.zip
 # prodlibs_070610_0841.zip
 #
 #           ** why purgeold2 (backupPurge2 vs backupPurge1) **
 #
 # backupPurge1 drops files older than specified dates
 # - based on directory entry dates using commands such as
 #
 #   find $BACKUP/Day   -mtime   +40 -exec rm -fr {} \;
 #   ==================================================
 #
 # But backupPurge1 method will not work if files copied w/o -p option
 # - which might happen if files moved or system moved to new machine
 # - backupPurge2 using this job & embedded dates is independent of dir entries
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 opr='uop=f0 - option defaults'
 opr='    f0 - interactive, prompts for remove (rm option -i)'
 opr='    f1 - no prompts, use force option (-f)'
 uop=q1f0      # option defaults
 fild1=?backupdir,typ=DIR,rcs=80
 @run
        opn    all                  open directory to prove present
 #
 # calc purge date yymmdd for current date - arg1 no of days
        mvn    d0(5),$arg1(5)       store no of days from $arg1
        mvc    d10(6),$yymmdd       store current date yymmdd
        datcn  d20(5),d10(6)        convert current yymmdd to days since 1900
        mvn    d30(5),d20(5)        move to 2nd area (in case debug display)
        sub    d30(5),d0(5)         current days since 1900 - days specified
        datnc  d40(6),d30(5)        convert days since 1900 to yymmdd
 #
 # begin loop to read filenames from directory & remove if older than spcfd
 man20  get    fild1,a0(80)         get next directory entry
        skp>   man90                (cc set > at EOD)
        skp<   man20                (cc set < if directory vs file)
        add    $ca1,1               count files
        scn    a0(50),'_'           scan to 1st '_' prior to yymmdd
        skp!   err1
        mvu    b0(7),ax1,'_'        move yymmdd until ending '_'
        skp!   err1
        cmcp   b0(6),'######'       6 numerics ?
        skp!   err1
 man28  cmc    b0(6),d40(6)         extracted date < older than current date ?
        skp=>  man20
 #
 # date extracted from filename is < (older than) calculated purge date
 man30  mvf    b0(250),'rm -f '      setup remove command & clear area
        cmn    $uopbf,1              rm -f option ? (vs prompt -i option)
        skp=>  1
        mvf    b0(250),'rm -i '      change to interactive prompt option
        mvu    b6(100),$fild1,x'00'  insert directory name
        cat    b0(200),'/'           append '/' separator
        cat    b0(200),a0(50)        append filename
 man36  sys    b0(200)               execute remove command
        add    $ca2,1                count remove (attempts)
        skp    man20                 return to get next filename
 #
 # EOD - close & end job
 man90  cls    all                   close all open
        msgv1  '$ca2 removes from $ca1 files in directory $fild1'
        eoj                          end of job
 #
 # Error if filename not as expected '_yymmdd_' embedded in filename
 # example: proddata_070501_0841.zip
 err1   msg    a0(50)            display current directory name
        msg    'proddata_070501_0841.zip  <-- sample format expected'
        msgw   'filename does not contain _yymmdd_, enter to bypass'
        skp    man20
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_5 ADMjobs - 'cron' automatic job scheduling

crontabs & scripts run by crontabs

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part5 cron scheduling - Contents continued

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5A1. ADMjobs: 'cron' - Introduction & Overview

'cron' is the Unix/Linux facility to automatically run scripts at specified times (daily, weekly, monthly, or whatever).

crontab files

Crontab files specify the times & the names of the scripts which are to be automatically executed at the specified times. Each user may have their own crontab file. See the 'crontab' activation procedures documented within the 'crontab_appsadm1' sample file (listed on the next page).

I recommend the applications administrator (appsadm) setup a 'crontab' to perform the applications backups for libraries & data files, and to run other jobs that can be auto scheduled (nightly, weekly, monthly, etc).

Note that 'cron' jobs run under the crontab owner userid, but the user profile is not automatically executed to setup PATH's etc. You will notice that the various sample scripts (nightly1,weekly1,monthly1) perform a dot '.' execution of /home/appsadm/.bash_profile to setup PATH's.

The appsadm cron jobs (backups & nightly scripts) do not need to run with 'root' privileges since appsadm is in the same group as the production operators who create the files. This means that appsadm can not back up the unix/linux system files.

It is probably not necessary to backup the unix/linux system nightly, since system files rarely change. But if you wish to backup the system nightly, you would set up a separate crontab owned by root to do this.

The system backups could be a separate archive at the end of the same tape just written by the appsadm crontab. Or they could be a different tape if you have a multi-tape system, or a different tape device if you have multiple tape drives.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5B1. ADMjobs: 'cron' automatic job scheduling

crontab_appsadm1 - sample crontab file for backups, etc

 # crontab_appsadm1 - crontab file to run various scripts
 #                    (backups,cleanups,application JCL/scripts)
 #                  - nightly1, weekly, monthly1
 #                  - store this file at /home/appsadm/sf/...
 #
 # - see ADMjobs.doc or www.uvsoftware.ca/admjobs.htm for crontab samples
 # - sample crontab file supplied in /home/uvadm/sf/adm/crontab_appsadm1
 # - I suggest you setup userid 'appsadm' to house your crontabs & cron scripts
 #   then copy to /home/appsadm/sf/crontab_appsadm1 & modify as required
 #
 # - you must log on as 'appsadm' to update crontabs
 #   (crontab affects crons only for the logged on user)
 # - the 'crontab' command (#4 below) copies the specified crontab file
 #   to the 'real' crontab file which would be: /var/spool/cron/appsadm
 # - Before 1st use, you must logon as root, &  add the 'appsadm'
 #   userid to: /etc/cron.allow if it exists. If it does not exist
 #   you can use cron unless your userid exists in /etc/cron.deny
 #
 # suggested procedures for updating crontab file (for appsadm) are:
 # 1. logon as appsadm --> /home/appsadm
 # 2. vi sf/crontab_appsadm1       - edit this file as required
 # 3. crontab -r                   - remove all old crontab lines for appsadm
 #                                   (OK since this file is always the source)
 # 4. crontab sf/crontab_appsadm1  - activate new crontab for appsadm
 # 5. crontab -l                   - list crontab onfile to confirm installation
 #
 #-----------------------------------------------------------------------
 # arguments to crontab are as follows:
 # minute hour day-of-mth mth-of-yr day-of-week <----command---->
 #
 00 3 * * 2-6 /home/appsadm/sf/backupTape # Nightly backup to tape 3AM
 #=======================================
 #
 30 3 * * 2-6 /home/appsadm/sf/nightly1   # Nightly cleanup 3:30 AM
 #=====================================
 # - nightly1 calls logfixN to fix console logs for viewing/printing
 # - cleanup tmp subdirs in homedirs, prodlibs, proddata
 #
 00 4 * * 0 /home/appsadm/sf/weekly1      # Weekly Sunday 4 AM
 #==================================
 # - remove report subdirs older than 15 days
 #
 00 5 01 * * /home/appsadm/sf/monthly1    # Monthly (1st day at 5 AM)
 #====================================
 # - monthly1 calls logfixM to copy /home/appsadm/log2/... to log3
 #   & clear log2/... subdirs
 #
 #------------------------ end crontab_appsadm1 --------------------------

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5C1. ADMjobs: 'cron' automatic job scheduling

nightly1 - script run by crontab_appsadm1

 #!/bin/ksh
 # nightly1 - nightly processing for user applications
 #          - by Owen Townsend, UV Software, updated Nov 2009
 #
 # This 'nightly1' script run by 'crontab_appsadm1'
 # - runs various scripts (cleantmps, logfixN, vtocrpts, etc)
 # - see scripts & crontabs stored in /home/appsadm/sf/...
 #
 export APPSADM=/home/appsadm
 cd $APPSADM                # change to /home/appsadm (above env/ & log/ subdirs
 . $APPSADM/.bash_profile   # '.' dot execute profile for PATH's,perms,etc
 #=======================   # (common_defines + RUNLIBS/RUNDATA + common_profile)
 export PATH=$PATH:$RUNLIBS/jcls # might need this if jcls not already in PATH
 #
 # clean out tmp subdir contents in homedirs, prodlibs, proddata, etc
 cleantmps
 #========
 # run 'logfixN' to process logfiles for any users who did not logoff
 # - killuser2 (run by crontab_root prior to this) has closed their logfiles
 logfixN
 #======
 #
 #Note - could run series of batch jobs here
 # jgl100.ksh  #<-- could run a demo job for testing
 # ==========     - see jgl100.ksh listed at www.uvsoftware.ca/admjobs.htm#5K5
 #
 #Note - console logs for jobs run under cron are 'mailed' to the crontab owner
 #       which is 'appsadm', so appsadm can login each morning & read his mail
 #       to see if the nigtly jobs had any errors ?
 # BUT - would be nice if we could save the mail as date/time stamped files
 #       in case appsadm forgets & we want to examine the history
 # YES - we can do it, see crontab2 & cronmailsave1 in ADMjobs.doc 5K1 & 5K4
 #     - crontab2 schedules cronmailsave1 to run after nightly jobs
 #       to read the mail & save in date/time stamped files in appsadm/cronlog2
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5C2. ADMjobs: 'cron' automatic job scheduling

cleantmps - script run by crontab_appsadm1

 #!/bin/ksh
 # cleantmps - clean out all 'tmp' subdirs
 #           - in homedirs, prodlibs, proddata, etc
 #           - by Owen Townsend, Dec26/05, at LNPF
 # - store these scripts & crontab_appsadm1 in /home/appsadm/sf/...
 #
 # - cleantmps called by 'nightly1', which is scheduled by crontab_appsadm1
 # - nightly performs '.' (source) execute of appsadm .bash_profile
 #   to setup $symbols $PRODLIBS, $PRODDATA for use below
 #
 # clean out tmp subdir contents in homedirs, prodlibs, proddata, etc
 rm -f  /home/*/tmp/*
 rm -f  $PRODLIBS/tmp/*
 rm -f  $PRODDATA/tmp/*
 rm -fr $PRODDATA/jobtmp/*
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5D1. ADMjobs: 'cron' automatic job scheduling

weekly1 - script run by crontab_appsadm1

 #!/bin/ksh
 # weekly1 - Korn shell script from UVSI stored in: /home/uvadm/sf/adm/
 # weekly1 - sample script run by 'cron' weekly Sunday at 3AM or whatever
 #         - makes weekly on-disc backup of PRODLIBS & PRODDATA
 #
 # - copy this (/home/uvadm/sf/adm/weekly1) to your /home/appsadm/sf/...
 #   & modify depending on site requirements
 # - this script can be auto scheduled by 'cron'
 # - see sample crontab file /home/uvadm/sf/adm/crontab_appsadm1, that you can
 #   copy to your /home/appsadm/sf/... & modify as required
 #
 # - perform '.' (source) execute of appsadm .bash_profile to setup $symbols
 # - see $symbols below ($PRODLIBS, $PRODDATA, $BACKUP)
 #
 export APPSADM=/home/appsadm
 cd $APPSADM                # change to /home/appsadm (above env/ & log/ subdirs
 . $APPSADM/.bash_profile   # '.' dot execute profile for PATH's,perms,etc
 #=======================   # (common_defines + RUNLIBS/RUNDATA + common_profile)
 export PATH=$PATH:$RUNLIBS/jcls # might need this if jcls not already in PATH
 #
 #----------------------------------------------------------------------
 # copy PRODLIBS & PRODDATA to weekly on-disc backup directories
 # - 1st remove all prior week backup files
 rm -rf $BACKUP/prodlibsBW/*
 cp -r $PRODLIBS $BACKUP/prodlibsBW
 rm -rf $BACKUP/proddataBW/*
 cp -r $PRODLIBS $BACKUP/proddataBW
 #
 #----------------------------------------------------------------------
 # clean out various temp subdirs
 # note - jobtmp & sysout are subdirectoried (use option 'r')
 rm -fr $PRODDATA/jobtmp/*   # clear all subdirs & files from jobtmp
 rm -fr $PRODDATA/sysout/*   # clear all subdirs & files from sysout
 rm -f $PRODDATA/tmp/*       # clear all files from tmp
 rm -f $PRODDATA/wrk/*       # clear all files from wrk
 #
 # clear files in subdir 'rpts' older than 15 days
 find $PRODDATA/rpts/* -ctime +15 -exec rm -r {} \;
 #=================================================
 #
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5E1. ADMjobs: 'cron' automatic job scheduling

monthly1 - script run by crontab_appsadm1

 #!/bin/ksh
 # monthly1 - Korn shell script from UVSI stored in: /home/uvadm/sf/adm/
 # monthly1 - sample script run by 'cron' early on the 1st of each month
 #
 # - copy this (/home/uvadm/sf/adm/monthly1) to /home/appsadm/sf/...
 #   & modify depending on site requirements
 # - this script can be auto scheduled by 'cron'
 # - see sample crontab file /home/uvadm/sf/adm/crontab_appsadm1
 # - copy to your site's /home/appsadm/sf/... & modify as required
 #
 # - this sample runs 'logfixM' logfile monthly processing
 # - see descriptions in /home/uvadm/sf/logfixM
 # - could add more monthly processing to this script
 #
 # establish appsadm PATH & execute common_profile_prod
 # - to define: PATH, PFPATH, PRODLIBS, PRODDATA, BACKUP dirs, etc
 #
 export APPSADM=/home/appsadm
 export APPSADM=/home/appsadm
 cd $APPSADM                # change to /home/appsadm (above env/ & log/ subdirs
 . $APPSADM/.bash_profile   # '.' dot execute profile for PATH's,perms,etc
 #=======================   # (common_defines + RUNLIBS/RUNDATA + common_profile)
 export PATH=$PATH:$RUNLIBS/jcls # might need this if jcls not already in PATH
 #
 logfixM    # save last months log files in log3 & clear log2 for this month
 #======
 backupBM   # make monthly backups of pdoddata & prodlibs
 #=======
 vtocshift  # shift vtoc report subdirs (vtoc2->vtoc3,vtoc1->vtoc2,clear vtoc1)
 #========
 #          # could add more scripts here ???
 exit 0
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5F1. ADMjobs: 'cron' automatic job scheduling

crontab_user - sample crontab file for users

 # crontab_user - Korn shell script from UVSI stored in: /home/uvadm/sf/adm/
 # crontab_user - sample crontab file for users
 #              - this sample only 'exit's in case you forgot to log off
 #              - to close the console logging file to prevent loss
 #              - required before crontab_appsadm1/nightly1 processes log files
 #
 # The 'console log' file is created by unix 'script' command at end of profile
 # - see 'console logging' at www.uvsoftware.ca/admjobs.htm#Part_6
 # - also see 'crontab_root' to kill users who did not log off
 #
 #     ** Op. Instrns. for console logging users **
 #
 # 1. login with your userid  --> /home/userid/
 # 2. mkdir sf                    - make directory if not already made
 # 3. cp /home/uvadm/sf/adm/crontab_user sf/crontab_userid
 #    ====================================================
 #    - copy supplied crontab_user to your subdir & rename with your userid
 #
 # 4. vi sf/crontab_userid        - edit this file if desired
 #                                - could add actions other than 'exit'
 # 5. crontab sf/crontab_userid   - activate new crontab for your userid
 #    =========================
 # 6. crontab -l                  - list crontab file to confirm installation
 #
 # minute hour day-of-mth mth-of-yr day-of-week <----command---->
 #
 51 01 * * * exit        #<-- 1st exit exits 'script' (console logging)
 #===============             (closes the console log file)
 52 01 * * * exit        #<-- 2nd exit exits your shell
 #===============
 #
 #BUT - this is obsoleted by crontab_root, which runs killuser2 script
 #      to kill all ksh & bash shell users who forgot to log off
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5F2. ADMjobs: 'cron' automatic job scheduling

crontab_root - sample crontab for root

 # crontab_root - crontab file to run under root
 #              - by Owen Townsend, UV Software, May 2008
 # - supplied in /home/uvadm/sf/adm/crontab_root
 # - copy to /home/appsadm/sf/crontab_root before customization & activation
 # - see details at www.uvsoftware.ca/admjobs.htm#Part_5
 #
 # Only for backups, cleanups, etc that require 'root'
 # - see crontab_apps1 for backups & applications (daily,weekly,monthly,etc)
 # - 'crontab_apps1' runs under 'appsadm' (not root) & much safer
 #
 # Actual crontab for root depends on the unix/linux OS
 # - you need to find it & add desired lines
 # OR, you could use the following procedure so you could maintain
 # all crontabs in 1 place /home/appsadm/sf/...
 # - after you have retrieved & combined root crontabs with this file
 # - for RHEL 5.1 there is no root crontab, so just copy this
 #
 # suggested procedures for updating root crontab:
 # 1. logon as root & cd to /home/appsadm
 # 2. crontab -l (if 1st setup)   - list to see if any crontab exists for root
 # 3. crontab -l >sf/crontab_root - ifso, redirect to sf/appsadm
 # 4. vi sf/crontab_root          - edit this file as required
 # 4a. :r /home/uvadm/sf/adm/crontab_root   <-- append this supplied file
 # 5. crontab -r                  - remove old crontab file for root
 # 6. crontab sf/crontab_root     - activate new crontab for root
 # 7. crontab -l                  - list crontab file to confirm installation
 #
 # minute hour day-of-mth mth-of-yr day-of-week <----command---->
 #
 15 00 * * * /home/appsadm/sf/killuser2 all  # kill users at 12:15AM every night
 #=========================================
 # - killuser2 kills all 'ksh' or 'bash' users who did not log off
 # - also see killuser1 to interactively kill any 1 specified userid
 #
 30 00 * * 2-6 /home/appsadm/sf/setperms1 all # set perms at 12:30AM Tues-Sat
 #===========================================
 # At 12:30 AM Tues-Sat, run setperms1 to set permissions on data & libraries
 # - ensure directories 775, data-files 664, script-files 775
 # - ensure owner:group appsadm:apps (see details in setperms1 script)
 #
 00 01 * * 0 /sbin/shutdown -g0 -y -i6  # reboot at 01:00AM Sunday
 #====================================    (above command for traditional unix)
 # reboot Sunday 1 AM -g0(no wait) -y(auto reply y to prompt) -i6(reboot)
 # 00 01 * * 0 /sbin/shutdown -r now    #<-- can use this for Linux
 # =================================
 #
 #Note: Add commands here to bring up software packages, such as:
 # - Micro Focus COBOL license manager
 # - Online systems (MTO, unikix, CICS6000, etc)
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5G1. ADMjobs: 'cron' automatic job scheduling

killuser2 - script run by crontab_root

 #!/bin/ksh
 # killuser2 - Korn shell script from UVSI stored in: /home/uvadm/sf/adm/
 # killuser2 - kill users who did not logout
 #           - by Owen Townsend, www.uvsoftware.ca, March 2002
 #
 # - intended to be run by crontab_root at 11:45 PM or whenever
 # - killuser2 is run before logfixN which processes user console logs
 # - this kills user 'script' & closes the script output file
 # - also see 'killuser1' script for interactive use to kill any 1 user
 #
 #usage: killuser2 all
 #       =============
 #
 # verify arg1 'all'
 if [ "$1" != "all" ]; then echo "killuser2 arg1 must be 'all' "; exit 1; fi
 #
 # redirect ps -f output to a tmp file
 # - use '-o' output option for 3 fields only (COMMAND, PID,& RUSER)
 ps -e -ocomm -opid -oruser >/tmp/psef
 #====================================
 #    ** sample output lines **
 # COMMAND            PID RUSER
 # init                 1 root
 # bash              9022 root
 # bash              9077 uvadm
 # bash              9118 uvbak
 # ps                9226 uvadm
 #
 # - open the file & read back into variables for easier manipulation
 exec 3< /tmp/psef            # open file #3
 #
 x=0; y=0
 while read -u3 comm pid ruser
 do if [[ ("$comm" == ksh || "$comm" == bash) && ("$ruser" != root) ]]
       then let x=x+1
            kill -9 $pid
            if [[ $? == 0 ]]; then kok=OK; let y=y+1; else kok=NAK; fi
            echo "#$x kill $comm $pid $ruser - $kok"
    fi
 done
 exec 3<&-                    # close file #3
 echo "$x kills attempted, $y killed OK $(date)"
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5H1. ADMjobs: 'cron' automatic job scheduling

 #!/bin/ksh
 # setperms1 - set permissions on subdirs & files under a specified superdir
 #           - using 'find' to process all levels of directories & files
 #           - followed by chmod 775 for any bin/* & script/* dirs
 #           - by Owen Townsend, UV Software, May 2008
 #           - see complete doc at www.uvsoftware.ca/admjobs.htm#Part_5
 #
 # This script run may be run by a root crontab prior to nightly batch runs
 # - script supplied in /home/uvadm/sf/adm/setperms1
 # - should setup user 'appsadm' & copy this to /home/appsadm/sf/setperms1
 #
 # This script run by crontab_root stored at /home/appsadm/sf/crontab_root
 # - see complete listing at www.uvsoftware.ca/admjobs.htm#5F1
 # - here is just the crontab command line to run this script:
 #
 # 30 00 * * 2-6 /home/appsadm/sf/setperms1 all
 # ============================================
 #
 # This script intended as part of the Vancouver Utility mainframe conversions
 # - to ensure no bad permissions get into the DATA & Library file systems
 # - see DATA & LIBS directories suggested in www.uvsoftware.ca/admjobs.htm#2C0
 #   (examples: p1/testlibs, p1/testdata, p2/prodlibs, p2/proddata)
 # - use $symbols $TESTLIBS, $TESTDATA, $PRODLIBS, $PRODDATA
 # - defined in /home/appsadm/env/common_defines
 #
 # After using find to set perms for all subdirs(775) & all files(664), within
 # library superdirs, we must follow with 'chmod 775' for any bin/script subdirs
 # - this script assumes these are called 'bin', 'sf',& 'jcls' (VU conversions)
 # - you must modify if you use different names or setup additional bin/scripts
 #
 # Ensure arg1 is 'all' (protection against inadvertent entry of 'setperms1')
 echo "setperms1 - set perms on all subdirs(775) & all files(664)"
 echo " - within \$PRODLIBS, \$PRODDATA, \$TESTLIBS, \$TESTDATA"
 echo " - as defined in /home/appsadm/env/common_defines"
 echo " - this script can be scheduled by /home/appsadm/sf/crontab_root"
 if [[ "$1" != "all" ]]; then
    echo "usage: setperms1 all"
    echo "       ============="
    echo " - arg1 must be 'all'"
    exit 90; fi
 #
 # '.' execute common_defines to get superdir locations
 . /home/appsadm/env/common_defines
 #=================================
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 echo "set perms for all subdirs(775) & files(664) in \$PRODDATA=$PRODDATA"
 test -d "$PRODDATA" || (echo "\$PRODDATA directory not defined"; exit 91);
 #
 find "$PRODDATA" -type d -exec chmod 775 {} \;
 find "$PRODDATA" -type f -exec chmod 664 {} \;
 #=============================================
 #
 echo "set perms for all subdirs(775) & files(664) in \$PRODLIBS=$PRODLIBS"
 test -d "$PRODLIBS" || (echo "\$PRODLIBS directory not defined"; exit 92);
 #
 find "$PRODLIBS" -type d -exec chmod 775 {} \;
 find "$PRODLIBS" -type f -exec chmod 664 {} \;
 #=============================================
 #
 # restore 775 for executable files (in bin, sf, jcls)
 echo "restore perms 775 for executables (bin,sf,jcls) in \$PRODLIBS=$PRODLIBS"
 test -d "$PRODLIBS"/bin && chmod 775 "$PRODLIBS"/bin/*
 test -d "$PRODLIBS"/sf && chmod 775 "$PRODLIBS"/sf/*
 test -d "$PRODLIBS"/jcls && chmod 775 "$PRODLIBS"/jcls/*
 #
 echo "set perms for all subdirs(775) & files(664) in \$TESTDATA=$TESTDATA"
 test -d "$TESTDATA" || (echo "\$TESTDATA directory not defined"; exit 93);
 #
 find "$TESTDATA" -type d -exec chmod 775 {} \;
 find "$TESTDATA" -type f -exec chmod 664 {} \;
 #=============================================
 #
 echo "set perms for all subdirs(775) & files(664) in \$TESTLIBS=$TESTLIBS"
 test -d "$TESTLIBS" || (echo "\$TESTLIBS directory not defined"; exit 94);
 #
 find "$TESTLIBS" -type d -exec chmod 775 {} \;
 find "$TESTLIBS" -type f -exec chmod 664 {} \;
 #=============================================
 #
 # restore 775 for executable files (in bin, sf, jcls)
 echo "restore perms 775 for executables (bin,sf,jcls) in \$TESTLIBS=$TESTLIBS"
 test -d "$TESTLIBS"/bin && chmod 775 "$TESTLIBS"/bin/*
 test -d "$TESTLIBS"/sf && chmod 775 "$TESTLIBS"/sf/*
 test -d "$TESTLIBS"/jcls && chmod 775 "$TESTLIBS"/jcls/*
 #
 #-------------------------------------------------------
 # Set Owner & Group - could #comment if sure not a problem
 # - this protects for somebody using root & forgetting to reset owner:group
 chown -R appsadm:apps "$PRODDATA"
 chown -R appsadm:apps "$PRODLIBS"
 chown -R appsadm:apps "$TESTDATA"
 chown -R appsadm:apps "$TESTLIBS"
 #
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5H2. ADMjobs: 'cron' automatic job scheduling

Preventing Nightly Batch Failures

Several of our customers have nightly batch jobs scheduled by cron. Some of them have reported batch shift failures due to file permissions. To understand how we use cron to schedule batch jobs, please see: http://www.uvsoftware.ca/admjobs.htm#Part_5

You could run a night shift from a crontab owned by 'root' & never have a permissions failure, but this would be extremely dangerous. One wrong use of 'rm *' could wipe out your system. Running as appsadm/apps protects your system.

Our suggested crontabs & scripts are owned by 'appsadm' in group 'apps' which is common to the group of operators & programmers who work with the production data & libraries. The permissions in this group must be 775 for directories & 664 for files (which extends security to the group level).

Batch failures can occur if a day shift operator/programmer creates a file with the wrong permissions or group & this file is later used by the nightly batch scripts. FTP'd files can have the wrong permissions. Somebody might use 'root' to fix something & forget to reset permissions/owner/group.

We can prevent these failures if we setup a cron script (setperms1) to set permissions & group before the nightly batch jobs are scheduled. The setperms1 script & the crontab used to schedule it must of course be run under 'root' to be able to change permissions & groups. See crontab_root example at http://www.uvsoftware.ca/admjobs.htm#5F2. Here is the essential line:

 30 01 * * 2-6 /home/appsadm/sf/setperms1 all  # fix permissions
 #===========================================    on data & libraries
 # minute hour day-of-mth mth-of-yr day-of-week <----command---->

The crontab above (owned by root) schedules setperms1 at 1:30 AM Tues-Sat. The crontab below (owned by appsadm) schedules nightly1 at 1:45 AM Tues-Sat.

 45 01 * * 2-6 /home/appsadm/sf/nightly1 all  # schedule nightly batch jobs
 #==========================================

See the full 'setperms1' script at http://www.uvsoftware.ca/admjobs.htm#5H1, but here are the essential lines:

 . /home/appsadm/env/common_defines
 #=================================
 find $PRODDATA -type d -exec chmod 775 {} \;
 #===========================================
 find $PRODDATA -type f -exec chmod 664 {} \;
 #===========================================
 chown -R appsadm:apps $PRODDATA
 #==============================

'PRODDATA' is the super directory containing all production subdirs & files, and is defined in 'common_defines' (see '1C1'), for example:

 export PRODDATA=/p2/proddata
 ============================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5I1. ADMjobs: job scheduling under 'cron'

'job logging' via 'mail' under 'cron'

Part_6 documents 'console logging' for logged on users, BUT, it does not work for jobs scheduled by cron, because the 'script command' (uncommented at the end of the stub_profile) is designed to work only for login sessions. See the script command documented on page '6C1'.

However, we have an alternate solution based on the fact that jobs scheduled by cron 'mail' any console output to the user who issued the 'crontab'. The following pages show you how to capture the mail into date_time stamped log files.

crontabs & scripts to demo log capture by mail

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5I2. 'job logging' under 'cron' via 'mail'

appsadm subdirs for cron logs by mail

 /home/appsadm
 :-----cronlog1     <-- 1 line status msgs (optional)
 :     :-----090423_174201_cronscript1
 :     :-----090423_174401_cronscript1
 :     :-----
 :-----cronlog2     <-- console logs captured by mail from cron
 :     :-----090423_174301_cronscript1
 :     :-----090423_174501_cronscript1
 :     :-----
 :-----env          <-- profiles called by cronscript1
 :     :-----common_defines (since cron does not execute profiles)
 :     :-----common_profile
 :     :-----stub_profile
 :     :-----stub_profile_appsadm_cronlogdemo
 :     :-----
 :-----sf           <-- crontabs & scripts for demo & production models
 :     :-----cronscript1
 :     :-----cronmailsave1
 :     :-----crontab2
 :     :-----crontabtest2

JCL/scripts & DATA files used for cron log mail tests

 /home/mvstest
 :-----testdata
 :     :-----gl
 :     :-----account.acntlist_000001  <-- jgl100.ksh writes GDG file
 :     :-----account.acntlist_000002  <-- existing generations
 :     :-----account.acntlist_000003
 :     :-----                         <-- observe creation of new GDGs
 :-----testlibs
 :     :-----cbls       <-- COBOL programs
 :     :-----cgl100.cbl
 :     :-----jcls       <-- JCL/scripts (converted from mainframe JCL)
 :     :-----jgl100.ksh
 :     :-----sfun       <-- functions called by JCL/scripts
 :     :-----exportgen1   - function to create new GDG
 :     :-----logcron1     - function to write 1 line status msgs
                            (to /home/appsadm/cronlog1/...)

• above illustrates only the files used by the cron job logging by mail demo
• see complete dtree subdir illustrations at MVSJCL.htm#1A1

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5I3. 'job logging' under 'cron' via 'mail'

Setup appsadm for cron logging by mail

• originally documented on pages '1D1' - '1D7'
• minimized here to bare requirements to demo logging by mail under cron
• note dtree subdirs illustrated on page '1D1'
• relevant subdirs are: ....

 #1. Login as appsadm --> /home/appsadm
     ================

 #2a. mkdir cronlog1  <-- subdir for 1 line msgs vis cronlog1 function
 #2b. mkdir cronlog2  <-- subdir date_time stamped logs captured from cron mail
 #2c. mkdir env       <-- subdir for profiles (copied from /home/uvadm/env/...)
 #2d. mkdir sf        <-- subdir for crontabs&scripts from /home/uvadm/sf/adm/...

 #3. cp /home/uvadm/sf/adm/cron* sf  <-- copy crontabs & scripts from $UV
     ==============================

 #4. cp /home/uvadm/env/* env   <-- copy profiles from $UV to /home/appsadm/env
     ========================     - for this demo & site specific customization

 #5. vi env/stub_profile_appsadm <-- examine profile for appsadm, listed on '1C8'
     ===========================   - customize if required for your site
     - will later change TESTLIBS/TESTDATA to PRODLIBS/PRODDATA for production

 #6. cp env/stub_profile_appsadm .bash_profile
     =========================================
     - copy 'appsadm' version of stub_profile to the actual '.bash_profile'
       (assuming bash/linux, copy to '.profile' for unix Korn shell)

 #7. logoff & back on to make new .bash_profile effective

 #8. vi env/stub_profile_appsadm_cronlogdemo <-- examine profile for cronlogdemo
     =======================================
     - should not need changes for testing in /home/mvstest/...

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5I4. 'job logging' under 'cron' via 'mail'

test cron job log capture via mail

The following instructions assume:

• 'appsadm' setup & files copied from $UV to /home/appsadm/... (see page '5I3')
• appsadm/env/stub_profile copied as per #5 '5I3' OR modified as per #6 '5I4'
• 'mvstest' setup, files copied from $UV/mvstest as per MVSJCL.htm#1P1 - 1P4.
• you have converted demo JCL/COBOL as per MVSJCL.htm#1Q1 - 1R5.
• you have proved you can execute demo JCL/COBOL as per MVSJCL.htm#1S1 - 1T4.

 #1. Login as appsadm --> /home/appsadm
     ================

 #2. crontab sf/crontabtest2   <-- start cron for appsadm
     =======================     - see listing on page '5K1'

 #3. Wait for next 'EVEN' minute & then check for log/mail files created

 #4a. l cronlog1  <-- list message files in cronlog1/ from sf/cronscript1
      ==========    - should see msgs from cronscript1 (run on EVEN minutes)

 #4b. l cronlog2  <-- list mail files captured by cronmailsave1 date_time stamped
      ==========    - will be none until ODD minute

 #5. Wait for next 'ODD' minute & then check for log/mail files created

 #6a. l cronlog1  <-- Re-list message files in cronlog1/ from sf/cronscript1
      ==========    - should now be some (after cronmailsave1 scheduled)

 #6b. l cronlog2  <-- Re-list mail files captured by cronmailsave1
      ==========    - will be more after next ODD minute

 #7. Wait for 1 more cycle of #3 - #6

 #8. crontab -r   <-- Remove/deactivate the crontab (for appsadm)
     ==========

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5I5. 'job logging' under 'cron' via 'mail'

observations in 'mvstest' directories

You can also login as 'mvstest' on another screen to observe results of running JCL/scripts by cron. The demo script 'cronscript1' runs 'jgl100.ksh' which writes a GDG file. You should be able to see a new generation created every 2 minutes.

 #1. Login as mvstest --> /home/mvstest
     ================

 #2. cdd  alias='cd $TESTDATA' --> /home/mvstest/testdata
     ===

 #3. l gl     <-- list gl subdir prior to 1st crontab execution of jgl100.ksh
     ====       - account.acntlist_ demo file distributed with 3 generations

 /home/mvstest
 :-----testdata
 :     :-----gl
 :     :-----account.acntlist_000001
 :     :-----account.acntlist_000002  <-- existing generations
 :     :-----account.acntlist_000003

 #4. l gl     <-- list gl subdir AFTER 1st crontab execution of jgl100.ksh
     ====

 :     :-----account.acntlist_000004  <-- jgl100.ksh writes 4th generation

 #5. l gl     <-- list gl subdir AFTER 2nd crontab execution of jgl100.ksh
     ====

 :     :-----account.acntlist_000005  <-- jgl100.ksh writes 5th generation

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5J1. 'job logging' under 'cron' via 'mail'

results after 2 cycles cronscript1/cronmailsave1

 #1. Login appadm --> /home/appsadm
     ============

 #2. ls cronlog1 cronlog2  <-- list log files captured from cronscript1
     ====================

      /home/appsadm
      :-----cronlog1     <-- 1 line status msgs (optional)
      :     :-----090423_174201_cronscript1
      :     :-----090423_174401_cronscript1
      :     :-----
      :-----cronlog2     <-- console logs captured by mail from cron
      :     :-----090423_174301_cronscript1
      :     :-----090423_174501_cronscript1
      :     :-----

 #3. cat cronlog1  <-- display contents of /home/appsadm/cronlog1/...
     ============    - 1 line status msgs created by logcron1 function
                     - optionally coded in scripts triggered by cron

 090423_174201_cronscript1: cronscript1 - test running scripts via crontab
 090423_174201_cronscript1: cronscript1 - end running scripts via crontab

 090423_174401_cronscript1: cronscript1 - test running scripts via crontab
 090423_174401_cronscript1: cronscript1 - end running scripts via crontab

Notes

      17:42:01 - cronscript1 begins (1st of 2 cycles tested)
      17:42:01 - cronscript1 ends (in same second)

      17:44:01 - cronscript1 begins (2nd of 2 scycles, 2 minutes later)
      17:44:01 - cronscript1 ends (in same second)

 #4. l cronlog2   <-- list log files from crontabtest2/cronscript1
     ==========     - date_time stamped by cronmailsave1

      090423_174301_cronscript1
      090423_174501_cronscript1

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5J2. 'job logging' under 'cron' via 'mail'

 #5a. vi cronlog2/*   <-- inspect contents of log files
      =============

      From appsadm@uvsoft4.uvsoftware.ca  Thu Apr 23 15:19:01 2009
      Return-Path: <appsadm@uvsoft4.uvsoftware.ca>
      Received: from uvsoft4.uvsoftware.ca (localhost [127.0.0.1])
 	by uvsoft4.uvsoftware.ca (8.13.8/8.13.8) with ESMTP id n3NMJ1w4006673
 	for <appsadm@uvsoft4.uvsoftware.ca>; Thu, 23 Apr 2009 15:19:01 -0700
      Received: (from appsadm@localhost)
 	by uvsoft4.uvsoftware.ca (8.13.8/8.13.8/Submit) id n3NMJ18c006672;
 	Thu, 23 Apr 2009 15:19:01 -0700
      Date: Thu, 23 Apr 2009 15:19:01 -0700
      Message-Id: <200904232219.n3NMJ18c006672@uvsoft4.uvsoftware.ca>
      From: root@uvsoft4.uvsoftware.ca (Cron Daemon)
      To: appsadm@uvsoft4.uvsoftware.ca
      Subject: Cron <appsadm@uvsoft4> /home/appsadm/sf/cronmailsave1
      Content-Type: text/plain; charset=UTF-8
      Auto-Submitted: auto-generated
      X-Cron-Env: <SHELL=/bin/sh>
      X-Cron-Env: <HOME=/home/appsadm>
      X-Cron-Env: <PATH=/usr/bin:/bin>
      X-Cron-Env: <LOGNAME=appsadm>
      X-Cron-Env: <USER=appsadm>
      Status: R

 Mail version 8.1 6/6/93.  Type ? for help.
 "/var/mail/appsadm": 2 messages 2 new
 >N  1 root@uvsoft4.uvsoftw  Thu Apr 23 15:17  26/1121  "Cron <appsadm@uvsoft4"
  N  2 root@uvsoft4.uvsoftw  Thu Apr 23 15:18  56/3469  "Cron <appsadm@uvsoft4"
 "/home/appsadm/cronlog2/cronscript1" [Appended]

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5J3. 'job logging' under 'cron' via 'mail'

 #5b. vi cronlog2/*   <-- inspect contents of log files
      =============     - continued

 stty: standard input: Invalid argument
 stty: standard input: Invalid argument
 rm: cannot lstat `/home/appsadm/mbox': No such file or directory
 090423_174201_cronscript1: cronscript1 - test running scripts via crontab
 clear TESTDATA temp subdirs before tests or batch shift
 - easier to investigate any problems (unclutered by old files)
 - jobtmp, tmp, wrk, rpts, sysout, joblog, jobctl
 - this script does 'cd $TESTDATA' so you can run from anywhere
 - clear TESTDATA=/home/mvstest/testdata y/n ?
 all files removed from: jobtmp,tmp,wrk,rpts,sysout,joblog,jobctl
 090423:174201:JGL100: Begin Job=JGL100
 090423:174201:JGL100: /home/mvstest/testlibs/jcls/jgl100.ksh
 090423:174201:JGL100: Arguments:
 090423:174201:JGL100: RUNLIBS=/home/mvstest/testlibs
 090423:174201:JGL100: RUNDATA=/home/mvstest/testdata
 090423:174201:JGL100: JTMP=/home/mvstest/testdata/jobtmp/JGL100 SYOT=/home/mvstest/testdata/sysout/JGL100
 090423:174201:JGL100: RUNDATE=20090423
 090423:174201:JGL100: ******** Begin Step S0010 cgl100 (#1) ********
 090423:174201:JGL100: EOF fili01 rds=3 size=75: /home/mvstest/testdata/jobtmp/JGL100/gtmp/0010I_gl_account.master_
 090423:174201:JGL100: EOF filr01 rds=1 upds=1 size=10240: /home/mvstest/testdata/ctl/gdgctl51I
 090423:174201:JGL100: EOF filo02 wrts=1 size=51: /home/mvstest/testdata/jobtmp/JGL100/gtmp/0010G0_gl_account.master_
 090423:174201:JGL100: gen0: ACCTMAS=gl/account.master_000003 insize=8720
 090423:174201:JGL100: EOF fili01 rds=3 size=81: /home/mvstest/testdata/jobtmp/JGL100/gtmp/0010O_gl_account.acntlist_
 090423:174201:JGL100: EOF filr01 rds=1 upds=1 size=10240: /home/mvstest/testdata/ctl/gdgctl51I
 090423:174201:JGL100: EOF filo02 wrts=1 size=128: /home/mvstest/testdata/jobtmp/JGL100/gtmp/0010G1_gl_account.acntlist_
 090423:174201:JGL100: gen+1: ACTLIST=/home/mvstest/testdata/jobtmp/JGL100/GDG/gl/account.acntlist_000004 gens=8
 090423:174201:JGL100: file: SYSOUT=/home/mvstest/testdata/sysout/JGL100/S0010_SYSOUT bytes=
 090423:174201:JGL100: Executing--> cobrun -F /home/mvstest/testlibs/cblx/cgl100
 090423:174201:JGL100: Job Times: Begun=17:42:01 End=17:42:01 Elapsed=00:00:00
 090423:174201:JGL100: moving /home/mvstest/testdata/jobtmp/JGL100/GDG/subdir/files back to $RUNDATA/subdirs/
 `/home/mvstest/testdata/jobtmp/JGL100/GDG/gl/account.acntlist_000004' -> `gl/account.acntlist_000004'
 090423:174201:JGL100: EOF filr01 rds=5 upds=1 size=10240: /home/mvstest/testdata/ctl/gdgctl51I
 090423:174201:JGL100: JobEnd=Normal, StepsExecuted=1, LastStep=S0010
 090423_174201_cronscript1: cronscript1 - end running scripts via crontab

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5K1. listings of crontabs & scripts used cron/log/mail demo

crontab2 - schedule cronscript1 & cronmailsave1

 # crontab2 - crontab file sample for user modification/implementation
 #          - schedules sample script 'cronscript1' at 2 AM Tues-Sat
 #          - ALSO schedules 'cronmailsave1' at 3 AM to capture mail for job log
 #          - by Owen Townsend, April 2009
 #          - see documentation at www.uvsoftware.ca/admjobs.htm#Part_5
 #
 # crontabtest2 - variation for testing at UV Software
 #              - test running JCL/scripts by cron & capturing logs in mail
 #              - schedules script 'cronscript1' every 2 minutes (even minutes)
 #                AND schedules 'cronmailsave1' every 2 minutes (odd minutes)
 #                for testing with minimal wait for results
 #
 # cronscript1 - script executing JCL/scripts to be logged
 #             - runs demo JCL/script /home/mvstest/testlibs/jcls/jgl100.ksh
 #             - jgl100.ksh writes a GDG file in $RUNDATA/gl/...
 #
 # cronmailsave1 - script called by this crontab2
 #                 to save mail from prior crontab2/cronscript1
 #               - script might be as follows:
 #                 echo "save * $APPSADM/cronlog2/$CRONDT_$JOBID1" | mail
 #
 #minute hour day-of-mth mth-of-yr day-of-week <----command---->
 00 2 * * 2-6 /home/appsadm/sf/cronscript1   # 2 AM Tues-Sat
 #========================================
 #
 #Note - jobs run under cron send mail to user (appsadm)
 #     - We will capture the mail for a joblog (into a date_time stamped file)
 #     - BUT, we have to do this AFTER cron session ends (with a separate cron)
 #     - NOW, run the mail capture script at 3 AM
 #
 #minute hour day-of-mth mth-of-yr day-of-week <----command---->
 00 3 * * 2-6 /home/appsadm/sf/cronmailsave1 # 3 AM Tues-Sat
 #==========================================
 # Also see crontabtest2 which schedules cronscript1 & cronmailsave1
 # - every 2 minutes for easy testing (until you disable via 'crontab -r')
 # See crontabs documented at www.uvsoftware.ca/admjobs.htm#Part_5
 #------------------------ end crontab2 --------------------------

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5K2. listings of crontabs & scripts used cron/log/mail demo

crontabtest2 - schedule cronscript1/cronmailsave1 every 2 minutes

 # crontabtest2 - test running JCL/scripts by cron & capturing logs in mail
 #              - schedule cronscript1(EVEN minutes) & cronmailsave1(ODD minutes)
 #              - every 2 minutes for testing with minimal wait for results
 #              - by Owen Townsend, April 2009
 #              - see documentation at www.uvsoftware.ca/admjobs.htm#Part_5
 #
 #*crontabtest2 - *THIS schedules cronscript1 every EVEN minute
 #                AND schedules cronmailsave1 every ODD minute
 #
 # cronscript1 - script executing JCL/scripts to be logged
 #             - runs demo JCL/script /home/mvstest/testlibs/jcls/jgl100.ksh
 #             - jgl100.ksh writes a GDG file in $RUNDATA/gl/...
 #
 #*crontabtest2 - *THIS ALSO schedules cronmailsave1 every odd minute
 #              - to save the 'mail' from prior crontab2/cronscript1
 #                in date_time stamped file
 #                ($APPSADM/cronlog2/yymmdd_HHMMSS_jobname)
 #
 # cronmailsave1 - script called by this crontab2/crontabtest2
 #                 to save mail from prior cronscript1
 #               - script might be as follows:
 #                 echo "save * $APPSADM/cronlog2/$CRONDT_$JOBID1" | mail
 #
 # schedule cronscript1 every even minute
 #minute hour day-of-mth mth-of-yr day-of-week <----command---->
 00,02,04,06,08,10,12,14 * * * * /home/appsadm/sf/cronscript1
 16,18,20,22,24,26,28,30 * * * * /home/appsadm/sf/cronscript1
 32,34,36,38,40,42,44,46 * * * * /home/appsadm/sf/cronscript1
 48,50,52,54,56,58       * * * * /home/appsadm/sf/cronscript1
 #
 #Note - jobs run under cron send mail to user (appsadm)
 #     - We will capture the mail for a joblog (into a date_time stamped file)
 #     - BUT, we have to do this AFTER cron session ends (with a separate cron)
 #     - SO NOW, run the mail capture script every 2 minutes on the odd minute
 #
 # schedule cronmailsave1 every odd minute
 #minute hour day-of-mth mth-of-yr day-of-week <----command---->
 01,03,05,07,09,11,13,15 * * * * /home/appsadm/sf/cronmailsave1
 17,19,21,23,25,27,29,31 * * * * /home/appsadm/sf/cronmailsave1
 33,35,37,39,41,43,45,47 * * * * /home/appsadm/sf/cronmailsave1
 49,51,53,55,57,59       * * * * /home/appsadm/sf/cronmailsave1
 # scripts run every 2 minutes (until you disable via 'crontab -r')
 # - see crontabs documented at www.uvsoftware.ca/admjobs.htm#Part_5
 #------------------------ end crontabtest2 --------------------------

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5K3. listings of crontabs & scripts used cron/log/mail demo

cronscript1 - executing JCL/script jgl100.ksh

 #!/bin/ksh
 # cronscript1 - test/demo running scripts via crontab
 #             - by Owen Townsend, April 2009
 # - demo running 'mvstest' JCL/scripts under 'cron' & capturing log via 'mail'
 # - script, crontab,& profile distributed in /home/uvadm/testlibs/sf/...
 # - setup user appsadm & copy test crontabs & scripts to /home/appsadm/sf/...
 # - also copy profiles from /home/uvadm/env/... to /home/appsadm/env/...
 # - must setup subdirs /home/appsadm/cronlog1, cronlog2
 # - all documented at www.uvsoftware.ca/admjobs.htm#Part_5
 #
 # crontab2 or crontabtest2 - schedules cronscript1
 #             - every 2 minutes for testing with minimal wait for results
 #*cronscript1 - *THIS script executing JCL/scripts to be logged
 #             - runs demo JCL/script /home/mvstest/testlibs/jcls/jgl100.ksh
 #             - jgl100.ksh writes a GDG file in $RUNDATA/gl/...
 #
 # Note - scripts scheduled by crontab do not have benefit of a login profile
 #      - must dot execute profile to set PATHs to find scripts & programs
 #      - also allows us to call using just the jobnames (vs full path names)
 #
 export APPSADM=/home/appsadm   # homedir for crontabs & scripts
 # . $APPSADM/env/stub_profile_appsadm  # uncmt for production cron job mail logs
 #====================================  # next line active for test/demo
 . $APPSADM/env/stub_profile_appsadm_cronlogdemo # for demo ADMjobs.doc#5I1-5K7
 #==============================================
 autoload logcron1              # function to log msgs to $APPSADM/cronlog1
 #================              # found via export FPATH=... set in profile
 export CRONDT=$(date +%y%m%d_%H%M%S) # date_time for logcron1 function
 export JOBID1=cronscript1      # jobid for date_time_jobid msg prefix
 #========================
 rm $APPSADM/mbox   # remove old mail for appsadm (owner crontab running this)
 #================  # jobs run under cron will send mail to user (appsadm)
 #Note - you can copy/rename this demo script for your production cron scripts
 #     - replace commands between BEGIN/END with your commands
 #     - replace all instances of demo name (cronscript1) with your scriptname
 #
 #---------------------------- BEGIN user jobs --------------------------------
 #
 logcron1 "$JOBID1 - test running scripts via crontab"
 testdatainit2 #<-- optional script to clear any old files in temp subdirs
 jgl100.ksh    # test/demo job documented at www.uvsoftware.ca/mvsjcl.htm#1E3
 #=========    # - runs demo JCL/script /home/mvstest/testlibs/jcls/jgl100.ksh
 #             # - jgl100.ksh writes a GDG file in $RUNDATA/gl/...
 #             # - you can replace these testjobs with your production jobs
 logcron1 "$JOBID1 - end running scripts via crontab"
 #----------------------------- END user jobs ---------------------------------
 #Note - jobs run under cron will send mail to user (appsadm)
 #     - save mail (as date_time stamped file) after cron session as follows:
 # echo "save * $APPSADM/cronlog2/$CRONDT_$JOBID1" | mail
 #=======================================================
 #     - BUT, you have to do this AFTER cron session ends (separate cron)
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5K4. listings of crontabs & scripts used cron/log/mail demo

cronmailsave1 - capture cron mail into date/time stamped file

 #!/bin/ksh
 # cronmailsave1 - script called by crontab2 or crontabtest2
 #                 to save mail from cronscript1
 #               - by Owen Townsend, April 2009
 #               - see doc at www.uvsoftware.ca/admjobs.htm#Part_5
 #
 # crontab2 or crontabtest2
 #             - schedules cronscript1 every EVEN minute
 #             - every 2 minutes for testing with minimal wait for results
 #
 # cronscript1 - script executing JCL/scripts to be logged
 #             - runs demo JCL/script /home/mvstest/testlibs/jcls/jgl100.ksh
 #             - jgl100.ksh writes a GDG file in $RUNDATA/gl/...
 #
 # crontab2 or crontabtest2
 #             - runs script to save the 'mail' from prior cronscript1
 #               in date_time stamped file every ODD minute
 #               in $APPSADM/cronlog2/yymmdd_HHMMSS_jobname
 #
 #*cronmailsave1 - script called by crontab2/crontabtest2 (every ODD minute)
 #                 to save mail from prior crontab2/cronscript1
 #
 APPSADM=/home/appsadm         #<-- location of mail save subdir (cronlog2)
 JOBID1=cronscript1            #<-- jobname for suffix on mail save file
 CRONDT=$(date +%y%m%d_%H%M%S) #<-- date_time stamp for mail save file
 #
 # use 'echo' to pipe 'save' & 'delete' commands to 'mail'
 #=======================================================
 echo "save * $APPSADM/cronlog2/${CRONDT}_$JOBID1" | mail
 echo "delete *" | mail
 #=======================================================
 #Note - jobs run under cron send mail to user (appsadm)
 #     - BUT, you have to do this AFTER cron session ends (via separate cron)
 #     - see more explanations at www.uvsoftware.ca/admjobs#Part_5
 #------------------------ end cronmailsave1 --------------------------
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5K5. listings of crontabs & scripts used cron/log/mail demo

jgl100.ksh - JCL/script executed by cronscript1

 #!/bin/ksh
 ##JGL100   JOB  (1234),'LIST GL MASTER CHART OF ACCOUNTS'
 export JOBID2=JGL100; scriptpath="$0"; args="$*"
 if [[ -z "$JOBID1" ]]; then JOBID1=JGL100; fi; export JGL100
 for arg in $args; do if [[ "$arg" == *=* ]]; then export $arg; fi; done
 integer JCC=0 SCC=0 LCC=0  # init step status return codes
 autoload jobset51 jobend51 jobabend51 logmsg1 logmsg2 stepctl51 exportfile
 autoload exportgen0 exportgenp exportgenq exportgenr exportgenall exportgenx
 autoload exportgen1 exportgen2 exportgen3
 jobset51  # call function for JCL/script initialization
 #jobset51 stores any restart step# in alias 'goto' (must be last on line)
 goto
 S0000=A
 ##STEPA    EXEC PGL100,HLQ=GL,YEAREND=2003               #<-- PROC1 exp
 HLQ="GL";YEAREND="2003";
 # * LIST G/L CHART OF ACCOUNTS FROM ACCOUNT.MASTER
 #1======================= begin step#S0010 CGL100 ========================
 S0010=A
 JSTEP=S0010; ((XSTEP+=1)); SCC=0; LCC=0; alias goto="";
 logmsg2 "******** Begin Step $JSTEP cgl100 (#$XSTEP) ********"
 stepctl51  # test oprtr jcpause/jcclear
 ##STEP010  EXEC PGM=CGL100,REGION=1024K,PARM=&YEAREND
 export PROGID=cgl100
 export PARM="$YEAREND"
 exportgen0 ACCTMAS gl/account.master_
 exportgen1 ACTLIST $JGDG/gl/account.acntlist_
 #exportgen1,2,3 GDG subdir/file, option g1 move to outdir at Normal EOJ
 exportfile SYSOUT $SYOT/${JSTEP}_SYSOUT
 ##PEND1 PGL100
 logmsg2 "Executing--> cobrun $ANIM $RLX/cgl100"
 #3----------------------------------------------------------------------
 cobrun $ANIM $RLX/cgl100
 #4----------------------------------------------------------------------
 LCC=$?; S0010C=$LCC; ((SCC+=LCC)); ((JCC+=LCC)); S0010R=1; alias goto="";
 if ((S0010C != 0))
    then logmsg2 "ERR: step#$JSTEP cgl100 abterm $SCC"
    alias goto="<<S9900=A"; fi
 goto
 #8======================================================================
 S9000=A
 jobend51 #move any GDG files from jobtmp/GDG/subdirs to RUNDATA/subdirs
 logmsg2 "JobEnd=Normal, StepsExecuted=$XSTEP, LastStep=$JSTEP"
 exit 0 #jclunix51 ver:20090313 b2c0d1g1e0f1i1j0k3l20m4n3o0p0r0s0t1u1w0x0y1z0
 #9======================================================================
 S9900=A
 logmsg2 "ERR: Terminated Abnormally,JCC=$JCC,StepsX/L=$XSTEP/$JSTEP" RV ACK
 jobabend51 #report GDGs NOT moved from jobtmp/GDG/subdirs to outdirs
 exit $JCC

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5K6. listings of crontabs & scripts used cron/log/mail demo

stub_profile_appsadm_cronlogdemo

 # stub_profile_appsadm_cronlogdemo - file distributed in /home/uvadm/env/...
 #                                  - to be copied to /home/appsadm/env/...
 # Special version of profile to demo capturing logs from jobs run by cron
 # - defines RUNLIBS & RUNDATA as /home/mvstest/testlibs & testdata
 #   ==============================================================
 # - see www.uvsoftware.ca/admjobs.htm#5I1 - 5K6
 #
 # This stub_profile_appsadm_cronlogdemo called directly by 'cronscript1'
 # - which is scheduled by 'crontab2' & 'crontabtest2'
 # - since 'cron' environment has NO profile to setup PATHs, etc
 #
 # We have dropped a lot of explanatory #cmts here in cronlogdemo version
 # - see explanatory #cmts in original /home/uvadm/env/stub_profile
 # - especially see '#2X Exception coding' below required for cron/log demo
 #
 # #1 - call 'common_defines'
 # . /home/uvadm/env/common_defines   #<-- call common_defines from 'uvadm'
 #=================================   #  - distributed in /home/uvadm/env/...
 . /home/appsadm/env/common_defines   #<-- copy to 'appsadm' for your conversions
 #=================================      - customize for your conversion
 #Note - this is stub_profile_appsadm_cronlogdemo, uvadm #cmtd, appsadm uncmntd
 #
 # #2X. Exception coding to demo capturing logs from JCL/scripts run by cron
 #      - see www.uvsoftware.ca/admjobs.htm#5I1 - 5K6
 #
 #=====================================
 export TESTLIBS=/home/mvstest/testlibs #<-- test cron mail log capture
 #=====================================    - these exports are VITAL to the demo
 export TESTDATA=/home/mvstest/testdata #  - demo in ADMjobs.doc#5I1 - 5K6
 #=====================================
 #
 # #2. define critical VARIABLES using $VARIABLES defined in common_defines
 #     - OR overridden above in stub_profile or stub_profile_appsadm
 export RUNLIBS=$TESTLIBS
 export RUNDATA=$TESTDATA
 #
 # #3. call common_profile (supplied in /home/uvadm/env/...)
 # . /home/uvadm/env/common_profile  #<-- call common_profile from 'uvadm'
 #=================================  #  - distributed in /home/uvadm/env/...
 . /home/appsadm/env/common_profile  #<-- copy to 'appsadm' for your conversions
 #=================================     - customize for your conversion
 #Note - end stub_profile_appsadm_cronlogdemo, uvadm is #cmtd, appsadm uncmntd

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

5K7. listings of crontabs & scripts used cron/log/mail demo

 # logcron1 - function to log msgs from jobs run by cron
 #          - by Owen Townsend, April 23/2009
 #
 # Prints to screen & append to file: $APPSADM/cronlog1/yymmdd_HHMMSS_$JOBID1
 # - prefix messages with date_time:$JOBID1
 # - use cron to remove $APPSADM/cronlog1/... older than 10 days ?
 # - calling script should define CRONDT & JOBID1 for output filename
 #
 # logcron1 "x---message---x"         <-- sample command in calling JOBXXX
 # 051011_124700_JOBXXX x---msg---x   <-- sample output (at 12:47 Oct 11/05)
 #
 function logcron1
 {
 if [[ $CRONDT == "" ]]; then CRONDT=$(date +%y%m%d_%H%M%S); fi
 NOWDT=$(date +%y%m%d_%H%M%S)
 msg="${NOWDT}_$JOBID1: $1"
 print "$msg"
 # append msg to the cronlog1/file
 print $msg >>$APPSADM/cronlog1/${CRONDT}_$JOBID1
 }

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_6. ADMjobs - Console Logging

Console Logging - Contents

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6A1. ADMjobs: Console Logging

Console Logging vs Job logging

'Console Logging' captures everything that happens on the screen, including operator commands & replies to prompts (highly recommended for production).

'Job Logging' captures the console output (only) for 1 job at a time, which is better for the programmers, because it can be inspected immediately.

'Console Logging' requires some setup (documented on following pages). Until you get console logging activated, anybody could use the joblog scripts to capture the console log for 1 job at a time.

3 versions of joblog script

example using joblog2

 #1. cdd                 <-- change to $TESTDATA superdir

 #2. joblog2 jar100.ksh  <-- use joblog2 to run script jar100.ksh
     ==================      & capture console log in joblog/jar100.log

 #3. uvlp12 joblog/jar100.log  <-- print the log
     ========================

These scripts capture the JCL/script screen displays into a file using the unix/linux 'tee' command. The log filename is created by dropping the '.ksh' extension from the jobname & then appending '.log'.

You can see 'joblog1' listed at ADMjobs.htm#6S1. You can see all scripts in /home/uvadm/sf/IBM/...

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6B1. ADMjobs: Console Logging

capturing logs via Unix/Linux 'script' command

Most UNIX systems have a 'script' command which can be used to capture all activity on any terminal.

This would be especially important to mainframe users as a replacement for the system console log, but 'script' can be employed by any user to capture their own console log, for example:

          script logfile
          ==============

All console I/O will be captured into the named file until you log off. To make effective use of this facility, I suggest the following:

 1 - Start the script automatically as the last command in your .profile
       (this way you can not forget)

 2 - Assign the script filename as the current date & time
       (using the $(date) command - see below)

3 - Setup a separate directory to hold your logfiles

 4 - Accumulate your logfiles for a month (or whatever period suits you).
     You might accumulate these in directory log1 for example.
   - at the end of the month copy log1 to log2, backup log1 to tape
     or diskette, and remove all files from log1 for the new month

5 - You should login only once on the userid you intend to use for logging, because your current logfile would be destroyed by the 2nd login. (see logfixA & logview scripts listed later in this section).

The last command in your .profile might then be:

          exec script log1/$(date +%y%m%d_%H%M%S)
          =======================================

          exec script log1/060219:075200     - sample expansion
          ==============================

Your site administrator might want to keep log files for all users in 1 directory, with sub-directories for each user, in which case the following might be appropriate:

          exec script $LOGDIR/log1/$LOGNAME/$(date +%y%m%d_%H%M%S)
          ========================================================

I suggest that LOGDIR=/home/appsadm (the site administrator's home dir). An export for LOGDIR is coded in the profile (listed in Part_1).

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6C1. ADMjobs: Console Logging

activating console logging

You activate console logging by uncommenting 4 lines marked near the end of the profile provided (in /home/uvadm/env/stub_profile).

See complete listing begining on page '1C1', but here are the last few lines:

 #--------------------------------------------------------------------------
 # Capture console log files (see details at www.uvsoftware.ca/admjobs.htm)
 # - APPSADM contains subdirs log1,log2,log3 (cur file,month,lastmonth)
 # - each of log1,log2,log3 must have subdirs matching $LOGNAME
 # ---> uncomment following '##' lines in profiles of users who require logging
  ## login1 || exit 2          # exit here if 2nd login
  ## logfixA $LOGNAME          # process log1 file to log2 (to allow read/print)
  ## echo "console logging requires aliases to be stored in .bashrc or .kshrc"
  ## echo "--> . aliases <-- 'dot' execute aliases NOW, if not in .bashrc/.kshrc"
  ## echo "--> logview   <-- execute logview script to see prior console logs"
  ## echo "script $APPSADM/log1/$LOGNAME/$(date +%y%m%d_%H%M%S)"
  ## exec script $APPSADM/log1/$LOGNAME/$(date +%y%m%d_%H%M%S)
 # - 'exec script' must be the last non-comment line in the profile
 # - 'script' disables 'aliases', put them in '.bashrc' to make them effective
 #--------------------------- end of stub_profile ---------------------------

You also need to setup a subdir for each user before the login for the 1st time after uncommenting the 4 lines at the end of their profile.

The 'appsadm' might setup directories as follows:

 #0. user appsadm login ---> /home/appsadm

 #1. mkdir log1 log2 log3    <-- 1 time only

 #2. mkdir log1/userxx log2/userxx log3/userxx
     =========================================
     - setup subdirs for each user to be console logged

See descriptions of log1,log2,log3 on the next page:

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6D1. ADMjobs: Console Logging

Directories to store console logs

The script files are created as $LOGDIR/log1/$LOGNAME/yymmdd_HHMMSS

$LOGDIR is defined in profiles as: export LOGDIR=/home/appsadm & contains 3 logfile directories as follows:

filtering log files for viewing & printing

'logfixA' in your profile to performs the filtering. This is activated simply by logging off & back on & then using the 'logview' script.

The advantage of logfixA in the .profile is that you can see your filtered log files without waiting for the nightly cron job or running logfix manually.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6E1. ADMjobs: Console Logging

filtering console logs to enable viewing & printing

The previous page explained how to capture console log files via a 'script' command in your .profile, for example:

       exec script $LOGDIR/log1/$LOGNAME/$(date +%y%m%d_%H%M%S)
       ========================================================

If you attempt to edit or print the log files directly, you will have problems due to screen control escape sequences from COBOL program displays, vi editor sessions, etc. Another problem is extraneous voluminous data from various commands (ls, vi, cat, more, programs).

'logfix1' is a uvcopy job that will solve these problems, by copying the log files dropping the escape sequences & inserting LineFeeds as required to ensure that no lines are longer than 80 columns.

'logfix1' has options to reduce voluminous output displays to 3 or 4 lines by scanning for known symbols '<@' at the beginning of each user prompt. PS1 is modified accordingly to: export PS1='<@${PWD}> '

'logfix1' has options to set max lines for certain commands. For 'vi' option 'v5' limits output to 5 lines, for any JCL/script (ID by .jcl suffix) option j1000 means unlimited, for all other commands 'm5' sets max 5 lines.

create script demo file

 #1a. mkdir tmp   (or rm tmp/*)
 #1b. script tmp/demolog1          --> Script Started

 #2a. ps
 #2b. ls -l /dev       (7500 lines)
 #2c. vi /etc/passwd   (200 lines)
 #2d. :q  (quit vi)
 #2e. jar200.jcl      <-- demo JCL from MVSJCL.doc (can omit if not yet setup)
 #2f. exit                         --> Script Done

filter logfile for viewing/printing

 #3. uvcopy logfix1,fili1=tmp/demolog1,filo1=tmp/demolog2,uop=f1j1000m5v5
     ====================================================================
     - see options (f1j1000m5v5) explained in logfix1 listing on page '6U1'.

Examine Input & Output files

 #4a. vi tmp/demolog1       - examine input file (about 7500 lines)
 #4b. vi tmp/demolog2       - examine output file (about 200 lines)
 #4c. uvlp12 tmp/demolog2   - output file can printed easily

See the results on the next page --->

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6E2. ADMjobs: Console Logging

sample input to logfix1 (BEFORE filtering)

 Script started on Sat 22 Nov 2003 01:03:06 PM PST
 <@:mvstest:/home/mvstest> ps
   PID TTY          TIME CMD
  2794 pts/1    00:00:00 ksh
  2795 pts/1    00:00:00 ps
 <@:mvstest:/home/mvstest> ls -l /dev
 total 228
 crw-------    1 root     root      10,  10 Jan 30  2003 adbmouse
 crw-r--r--    1 root     root      10, 175 Jan 30  2003 agpgart
 crw-------    1 root     root      10,   4 Jan 30  2003 amigamouse
 .................... 7500 lines removed ..........................
 crw-rw----    1 root     disk      27,  19 Jan 30  2003 zqft3
 <@:mvstest:/home/mvstest>  vi /etc/passwd
 ^[?1h[=^[1;25r^[m^[m^[m.^[H^[J^[25;1H[1;1Hroot:x:0:0:root:/root:/bin/bash
 bin:x:1:1:bin:/bin:/sbin/nologin
 daemon:x:2:2:daemon:/sbin:/sbin/nologin
 adm:x:3:4:adm:/var/adm:/sbin/nologin
 .................... 200 lines removed ..........................
 :q ^[?1l[>^[25;1H^[K^[25;1H<@:mvstest:/home/mvstest> jar200.jcl
 JAR200:031122:130353: JOB begun=JAR200
 JAR200:031122:130353: RUNDATA=/home/mvstest/testdata RUNLIBS=/home/mvstest
 JAR200:031122:130353: DW=wrk,DP=prt/031122
 JAR200:031122:130353: begin step# 1/1 iebgener
 JAR200:031122:130353: gen0: SYSUT1=ar/customer.master_0005
 JAR200:031122:130353: file: SYSUT2=wrk/JAR200_tempcm
 JAR200:031122:130353: uvcp fili1=ar/customer.master_0005,filo1=wrk/JAR200_tempcm
 JAR200:031122:130353: EOF input, 32 recs, 8192 bytes: ar/customer.master_0005
 JAR200:031122:130353: close out, 32 recs, 8192 bytes written: wrk/JAR200_tempcm
 JAR200:031122:130353: begin step# 2/2 idcams
 JAR200:031122:130353: file: TEMPCM=wrk/JAR200_tempcm
 JAR200:031122:130353: gen+1: CUSMAS=ar/customer.master_0006 gens=10
 JAR200:031122:130353: JOB completed=JAR200
 <@:mvstest:/home/mvstest> exit
 Script done on Sat 22 Nov 2003 01:03:58 PM PST

Notes

1. Please compare this with the 'AFTER' results on the next page --->
   
   
2. Here I have removed about 7600 lines (manually), but logfix1 has an option to do that automatically by limiting outputs from some commands (such as vi) to a specified limited number of lines.
   
   
3. Here I have replaced the escape control codes x'1B' by circumflexes '^'s manually (escape codes can interfere with viewing & printing).
   
   
4. Note the many control codes (^[?1h^=^[1;25r;1H...etc...) here which are removed automatically by logfix1 (see next page).
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6E3. ADMjobs: Console Logging

sample output from logfix1 (AFTER filtering)

 <@:mvstest:/home/mvstest> ps
   PID TTY          TIME CMD
  2794 pts/1    00:00:00 ksh
  2795 pts/1    00:00:00 ps

 <@:mvstest:/home/mvstest> ls -l /dev
 total 228
 crw-------    1 root     root      10,  10 Jan 30  2003 adbmouse
 crw-r--r--    1 root     root      10, 175 Jan 30  2003 agpgart
 crw-------    1 root     root      10,   4 Jan 30  2003 amigamouse

 <@:mvstest:/home/mvstest>  vi /etc/passwd
 [1;1Hroot:x:0:0:root:/root:/bin/bash
 bin:x:1:1:bin:/bin:/sbin/nologin
 daemon:x:2:2:daemon:/sbin:/sbin/nologin
 adm:x:3:4:adm:/var/adm:/sbin/nologin

 <@:mvstest:/home/mvstest> jar200.jcl
 JAR200:031122:130353: JOB begun=JAR200
 JAR200:031122:130353: RUNDATA=/home/mvstest/testdata RUNLIBS=/home/mvstest
 JAR200:031122:130353: DW=wrk,DP=prt/031122
 JAR200:031122:130353: begin step# 1/1 iebgener
 JAR200:031122:130353: gen0: SYSUT1=ar/customer.master_0005
 JAR200:031122:130353: file: SYSUT2=wrk/JAR200_tempcm
 JAR200:031122:130353: uvcp fili1=ar/customer.master_0005,filo1=wrk/JAR200_tempcm
 JAR200:031122:130353: EOF input, 32 recs, 8192 bytes: ar/customer.master_0005
 JAR200:031122:130353: close out, 32 recs, 8192 bytes written: wrk/JAR200_tempcm
 JAR200:031122:130353: begin step# 2/2 idcams
 JAR200:031122:130353: file: TEMPCM=wrk/JAR200_tempcm
 JAR200:031122:130353: gen+1: CUSMAS=ar/customer.master_0006 gens=10
 JAR200:031122:130353: JOB completed=JAR200

<@:mvstest:/home/mvstest> exit Script done on Sat 22 Nov 2003 01:03:58 PM PST

Notes re logfix1 output above

The intention is to highlight the original console commands & reduce extraneous voluminous output displays. Note that unprintable characters have been eliminated. for example the escape control code x'1B' preceding screen control sequences.

Note that the 7500 lines (from ls -l /dev) have been reduced to 5 lines, the 'vi' display has been reduced to 5 lines, BUT the JCL/script output has not been reduced since this displays are vital. These limits are set by options m5, v5,& j1000 in the logfix1 job (see listing on page '6U1').

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6S0. ADMjobs: Console Logging

scripts & uvcopy jobs to process console logs

Some of the logfix scripts & uvcopy jobs are listed on the following pages:

You can use various UNIX utilities to investigate the console log files, if you explore processed files in log2 vs the unprocessed files in log1.

           uvlp12 /home/appsadm/log2/uvadm/031115:092400
           =============================================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6S1. ADMjobs: Console Logging

joblog1 - capture log for 1 job at a time

 # joblog1 - run a JCL/script & capture a log file (via tee)
 #         - names the log file by dropping the .ksh & appending .log
 # - writes the logfile into subdir 'joblog' (creates it if not present)
 # - prompts for command to view/print (vi,cat,more,uvlp12,etc)
 #   &/or optionally (save) with a date/time stamp
 #
 #                ** 3 versions of this script **
 #
 #*joblog1 - writes the log file to $RUNLIBS/joblog/jobname.log
 # joblog2 - writes the log file to $RUNDATA/joblog/jobname.log
 # joblog3 - writes the log file to $HOME/joblog/jobname.log
 #
 # This script intended only for test/debug when console logging not activated
 # - Console logging is better because it captures everything that happens,
 #   including operator commands & replies to prompts.
 # - to activate console logging, see: www.uvsoftware.ca/admjobs.htm#Part_6
 #
 sf="$1"     # capture the script filename with extension (jclname.ksh)
 if [[ ! -f $RUNLIBS/jcls/$sf ]]
    then
    echo "usage: joblog1 jclname.ksh [args] <-- file not found $RUNLIBS/jcls/$sf"
    echo "       ========================="
    exit; fi
 #
 # setup joblog directory pathname & make joblog subdir if not existing
 ld=$RUNLIBS/joblog    # setup joblog directory pathname
 if [[ ! -d $ld ]]; then mkdir $ld; fi
 #
 # create logfilename by dropping .ksh & appending .log
 f=${sf%\.*}           # drop extension .ksh from JCL/script filename
 lf=$f.log             # add extension .log to create logfilename
 #
 $sf $2 2>&1 | tee $ld/$lf
 #========================
 echo "enter command to view, print, and/or save logfile"
 echo "logfile: $ld/$lf"
 echo "--> vi,cat,more,uvlp12,etc, and/or 'save' to date/time stamp"
 read reply
 if [[ "$reply" == *save* ]]; then
    cp $ld/$lf $ld/${lf}_$(date +%y%m%d_%H%M%S); fi
 cmd=${reply%save*}
 if [[ -n "$cmd" ]]; then $cmd $ld/$lf; fi
 exit 0
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6S2. ADMjobs: Console Logging

logfixA - filter console logs to facilitate viewing & printing

 #!/bin/ksh
 # logfixA - Korn shell script from UVSI stored in: /home/uvadm/sf/adm/
 # logfixA - convert all script log files from log1 to log2 & remove log1 file
 #         - removes control chars & limits 'vi' & other lengthy outputs
 #
 #usage:    logfixA logsubdir
 #default:  logfixA $LOGNAME      <-- usually subdir=$LOGNAME
 #
 # - logfixA is run from the .profile at login time, to process the prior logfile
 #   which was closed when you logged out, so you can display it via logview
 # - the last 3 lines in the .profile should be as follows:
 #
 # 1. login1 || exit 2          # prevents 2nd login (would destroy logfiles)
 # 2. logfixA $LOGNAME          # process log1 file to log2 (to allow read/print)
 # 3. exec script $LOGDIR/log1/$LOGNAME/$(date +%y%m%d:%H%M%S)
 #
 # - logfixA allows only 1 login per userid (recommended for console logging)
 # - but see 'logfixB' if you really want to allow multiple logins per userid
 # - use the 'logview' script to view your logfiles (requires no arguments)
 # - logview lists your logfiles & prompts for sequence# of logfile to vi
 # - 1st logoff & back on if you want to see your latest activity
 #
 job=logfixA     # setup jobname for echo msgs
 # ensure logfile parent directory defined & change to it
 if [ ! -d "$LOGDIR" ]; then
    echo "$job - LOGDIR undefined (usually = \$APPSADM)"; exit 9; fi
 cd $LOGDIR      # change to parent directory of logfile subdirs
 #=========
 subdir="$1"     # capture arg1 (default to $LOGNAME if not specified)
 echo "$job - convert $subdir log files for viewing & printing "
 ## if [[ -z "$subdir" ]]; then subdir=$LOGNAME; fi
 if [[ ! -d "log1/$subdir" ]]; then
    echo "usage: $job $subdir <-- arg1 must be subdir in $LOGDIR/log1/ & log2/"
    echo "       ==========="
    exit 99; fi
 #
 echo "files in log1/$subdir listed below:"
 ls -l log1/$subdir
 for i in log1/$subdir/*
 do
   if [[ -s $i ]]; then
      f=${i##*/}
      uvcopy logfix1,fili1=$i,filo1=log2/$subdir/${f},uop=q0i7e1j1000m5v5
      #==================================================================
      let x=x+1
      echo "file# $x  $i converted to log2/$subdir/$f"
      # remove input file from log1, unless LOGFIXDEBUG=Y
      if [[ "$LOGFIXDEBUG" != "Y" ]]; then rm -f $i; fi
  fi
 done
 echo "$job - $x files converted from log1/$subdir to log2/$subdir for vi/uvlp12"
 echo "$job - use 'logview' script to list & view logfiles (no prmtrs reqd)"
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6S3. ADMjobs: Console Logging

logview - list logfiles & allow pick by number

 #!/bin/ksh
 # logview - Korn shell script from UVSI stored in: /home/uvadm/sf/adm/
 # logview - list user's log2 files & view specified files
 #
 #usage: logview [ user ]    <-- user defaults to $LOGNAME
 #       ================
 #
 # This script does an 'ls -l' of log2/user files
 # & prompts for seq# to 'vi' (counting backwards on ls -l list)
 # Repeats the ls -l & re-prompts until user enters 'q'
 # 2nd arg optional to print to specified destination (or default)
 # - must be no space between '-d' & printer destination
 # -> 1         - vi last log file, :q to quit vi & redisplay file list
 # -> 3 -d      - print 3rd last log file to default printer
 # -> 3 -dlp09  - print 3rd last log file on lp09
 # -> q         - quit logview
 #
 if [[ "$1" = "" ]]; then user=$LOGNAME; else user="$1"; fi
 logdir=$LOGDIR/log2/$user              # setup logdir for use below
 tmpd=$HOME/tmp
 if [ ! -d $tmpd ]; then mkdir $tmpd; fi
 typeset lognames[200]                  # array for up to 200 files
 integer fil=1                          # force at least 1 list & prompt
 while (($fil > 0))                     # if user reply > 0
   do ls -l $logdir                     # show log files to user
      ls -1r $logdir > $tmpd/logfiles   # logfile names in reverse order
      # read logfile names into an array
      integer n=1
      exec 3< $tmpd/logfiles            # open log
      while read -u3 logname            # read current name
      do lognames[$n]=$logname          # add current filename to array
         n=n+1
      done
      exec 3<&1                         # close file descriptor 3
      echo "enter 1,2,3,etc to vi logfile (counting backwards) q to quit"
      echo " - follow file# with -ddest to print for example--> 1 -dlpt1 "
      read fil prt   # read users response fil=file#, prt dest (optional)
      if (( $fil < 1 || $fil > 9 )); then break; fi
      if [[ "$prt" = "" ]]; then
         vi $logdir/${lognames[$fil]}   # vi specified file#
      else
         if [[ "$prt" == -d* ]]; then dest="$prt"; else dest=$UVLPDEST; fi
         if [[ "$dest" == *lpt* ]]; then I=i1; else I=i0; fi
         uvlist $logdir/${lognames[$fil]} p60$I t1c13 | lp -onobanner $dest
      fi
   done
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6S4. ADMjobs: Console Logging

logfixM - rotate console logs - cron monthly

 #!/bin/ksh
 # logfixN - convert all script log files from $APPSADM/log1 to $APPSADM/log2
 #
 # This 'logfixN' run by cron to fix logs of users who did not logoff
 # - part of 'nightly1' which is scheduled by 'crontab_1'
 # - these scripts & crontabs distributed by UVSI in /home/uvadm/sf/adm/...
 # - should be customized & stored for execution at /home/appsadm/sf/...
 #
 # Note - 'logfixA' is run by user profiles to fix logs during the day
 # - users logoff & backon to process logs, for viewing by logview script
 # - 'logfixN' run by crontab_appsadm1 fixes logs of users who did not logoff
 # - crontab_root has already KILLed users who did not logoff
 #
 #usage: logfixN   #<-- no arguments required
 #       =======
 #
 # - removes control chars & inserts LFs to enable log file view & print
 # - copies all files for all users from log1/users/... to log2/users/...
 # - then removes all from log1/users/...
 # - this script is run by 'nightly1' which is scheduled by crontab_appsadm1
 #
 # To initially setup logging for a user:
 # - setup a subdir matching his login/userid in log1/...,log2/...,log3/...
 # - remove '#' from last 4 lines of his .bash_profile (stub_profile_prod)
 # - see stub_profiles & common_profiles supplied in /home/uvadm/env/...
 #
 export APPSADM=/home/appsadm  # define APPSADM
 cd $APPSADM                   # change to parent directory of logfile subdirs
 #
 JOBID=logfixN       # setup scriptname for echos
 echo "$JOBID - convert script log files for editing & printing"
 echo "- convert all files from $APPSADM/log1/user/... to $APPSADM/log2/user/..."
 x=0
 for d in log1/*
  do s=${d##*/}             # capture subdir
     let x=x+1; y=0         # count subdirs & reset file ctr for subdir
     for f in $d/*          # for each file in current subdir
      do if [[ -s $f ]]; then
            g=${f##*/}                        # capture file element name
            h=log2/$s/$g                      # setup output filename
            uvcopy logfix1,fili1=$f,filo1=$h  # convert current logfile
            rm -f  $f                         # remove file from log1
            let  y=y+1
            echo "$x/$y - $f converted to log2/$s & deleted from $d"
         fi
      done
  done
 echo "$JOBID - $x subdirs converted from $APPSADM/log1 to $APPSADM/log2"
 exit

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6S5. ADMjobs: Console Logging

logfixM - rotate console logs - cron monthly

 #!/bin/ksh
 # logfixM - Korn shell script from UVSI stored in: /home/uvadm/sf/adm/
 # logfixM  - console log file processing - called by monthly1 script
 #            which is scheduled by CRON on the 1st of each month
 #          - moves all log files (created by UNIX 'script' command)
 #            from log2 to log3 & then removes all files from log2
 #          - UVSI suggests you setup userid 'appsadm' for aplctns admin
 #
 export APPSADM=/home/appsadm       # define apps admin homedir
 #===========================
 # $APPSADM should be defined in all user homedirs as the LOGDIR
 # - with subdirs as follows:
 #
 # log1 - daily log written here by UNIX script commands
 #      - each user has a subdir containing his date/time stamped logfiles
 #        for example: log1/$LOGNAME/yymmdd:HHMM
 #                     log1/owen/950615:0710
 #                     log1/gordon/950615:1030
 # log2 - current month log files
 #      - converted by logfixA to allow editing & printing
 #      - copied over to log3 on the 1st of each month
 #      - then cleaned out for re-accumulation
 # log3 - last month log files
 #      - provides log history up to 2 months ago (at end of mth)
 #
 cd $APPSADM    # change to $APPSADM homedir
 JOBID=logfixM    # setup JOBID for multi use below (easier to clone/rename)
 #
 echo "$JOBID - remove all subdirs/files from $APPSADM/log3"
 rm -rf log3/*         # remove all subdirs & files from log3
 echo "$JOBID - copy all subdir/files in $APPSADM/log2 to $APPSADM/log3"
 cp -rp log2/* log3    # copy all log2 subdirs & files to log3
 echo "$JOBID - remove all files from all user subdirs in $APPSADM/log2"
 rm -f  log2/*/*       # remove all files from all log2 users subdirs
 echo "$JOBID - remove all files from all user subdirs in $APPSADM/log1"
 rm -f  log1/*/*       # remove all files from all log1 users subdirs
 exit 0
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

6U1. ADMjobs: Console Logging

logfix1 - uvcopy job to filter log files

 # logfix1 - uvcopy Parameter File from UVSI stored in: /home/uvadm/pf/util/
 # logfix1 - fix the console log created by the 'script' command
 #         - called by script 'logfixA' invoked at end of user profile's
 #
 #example: uvcopy logfix1,fili1=tmp/demolog1,filo1=tmp/demolog2
 #         ====================================================
 #
 # - see console logging documented in ADMjobs.doc
 #   'script' is used to capture console logs for each user
 # - this job makes the log files much easier to use as follows:
 # - removes screen control 'escapes' that interfere with viewing/printing
 #   (causing overprinting, long lines,& lost data)
 # - unprintable characters are translated to x'00' & squeezed out
 # - drop blank lines
 #
 # - option 'f' to remove screen control codes (in addition to escapes)
 #   (drop data from 1st '[' on a line to the last '[' on a line)
 # - depend on unique PS1 prompt pattern in the profile
 #   must have: export PS1='<@${PWD}> '
 #
 # - option j# to limit '*.jcl' displays to spcfd max lines
 # - option v# to limit 'vi' displays to spcfd max lines
 # - option m# to limit all other command displays to spcfd max lines
 # Note - must have option f1 or f3 to activate options j,v,m
 # - option 'f0' processes all lines (ensures you don't lose anything)
 #
 opr='$jobname - fix script/console log, remove escapes & vi displays'
 opr='uop=q1f0j1000m50v5   - option defaults'
 opr='    q0               - do not prompt to allow option changes'
 opr='    q1               - prompt to allow option changes'
 opr='      f0             - filter option off (show 1st 256 all lines)'
 opr='      f1             - filter option on - limit output for JCL,vi,other'
 opr='      f2             - drop vi escape codes'
 opr='                       from 1st "[" on a line to last "[" on a line'
 opr='        j1000        - for "*.jcl" cmds - limit output to 1000 lines'
 opr='             v5      - for " vi " cmds  - limit output to    5 lines'
 opr='               m50   - for other cmds   - limit output to    5 lines'
 opr='options j,v,m require option f1+ & depend upon unique pattern in PS1'
 opr='export PS1="<@${PWD}> " <-- PS1 in the profile must contain "<@" ID'
 opr='IE - option f0 disables optns j,v,m & all lines are processed'
 uop=q1f0j1000m50v5      # option defaults
 was=g20000              # allow 20000 input area in case of long lines w/o LFs
 fili1=?input,rcs=16000,typ=LSTe1  #<-- see note in getr re optn e1/n1
 filo1=?tmp/output,rcs=512,typ=LSTt
 @run
        opn     all
 # setup translate table for later use to remove unprintable characters
        mvc     p0(256),$trt          neutral translate tbl to w/s
        clr     p0(32),x'00'          clear ctls low
        clr     p126(130),x'00'       clear ctls high including tilde & del
        mvc     p10(1),x'0A'          restore LineFeed
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 # begin loop to get/process/put records until EOF
 # getr subrtn gets lines up to 8000 bytes, but discards anything > 256
 man20  bal     getr                  get next line
        skp>    man80
 #
 # test option f1 to limit outputs for vi, JCL, other
 man22  cmn     $uopbf,1             limit outputs for vi,JCL,other ?
        skp<    man66
 #
 # test for PS1 prompt & options to limit outputs depending on command
 man24  scne1z1 aa0(256),'<@'        PS1 prompt ?
        skp!    man60
 man25  mvn     $ca1,0               clear ctr lines for current PS1
        mvn     $ca2,$uopbm          presume max lines for other cmds
        scne1   aa0(256),'> '        scan to end of PS1 prompt
        skp!    man60
        scn     aa0(10),' vi '       ' vi ' within 10 bytes of prompt ?
        skp=    man30
        scn     aa0(20),'.ksh'       JCL/ksh/script (within 20 bytes) ?
        skp=    man34
        skp     man60
 #
 # vi - set max lines from option v
 man30  mvn     $ca2,$uopbv
        skp     man60
 #
 # JCL/script - set max lines from option j
 man34  mvn     $ca2,$uopbj
        skp     man60
 #
 # common point to output & return to get next line
 # - output inhibited if line ctr for current cmd > max set at PS1 prompt
 man60  add     $ca1,1                count lines since last PS1 prompt
        cmn     $ca1,$ca2             lines > max ?
        skp>    man20
 man66  put     filo1,a0(256)         write current line to output log
        skp     man20                 return to get next
 #
 # end of file - close files & end job
 man80  cls     all
        eoj
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 #---------------------------- getr --------------------------------------
 # getr - subrtn to get next line
 #      - allow 16000 bytes & discard anything > 256
 #Dec2003 - option LSTe1 changed to n1 & n1 added to get in getr subrtn
 #        - option e1/n1 inhibits errmsg if get data > op2 area
 #        - option e1 left on LSTe1 for Henrico new logfix w/o new uvcopy
 getr   getn1   fili1,g0(16000)      optn n1 no errmsgstop if data > op2
        skp>    getr9
 #
 # translate any remaining control chars to nulls & squeeze left
        mvc     h0(80),g0            save 1st 80 for escape test below
        trt     g0(8000),p0          translate ctl chars to nulls
        sqz     g0(8000),x'00'       squeeze out nulls
        sqzc1   g60(8000),' '        squeeze multi blanks to 1
 # sqzc1 above starts at col 60 to retain spacing for most lines, but
 # squeeze out multiple blanks for long lines (screen displays?)
 #
 # drop any blank lines
        cmc     g0(256),' '          all blank line ?
        skp=    getr
 #
 # ensure PS1 prompt (ID by <@) starts on a new line (inserts blank if alrdy)
 # if PS1 was not near begin line, insert a 2nd LF to create blank line
        rep     g0(256),'<@',x'0A3C40'   insert LineFeed prior to PS1 prompt
        scn     g0(10),'<@'              PS1 near begin line ?
        skp=    getr4
        rep     g0(256),'<@',x'0A3C40'   insert 2nd LF to create blank line
 #
 # optionally drop escape codes (escape itself x'1B' drop by above trt/sqz)
 # not perfect - drops from 1st '[' on line to last '[' on a line
 getr4  mvc     a0(256),g0           presume option off & move all
        tsb     o6(1),x'02'          drop escape sequences ?
        skp!    getr8
 getr5  cmn     $uopbf,2             drop escape codes [....[ ?
        skp<    getr8
        scn     h0(80),x'1B'         any escapes in current line ?
        skp!    getr8
 getr6  clr     a0(512),' '          clear delivery area
        mvu     a0(256),g0,'['       move until '[' found (if any)
        skp!    getr8
        mvn     $ra,$rx              save ptr to 1st [ in perm rgstr 'a'
        scnr    g0(256),'['          scan from right for last '[' on line
        mvn     $rg,$rx              save ptr to last [ in perm rgstr 'g'
        mvc     aa0(256),gg0         move rmndr of line (dropping [...[)
 #
 getr8  ret=
 getr9  ret>
 #-------------------------- end of logfix1 ------------------------------

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_7. ADMjobs.doc - Applications Administration - useful scripts

useful admin scripts - Contents

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

ADMjobs.doc - Applications Administration - useful scripts

useful admin scripts - Contents (continued)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7A1. ADMjobs.doc - Applications Administration - useful scripts

scripts for unix/linux administrators

There are over 500 Korn shell scripts included in the Vancouver Utilities. After installation these will be in /home/uvadm/sf, which is sub-directoried as shown in the PATH below (extracted from the common_profile).

export UV=/home/uvadm

 export PATH=$PATH:$UV/sf/adm:$UV/sf/demo:$UV/sf/util:$UV/sf/IBM

For Part 7 of ADMjobs, we have selected a few of the scripts that are most useful to unix/linux/uvadm administrators.

These scripts can save you a lot of time. They can do in seconds what could take you hours to do manually.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7B1. ADMjobs: scripts for unix/linux administrators

chmod1 - change permissions on all files & subdirs

This script will change permissions on entire directory trees, using 'find' to process all levels of sub-directories. This script can save hours of manual investigation & correction.

For projects such as mainframe conversions, there is usually a team of programmers who must be able to read & write on a common set of directories & files (JCL/COBOL libraries & datafiles).

We recommend 775 for directories & 664 for files. This extends security to the 'group' level & all team members must be in the same group. Scripts would be 775 since they are files with the execute bit on.

This script is the solution to a very significant problem, that I frequently encounter when I arrive onsite to assist customer conversions.

If the site administrator did not initially setup the profiles with umask 002 (permissions 775/664 for directories/files), then the other programmers will be very frustrated when they attempt to work on the shared directories of JCL, COBOL,& DATA files.

If you are interested, you can see my recommended profiles in 'Part_1'. The profiles consist of 4 files (common_defines, stub_profile, common_profile, & bashrc). 'umask' is specified in both the stub_profile & bashrc. Note that the stub_profile is copied to the homedirs & renamed '.bash_profile' & bashrc is copied & renamed as '.bashrc'.

Operating Instructions

You must login or su to 'root' to run this script since it changes permissions. In the instructions below, I have included an 'export' to add the appsadm & uvadm script subdirs to root's PATH (or you could add this permanently in root's profile for future use).

 #1. su root                     <-- switch to root

 #2. export PATH=$PATH:/home/appsadm/sf:/home/uvadm/sf/adm
     =====================================================
     - add to PATH, so root can find chmod1 (in either appsadm or uvadm)

 #3. chmod1 directory dir-perms file-perms  <-- command format
     =====================================

 #3a. chmod1 directory 775 664              <-- recommended permissions
      ========================

 #4. exit         <-- exit from root asap
     ====

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7B2. ADMjobs: scripts for unix/linux administrators

chmod1 test/demo

We have provided a test directory '/home/uvadm/tf/test_chmod' that you can use to test/demo the chmod1 script. If you don't have the root password, you could test in your own home dir as follows:

 #1. login as yourself --> /home/userxx

 #2. mkdir tmp1               <-- make a subdir to receive demo files
     ==========

 #3. cp -pr /home/uvadm/tf/test_chmod tmp1 <-- copy test dirtree to tmp1/...
     =====================================
     - option 'p' Preserves current permissions
     - option 'r' (Recursive) copies all levels of sub-directories

 #4. ls -lr tmp1            <-- display test dir tree BEFORE chmod1
     ===========

 tmp1:
 total 16
 drwxr-xr-x    2 uvtest   users        4096 Dec 22 17:32 data
 -rw-r--r--    1 uvtest   users           7 Sep 22 23:54 file1
 drwxr-x---    2 uvtest   users        4096 Dec 22 17:29 programs
 drwxr-x---    2 uvtest   users        4096 Dec 22 17:29 scripts

 tmp1/data:
 total 8
 -rw-r-----    1 uvtest   users          11 Sep 22 23:54 datafile1
 -rw-r----x    1 uvtest   users          11 Dec 22 17:32 datafile2

 #5. chmod1 tmp1 775 664   <-- execute chmod1
     ===================

 #6. ls -lr tmp1           <-- display test dir tree AFTER chmod1
     ===========

 tmp1:
 total 16
 drwxrwxr-x    2 uvtest   users        4096 Dec 22 17:32 data
 -rw-rw-r--    1 uvtest   users           7 Sep 22 23:54 file1
 drwxrwxr-x    2 uvtest   users        4096 Dec 22 17:29 programs
 drwxrwxr-x    2 uvtest   users        4096 Dec 22 17:29 scripts

 tmp1/data:
 total 8
 -rw-rw-r--    1 uvtest   users          11 Sep 22 23:54 datafile1
 -rw-rw-r--    1 uvtest   users          11 Dec 22 17:32 datafile2

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7B3. ADMjobs: scripts for unix/linux administrators

chmod1 - change perms of subdirs & files - using find

 #!/bin/ksh
 # chmod1 - change permissions on directories & files
 #        - using 'find' to process all levels of directories & files
 #          from a specified starting directory
 #        - by Owen Townsend, April 26/2006
 #
 # chmod1 directory dir-perms file-perms   <-- command format
 # ======================================
 # chmod1 directory 775 664                <-- recommended permissions
 # =========================
 #
 # - also see alternatives chmod2 & chmod3 (to this *chmod1)
 #*chmod1 - change perms on dirs & files, using 'find'
 # chmod2 - change perms on dirs & files, using 'recursion'
 # chmod3 - change perms on dirs & files, using 'recursion'
 #        - set x perm on bin & script files via subdir names
 #
 # After running chmod1 or chmod2, you must manually fix permissions on
 # executable programs & scripts via -->chmod 775 bin/*; chmod 775 scripts/*
 #
 # capture arguments & force perms integers
 dir="$1";
 typeset i dperm="$2"; typeset i fperm="$3";
 #
 # ensure arg1 is directory & length of perms are 3 digits
 dpl=${#dperm}; fpl=${#fperm};
 #
 if [[ -d "$dir" ]] && ((dpl==3 && fpl==3)); then :
    else echo "usage: chmod1 directory dir-perms file-perms"
         echo "       ====================================="
         echo "example: chmod1 dirxx 775 664"
         echo "         ===================="
         echo " - arg1 must be dir, args 2 & 3 must be 3 digits"
         exit 90; fi
 #
 echo -n "chmod1: set perms $dperm/$fperm from: $dir - enter to continue"
 read reply
 #
 find $dir -type d -exec chmod $dperm {} \;
 #=========================================
 #
 find $dir -type f -exec chmod $fperm {} \;
 #=========================================
 #
 echo "chmod1: perms set $dperm/$fperm for all subdirs & files within: $dir"
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7B4. ADMjobs: scripts for unix/linux administrators

chmod3 test/demo

One problem with 'chmod1' is that files in any executable program or script directories will get permissions 664 & they must be 775 to execute.

'chmod3' is an alternate script provided for this problem, but it's not perfect. It looks for any subdirs whose names contain 'bin', 'program', or 'script', or 'sf' (my preferred short name for Script File subdirs).

 #1. login as yourself --> /home/userxx

 #2. mkdir tmp3               <-- make a subdir to receive demo files
     ==========

 #3. cp -pr /home/uvadm/tf/test_chmod tmp3 <-- copy test dirtree to tmp3/...
     =====================================

 #4. ls -lr tmp3            <-- display test dir tree BEFORE chmod1
     ===========

 tmp3:
 total 16
 drwxr-xr-x    2 uvtest   users        4096 Dec 22 17:32 data
 -rw-r--r--    1 uvtest   users           7 Sep 22 23:54 file1
 drwxr-x---    2 uvtest   users        4096 Dec 22 17:29 programs
 drwxr-x---    2 uvtest   users        4096 Dec 22 17:29 scripts

 tmp3/data:
 total 8
 -rw-r-----    1 uvtest   users          11 Sep 22 23:54 datafile1
 -rw-r----x    1 uvtest   users          11 Dec 22 17:32 datafile2

 tmp3/programs:
 total 8
 -rwxr-x---    1 uvtest   users         127 Dec 22 16:53 program1
 -rwxr-x--x    1 uvtest   users         127 Dec 22 17:29 program2

 tmp3/scripts:
 total 8
 -rwxr-x---    1 uvtest   users         125 Dec 22 16:53 script1
 -rwxr-x--x    1 uvtest   users         125 Dec 22 17:29 script2

 #5. chmod1 tmp3 775 664 775   <-- execute chmod1
     =======================

 #6. ls -lr tmp3               <-- display test dir tree AFTER chmod1
     ===========

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7B5. ADMjobs: scripts for unix/linux administrators

test dir tree AFTER chmod3 fix perms on programs & scripts

 tmp3:
 total 16
 drwxrwxr-x    2 uvtest   users        4096 Dec 22 17:32 data
 -rw-rw-r--    1 uvtest   users           7 Sep 22 23:54 file1
 drwxrwxr-x    2 uvtest   users        4096 Dec 22 17:29 programs
 drwxrwxr-x    2 uvtest   users        4096 Dec 22 17:29 scripts

 tmp3/data:
 total 8
 -rw-rw-r--    1 uvtest   users          11 Sep 22 23:54 datafile1
 -rw-rw-r--    1 uvtest   users          11 Dec 22 17:32 datafile2

 tmp3/programs:
 total 8
 -rwxrwxr-x    1 uvtest   users         127 Dec 22 16:53 program1
 -rwxrwxr-x    1 uvtest   users         127 Dec 22 17:29 program2

 tmp3/scripts:
 total 8
 -rwxrwxr-x    1 uvtest   users         125 Dec 22 16:53 script1
 -rwxrwxr-x    1 uvtest   users         125 Dec 22 17:29 script2

1. All directories have been changed to 775
   
   
2. Data directory files have been changed to 664
   
   
3. Program & script files have been changed to 775, because chmod3 recognized subdir names containing 'bin', 'program', 'script',& 'sf'.
   
   
4. You would have to modify the script if your program & script subdirs were named differently. See script listed on the next page:
   
   
5. Note that 'chmod3' uses 'Recursion' - it calls itself when it finds a subdir within the current directory.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7B6. ADMjobs: scripts for unix/linux administrators

chmod3 - change perms on subdirs/files & programs/scripts

 #!/bin/ksh
 # chmod3 - change permissions on directories, files,& executable files
 #         - using 'recursion' to process all levels of subdirs
 #         - by Owen Townsend, June 13/2006
 #
 # - also see alternatives chmod1 & chmod2 (to this *chmod3)
 # chmod1 - change perms on dirs & files, using 'find'
 # chmod2 - change perms on dirs & files, using 'recursion'
 #*chmod3 - change perms on dirs & files, using 'recursion'
 #        - set x perm on bin & script files via subdir names
 #
 # After running this script, you must manually fix permissions on executable
 # programs & scripts via -->chmod 775 bin/*; chmod 775 scripts/*
 # - if you can't modify the script to recognize all your bin & script dirs
 #
 # capture arguments & force perms integers
 dir="$1";
 typeset i dperm="$2"; typeset i fperm="$3"; typeset i xperm="$4";
 # ensure arg1 is directory & length of perms are 3 digits
 dpl=${#dperm}; fpl=${#fperm}; xpl=${#xperm};
 if [[ -d "$dir" ]] && ((dpl==3 && fpl==3 && xpl==3)); then :
    else echo "usage: chmod3 directory dir-perms file-perms xfile-perms"
         echo "       ================================================="
         echo "example: chmod3 dirxx 775 664 775"
         echo "         ========================"
         echo " - arg1 must be dir, args 2,3,4 must be 3 digits"
         exit 90; fi
 echo -n "chmod3: Begin directory $dir - enter to continue "
 read reply
 #note - remove the pause (read reply) above if desired
 integer nd=0 nf=0 nxf=0
 for df in $dir/*
   { if [[ -d $df ]] then chmod $dperm $df; ((nd+=1))
        elif [[ $df == *script* || $df == *sf* || $df == *bin*
                                               || $df == *program* ]]
             then chmod $xperm $df; ((nxf+=1))
        else chmod $fperm $df; ((nf+=1))
     fi
     # if current entry in current dir is a subdir
     # - call this script again to process its files & subdirs
     # - using 'recursion' to process all existing levels of subdirs
     # - but do not call if directory is empty
     if [[ -d $df ]]; then
        ls $df >/tmp/chmod3
        if [[ -s /tmp/chmod3 ]]; then
           chmod3 $df $dperm $fperm $xperm
           #==============================
        fi
     fi
   }
 echo "chmod3: End dir $dir, subdirs=$nd, files=$nf, xfiles=$nxf"
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7B7. ADMjobs: scripts for unix/linux administrators

customizing chmod3

Note that line 39 of chmod3 (listed above) identifies subdirs with executable files if their names contain 'bin', 'program', 'script',or 'sf'. You could change &/or add patterns depending on the conventions at your site.

Also note that when you wish to change any script in /home/uvadm/sf/..., you should 1st copy it to /home/appsadm/sf/... and modify it there.

/home/appsadm/sf is in the PATH prior to /home/uvadm/sf, so your modified script will be found before the original uvadm script. This way, you will not lose your changes when you install a new version of uvadm.

umask in .bashrc for logging

If you are using 'console logging' (as documented in 'Part_6'), then you must ensure you have setup '.bashrc' with umask 002 in the user homedirs.

The umask 002 in the .bash_profile is lost when console logging is activated, because logging uses the unix/linux 'script' command which is another level of the shell. '.bashrc' solves this problem (see listing on page '1C4').

downloading chmod3

Even if you are not currently a customer of UV Software, you are welcome to download scripts such as chmod3 that are listed in the web documentation.

Your web browser will allow you to save this page & you can then cut it out & store it in your scripts directory. Remember to change permissions to 775.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Nightly batch jobs could fail due to files with bad permisions or group. Nightly batch jobs are scheduled by a crontab owned by 'appsadm' (see crontabs in 'Part_5'). Files with bad permissions migt be FTP to the site or somebody may have used 'root' to copy a file & forgot to fix permissions.

See 'chmod_custom1' '7B8' sample script that could be run before the nightly batch jobs to ensure permissions on all data directories/files 775/664 and group 'apps'. You could also reset owner to 'appsadm' if you want to see who changed what files during the day (or reset owner more infrequently). This sample script has hard-coded directories & permissions for reliability. You would customize for your site.

Note that 'root' should be used only when necessary (fixing permissions, etc). It is too dangerous to run application scripts with root privileges. Of course the chmod_custom1 script must be scheduled by a root crontab, but all batch jobs would be scheduled by 'appsadm' crontabs. And appsadm shares group 'apps' with all operators & programmers who access the data files.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7B8. ADMjobs: scripts for unix/linux administrators

chmod_custom1 - fix perms before night cron jobs

 #!/bin/ksh
 # chmod_custom1 - script to fix perms/owner/group in a directory tree
 #               - fixes all subdirs & files at all levels down the tree
 #               - manually restore execute perms on any script/program subdirs
 #               - by Owen Townsend, UV Software, January 2010
 # See hard-coded perms/owner/group in code further below
 # This script must run as 'root', BUT do NOT forget to exit root after use
 # - could schedule via a 'crontab' owned by root see ADMjobs.htm#Part_5
 # - assumes your root profile PATH modified to access UV scripts
 #
 # 1. Login as root
 #
 # 2. chmod_custom1 directory   <-- arg1 must be a directory
 #
 # 2a. chmod_custom1 /p2/proddata  <-- examples
 #     ==========================
 # 2b. chmod_custom1 /p2/prodlibs
 #     ==========================
 #
 # 3a. chmod 775 /p2/prodlibs/jcls <-- restore execute perms on JCL/scripts
 #     ===========================
 # 3b. chmod 775 /p2/prodlibs/sf   <-- restore on any other script/program dirs
 #     ===========================
 #
 # 4. exit    <-- exit root
 #
 # Sample script to fix perms/owner/group on all subdirs/files in directory tree
 # - with hard-coded perms/owner/group
 #   (chown to appsadm optional & #commented out below)
 # - might use as emergency fix if somebody used root to copy files
 #
 #Note - UV conversion sites should be run by users, NOT 'root' (too dangerous)
 #     - root perms/group would cause batch jobs to fail (cron nightly jobs)
 #     - all directories should be 775
 #     - all files should be 664, except programs & scripts must be 775 (execute)
 #     - user profiles umask 002 (vs dflt 022) allow group share read/write
 #     - converted JCL/scripts run by users with recommended profile/umask
 #       should always write directories/files with 775/664 permsions
 #     - all users running scripts must be setup with common group (ex: appsadm)
 #       so any user in the group has read/write access to directories & files
 #
 #Potential Problems:
 # 1. Files imported (by email,FTP,CD,etc) may have wrong permissions
 # 2. Somebody might operate as 'root' & create files with wrong perms & group
 #
 #VITAL - use 'root' ONLY to fix permissions,owners,groups on imported files
 #      - NEVER use 'root' to create any files in the application directories
 #        (will cause batch jobs to fail, possibly run by cron at night)
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 #          ** sample hard-coded script to fix perms/groups **
 #
 dperm="775"; fperm="664";  #<-- hard-coded perms for Dirs & Files
 owner="appsadm";           #<-- hard-coded owner (optional may #cmt out below)
 group="apps";              #<-- hard-coded group
 dir="$1";                  #<-- capture directory from arg1
 #
 # ensure arg1 is a directory
 if [[ ! -d "$dir" ]]; then
    echo "usage: chmod1 directory     <-- arg1 must be a directory"
    echo "       ================="
    echo "ex:    chmod1 /p2/proddata  <-- example"
    echo "       ==================="
    echo "DIRperms=$dperm, FILEperms=$fperm, owner=$owner, group=$group"
    exit 90; fi
 #
 echo "DIRperms=$dperm, FILEperms=$fperm, owner=$owner, group=$group"
 echo "set perms, owner, group on $dir - enter to continue"
 read reply
 #
 find $dir -type d -exec chmod $dperm {} \;
 #=========================================
 # - set perms ($dperm) on all subdirs in the directory tree
 #
 find $dir -type f -exec chmod $fperm {} \;
 #=========================================
 # - set perms ($fperm) on all files in the directory tree
 #
 chgrp -R $group $dir  #<-- set group on all subdirs/files in tree
 #===================
 #
 chown -R $owner $dir  #<-- set owner on all subdirs/files in tree
 #===================
 #Note - might #comment out above chown (to see who created files)
 #
 echo "perms set for all subdirs & files within: $dir"
 echo "DIRperms=$dperm, FILEperms=$fperm, owner=$owner, group=$group"
 echo "Note - must restore execute perms on any program/script subdirs"
 echo "     - examples below, could hard-code here ??"
 echo "chmod 775 /p2/testlibs/cblx/*  <-- COBOL programs"
 echo "chmod 775 /p2/testlibs/rpgx/*  <-- RPG programs"
 echo "chmod 775 /p2/testlibs/jcls/*  <-- JCL/scripts"
 echo "chmod 775 /p2/testlibs/sf/*    <-- other misc scritps"
 echo "- enter to acknowledge & end script"; read reply
 exit 0
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7C1. ADMjobs: scripts for unix/linux administrators

dtree - draw directory tree

The 'dtree' script will draw a directory tree from any specified starting directory (great for documentation).

For example here is the 1st 40 lines of the dtree for /lib on my Red Hat system.

 #1. dtree /lib >tmp/lib      <-- create dtree diagram for /lib
     ===================

 #2. more tmp/lib             <-- display /lib dtree
     ============

 /lib
 :-----evms
 :-----i686
 :-----iptables
 :-----kbd
 :     :-----consolefonts
 :     :     :-----partialfonts
 :     :-----consoletrans
 :     :-----keymaps
 :     :     :-----amiga
 :     :     :-----atari
 :     :     :-----i386
 :     :     :     :-----azerty
 :     :     :     :-----dvorak
 :     :     :     :-----fgGIod
 :     :     :     :-----include
 :     :     :     :-----qwerty
 :     :     :     :-----qwertz
 :     :     :-----include
 :     :     :-----mac
 :     :     :     :-----all
 :     :     :     :-----include
 :     :     :-----sun
 :     :-----unimaps
 :-----lsb
 :-----modules
 :     :-----2.4.21-4.EL
 :     :     :-----kernel
 :     :     :     :-----arch
 :     :     :     :     :-----i386
 :     :     :     :     :     :-----kernel
 :     :     :     :-----crypto
 :     :     :     :-----drivers
 :     :     :     :     :-----addon
 :     :     :     :     :     :-----aep
 :     :     :     :     :     :-----bcm
 :     :     :     :     :     :-----cipe
 :     :     :     :     :     :-----megarac
 :     :     :     :     :     :-----qla2200
 :     :     :     :     :-----block
 :     :     :     :     :-----cdrom

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7C2. ADMjobs: scripts for unix/linux administrators

dtree - draw directory tree

 #!/bin/ksh
 # dtree - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # dtree - list a directory tree
 #       - contributed by Howard Lobsinger (Peacock Engineering, Montreal)
 #
 #usage: dtree directory
 #       ===============
 #
 D=${1:-`pwd`}
 (cd $D; pwd)
 find $D -type d -print | sort |
 sed -e "s,^$D,,"\
     -e "/^$/d"\
     -e "s,[^/]*/\([^/]*\)$,\:-----\1,"\
     -e "s,[^/]*/,:     ,g" | more
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7D1. ADMjobs: scripts for unix/linux administrators

'rename' scripts

These scripts will rename all files in a directory saving hours of manual 'mv' commands. We will present 20 scripts to perform various renames, (UPPER case, lower case, add/remove/change extensions, prefixes,& patterns).

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7D2. ADMjobs: scripts for unix/linux administrators

sample rename script - renameL

When we convert mainframe files to unix/linux, we will change filenames from UPPER case to lower case. This would be very laborious if we had to manually enter an 'mv' command for each file, but very quick using the 'renameL' script.

We will illustrate using just 3 filenames, but the directory could contain hundreds of long filenames.

 #1. ls datadir
     ==========

     E2123001.ITAXE.BANQTAXE
     E2123002.ITAXE.TAXATION
     E2123003.ITAXE.TRANSDAM

 #2. mv E2123001.ITAXE.BANQTAXE e2123001.itaxe.banqtaxe  <-- the HARD way
     mv E2123002.ITAXE.TAXATION e2123002.itaxe.taxation
     mv E2123003.ITAXE.TRANSDAM e2123003.itaxe.transdam

 #2a. renameL datadir                                     <-- the EASY way
      ===============

 #3. ls datadir
     ==========

e2123001.itaxe.banqtaxe e2123002.itaxe.taxation e2123003.itaxe.transdam

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7D3. ADMjobs: scripts for unix/linux administrators

Here is a listing of just 1 of the 'rename' scripts provided. You can see the others in /home/uvadm/sf/util/... You can also see most of these on the web site at: www.uvsoftware.ca/scripts.htm

renameL - sample rename script

 #!/bin/ksh
 # renameL - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # renameL - rename an entire directory of filenames to lower case
 #
 echo "rename all filenames in subdir to lower case"
 if [ -d "$1" ]; then :
    else echo "usage: renameL directory  <-- arg1 must be a directory"
         echo "       ================="
         exit 1; fi
 #
 reply="n"
 until [ "$reply" = "y" ]
 do echo "will rename all files in $1 to lower case OK ? y/n"
    read reply
 done
 #
 x=0; y=0
 for i in $1/*
 do let x=x+1
    f=${i##*/}
    typeset -l g=$f
    if [[ $g != $f ]]; then
       mv -i $1/$f $1/$g
       let y=y+1
       echo "file# $y (of $x) $1/$f - renamed to: $1/$g"
    fi
 done
 echo "total $y files in ${1}, $x renamed to lower case"
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7E1. ADMjobs: scripts for unix/linux administrators

Recommended Aliases in profile

Here are 'alias' commands extracted from the common_profile which is listed on page '1C3'.

 # alias UNIX commands to prompt for overwrite (highly recommended)
 # - use \rm, \mv, \cp, when you have many files & know what you are doing
 # - '\' tells UNIX to ignore the alias & use native UNIX command
 alias rm='rm -i'              # confirm removes
 alias mv='mv -i'              # confirm renames
 alias cp='cp -i'              # confirm copy overwrites
 alias l='ls -l'               # saves a lot of keystrokes
 alias cdd='cd $RUNDATA'       # quick access to data dir
 alias cdl='cd $RUNLIBS'       # quick access to libs (same as cd)
 alias cdc='cd $CNVDATA'       # quick access to data conversion superdir

The 1st 3 (rm='rm -i',etc) are recommended for unix/linux beginners & experts since it is so easy to wipe out files unintentionally. When you do have multiple files to remove (using the '*' wildcard), you can disable the prompt by preceding the command with a backslash or using option '-f'.

cd aliases to RUNDATA, RUNLIBS,& CNVDATA

These are a great convenience for Your programmers & operators, because the recommended directory design has libraries & data in different filesystems & the paths can be long & awkward if you had to key them often.

 export TESTLIBS=/p1/testlibs     #<-- defs in common_defines
 export TESTDATA=/p1/testdata
 export PRODLIBS=/p2/prodlibs
 export PRODDATA=/p2/proddata

 export RUNLIBS=$TESTLIBS         #<-- defs in .profile or .bash_profile
 export RUNDATA=$TESTDATA
         --- OR ---
 export RUNLIBS=$PRODLIBS
 export RUNDATA=$PRODDATA

 alias cdd='cd $RUNDATA'          #<-- defs in common_profile
 alias cdl='cd $RUNLIBS'

With the above aliases in your profile, you can switch between your libraries & data with 3 character commands:

 cdd     <-- change to your data files superdir
 ===
 cdl     <-- change to your library files superdir
 ===

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7F1. ADMjobs: scripts for unix/linux administrators

alldiff2 - diff report for all files in 2 dirs

The 'alldiff2' script will run the unix 'diff' utility on all pairs of files in 2 subdirs. The unix 'diff' command is a marvelous thing. It shows you the differences between any 2 text files.

I recommend you use 'alldiff2' whenever you make changes to the JCL converter or the control files (same applies to the COBOL converter & its search/replace).

First save the existing scripts as jcl3.old, remake jcl3, rerun the converter, & then run alldiff2 to verify that the changes made are the changes you intended & nothing else has gotten screwed up.

You can use the 'newold' script to change the name & make the new subdir.

 #1a. Login as yourself or appsadm
 #1b. cdl ---> $TESTLIBS

2. Modify converters or control files. For this example, we added 'topi=$TAPE2' on the control file line for data file 'tu.f01.e212801.adrpos'. We then reloaded the indexed control file & reconverted all JCL as follows:
   

 #3. newold jcl3   <-- script to 'mv jcl3 jcl3.old' & 'mkdir jcl3'
     ===========

 #4. jclxx41 jcl2 jcl3       <-- reconvert all JCL in jcl2 to ksh in jcl3
     =================

 #5. alldiff2 jcl3.old jcl3  <-- run alldiff2 script
     ======================    - saves jcl3.dif in the tmp subdir

 #6. uvlp12 tmp/jcl3.dif     <-- print the jcl3.dif file
     ===================       - or vi, more, etc

sample jcl3.dif

 37c37
 < exportgen0 E211801 $TAPE/tu.f01.e211801.adrpos_
 ---
 > exportgen0 E211801 $TAPE2/tu.f01.e211801.adrpos_
 diff file# 10 - jcl3.old/... vs jcl3/24599j04.ksh

 97c97
 < exportgen1 E212990 $TAPE/tu.f01.e211801.adrpos_
 ---
 > exportgen1 E212990 $TAPE2/tu.f01.e211801.adrpos_
 diff file# 78 - jcl3.old/... vs jcl3/28401j04.ksh

2 different of 280 files compared jcl3.old to jcl3

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7F2. ADMjobs: scripts for unix/linux administrators

alldiff2 - diff report for all files in 2 dirs

 #!/bin/ksh
 # alldiff2 - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # alldiff2 - script to compare all text files in 1 directory to a 2nd directory
 #          - bypass non-text files & provide audit trail with file-counts
 #
 # alldiff variations:
 # alldiff  - displays filename only if differences exist
 # alldiff1 - displays filename even if no differences
 #*alldiff2 - same as alldiff, but in addition
 #          - redirects output to tmp/dir2.dif & prompts to view/print/etc
 #
 d1="$1"; d2="$2";
 if [[ -d "$d1" && -d "$d2" ]]; then :
 else echo "USAGE: alldiff2 dir1 dir2"
      echo "       =================="
      exit 1; fi
 d2b=$(basename $d2)   # get basename of dir2 (drop any preceding /path/...)
 log=tmp/$d2b.dif      # make name for output log file
 >$log                 #init logfile in tmp subdir w same name as dir2 + .dif
 x=0; y=0;
 for i in $d1/*
 do
   let x=x+1
   typ=$(file $i)
   if [[  $typ == *text* || $typ == *script* || $typ == *spreadsheet* ]]; then
      f=${i##*/}
      diff -b $d1/$f $d2/$f >>$log
      if [[ $? -gt 0 ]]; then
         echo "diff file# $x - $d1/... vs $d2/$f" >>$log
         echo " " >>$log
         let y=y+1
      fi
   else
      echo "   file# $x $i - NOT a text/script file" >>$log
   fi
 done
 lines=$(wc -l $log)      # capture line count
 echo "$y different of $x files compared $d1 to $d2" >>$log
 echo "$y diff of $x files in $d1 & $d2, report is: $lines"
 echo "--> use uvlp12,uvlp14,uvlp16 to laser print at 12,14,16 cpi"
 echo "--> enter command (vi,cat,more,uvlp12,etc, or null)"
 read ans
 if [[ ! "$ans" = "" ]]; then
    $ans $log
 fi
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7G1. ADMjobs: scripts for unix/linux administrators

cfl - Count Files & Lines in a directory

 #!/bin/ksh
 # cfl - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # cfl - Count Files & Lines in a directory (option to print via uvlp12)
 #     - option to count only files matching suffix supplied by arg#2
 d="$1"; p="$2"; db=$(basename $d); tmpf=tmp/$db.cfl
 if [ -d "$d" ]; then :
    else echo "usage: cfl directory [suffix]"
         echo "       ======================"
         echo "- arg1=directory, arg2=prefix file select (optional)"
    exit 1; fi
 echo "cfl Count Files & Lines in subdir: $d (with optional prefix=$p )" >$tmpf
 integer fs=0 fsp=0 lnt=0 ln2=0;
 for df in $d/*
 do
   ((fs+=1))
   f=${df##*/}
   if [[ "$p" == "" || "$f" == "$p"* ]]; then
      ((fsp+=1))
      ln1=$(wc -l $df); ln2=${ln1% *}; ln3=${ln2##* };
      fdus=$(\du -s -h $df | tr '\t' ' '); fdu=${fdus%% *};
      printf "#%03d" $fsp  >>$tmpf
      if [[ -d $df ]]; then printf "/ " >>$tmpf; else printf "  " >>$tmpf; fi
      printf "%8s " $fdu   >>$tmpf
      printf "%8d " $ln3   >>$tmpf
      printf "$df \n"      >>$tmpf
      ((lnt+=ln3))                # accum total lines
   fi
 done
 echo "Totals: $fs files & $lnt lines in subdir: $d" >>$tmpf
 if [ -n "$p" ]; then echo "$fsp files with prefix=$p (null=all)" >>$tmpf; fi
 more   $tmpf
 echo "enter 'p' to print"; read reply
 if [[ "$reply" == "p" ]]; then uvlp12 $tmpf; fi
 exit 0

 cfl /home/uvadm/env   <-- example, count files & Lines in env/ (profiles)
 ===================     - output shown below:

 cfl Count Files & Lines in subdir: env (with optional prefix= )
 #001      4.0K       36 env/bashrc
 #002      8.0K      133 env/common_defines
 #003       16K      272 env/common_profile
 #004      4.0K       36 env/kshrc
 #005/     100K        0 env/other_profiles
 #006      4.0K       49 env/root_profile
 #007      4.0K       57 env/smb.conf
 #008      8.0K      147 env/stub_profile
 #009      4.0K       50 env/stub_profile_appsadm
 #010      4.0K       36 env/stub_profile_appsadm_cronlogdemo
 #011      4.0K       54 env/utilities_profile
 Totals: 11 files & 870 lines in subdir: env

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7G2. ADMjobs: scripts for unix/linux administrators

statdir1 - report file counts & KB used in a Parent dir

This script should be very useful to any unix/linux site:

1. Helps control disc storage used. Note the sample report shown below for all user /home/... dirs. You can quickly spot any users with extreme Total KB of storage.
   
   
2. Helps verify results of batch runs, you might know how many files & approx how much disc usage there should be for each subdir.
   
   
3. Helps maintain the integrity of your directories & files. The report shows if there are any subdirs within a subdir, which might be a problem if you have batch jobs that process all entries in a directory & might be upset by the appearance of an unintended subdir created by mistake.
   

Here is the report I created for all /home/... dirs on my machine:

 statdir1 /home      <-- call script, arg1 is directory to be examined
 ==============        - sample report below

 stats/home.rpt - FileCounts & DiscUsage for SubDirs in ParentDir: /home
 #1 Files=0000063 SubDirs=0049 TotKB=0004308 - /home/appsadm
 #2 Files=0000126 SubDirs=0022 TotKB=0001180 - /home/aquitec
 #3 Files=0000266 SubDirs=0057 TotKB=0018216 - /home/berks
 #4 Files=0000036 SubDirs=0037 TotKB=0000372 - /home/callatay
 #5 Files=0000070 SubDirs=0061 TotKB=0025964 - /home/carol
                  - - - - - etc - - - - -
 #82 Files=0003280 SubDirs=0252 TotKB=0060484 - /home/uvadm
 #83 Files=0003995 SubDirs=0297 TotKB=0070032 - /home/uvext
 #84 Files=0000015 SubDirs=0006 TotKB=0000100 - /home/uvtest1
 #85 Files=0000407 SubDirs=0079 TotKB=0003168 - /home/vsetest
 Total Dirs=85, Files=33711, SubDirs=3646, KB=1550852, for ParentDir=/home

You can rerun for homedirs using a lot of storage, for example:

 statdir1 /home/uvadm     <-- rerun for selected homedirs
 ====================

'statdir1' is a very powerful & interesting script. You can see it listed at http://www.uvsoftware.ca/CNVaids.htm#2G1

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7G3. ADMjobs: scripts for unix/linux administrators

statdir1 - report file counts & KB used in a Parent dir

 #!/bin/ksh
 # statdir1 - Korn shell script from UVSI stored in: /home/uvadm/sf/util/
 # statdir1 - report file count & total KB for each subdir in a parent dir
 #          - display 1 line per subdir & write report into stats/statdir1.rpt
 #            (subdir 'stats' must be present in cwd to receive report)
 #          - ignore any files in the parent dir
 #          - by Owen Townsend, November 2007
 #
 if [[ -d "$1" && -d stats ]]; then :
    else echo "usage: statdir1 parentdirectory"
         echo "       ========================"
         echo "- arg1 must be a directory with subdirs to be analyzed"
         echo "- subdir 'stats' must exist in cwd to receive report"
         echo "- report name will be stats/ParentDir.rpt"
         exit 99; fi
 #
 pd="$1";               #capture parentdir name
 pdb=$(basename $pd)    #extract basename ParentDir for rpt name in stat subdir
 statrpt=stats/$pdb.rpt #setup output report name
 echo "$statrpt - FileCounts & DiscUsage for SubDirs in ParentDir: $pd"
 echo "$statrpt - FileCounts & DiscUsage for SubDirs in ParentDir: $pd">$statrpt;
 integer lines=0;       #init count subdirs in pd
 integer pdsdc=0;       #init count total files in parentdir
 integer pdsdu=0;       #init count total disc-usage in parentdir
 integer pdssdc=0;      #init count total subsubdirs in parentdir
 typeset -RZ7 sdcnz     #set fixlth for file count (align display)
 typeset -RZ7 sdduz     #set fixlth for total KB (keep display aligned)
 typeset -RZ4 ssdcnz    #set fixlth for SubSubDirs
 #
 for sd in $pd/*
   { # we expect ParentDir to have no files (all files s/b in the subdirs)
     # Indicate any files within ParentDir with 'File' vs 'Files'
     if [[ -d $sd ]]; then Files="Files"; else Files="File"; fi
     ((lines+=1))                #increment count of subdirs within parentdir
     sdcnt=$(find $sd -type f | wc -l)  # capture file count string
     sdcnz=${sdcnt##* }                 # drop leading blanks
     sddus=$(du -s $sd | tr '\t' ' ')   # capture du string, trnslt tab=space
     sdduz=${sddus%% *}                 # drop blank(tab) & subdir-name
     ssdcnt=$(find $sd/* -type d | wc -l) # capture sub-subdir count string
     ssdcnz=${ssdcnt##* }               # drop leading blanks
     echo "#$lines $Files=$sdcnz SubDirs=$ssdcnz TotKB=$sdduz - $sd"
     echo "#$lines $Files=$sdcnz SubDirs=$ssdcnz TotKB=$sdduz - $sd" >>$statrpt
     ((pdsdc+=sdcnz))            #acum total files in parentdir
     ((pdsdu+=sdduz))            #acum total Disc-Usage in parentdir
     ((pdssdc+=ssdcnz))          #acum total subsubdirs in parentdir
   }
 echo "Total Dirs=$lines, Files=$pdsdc, SubDirs=$pdssdc, KB=$pdsdu, for ParentDir=$pd"
 echo "Total Dirs=$lines, Files=$pdsdc, SubDirs=$pdssdc, KB=$pdsdu, for ParentDir=$pd" >>$statrpt
 echo "view   $statrpt    <-- view report"
 echo "uvlp12 $statrpt    <-- print report"
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7H1. ADMjobs: scripts for unix/linux administrators

statlogin1 - table summary of logins by user

 # statlogin1 - table summary of user logins by month & userid
 #            - based on /var/log/messages
 #            - by Owen Townsend, UV Software, Nov 26/2007
 #
 # uvcopy statlogin1,fili1=/var/log/messages,filo1=stats/login.rpt1
 # ================================================================
 # uvcopy statlogin1   <-- same & easier (files default as shown above)
 # =================
 #
 #Note - above works if you have updated root's profile to run uvcopy
 #     - if not, use following procedure:
 #1. mkdir stats tmp           <-- make subdirs in your working directory
 #2. su root                   <-- Switch User to root & enter password
 #3. cp /var/log/messages tmp  <-- copy messages file to tmp subdir
 #4. chmod 777 tmp/messages    <-- change permissions on messages file
 #5. uvcopy statlogin1,fili1=tmp/messages,filo1=stats/login.rpt1
 #   ===========================================================
 #6. vi stats/login.rpt1       <-- view report
 #7. uvlp12 stats/login.rpt1   <-- print report
 #
 #                  ** sample input /var/log/messages **
 #
 # Oct 23 06:11:51 uvsoft3 login[16341]: session opened for user laval4 by LOGIN(uid=0)
 # Oct 23 06:11:51 uvsoft3  -- laval4[16341]: LOGIN ON tty5 BY laval4
 # Oct 23 06:13:08 uvsoft3 ftpd[16342]: FTP LOGIN FROM 192.168.0.2, uvsoft2.uvsoft.ca (laval4)
 # Oct 23 07:15:00 uvsoft3 login[16341]: session closed for user laval4
 # Oct 23 07:15:23 uvsoft3 login[16516]: authentication failure; logname=LOGIN uid=0 euid=0 tty=tty5 ruser= rhost=  user=mvstest
 # Oct 23 07:15:33 uvsoft3 login[16516]: session opened for user mvstest by LOGIN(uid=0)
 # Oct 23 07:15:33 uvsoft3  -- mvstest[16516]: LOGIN ON tty5 BY mvstest
 # Oct 23 21:00:42 uvsoft3 shutdown: shutting down for system halt
 #
 #Note - the code (on next page) scans for 'session opened' & if found
 #     - then scans for ' user ', extracts following word (userid)
 #     - moves month (1st 3 bytes) & userid together for table argument
 #     - see vital instructions 'tbl' (build table) & 'tbp' (print table)
 #
 #                   ** sample output report **
 #
 # statlogin1  2007/11/26_21:24:24  logins by month & userid
 # tbl#001 pg#001     -argument-
 # line#  count    %  mth login
 #     1      13   5  Nov efunds2
 #     2      13   5  Nov laval4
 #     3      23  10  Nov mvstest
 #     4      27  12  Nov root
 #     5      26  11  Nov uvadm
 #     6      26  11  Nov uvbak
 #     7       8   3  Oct efunds2
 #     8       7   3  Oct laval4
 #     9      10   4  Oct mvstest
 #    10      11   4  Oct root
 #           222*100   *TOTAL*

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

 rop=r1    # option to prompt for report disposition at EOJ
 fili1=/var/log/messages,typ=LST,rcs=256
 filo1=stats/login.rpt1,typ=LSTt,rcs=128
 @run
        opn     all
 #
 # begin loop to get & process messages lines until EOF
 man20  get     fili1,a0              get next line of messages
        skp>    man90                 (cc set > at EOF)
        sqzc1   a0(256),' '           ensure only 1 blank between words
 #
 # scan for 'session opened' & 'user' login
 # table user logins by month & by month+day
        scn     a0(100),' session opened '
        skp!    man20
        scn     a0(100),' user '
        skp!    man20
        clr     b0(500),' '           clear workarea
        mvu     b0(25),ax6,' '        store user login until ending blank
        mvc     b100(3),a0            store mth (1st 3 bytes)
        mvc     b104(25),b0           follow with userid
        tblt1f4 b100(32),'mth login'
        skp     man20                return to get next line
 #
 # EOF - dump tables, close files, prompt for report view (rop=r1), end job
 man90  tbpt1   filo1,'logins by month & userid'
        cls     all
        eoj

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7I1. ADMjobs: scripts for unix/linux administrators

devicemod1 - allow user access to tape & diskette

 #!/bin/ksh
 # devicemod1 - allow user access to some devices (tape & diskette)
 #            - by Owen Townsend, June 2008
 #
 # This script distributed with Vancouver Utilities at
 # /home/uvadm/sf/adm/devicemod1
 #
 # Setup to run script at boot time as follows:
 # 1. login as root --> /root
 # 2. mkdir sf          # make /root/sf  (if not already present)
 # 3. cp /home/uvadm/sf/adm/devicemod1 sf  # copy to /root/sf subdir
 # 4. cd /etc/rc5.d     # change to run level 5 init script directory
 # 5. ln -s /root/sf/devicemod1 S99xxdevicemod1
 #    =========================================
 #
 chmod 666 /dev/st0        # SCSI tape (rewind)
 chmod 666 /dev/nst0       # SCSI tape non-rewind
 chmod 666 /dev/sde        # diskette on USB
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

7I2. ADMjobs: scripts for unix/linux administrators

udev rules

Red Hat recommends using 'udev rules' to set desired modes on devices (alternative to devicemod1 init script above).

The udev default rules are stored at /etc/udev/rules.d/50-udev-rules. Red Hat recommend you do NOT change that file, but rather create a new file with overrides (see sample '/etc/udev/rules.d/70-local.rules' listed below).

 # 70-local.rules - by Owen Townsend, UV Software, June 2008
 #                - allow user access to DAT tape & USB diskette
 # /etc/udev/rules.d/50-udev.rules  <-- system default rules stored here
 # /etc/udev/rules.d/70-local.rules <-- this file overrides defaults 0660
 #
 KERNEL=="st0",   GROUP="disk", MODE="0666"
 KERNEL=="nst0",  GROUP="disk", MODE="0666"
 KERNEL=="st0m",  GROUP="disk", MODE="0666"
 KERNEL=="nst0m", GROUP="disk", MODE="0666"
 KERNEL=="sde",   GROUP="disk", MODE="0666"
 #

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Part_8. ADMjobs.doc - Applications Administration

Networking & System Administration

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Networking & System Admin - Contents continued

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8A1. ADMjobs: Networking & System Administration

UV Software Network example

As an example, I will describe the small network I use at UV Software (3 PC's on a LAN/router with a DSL modem to my ISP).

            google.com   yahoo.com     msn.com    etc.com
                |            |            |          |
                --------------- Internet ---------------
                                   |
                                   |
            Web server ------- DNS server ------- Mail server
          uvsoftware.ca       uniserve.com     owen@uvsoftware.ca
                                (my ISP)
                                   |
 WAN DNS Dynamic IPs               |
 =========================================================================
 LAN No DNS static IPs             |
                                   |
                                 modem
                                   |
       ------------------------- router ---------------------------
       |                |     192.168.0.1       |                 |
       |                |                       |                 |
  192.168.0.4     192.168.0.3               192.168.0.2     192.168.0.101
   HP XW9400       HP Kayak                 HP Pavilion      Lexmark T642
    RHEL 5.1       RHEL 3.0                  Windows XP       45 ppm Laser

UV Software Network Hardware/Software

1. 8 port router, D-Link DI-808HV, IP 192.168.0.1
   
   
2. DSL Ethernet modem, D-Link DSL-3001,
   
   
3. RHEL 5.1, HP XW9400 workstation, IP 192.168.0.4
   
   
4. RHEL 3.0, HP Kayak workstation, IP 192.168.0.3
   
   
5. Windows XP, HP Pavilion a1477c, IP 192.168.0.2
   
   
6. Laser Printer, Lexmark T642 45 ppm, IP 192.168.0.101
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8A2. ADMjobs: Networking & System administration

/etc/hosts file on RHEL 5.1

 # /etc/hosts - on 'uvsoft4' RHEL 5.1 at UV Software 2008
 # Do not remove the following line, or various programs
 # that require network functionality will fail.
 ::1	localhost6.localdomain6	localhost6
 127.0.0.1       localhost
 192.168.0.4	uvsoft4	uvsoft4.uvsoftware.ca
 192.168.0.3	uvsoft3	
 192.168.0.2	uvsoft2	
 192.168.0.1	gateway	router

resolv.conf

 nameserver     216.113.192.5
 nameserver     216.113.192.6

Notes

1. Either 'IP addresses' or 'host names' may be specified for the various network programs (FTP, TELNET, SSH, web browsers), but obviously 'host names' are easier to remember.
   
   
2. /etc/hosts is used to convert 'host names' (uvsoft4, etc) into IP addresses on the 'LAN' (Local Area Network). My LAN consists of a router & 3 PCs.
   
   
3. /etc/resolv.conf specifies the DNS (Domain Name Server) used to convert host names to IP addresses for the 'WAN' (Wide Area Network). This means the outside world internet. My ISP (uniserve.com) has 2 DNS servers whose IP addresses are shown in the resolv.conf file above.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8A3. ADMjobs: Networking & System Administration

setup Router access to ISP

Use a web browser (FireFox or IE) to setup the router to access your ISP. Here is the procedure for my D-Link router:

2. enter userid 'admin' (no password by default on new router)
   
   
3. run the setup wizard
   
   
4. select 'Dynamic IP Address' to obtain IP address from ISP
   
   
5. select 'PPP over ethernet' (for DSL users)
   
   
6. select 'clone Mac Address'
   
   
7. enter the DNS (Domain Name Server) IP addresses (in my case 216.113.192.5 & 216.113.192.6)
   
   
8. enter the PPPoE Account-name & password assigned by your ISP (example: uvsoft4@uniserve.com).
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8B1. ADMjobs: Networking & System Administration

nslookup - get IP adrs from domain name

Ever wonder what IP address was assigned to you by your ISP ? For example my IP address is currently '174.133.234.43'. This is dynamically assigned, but might never change if I never turn off my router & modem. Note my computers on my LAN address the router/gateway as 192.168.0.1, but my ISP addresses my router from the outside world as 174.133.234.43.

nslookup unix/linux command

Here are some examples using the 'nslookup' command entered from a unix/linux login prompt (vs an internet browser & whatismyipadress below).

 nslookup uvsoftware.ca       --> 174.133.234.43
 nslookup uniserve.com        --> 204.239.42.63   <-- uniserve   OLD ISP
 nslookup webfaction.com      --> 75.125.202.115  <-- webfaction NEW ISP
 nslookup smtp.webfaction.com --> 74.55.86.74     <-- webfaction Mail Server
 nslookup ns1.webfaction.com  --> 207.44.144.46   <-- webfaction Name Server
        webfaction DNS Server --> 4.2.2.1         <-- not sure what this is ?

whatismyipaddress.com

'whatismyipaddress.com' is an alternative method using an internet browser. It also provides other lookups such as 'IP to hostname'. Here are some examples:

 216.113.194.1     --> www6.uvsoftware.ca
 204.239.42.63     --> smtp-relay-uniserve.com
 216.113.203.78    --> fatwire-203-77.uniserve.ca

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8C1. ADMjobs: Networking & System Administration

'ping' to investigate network problems

'ping' is usually the 1st tool used to investigate problems communicating with other computers. For example if I had problems FTP'ing or browsing to my web site I might use the following ping command:

 ping uvsoftware.ca
 ==================

 PING uvsoftware.ca (216.113.194.1) 56(84) bytes of data.
 64 bytes from www6.uniserve.com (216.113.194.1): icmp_seq=1 ttl=61 time=7.27 ms
 64 bytes from www6.uniserve.com (216.113.194.1): icmp_seq=2 ttl=61 time=8.41 ms
 64 bytes from www6.uniserve.com (216.113.194.1): icmp_seq=3 ttl=61 time=7.75 ms
 --- uvsoftware.ca ping statistics ---
 3 packets transmitted, 3 received, 0% packet loss, time 2000ms
 rtt min/avg/max/mdev = 7.275/7.814/8.414/0.466 ms

If the above ping using the host-name failed, then I would use the IP address (to see if the Domain Server might be down vs a connection problem).

 ping 216.113.194.1
      =============

 PING 216.113.194.1 (216.113.194.1) 56(84) bytes of data.
 64 bytes from 216.113.194.1: icmp_seq=1 ttl=61 time=7.52 ms
 64 bytes from 216.113.194.1: icmp_seq=2 ttl=61 time=7.84 ms
 64 bytes from 216.113.194.1: icmp_seq=3 ttl=61 time=6.92 ms
 --- 216.113.194.1 ping statistics ---
 3 packets transmitted, 3 received, 0% packet loss, time 2002ms
 rtt min/avg/max/mdev = 6.924/7.430/7.840/0.380 ms

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8D1. ADMjobs: Networking & System Administration

FTP to 2nd computer on LAN

Here is a sample FTP session to transfer a file between my 2 Linux computers on my LAN (3 PCs & a router). I captured the session by turning on 'console logging' as explained in Part_6.

 Script started on Sat 14 Jun 2008 09:29:07 AM PDT

 <@:owen:/home/owen> ftp 192.168.0.3
                     ===============
 Connected to 192.168.0.3.
 Name (192.168.0.3:owen): owen
 331 Password required for owen.
 Password:
 230 User owen logged in.
 Remote system type is UNIX.
 Using binary mode to transfer files.

 ftp> put stub_profile
      ================
 local: stub_profile remote: stub_profile
 200 PORT command successful.
 150 Opening BINARY mode data connection for stub_profile.
 226 Transfer complete.
 8538 bytes sent in 8.2e-05 seconds (1e+05 Kbytes/s)

 ftp> dir
      ===
 200 PORT command successful.
 150 Opening ASCII mode data connection for /bin/ls.
 total 60
 -rw-------    1 owen          349 Jun 13 20:50 .bash_history
 -rwxrwxr-x    1 users        8858 Apr 26 14:17 .bash_profile
 -rw-r-----    1 owen         8538 Jun 14 09:26 stub_profile
 drwxrwxr-x    2 owen         4096 Jun 13 12:35 tmp
 226 Transfer complete.

 ftp> bye
      ===
 221 Goodbye.
 <@:owen:/home/owen> exit
                     ====
 Script done on Sat 14 Jun 2008 09:30:48 AM PDT

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8D2. ADMjobs: Networking & System Administration

FTP to my web site

 Script started on Sat 14 Jun 2008 05:16:01 PM PDT

 <@:owen:/home/owen> ftp www.uvsoftware.ca
                     =====================
 Connected to uvsoftware.ca.
 220 www6.uniserve.ca FTP server ready
 Name (www.uvsoftware.ca:owen): wd-uvsoft
 331 Password required for wd-uvsoft.
 Password:
 230 User wd-uvsoft logged in.
 Remote system type is UNIX.
 Using binary mode to transfer files.
 ftp> cd public_WWW
      =============
 250 CWD command successful.
 ftp> put uvweb.zip
      =============
 local: uvweb.zip remote: uvweb.zip
 150 Opening BINARY mode data connection for uvweb.zip
 226 Transfer complete.
 2765834 bytes sent in 38 seconds (72 Kbytes/s)
 ftp> dir
      ===
 150 Opening ASCII mode data connection for file list
 -rw-rw-r--   1 wd-uvsoft wd-uvsoft   465893 Jun 13 19:07 admjobs.htm
 -rw-rw-r--   1 wd-uvsoft wd-uvsoft   483264 Jun 13 19:07 cmpjobs.htm
 -rw-rw-r--   1 wd-uvsoft wd-uvsoft   340703 Jun 13 19:07 cnvaids.htm
                   - - - 70 lines omitted - - -
 -rw-rw-r--   1 wd-uvsoft wd-uvsoft  2765834 Jun 14 17:15 uvweb.zip
                   - - - 10 lines omitted - - -
 -rw-rw-r--   1 wd-uvsoft wd-uvsoft   132637 Jun 13 19:07 windowsdos.htm
 -rw-rw-r--   1 wd-uvsoft wd-uvsoft    47649 Jun 13 19:07 wordjobs.htm
 -rw-rw-r--   1 wd-uvsoft wd-uvsoft    77710 Jun 13 19:07 xrefjobs.htm
 226 Transfer complete.
 ftp> bye
 221 Goodbye.

<@:owen:/home/owen> exit Script done on Sat 14 Jun 2008 05:18:27 PM PDT

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8E1. ADMjobs: Networking & System Administration

SSH

To reload my web-site I first create a 'zip' file (about 100 HTML documents). Next I FTP the zip file into the public_WWW directory for www.uvsoftware.ca. Then I need to 'unzip' the zip file.

My ISP does not allow 'TELNET', but does allow 'SSH' (which is more secure). Here is the SSH session captured by console logging (see Part_6).

 Script started on Sat 14 Jun 2008 09:34:16 AM PDT

 <@:owen:/home/owen> ssh wd-uvsoft@www.uvsoftware.ca
 Password:
 wd-uvsoft@www6$ ls -l
                 =====
 total 3
 lrwxr-xr-x  1 root       wheel        27 Aug  2  2003 public_CGI -> /u/www/public_CGI/wd-uvsoft
 drwxr-x--x  4 wd-uvsoft  wd-uvsoft  3072 Jun 14 19:10 public_WWW
 wd-uvsoft@www6$ cd public_WWW
                 =============
 wd-uvsoft@www6$ unzip uvweb.zip
                 ===============
 wd-uvsoft@www6$ ls -l
                 =====
 total 16435
 -rw-rw-r--  1 wd-uvsoft  wd-uvsoft   465893 Jun 14 19:07 admjobs.htm
 -rw-rw-r--  1 wd-uvsoft  wd-uvsoft   483264 Jun 14 19:07 cmpjobs.htm
 -rw-rw-r--  1 wd-uvsoft  wd-uvsoft   340703 Jun 14 19:07 cnvaids.htm
                - - - many lines omitted - - -
 -rw-rw-r--  1 wd-uvsoft  wd-uvsoft   132637 Jun 14 19:07 windowsdos.htm
 -rw-rw-r--  1 wd-uvsoft  wd-uvsoft    47649 Jun 14 19:07 wordjobs.htm
 -rw-rw-r--  1 wd-uvsoft  wd-uvsoft    77710 Jun 14 19:07 xrefjobs.htm

 wd-uvsoft@www6$ exit
                 ====
 logout Connection to www.uvsoftware.ca closed.

 <@:owen:/home/owen> exit
                     ====
 Script done on Sat 14 Jun 2008 09:38:32 AM PDT

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8F1. ADMjobs: Networking & System Administration

putty - SSH Terminal Emulator

'putty' is a free terminal emulator for windows, that may be downloaded from http://www.chiark.greenend.org.uk written by Simon Tatham.

'putty' uses SSH (Secure SHell) protocol which is more secure that 'telnet'. We recommend putty for mainframe conversion projects where programmers & operators will be using Windows PCs online to a Unix/Linux system.

download & install putty

1. On each Windows PC where putty is to be installed, use an internet browser (IE or FireFox) to download/save from:
   
   
   

   	http://www.chiark.greenend.org.uk/~sgtatham/putty/
   	====================================================
   	- putty.zip <-- download
   	- putty-0.60-installer.exe <-- download

2. Run Windows Explorer to goto the download directory & execute (double click)
   

 #3. putty-0.60-installer.exe  <-- execute
     ========================
     - installs the following programs into C:\program files\putty\...

The install creates a putty icon on the desktop & adds menu items to programs.

4. Add the putty program directory to your PATH (so you can execute psftp from a command line window.
   

      control panel --> system --> advanced --> environmental variables

      system variables --> PATH --> edit append ';C:\program files\putty'

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8F2. putty - SSH Terminal Emulator

configuring putty

Executing putty (double click putty icon on desktop) displays the configuration screen. I have listed the menu below with items to be modified in UPPER case.

      Session    - Logging
      Terminal   - Keyboard, Bell. Features
      Window     - APPEARANCE, Behaviour, Translation, Selection, COLOURS
      Connection - Data, Proxy, Telnet, Rlogin, SSH, Serial

      HOST-NAME or IP ADDRESS ______________ <-- enter '192.168.0.4'
      Port: 22
      Connection Type: raw, telnet, rlogin, *SSH, serial
      Saved session: _______________________ <-- enter 'uvsoft4'
      Load, Save, Delete                     <-- enter Save, then Load
      Open, Cancel                           <-- enter Open

Configuration Example

1. enter Host-Name/IP address --> 192.168.0.4
   
   
2. enter Saved Sessions --> uvsoft4
   
   
3. click SAVE
   
   
4. highlight saved session 'uvsoft4'
   
   
5. click LOAD
   
   
6. click OPEN --> displays black screen with login prompt
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8F3. putty - SSH Terminal Emulator

setting Appearance & Colour

The default login screen is small, black & white, and ugly. We can make putty much more pleasant to work with as follows:

1. click upper left corner of screen ---> displays 'putty Re-configuration' screen
   

2a. click Appearance --> Change

2b. select Courier, Bold, 14 point

3a. click Colours --> Modify

3b. select Default Background --> Yellow

3c. select Default Foreground --> Black

3d. click Apply

4. Position & Expand the putty window Depth & Width as desired (allow for overlapping multiple sessions)
   
   
5. Save your configuration
   

5a. click upper left corner of screen --> displays config menu

5b. click Change Settings --> displays list of saved sessions

5c. highlight the session name just modified

5d. click Save & exit

On future sessions, loading the saved session (uvsoft4 in my example) will bring up the login screen with your desired colours & size.

For multiple sessions, use the 'Duplicate Sessions' option on the putty reconfig menu.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8G1. ADMjobs: Networking & System Administration

setup Samba Linux File-Server

Some of UV Software's customers use Samba to allow Windows PC access to files on the Linux machine. Mainframe conversion sites might provide access to the COBOL reports now created on the Linux machine.

Please see the sample Samba configuration file listed 2 pages ahead -->

setup Samba on Linux

 #1. Login as root

 #2. cd /etc/samba

 #3. mv smb.conf smb.conf.orig       <-- save original
     =========================

 #4. cp /home/uvadm/env/smb.conf .   <-- copy VU sample
     =============================

 #5. vi smb.conf <-- modify smb.conf for your site
     ===========   - change user-names, directories, etc for your site

 #6. smbpasswd -a userid    <-- add Samba users
     ===================      - userid already existing as a unix user

 #7. service smb start      <-- start Samba service now for testing
     =================

 #8. chkconfig smb on       <-- setup to start Samba whenever system booted
     ================           (run levels 2,3,4,5)

 9. chkconfig --level 2345 iptables off
    ===================================

 10. System --> Administration --> SELinux mngmnt --> disable
     ========================================================
     - check the box 'Relabel on next boot' & reboot Linux

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8G2. ADMjobs: Networking & System Administration

setup Windows for Samba access

 #1. start Windows Explorer

 #2. Tools --> Map Network drive

      Drive:  Z: \\uvsoft4\root
      Folder: root
      Reconnect at logon: check

      Drive:  Y: \\uvsoft4\home
      Folder: home
      Reconnect at logon: check

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8G3. ADMjobs: Networking & System Administration

Samba configuration file

 # smb.conf - Samba configuration file
 # /etc/samba/smb.conf      <-- location on Red Hat Enterprise Linux
 # /home/uvadm/env/smb.conf <-- sample distributed with Vancouver Utilities
 # - this shorter sample created June 2008, by Owen Townsend (UV Software)
 [global]
 workgroup = uvsoftware
 printing = CUPS
 printcap name = CUPS
 disable spoolss = yes
 show add printer wizard = no
 passdb backend = smbpasswd
 [home]
 comment = uvsoft4 /home directories
 path = /home
 valid users = uvadm, uvbak, uvext
 read only = no
 create mask = 0775
 directory mask = 0775
 force group = apps
 [root]
 comment = uvsoft4:root files
 path = /
 valid users = uvadm, uvbak, uvext
 read only = yes
 #
 # Install & modify for your site as follows:
 #
 # 1. Login as root
 # 2. cd /etc/samba
 # 3. mv smb.conf smb.conf.orig     <-- save original
 # 4. cp /home/uvadm/env/smb.conf . <-- copy VU sample
 # 5. vi smb.conf  <-- modify smb.conf for your site
 #                   - change user-names, directories, etc for your site
 #
 # 6. smbpasswd -a userid    <-- to add new Samba users
 #    ===================      - userid already existing as a unix user
 #
 # 7. service smb start      <-- restart after config file changes
 #    =================          (easier than using separate stop/start)
 #
 # 8. chkconfig smb on       <-- start Samba when system booted
 #    ================           (run levels 2,3,4,5)
 #
 # 9. chkconfig --level 2345 iptables off
 #    ===================================
 #
 # I had to disable iptables (firewall) before samba would work properly
 # - my router has a built in firewall
 # Also had to turn off SELinux, using the GUI mngmnt tool as follows:
 #
 #10. System --> Administration --> SELinux mngmnt --> disable
 #    ========================================================
 #    - check the box 'Relabel on next boot' & reboot Linux

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8H1. ADMjobs: Unix/Linux System Administration

investigating /var/log/dmesg bootup file

When I installed a DAT tape device & an Adaptec SCSI controller, I checked to ensure the system recognized these devices at system boot time & to determine the device name assigned to the DAT tape drive.

 grep -i 'adaptec' /var/log/dmesg
 ================================

 Unix/Linux saves the last boot messages in /var/log/dmesg (563 lines on my
 Red Hat 5.1). Here is a sample from my last boot - lines 1-12 & 461-472).

sample boot messages from /var/log/dmesg

 Bootdata ok (command line is ro root=LABEL=/ rhgb quiet)
 Linux version 2.6.18-92.el5xen
 (brewbuilder@ls20-bc2-13.build.redhat.com)
 (gcc version 4.1.2 20071124 (Red Hat 4.1.2-41))
 #1 SMP Tue Apr 29 13:31:30 EDT 2008
 BIOS-provided physical RAM map:
  Xen: 0000000000000000 - 00000000f39b5000 (usable)
 On node 0 totalpages: 997813
   DMA zone: 997813 pages, LIFO batch:31
 DMI 2.5 present.
 ACPI: RSDP (v002 HP                                    ) @ 0x00000000000e9e10
 ACPI: XSDT (v001 HPQOEM SLIC-WKS 0x20070625  0x00000000) @ 0x00000000dffca474

                  - - - - - 400 lines omitted - - - - -

 scsi8 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 7.0
         <Adaptec 29160N Ultra160 SCSI adapter>
         aic7892: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
   Vendor: HP        Model: C1537A            Rev: L111
   Type:   Sequential-Access                  ANSI SCSI revision: 02
  target8:0:3: Beginning Domain Validation
  target8:0:3: FAST-10 SCSI 10.0 MB/s ST (100 ns, offset 32)
 scsi 8:0:3:0: Attached scsi generic sg6 type 1
 ACPI: PCI Interrupt Link [LMC0] enabled at IRQ 16
 GSI 24 sharing vector 0x29 and IRQ 24
 ACPI: PCI Interrupt 0000:00:08.0[A] -> Link [LMC0] -> GSI 16 (level, high) -> IRQ 24
 st 8:0:3:0: Attached scsi tape st0
                  - - - - - 100 lines omitted - - - - -

grep output

 grep -i 'adaptec' /var/log/dmesg
 ================================

 scsi8 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 7.0
         <Adaptec 29160N Ultra160 SCSI adapter>

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8I1. ADMjobs: Unix/Linux System Administration

mounting USB memory devices

Unix/Linux systems might not automatically mount USB memory devices (as does Windows). You can mount devices manually if you know the device name & are logged in as root. For example if you know the device name is 'sdf1' you could mount as follows:

 mount /dev/sdf1 /mnt    (/dev/sdf1 is a USB memory device on Owen's system)
 ====================

 ls -l /mnt            <-- display any files on the memory stick
 ==========

 cp /etc/passwd /mnt   <-- copy a file to the memory stick
 ===================

determining USB device name - method #1

If you have some understanding of Unix/Linux devices you can list /dev/... & probably guess which device is the USB memory stick. I know that USB devices are treated as SCSI devices & that SCSI device names are /dev/sd... On my system I can list all SCSI devices & get the response shown below:

 ls -l /dev/sd*
 ==============

 /dev/sda     <-- 1st hard disc (a)
 /dev/sda1    <-- 1st partition on 1st disc
 /dev/sda2    <-- 2nd partition on 1st disc
 /dev/sda3
 /dev/sda4
 /dev/sdb     <-- 2nd hard disc (b)
 /dev/sdb1    <-- 1st partition on 2nd disc
 /dev/sdb2
  --etc--
 /dev/sdd4    <-- 4th partition on 4th disc (d)
 /dev/sde     <-- floppy disc
 /dev/sdf     <-- USB
 /dev/sdf1    <-- 1st USB device

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8I2. mounting USB memory devices

determining USB device name - method #2

Another way is to plug in the device & watch for a system response on the system console (console screen tty1 Alt-Function-1).

If you do not have access to the system console, you can use 'dmesg' which displays all system messages since the last boot (from /var/log/dmesg). To see just the last few lines pipe the output into 'tail'.

 dmesg | tail
 ============

 Vendor: SanDisk   Model: Cruzer Mini       Rev: 0.2
   Type:   Direct-Access  ANSI SCSI revision: 02
 SCSI device sdf: 1000944 512-byte hdwr sectors (512 MB)
 sdf: Write Protect is off
 sdf: assuming drive cache: write through
 SCSI device sdf: 1000944 512-byte hdwr sectors (512 MB)
 sdf: Write Protect is off
 sdf: assuming drive cache: write through
 sdf: sdf1
 sd 12:0:0:0: Attached scsi removable disk sdf
 sd 12:0:0:0: Attached scsi generic sg7 type 0

From the above, you can guess that the USB device is '/dev/sdf1'.

determining USB device name - method #3

Another alternative might be to search (vi,grep,tail) /var/log/dmesg or /var/log/messages for patterns you know represent the USB devices. BUT, you need 'root' permissions to access those files directly.

 grep 'USB' /var/log/dmesg      <-- requires root access
 =========================

 grep 'USB' /var/log/messages   <-- requires root access
 ============================

USB automatic mount

In fact on my Red Hat Enterprise 5.1, the USB memory device is automatically mounted on /media/disk, but I would not have know this by searching messages because it is not recorded.

 mount /dev/sdf1 /media/disk   <-- automatic mount on my RHEL 5.1
 ===========================

 mount /dev/sdf1 /mnt           <-- can also mount like this
 ====================               (if you don't know auto mount point)

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8J1. ADMjobs: Unix/Linux System Administration

Unix/Linux system log files

Here is a summary of system log files that are most relevant to the casual Unix/Linux user (vs system administrator geeks).

 /var/log/messages - records system events at boot time & ongoing
                   - boot, shutdown, device recognitions, hardware faults,
                     software faults, server events (FTP, Samba, etc)
                   - text file (may inspect/search with vi, grep, etc)

 /var/log/dmesg - records system boot/restart messages
                - the info you see flying up the screen on bootup
                - text file (may inspect/search with vi, grep, etc)

 /var/run/utmp - currently logged in users (1 record per user)
               - userid, login date/time, terminal, process id
               - binary file (many x'00' bytes, fixed recsize 384)
               - can NOT inspect/search with vi, grep, etc
               - interrogated by several Unix/Linux system utilities
                 (who, w, finger, utmp-dump)
               - may also use 'uvhd' to investigate (see page '8K1')

 /var/log/wtmp - same as /var/run/utmp, but includes last 300 logins & reboots
                 in date/time sequence
               - interrogated by Unix/Linux system utilities 'last' & 'lastlog'
               - may also use 'uvhd' to investigate (see page '8K1')

commands to access Unix/Linux log files

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8J2. log file commands - sample outputs

who - sample output

 who     <-- display users currently logged in
 ===

 root     tty1         2008-08-31 07:00
 uvadm    tty2         2008-08-31 07:00
 uvbak    tty3         2008-08-31 07:00
 mvstest  tty4         2008-08-31 07:19
 root     :0           2008-08-31 06:47
 root     pts/1        2008-08-31 10:04 (:0.0)

w - sample output

 w       <-- display users currently logged in & what they are doing
 ===

  10:22:03 up  3:35,  6 users,  load average: 0.04, 0.04, 0.00
 USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
 =====================================================================
 root     tty1     -                07:00    3:38   0.03s  0.03s -bash
 uvadm    tty2     -                07:00    0.00s  0.01s  0.00s w
 uvbak    tty3     -                07:00    3:21m  0.00s  0.00s -bash
 mvstest  tty4     -                07:19    2:59m  0.00s  0.00s -bash
 root     :0       -                06:47   ?xdm?  13.74s  0.04s /usr/bin/gnome-
 root     pts/1    :0.0             10:04   17:04   0.01s  0.01s bash

finger - sample output

 finger  <-- display user info (userid, terminal, login time)
 ======

 Login     Name       Tty      Idle  Login Time   Office     Office Phone
 ========================================================================
 mvstest              tty4     3:05  Aug 31 07:19
 root      root       tty1        9  Aug 31 07:00
 root      root      *:0             Aug 31 06:47
 root      root       pts/1      23  Aug 31 10:04 (:0.0)
 uvadm                tty2           Aug 31 07:00
 uvbak                tty3     3:27  Aug 31 07:00

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8J3. log file commands - sample outputs

last - sample output

 last    <-- display login history (last 300 logins, multiple lines per user)
 ====

 root     pts/1        :0.0             Sun Aug 31 10:04   still logged in
 mvstest  tty4                          Sun Aug 31 07:19   still logged in
 uvbak    tty3                          Sun Aug 31 07:00   still logged in
 uvadm    tty2                          Sun Aug 31 07:00   still logged in
 root     tty1                          Sun Aug 31 07:00   still logged in
             - - - 290 lines omitted - - -
 uvadm    tty2                          Mon Jul 21 10:37 - down   (10:50)
 root     tty1                          Mon Jul 21 09:39 - down   (11:48)
 reboot   system boot  2.6.18-92.1.6.el Mon Jul 21 07:38          (13:49)
 wtmp begins Sat Jul 19 19:57:15 2008

lastlog - sample output

 lastlog  -t7   <-- display last login info (1 line per user)
 ============     - for users login within past 7 days

 Username         Port     From             Latest
 =========================================================================
 root             tty1                      Sun Aug 31 07:00:45 -0700 2008
 uvadm            tty2                      Sun Aug 31 07:00:50 -0700 2008
 mvstest          tty4                      Sun Aug 31 07:19:05 -0700 2008
 uvbak            tty3                      Sun Aug 31 07:00:55 -0700 2008
 vsetest          tty5                      Sat Aug 30 07:43:39 -0700 2008
 uvext            tty4                      Thu Aug 28 20:50:49 -0700 2008

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8J4. log file commands - sample outputs

utmpdump - sample output

'utmpdump' dumps /var/run/utmp or /var/log/wtmp (binary files) to ASCII text files, but the result has many blanks between fields, & looks ugly. So, we will redirect the output of utmpdump to a tmp/file & then use 'uvlist' (as a filter), with option 'c9' to reduce multi-consecutive blanks to 1.

 #1. /usr/sbin/utmpdump /var/run/utmp >tmp/utmp
     ==========================================
     - dump /var/run/utmp to an ASCII file (tmp/utmp)

 #2. uvlist tmp/utmp -c9i1 >tmp/utmp2
     ================================
     - use uvlist with option 'c9' to reduce multi-consecutive blanks to 1
     - option 'i1' inhibits uvlist output 1st line laser printer control codes

 #3. cat tmp/utmp2    <-- display filtered output
     =============

 /home/uvadm/tmp/utmp1 now=080831:1721 uvadm pg#  1

 [8] [00420] [si ] [ ] [ ] [ ] [0.0.0.0 ] [Sun Aug 31 06:47:05 2008 PDT]
 [2] [00000] [~~ ] [reboot ] [~ ] [ ] [0.0.0.0 ] [Sun Aug 31 06:47:05 2008 PDT]
 [1] [20021] [~~ ] [runlevel] [~ ] [ ] [0.0.0.0 ] [Sun Aug 31 06:47:05 2008 PDT]
 [8] [02339] [l5 ] [ ] [ ] [ ] [0.0.0.0 ] [Sun Aug 31 06:47:30 2008 PDT]
 [7] [03974] [1 ] [root ] [tty1 ] [ ] [0.0.0.0 ] [Sun Aug 31 07:00:45 2008 PDT]
 [7] [03980] [2 ] [uvadm ] [tty2 ] [ ] [0.0.0.0 ] [Sun Aug 31 07:00:50 2008 PDT]
 [7] [03982] [3 ] [uvbak ] [tty3 ] [ ] [0.0.0.0 ] [Sun Aug 31 07:00:55 2008 PDT]
 [7] [03985] [4 ] [mvstest ] [tty4 ] [ ] [0.0.0.0 ] [Sun Aug 31 07:19:05 2008 PDT]
 [6] [03988] [5 ] [LOGIN ] [tty5 ] [ ] [0.0.0.0 ] [Sun Aug 31 06:47:30 2008 PDT]
 [6] [03989] [6 ] [LOGIN ] [tty6 ] [ ] [0.0.0.0 ] [Sun Aug 31 06:47:30 2008 PDT]
 [5] [03990] [x ] [ ] [ ] [ ] [0.0.0.0 ] [Sun Aug 31 06:47:30 2008 PDT]
 [8] [04030] [mF ] [ ] [ ] [ ] [0.0.0.0 ] [Sun Aug 31 06:47:31 2008 PDT]
 [7] [04110] [:0 ] [root ] [:0 ] [ ] [0.0.0.0 ] [Sun Aug 31 06:47:40 2008 PDT]
 [8] [00000] [/1 ] [root ] [pts/1 ] [ ] [0.0.0.0 ] [Sun Aug 31 10:32:28 2008 PDT]

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8K1. ADMjobs: Unix/Linux system log files

using 'uvhd' to investigate /var/log/wtmp

Now let's investigate /var/log/wtmp using 'uvhd' (the Vancouver Utility for investigating binary files). Note the difference between 'utmp' & 'wtmp'. Both files store user logins & events such as reboot, shutdown,& runlevel changes. Both are binary files with fixed record size 384 bytes.

/var/run/utmp - stores logins only for currently logged in users

/var/log/wtmp - stores login HISTORY (last 300 logins & shutdown/reboots)

 uvhd /var/log/wtmp r384   <-- investigate wtmp (recsize=384)
 =======================

                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ....05..~...............................~~..runlevel............
             0000330070000000000000000000000000000000770077666766000000000000
             10000500E0000000000000000000000000000000EE0025EC565C000000000000
          64 ............2.6.18-92.1.6.el5xen................................
             0000000000003232332332323266376600000000000000000000000000000000
             0000000000002E6E18D92E1E6E5C585E00000000000000000000000000000000
         128 ................................................................
             0000000000000000000000000000000000000000000000000000000000000000
             0000000000000000000000000000000000000000000000000000000000000000
         192 ................................................................
             0000000000000000000000000000000000000000000000000000000000000000
             0000000000000000000000000000000000000000000000000000000000000000
         256 ................................................................
             0000000000000000000000000000000000000000000000000000000000000000
             0000000000000000000000000000000000000000000000000000000000000000
         320 .......................H<.......................................
             000000000000000000008A843900000000000000000000000000000000000000
             00000000000000000000B928C0B0000000000000000000000000000000000000

1. On the 1st record above, the event is a runlevel change (on a reboot). 'runlevel' is at 44(8) (dsplcmnt 44 0 relative, length 8). On other records this field might be: shutdown, LOGIN, userids, etc).
   
   
2. Note that the time is coded in binary in bytes 340(4b), the Unix epoch time, the number of seconds since Jan 1, 1970. x'8BA98248' is 2008/07/19_19:57:15. The starting displacement is 320+20=340 since the row starts at 320 and the x'8B' lines up under 20 on the scale.
   
   
3. You can use the 'utmpdump' system command to display the file contents
   
   
4. Unix/Linux systems provide the 'utmpdump' utility to display /var/run/utmp and /var/log/wtmp records in a user friendly readable format. utmpdump will convert the binary Unix-times to a human readable format. Try 'utmpdump /var/log/wtmp | more'.
   
   
5. On the next page, we will use uvhd to select records for a desired userid and then use utmpdump to display the contents.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8K2. using 'uvhd' to investigate /var/log/wtmp

Search /var/log/wtmp for userid uvadm

 uvhd /var/log/wtmp r384   <-- startup uvhd for /var/log/wtmp (recsize 384)
 =======================     - will display 1st record (same as above)
                             - not shown here to save space

 --> s 44(5),'uvadm'     <-- search for records with userid 'uvadm'
                           - displays 1st record found as follows:

 r#       74 0123456789012345678901234567890123456789012345678901234567890123
       28032 ........tty2............................2...uvadm...............
             0000800077730000000000000000000000000000300077666000000000000000
             70006F004492000000000000000000000000000020005614D000000000000000
          64 ................................................................
             0000000000000000000000000000000000000000000000000000000000000000
             0000000000000000000000000000000000000000000000000000000000000000
         128 ................................................................
             0000000000000000000000000000000000000000000000000000000000000000
             0000000000000000000000000000000000000000000000000000000000000000
         192 ................................................................
             0000000000000000000000000000000000000000000000000000000000000000
             0000000000000000000000000000000000000000000000000000000000000000
         256 ................................................................
             0000000000000000000000000000000000000000000000000000000000000000
             0000000000000000000000000000000000000000000000000000000000000000
         320 ....................A..H9.......................................
             000000000000000080004C843E00000000000000000000000000000000000000
             00000000000000006F0019489640000000000000000000000000000000000000

 found--> s 44(5),'uvadm' <--at byte# 44 of record# 74
 =====================================================

 --> ss      <-- may use 'ss' to repeat the search
     ===

You could use 'ss' to repeat the search for the next matching record, until yor reach the end of the file. We will not show you any more matching records.

We will demo the select/write command on the next page -->

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8K3. using 'uvhd' to investigate /var/log/wtmp

Select all records for userid 'uvadm'

 --> w9999 44(5),'uvadm'    <-- Write all records with 'uvadm' in bytes 44-48
     ===================        to a tmp/file

                      10        20        30        40        50        60
 r#     1340 0123456789012345678901234567890123456789012345678901234567890123
      514176 ........tty2............................2...uvadm...............
             0000800077730000000000000000000000000000300077666000000000000000
             70005F004492000000000000000000000000000020005614D000000000000000
                      ----- bytes 64-319 omitted to save space -----
         320 .......................H.8......................................
             0000000000000000000011A4E300000000000000000000000000000000000000
             0000000000000000000050888890000000000000000000000000000000000000

 w9999 44(5),'uvadm' 30 written, tmp/wtmp_080817_151157W
 =======================================================

1. The 'w'rite command writes selected records to the 'tmp/' subdir within your current working directory, with a date/time stamp as shown above, and with 'W' suffix to identify as a Write command output.
   
   
2. Note tmp/... is in your current working directory (NOT /tmp). If ./tmp is not present uvhd will make it.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8K4. using 'uvhd' to investigate /var/log/wtmp

Examine records selected by Write

We can now examine the selected records as follows:

 uvhd tmp/wtmp_080817_153211W r384  <-- examine selected records (user 'uvadm')
 =================================

 uvhd filename=/home/uvadm/tmp/wtmp_080817_153211W
 options=r384 lastmod=2008081715 today=20080817153306 print=p1
 rec#=1 rcount=30 filesize=11520 recsize=384 fsize%rsize(remainder)=0
                      10        20        30        40        50        60
 r#        1 0123456789012345678901234567890123456789012345678901234567890123
           0 ........tty2............................2...uvadm...............
             0000800077730000000000000000000000000000300077666000000000000000
             70006F004492000000000000000000000000000020005614D000000000000000
                      ----- bytes 64-319 omitted to save space -----
         320 ....................A..H9.......................................
             000000000000000080004C843E00000000000000000000000000000000000000
             00000000000000006F0019489640000000000000000000000000000000000000

We can now use 'utmpdump' to display the selected records in user friendly format with the binary times converted to a readable format.

 utmpdump tmp/wtmp_080817_153211W
 ================================

 [7] [03974] [2   ] [uvadm   ] [tty2   [Mon Jul 21 10:37:05 2008 PDT]
 [7] [03936] [2   ] [uvadm   ] [tty2   [Tue Jul 22 07:39:50 2008 PDT]
 [7] [03974] [2   ] [uvadm   ] [tty2   [Wed Jul 23 10:07:30 2008 PDT]
         - - - - - 24 records omitted to save space - - - - -
 [7] [03971] [2   ] [uvadm   ] [tty2   [Fri Aug 15 06:32:59 2008 PDT]
 [7] [03973] [2   ] [uvadm   ] [tty2   [Sat Aug 16 08:06:10 2008 PDT]
 [7] [03973] [2   ] [uvadm   ] [tty2   [Sun Aug 17 04:48:37 2008 PDT]

linux Gazette article

This item (using 'uvhd' to investigate Unix/Linux system files) was published in the Linux Gazette in Sept 2008. See http://linuxgazette.net.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8L1. ADMjobs: Disc Monitoring Commands

df = Disc Free (Disc FileSystems)

 df          <-- disk free (Owen's HP XW9400)
 ===           - showing only Scsi Disk 'A' (omitting sdb,sdc,sdd)

 Filesystem             Size   Used  Avail Use% Mounted on
 =========================================================
 /dev/sda2              8.2G   4.4G   3.4G  57% /
 /dev/sda1              200M    30M   160M  16% /boot
 /dev/sda5               11G   436M   9.2G   5% /home
 /dev/sda6               11G   1.1G   8.7G  11% /home2
 /dev/sda7               11G   6.1G   3.7G  63% /home3
 /dev/sda8               11G   391M   9.3G   5% /home4
 /dev/sda9               11G   193M   9.5G   2% /home5
 /dev/sda10             4.1G   292M   3.6G   8% /var

du - Disc Usage

 du /home/uvadm/*    <-- disc usage for all subdirs in uvadm/...
 ================

      439k    archive
      132k    batDOS
      4.2M    bin
       --- 21 subdirs omitted ---
      7.5M    src
      8.2k    tmp
      3.3M    vsetest
      66M     total

statdir1 - script alternative to du

 statdir1 /home/uvadm   <-- Vancouver Utility script
 ====================       (more info than du)

 statdir1 - FileCounts & DiscUsage for SubDirs in ParentDir: /home/uvadm
 statdir1 uvadm >stats/uvadm.stats   Tue Sep  2 06:07:48 PDT 2008
 ===============================================================
 #1 Files=0000062 SubDirs=0003 KB=0000428 - /home/uvadm/archive
 #2 Files=0000031 SubDirs=0000 KB=0000128 - /home/uvadm/batDOS
 #3 Files=0000021 SubDirs=0000 KB=0004044 - /home/uvadm/bin
               - - - 21 subdirs omitted - - -
 #25 Files=0000057 SubDirs=0000 KB=0007312 - /home/uvadm/src
 #26 Files=0000003 SubDirs=0000 KB=0000016 - /home/uvadm/tmp
 #27 Files=0000188 SubDirs=0095 KB=0003156 - /home/uvadm/vsetest
 Total Files=2845, SubDirs=217, KB=63824, for ParentDir=/home/uvadm

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8L2. ADMjobs: System Information

free - display RAM memory free & used

 free     <-- system command to show internal memory free & used
 ====

              total       used       free     shared    buffers     cached
 =========================================================================
 Mem:       3983360    1059856    2923504          0      84444     460028
 -/+ buffers/cache:     515384    3467976
 Swap:      6144852          0    6144852

uname - Unix Name (System Info)

 uname -a   <-- display Unix/Linux system information
 ========     - OS version, etc

 Linux uvsoft4 2.6.18-92.1.10.el5xen #1 SMP x86_64 x86_64 x86_64 GNU/Linux
 =========================================================================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8M1. ADMjobs: Killing hung-up processes

ps & kill demo

Sometimes you need to kill a hung-up process. This is most easily done if you can login as root on another screen. Here is a demo:

 #1. Login as youself (uvadm in my case)

 #2. sleep 300      <-- run something
     =========

 #3. Login as root

 #4. ps -u uvadm    <-- display processes for user 'uvadm'
     ===========

   PID TTY          TIME CMD
  4845 tty2     00:00:00 bash
  6280 tty2     00:00:00 sleep

 #5a. kill 6280     <-- kill process hung up
      =========

 #5b. kill 4845     <-- OR kill the user's shell
      =========

Notes

1. Without root access, you could only kill your own jobs
   
   
2. If you did not run the job in the background, you would have to login on a 2nd screen to be able to kill your own job
   
   
3. OR you could put the job into the background via control-Z. Then you could do 'ps' to see process#, & then kill -9
   
   
4. Sometimes you need the '-9' option to kill & sometimes you don't ?
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8N1. ADMjobs: Messaging (wall, write, mail)

wall - broadcast a message to All logged on users

 wall    <-- initiate wall, type message, enter control-D to end
 ====
 Hello All users
 - this is a this is a 'wall' message from uvadm
 - testing wall for my documentation
 Thanks, Owen
 ^D

 Broadcast message from uvadm (tty2) (Tue Sep 2 12:04:57 2008)

write - send a message to logged on users

 write appsadm    <-- initiate write, type message, control-D to end
 =============
 Hello appsadm, this is a 'write' from uvadm
 - testing 'write' for my documentation
 Bye, Owen
 ^D

mail - mail a message to local or internet users

 mail -s'testing mail' appsadm <-- initiate mail, type message, control-D to end
 =============================
 Hello appsadm, this is mail from uvadm
 - testing 'mail' for my documentation
 Bye Owen
 ^D

1. You would not use this command line 'mail' program in place of the GUI Thunderbird, Outlook Express, etc.
   
   
2. 'mail' can get to the internet (if the unix/linux machine is connected) but many features of Thunderbird, etc are missing or inconvenient.
   

• address books, sent folders, local folders, etc
• need to remember to specify subject with '-s' option
• return address would not work & no option to specify

3. You might use 'mail' within a JCL/script to 'mail' an error-message to your home from an overnight job scheduled by 'cron'.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8O1. ADMjobs: Unix/Linux System Administration

'top' - system performance analysis tool

 top   <-- initiate 'top' (display shown below)
 ===

 top - 12:36:28 up  5:42,  6 users,  load average: 0.56, 0.22, 0.11
 Tasks: 169 total,   3 running, 166 sleeping,   0 stopped,   0 zombie
 Cpu(s):  0.4%us,  0.1%sy,  0.0%ni, 99.2%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st
 Mem:   3983360k total,  1816140k used,  2167220k free,   115316k buffers
 Swap:  6144852k total,        0k used,  6144852k free,  1146996k cached

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  7516 mvstest   25   0  4360  868  536 R   99  0.0   0:18.78 uvcopy (loop1)
  7517 vsetest   22   0  4360  892  552 R   47  0.0   0:06.57 uvcopy (loop2)
     1 root      15   0 10328  708  592 S    0  0.0   0:00.40 init
     2 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/0
     3 root      34  19     0    0    0 S    0  0.0   0:00.01 ksoftirqd/0
     4 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/0
     5 root      RT  -5     0    0    0 S    0  0.0   0:00.01 migration/1
     6 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/1
     7 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/1
     8 root      10  -5     0    0    0 S    0  0.0   0:00.06 events/0
     9 root      10  -5     0    0    0 S    0  0.0   0:00.00 events/1
    10 root      10  -5     0    0    0 S    0  0.0   0:00.00 khelper
    11 root      10  -5     0    0    0 S    0  0.0   0:00.00 kthread
    13 root      10  -5     0    0    0 S    0  0.0   0:00.00 xenwatch
    14 root      10  -5     0    0    0 S    0  0.0   0:00.00 xenbus
    17 root      10  -5     0    0    0 S    0  0.0   0:00.00 kblockd/0
    18 root      10  -5     0    0    0 S    0  0.0   0:00.00 kblockd/1
    19 root      20  -5     0    0    0 S    0  0.0   0:00.00 kacpid

notes

1. Before running 'top', I started the 2 jobs for users mvstest & vsetest, in order to show some processes consuming significant %CPU time.
   
   
2. The 2 uvcopy jobs are listed on page '8O3'. 'loop1' is a code loop & uses 99% of CPU. 'loop2' is performs I/O & uses less (47%) CPU since it is often waiting on I/O.
   
   
3. I ran the 2 load jobs & captured the top display as shown on the next page:
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8O2. ADMjobs: 'top' - system performance analysis

creating some load & capturing top display

 #1a. Login as mvstest

 #1b. uvcopy loop1       <-- run cpu bound job
      ============

 #2a. Login as vsetest

 #2b. uvcopy loop2       <-- run I/O bound job
      ============

 #3. Login as uvadm

 #4. top >tmp/top1       <-- run top & redirect output to a file
     =============
     --> q  <-- quit after 1 or 2 seconds

 #5. uvcopy unscreen1[,fili1=tmp/top1,filo1=tmp/top2]
     ================================================

 #6. vi tmp/top2         <-- inspect output
     ===========

See the 3 uvcopy jobs listed on the following pages:

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8O3. ADMjobs: 'top' - system performance analysis

jobs to create load for 'top' demo

 # loop1 - uvcopy job to do nothing but hang up in code loop
 #       - to test/demo 'top' performance analysis tool
 #
 # uvcopy loop1[,uop=t300]
 # =======================
 #
 opr='uop=t300 - option default'
 opr='    t300 - loop for 300 seconds'
 uop=q1t300    # option defaults
 @run
 # begin loop to test loop time reached
 # $time1 = time job starts (unix epoch time, seconds since 1970)
 # $time2 = time updated by the 'tim' instruction
 man20  tim                   update $time2 with current time
        mvn    $ca1,$time2    current time to work ctr
        sub    $ca1,$time1    - time job started
        cmn    $ca1,$uopbt    compare diff to option time ?
        skp<   man20
        eoj

 # loop2 - uvcopy job to generate I/O activity
 #       - to test/demo 'top' performance analysis tool
 #       - this job writes a specified no of records
 #       - writes a neutral translate table 256 bytes codes x'00' - x'FF'
 #       - output file at UVSI /h24/tmp/loop2_output
 #       - /h24 is an empty 35 gig file system (other than tmp subdir)
 #
 # uvcopy loop2[,uop=n4000000][,filo1=/h24/tmp/loop2_testfile]
 # ===========================================================
 #
 opr='uop=n4000000 - option default'
 opr='    n4000000 - write 4 million records (1 Gig)'
 uop=q1n4000000    # option defaults
 filo1=?/h24/tmp/loop2_testfile,rcs=256,typ=RSF
 @run
        opn    filo1               open the output file
 # begin loop to write records until spcfd# (option n) reached
 man20  put    filo1,$trt          write neutral translate table
        add    $ca1,1              count records written
        cmn    $ca1,$uopbn         reached spcfd# ?
        skp<   man20
        cls    filo1               close file
        eoj                        end job

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8O4. ADMjobs: 'top' - system performance analysis

job to remove escapes from top display

 # unscreen1 - remove screen control escape sequences
 #           - by Owen Townsend, UV Software, Sept 3/2008
 #
 # This job searches for the escape x'1B' start char & removes until 'm' or blank
 # I created this job so I could show screens in my text documentation
 # For example, the 'top' utility creates a screen loaded with escape sequences
 #
 # 1. top >tmp/top1
 #    --> q   <-- quit top
 #
 # 2. uvcopy unscreen1,fili1=tmp/top1,filo1=tmp/top2
 #    ==============================================
 #    - 2 lines of input/output shown below (escapes shown as '!')
 #
 # ![7m PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND ![0;10m![39;49m![K
 # ![0;10m 1 root 15 0 10328  708  592 S   0  0.0   0:00.40 init ![0;10m![39;49m
 #
 #   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 #     1 root      15   0 10328  708  592 S    0  0.0   0:00.40 init
 #
 was=a5000b5000
 fili1=tmp/top1,rcs=1024,typ=LST
 filo1=tmp/top2,rcs=1024,typ=LSTt
 @run
        opn    all
 # begin loop to get/process/put records until EOF
 man20  get    fili1,a0(1024)       get next input record
        skp>   man90                (cc set > at EOF)
 #
 # copy input area 'a' to output area 'b' removing escape sequences
        clr    b0(1024),' '         clear output area
        mvn    $ra,0                init rgstr 'a' ptr to area 'a'
        mvn    $rb,0                init rgstr 'b' ptr to area 'b'
 #
 # begin loop to copy data until escape x'1B' found
 man30  mvue3  bb0(1024),aa0,x'1B'  move until next escape found
        skp!   man40
        scne1m aa0(20),'m '         scan to end code 'm' or ' '
        add    $ra,1                bypass the 'm' or ' '
        skp    man30                repeat loop til no more escapes found
 #
 man40  put    filo1,b0(1024)       write out result
        skp    man20                return to get next line
 #
 # EOF - close files & end job
 man90  cls    all
        eoj

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8P1. ADMjobs: meminfo - system memory & usage

You can obtain system memory & memory usage by displaying /proc/meminfo. Here is the results from my Red Hat Linux Enterprise 5.1 installed on my HP xw9400 workstation with 4 GB memory:

 cat /proc/meminfo
 =================

 MemTotal:      3983360 kB
 MemFree:       2802296 kB
 Buffers:        143692 kB
 Cached:         502656 kB
 SwapCached:          0 kB
 Active:         433628 kB
 Inactive:       454720 kB
 HighTotal:           0 kB
 HighFree:            0 kB
 LowTotal:      3983360 kB
 LowFree:       2802296 kB
 SwapTotal:     6144852 kB
 SwapFree:      6144852 kB
 Dirty:             104 kB
 Writeback:           0 kB
 AnonPages:      242104 kB
 Mapped:          67024 kB
 Slab:            90192 kB
 PageTables:      22640 kB
 NFS_Unstable:        0 kB
 Bounce:              0 kB
 CommitLimit:   8136532 kB
 Committed_AS:  2032136 kB
 VmallocTotal: 34359738367 kB
 VmallocUsed:      3740 kB
 VmallocChunk: 34359734559 kB

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8Q1. ADMjobs.doc - Applications Administration

ssmtp - send email from scripts

The objective here is to send email from a script, scheduled by cron at night, to managers at home, to alert them of serious errors.

In this section, we will show you how to use 'ssmtp' to send 1 line error messages from cron scripts to managers at home using their internet email addresses (vs their unix/linux system login accounts, which are the usual destinations of the 'mail' utility without special 'sendmail' configurations).

It is easy to send mail from a unix/linux script to any other user account on the unix/linux system, but this mail will not be delivered to the internet without special configuration of the mail transport agent (sendmail by default).

mail/sendmail vs ssmtp

We do not want our implementation of 'ssmtp' to interfere with mail & sendmail. 'mail' (user interface) passes the mail to 'sendmail' (transport agent), which (without special configuration), delivers the mail ONLY to other login user accounts on the local unix/linux system, and NOT to the internet.

In Part_5, we illustrated how to schedule JCL/scripts with 'cron', which will automatically mail the console messages to the owner of the 'crontab' file, which we suggest should be 'appsadm'.

Each morning appsadm can read his mail to see the console log from nightly jobs. Pages '5I1' - '5K4' show you how to save the cron mail as date/time stamped files.

script 'ssmtp1' calling ssmtp

The 'ssmtp' utility is a simple way to send mail to the internet (vs mail & sendmail which send only to local users without complex configurations).

But to use 'ssmtp' directly you have to specify the 'To: ..., From: ...,& Subject: ...' lines (which are automatically created by the 'mail' utility).

We provide script 'ssmpt1' which creates these lines, appends the 1 line error message,& sends email to your ISP. Our example will send the email thru a gmail account to any desired email address on the internet.

Some ssmtp setups show you how to disable 'sendmail' & have 'mail' call 'ssmtp', but we need the standard mail/sendmail to deliver console logs from cron jobs to appsadm (as described above & on pages '5I1' - 5K4).

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8Q2. ssmtp - send email from cron scripts to mgrs at home

download/install 'ssmtp'

1. Login as root --> /root (homedir on Red Hat systems)
   
   
2. mkdir ssmtp <-- make subdir to store rpm pkg
   
   
3. Switch to a GUI screen & use FireFox browser to search the internet for the latest version of 'ssmtp' compatible with your operating system.
   

 http://www.rpmfind.net    <-- a good site for Red Hat 'rpm' packages
 ======================

 ssmtp rpm x86_64   <-- keywords I used to find pkg for my 64 bit Red Hat OS
 ================

 ssmtp-2.61.11.9.fc11.x86_64.rpm  <-- latest version found (as of Oct 2009)
 ===============================    - I downloaded to /root/ssmtp

4. switch back to your root command line login screen
   
   
5. cd /root/ssmtp <-- change into subdir where you downloaded rpm
   

 6. rpm -ivh ssmtp-2.61.11.9.fc11.x86_64.rpm
    ========================================
    - install downloaded package
    - stores 'ssmtp' executable in /usr/sbin/ssmtp
    - stores config file in /etc/ssmtp/ssmtp.conf

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8Q3. ssmtp - send email from cron scripts to mgrs at home

ssmtp.conf config file for ssmtp

We supply a minimal config file at /home/uvadm/sf/adm/ssmtp.conf, and I suggest you simply copy it to /etc/ssmtp/ssmtp.conf. You can use 'man ssmtp.conf' to see the descriptions that we omit from our minimal file).

 # /etc/ssmtp.conf - config file for sSMTP sendmail.
 # - ssmtp may be called from 'cron' jobs to email errors to managers at home
 # - minimum config for tests by Owen Townsend, UV Software, Oct 2009
 # - see ssmtp.conf(5) man page for descriptions of all available options.
 # - I downloaded/installed ssmtp on my Red Hat Enterprise 5.1 as follows:
 #   rpm -ivh ssmtp-2.61-11.8.el5.x86_64.rpm
 # - see www.uvsoftware.ca/admjobs.htm#8Q1 for 'ssmtp' download/install
 #   & for related script 'ssmtp1' that makes ssmtp easy to use
 #
 root=postmaster
 mailhub=smtp.gmail.com:587
 FromLineOverride=YES
 UseSTARTTLS=YES
 AuthUser=owen.townsend88@gmail.com
 AuthPass=owen1234     # password changed

modifying /etc/ssmtp/ssmtp.conf

I setup a 'gmail' account with userid 'owen.townsend88@gmail.com' & password 'owen1234'. You could setup a similar account at gmail (or at your own ISP, if you change 'smtp.gmail.com:587' to 'smtp.YOURISP.587').

1. Login root
   

 #3. cp /home/uvadm/sf/adm/ssmtp.conf /etc/ssmtp/. <-- copy supplied config file
     =============================================

 #4. vi /etc/ssmtp/ssmtp.conf  <-- change mailhub, AuthUser, AuthPass as desired
     ========================

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8Q4. ssmtp - send email from cron scripts to mgrs at home

ssmtp1 script to call ssmtp

 # ssmtp1 - call ssmtp to send email with 1 line message
 #        - by Owen Townsend, UV Software, October 2009
 # - intended for use by 'cron' scripts to send ERR msgs to managers at home
 # - destination email adrs hard-coded in this script for easy change in 1 place
 # - may send errmsg to multple destinations to ensure managers alerted
 # - I downloaded/installed ssmtp on my Red Hat Enterprise 5.1 as follows:
 #   rpm -ivh ssmtp-2.61-11.8.el5.x86_64.rpm
 # - see more at www.uvsoftware.ca/admjobs.htm#8Q1
 #
 msg="$1"
 if [[ -z "$msg" ]]; then
    echo "usage: ssmtp1 \"--- 1 line message ---\""
    echo "       =============================="
    echo "       - arg1 not specified"
    exit 99; fi
 #
 # concat (email headers + message) into temp file for <input to ssmtp
 # note - must have blank line after email headers & prior to arg1 message
 msgfile=tmp/ssmtp1_$(date +%y%m%d_%H%M%S)
 cat >$msgfile <</*EOF
 To: owen.townsend88@gmail.com
 From: owen@uvsoftware.ca
 Subject: Error detected by unattended cron script

 $msg
 /*EOF
 #
 ssmtp owen.townsend88@gmail.com <$msgfile
 ssmtp owen@uvsoftware.ca        <$msgfile
 # may send to multiple managers to ensure error message is received
 exit 0

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8Q5. ssmtp - send email from cron scripts to mgrs at home

test ssmtp1 - from command line

Here is how I tested the 'ssmtp1' script. From the listing above, you can see that my version of the script sends mail to both 'owen.townsend88@gmail.com' & 'owen@uvsoftware.ca'.

1. Login as any user whose profile provides access to Vancouver Utilities
   

 #2. ssmtp1 "--- testing ssmtp1 from command line ---"
     =================================================

3. I switched to a GUI screen & internet browser on same or any computer. I logged in to my gmail account & verified I got the email.
   
   
4. I switched to a GUI screen & my Thunderbird email reader & verified I got the email.
   

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

8Q6. ssmtp - send email from cron scripts to mgrs at home

test ssmtp1 - from another script

Here is how I tested calling 'ssmtp1' from another script, such as a JCL/script that might be scheduled by 'cron' at night. I will use 'jar100.ksh' which is a demo job listed in full at MVSJCL.htm#1C2.

Here are the last few lines containing the 'Normal' & 'Abnormal' job termination points. These lines are common to all JCLs converted to scripts by the JCL converter. I have inserted the 'ssmtp1' script call as marked by '-->'.

      #!/bin/ksh
      # jar100.ksh - accounts receivable processing
      #                              ---- 25 lines omitted ----
      LCC=$?  <-- capture COBOL return code, set to 99 to force failure
      #                               ---- 5 lines omitted ----
      S9000=A
      jobend51
      logmsg2 "JobEnd=Normal, StepsExecuted=$XSTEP, LastStep=$JSTEP"
      exit 0
      S9900=A
      logmsg2 "ERR: Terminated Abnormally,JCC=$JCC,Step=$JSTEP" RV ACK
 -->  ssmtp1  "$JOBID2 Terminated Abnormally, step=$JSTEP, ERRcode=$JCC"
      #=================================================================
      jobabend51
      exit $JCC

I used the 'mvstest' user & test/demo files as documented at MVSJCL.htm#1A1.

 #1. Login as 'mvstest' --> /home/mvstest

 #2. cdl --> /home/mvstest/testlibs (cdl is alias 'cd $RUNLIBS')

 #3. vi jcls/jar100.ksh  <-- edit jar100.ksh 2 changes as follows:
     ==================
 #3a. ssmtp1 ...   <-- insert call to 'ssmtp1' as shown in listing above
 #3b. LCC=99       <-- change COBOL return code to 99 to force failure
 #3c. :wq!         <-- write & quit editor

 #4. jar100.ksh    <-- run the job
     ==========

 #5. switch to a GUI internet browser & login to gmail account to see if
     error message sent & receieved OK.

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

Permuted Index of Keywords from ** Headings **

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z 

Goto:   Begin this doc ,  End this doc ,  Index this doc ,  Contents this library ,  UVSI Home-Page

A  B  C  D  E  F  G  H  I  J  K  L  M  N  O  P  Q  R  S  T  U  V  W  X  Y  Z