Housekeeping of an IBM Security Identity Governance and Intelligence (IGI) deployment is normally dependent on the operational practices of where it’s deployed. The following article is an example of a set of housekeeping scripts that were developed by the IBM Security Product Professional Services (PPS) team in Germany to support a customer deployment. They are shared here as a sample and could easily be adapted to any IGI deployment (but are written as shell scripts for a Unix/Linux system).
The article will discuss the scripts first, then define the implementation.
The scripts, in tar format, can be downloaded here.
NOTE: These scripts are provided as-is, and are not supported by IBM. They do leverage standard IBM product utilities (such as the Directory Server utilities) which are supported.
The Housekeeping Scripts
There are four scripts in the tar file;
- igiHousekeepingDaily.sh – this is the main script that is run and it orchestrates the different tasks
- igiva-setEnv.sh – this defines the environment variables used by the other scripts
- igiva-operateServer.sh – this runs the start/stop commands for the IGI application on the virtual appliance
- db2ReorgIGIDb.sh – this performs the DB2 runstats and reorg commands
These are explored in the following sections.
The igiHousekeepingDaily.sh Script
This is the main script, the one that is run to perform the housekeeping tasks.
The script will do the following:
- Stop the IGI server – this will leave the VA and DB2 running, just stop the IGI and Broker applications. It uses the igiva-operateSever.sh script to do this.
- Stop the IGI directory server – this will stop the TDS instance used by the Broker as a cache. It uses the ibmslapd command (with a -k argument) to do this.
- Export the IGI directory server data – this will export all TDS data using an idsdb2ldif command to a backup file.
- Restart the IGI directory server – using the ibmslapd command.
- Backup the IGI DB2 – this section will close down any application connections to the DB (“db2 force applications all” command, a few times to make sure) then backup the database (“db2 backup db” command).
- Delete obsolete IGI audit records – this will clean up unwanted entries from the audit table, using the “db2 delete from igacore.event_account where state = 1” query.
- Run a DB2 runstats and reorg – this will update the DB2 statistics on the DB and do a reorg using the db2ReorgIGIDB.sh script. NOTE – this should not be neccessary if using DB2 10.5 or later as it should be doing automatic tuning. You may want to consider disabling this part of the script if your DB is autotuning. As always, check the Performance Tuning Guide.
- Restart the IGI server – this will use the igiva-operateServer.sh command to restart the IGI and Broker applications.
- Cleanup old backup and log files – this will remove any backup files more than three (3) days old and any log files (from these scripts) more than seven (7) days old.
It needs to be run as root. It is designed to work on a standard install of IGI, DB2 and TDS (i.e. some command filepaths are hardcoded) and use DB2 commands.
If you are happy with the flow above, you do not need to edit this script for your specific environment.
The igiva-setEnv.sh Script
This script contains all of the environment variables used by the other scripts. You will need to modify this to suit your environment.
The shipped script looks like this:
#!/bin/sh
#
#set -x
export IGIHOST=192.168.42.61:9443
creds="admin:Passw0rd!"
export CONF1="curl -s -k -H Accept:application/json -H Content-type:application/json --user $creds --data-binary"
export CONF2="curl -s -k -H Accept:application/json -H Content-type:application/json --user $creds"
export CONF3="curl -s -k -H Accept:application/json --user $creds"
export SERVERURL=https://$IGIHOST
export RUNUSER=root
HOSTNAME=`hostname -s`
LOGDIR=/tmp
LOGPREFIX=${LOGDIR}/IGIHousekeeping.$HOSTNAME
LOGPREFIX2=${LOGDIR}/IGIHousekeeping.$HOSTNAME.returncode
export LOGFILE=${LOGPREFIX}.`date +"%Y%m%d%H%M"`.log
export LOGFILE2=${LOGPREFIX2}.`date +"%Y%m%d%H%M"`.log
export IGIBKUPDIR=/var/igibkups
export IGILDAPBACKUP=$IGIBKUPDIR/igi_ldap/IGILdapBackup.$HOSTNAME.`date +"%Y%m%d"`.ldif
export IGILDAPINST=igildap
export IGIDBINST=igiinst
export IGIDB=IGI_DB
export IGIDBBACKUP=$IGIBKUPDIR/igi_db/
# export smtpGW=192.168.42.69
You will need to set the following variables to suit your envrionment:
- IGIHOST – the hostname and port (note this is the VA/management hostname or IP and port, not the IGI application one)
- creds – these are the creds to get into the VA LMI/CLI
- LOGDIR – the location of the script execution logs
- IGIBKUPDIR – location for all of the backup files the script will generate
- IGILDAPBACKUP – the filepath and filename structure for the LDIF backups for the IGI directory
- IGILDAPINST – DB2 instance name for the LDAP
- IGIDBINST – DB2 instance name for the DB2
- IGIDBBACKUP – the filepath for the DB2 backups
There should be no need to change other settings.
The igiva-operateServer.sh Script
This simple script is used to stop, start and restart the IGI/Broker applications.
It uses a command and arguments specified in the igi-setEnv.sh script;
CMDOUT=$($CONF2 -X POST $SERVERURL/widgets/server/$operation/$ServerID)
This uses curl to send a REST API call to stop/start/restart the application server.
You do not need to modify this script.
The db2ReorgIGIDb.sh Script
This script will perform a series of DB2 operations on the IGI database:
- Revalidate the DB with a “CALL SYSPROC.ADMIN_REVALIDATE_DB_OBJECTS(NULL, NULL, NULL);” command
- Reorg all tables
- Reorg all indexes for tables
- Perform a reorgchk to update statistics on all tables
- Peform a runstats for all application tables (i.e. not SYS% tables)
- Rebind all packages with a db2rbind (not that the IGI DB of “igi_db” is hardcoded here)
This script is called from the main script with the “su – $IGIDBINST -c “$IGIBKUPDIR/db2ReorgIGIDb.sh” command. Note that it su’s to the IGIDBINST (i.e. instance owner) and runs this script from the backup directory you have defined. As the main script is run as root, there is no problem with su’ing to the DB instance owner.
This script could be placed anywhere on the database server, but if you put it elsewhere you will need to update this script.
Installing, Configuring and Running the Scripts
This section will look at installing, configuring and running the scripts.
Prereqs and Script Installation and Configuration
To install and run the scripts you will need:
- Somewhere to run the scripts from, such as a folder where you store all housekeeping scripts
- A backup directory that will hold the DB2 script and all the backups. It’s recommended to have the same common IGI backup directory, but separate directories for the DB and TDS backups
- A logging directory (like /tmp)
- The ability to run the script as root, preferably in a cron job
- Access to all of the IGI arguments set in the igi-setEnv.sh script (see above).
Installation is straightforward;
- Copy the tar file to the DB server and untar it
- Copy/move the igiHousekeepingDaily.sh, igiva-operateServer.sh and igiva-setEnv.sh scripts to the directory you will run them from, and set file permissions as needed
- Copy/move the db2ReorgIGIDb.sh script to the (new) backup directory and set any file permissions needed
Finally you need to edit the igiva-setEnv.sh script to match your IGI environment. As mentioned earlier you will probably need to set; IGIHOST, creds, LOGDIR, IGIBKUPDIR, IGILDAPBACKUP, IGILDAPINST, IGIDBINST and IGIDBBACKUP.
Running the Scripts
Running the scripts is trivial – login or su to root and run the igiHousekeepingDaily.sh script. If you have configured the environment variables correctly it should run through the steps outlined above.
The script will produce two log files; a summary log and a detailed log.
-rw-r--r-- 1 root root 707 Nov 28 03:12 IGIHousekeeping.igidb.returncode.201811280304.log
-rw-r--r-- 1 root root 4074825 Nov 28 03:12 IGIHousekeeping.igidb.201811280304.log
In the example above the IGIHouskeeping.igidb.returncode.<datetime>.log file is the summary log and the IGIHouskeeping.igidb.<datetime>.log file is the detailed log.
The summary log shows the major steps and return codes. For example:
2018.11.28 03:04:35 1 stop IGI server (leave DB running) +++ exit code: 0
2018.11.28 03:04:36 2 stop IGI TDS server (leave TDS DB running) +++ exit code: 0
2018.11.28 03:04:39 3 export IGI LDAP data for backup purpose -k=seed t=salt +++ exit code: 0
2018.11.28 03:04:41 4 restart IGI TDS server +++ exit code: 0
2018.11.28 03:06:09 8 run backup on IGI DB +++ exit code: 0
2018.11.28 03:06:10 9 delete obsolete audit events +++ exit code: 0
2018.11.28 03:11:35 10 run IGI DB runstat and reorg +++ exit code: 0
2018.11.28 03:12:41 11 restart IGI server +++ exit code: 0
2018.11.28 03:12:41 13 removing old DB backup files +++ exit code: 0
2018.11.28 03:12:41 14 removing old log files +++ exit code: 0
The detailed log includes websphere, db2 and slapd logging information.
This concludes the blog on the IGI housekeeping scripts. Feel free to leave comments or questions below.