The RMF Monitor III data gatherer writes records (sets of samples) it uses for creating interactive RMF reports to an in-storage buffer, or optionally, to pre-allocated VSAM data sets. Without the VSAM data sets, the data is overwritten as soon as the buffer is filled. By defining enough VSAM data sets, you can save large amounts of information, and you can reuse the VSAM data sets as RMF continuously records data over time.
The RMF Monitor III Data Index displays information about the recorded data available to RMF and the datasets used to contain them. You can get to Data Index by selecting S on the RMF Monitor III Primary menu, and then D on the Sysplex Report menu, or you can enter the DI command at the Primary Menu:
As shown above, we see that less than one day’s worth of records are saved in the two VSAM data sets we previously configured for system JA0 in our sysplex, in addition to about 3 minutes in the In-storage buffer.
Sometimes we may want to see short term changes over more than 1 days’ worth of activity but from the real time perspectives that Monitor III provides. For example does a spike in CPU for a specific address space happen at the same time every day for the last week?
We’ve found that one day’s worth of records is not enough to allow for this type of trend analysis using RMF Monitor III, and users need to dump from SMF offload data sets or SMF log streams and create RMF Postprocessor reports if they want to analyze performance records for two or more days. We decided to modify our configuration to allow at least 5 days’ worth of Monitor III records. Here’s how we did this.
First, we checked the current VSAM data set definitions for all of our systems; the following is a snapshot from system JA0 (other systems are similar). We can see the current VSAM data sets are 45000 tracks which is 3000 cylinders. From the RMF MonIII Data Index, we know that this can hold about14 hours of JA0’s data.
Setting up and controlling RMF data set recording is documented in several places in the z/OS RMF bookshelf in the z/OS Knowledge center, with the best reference for this exercise in the Administration chapter of the z/OS RMF Data Gatherer User’s Guide. There we found that RMF lets you define up to 100 data sets for use with the data gatherer. RMF can also keep at most 1100 sets of samples in one data set, regardless of the size of data set. And the size of each sample differs depending on system activity. So we needed to trim the data set size according to our environment. To discover a data size that worked for us, we changed JA0 VSAM data sets to different sizes and monitored the time duration of the records that each data set could hold. The following table summarizes the different time durations we saw for each data set size for JA0:
As shown, when the data set is defined with more tracks (in this case, 50000 tracks), it can hold records for a longer duration of time. Smaller size data sets below 45000 tracks kept records for a shorter time. Since a dataset can only hold up to 1100 sets of samples, the samples in a 45000 tracks data set will not reach the 1100 limit, and the data set with 50000 tracks will hold records for a longer duration of time. There's no waste for the 45000 tracks data sets to hold the samples, so we continue to use the size of 45000 tracks for the VSAM data sets definition.
If we want to keep roughly 5 days of data, we can calculate how many new VSAM data sets need to be defined: (5 days * 24 hours / 14) - 2 ~= 6
We then defined DATASET3/4/5/6/7/8 for each of our systems.
Below is the procedure we used to define 6 more VSAM data sets and enable them in our sysplex:
Step 1: Defined 6 more VSAM data sets, each with 3000 cylinders for each system using IDCAMs:
//DFRMFJ1 JOB ' ',MSGCLASS=H,
// CLASS=A,MSGLEVEL=(1,1),NOTIFY=&SYSUID
//DFMANDS EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=A
//SYSIN DD *
DEFINE CLUSTER (NAME(RMF.MONITOR3.JA0.DATASET3) -
VOLUME(RMF003) NUMBERED -
TRACKS(45000) REUSE -
RECORDSIZE(32752,32752) -
CISZ(32768) -
BUFFERSPACE(65536) SHAREOPTIONS(2,3)) -
DATA(NAME(RMF.MONITOR3.JA0.DATASET3.DATA) CISZ(32768) BUFSP(65536))
We repeated this DEFINE statement for each of the other 5 VSAM data sets for JA0, and
we defined 6 more for all systems using the same naming convention of RMF.MONITOR3.&SYSNAME..DATASET*.
Note: It is recommended that adjacent data sets are put on different volumes.
Step 2. Modified the RMF monitor III parameter member SYS1.PARMLIB(ERBRMFxx) to specify the new VSAM data sets for Monitor III gather session:
DATASET(ADD(RMF.MONITOR3.&SYSNAME..DATASET1)) /* previously defined */
DATASET(ADD(RMF.MONITOR3.&SYSNAME..DATASET2)) /* previously defined */
DATASET(ADD(RMF.MONITOR3.&SYSNAME..DATASET3)) /* newly added */
DATASET(ADD(RMF.MONITOR3.&SYSNAME..DATASET4)) /* newly added */
DATASET(ADD(RMF.MONITOR3.&SYSNAME..DATASET5)) /* newly added */
DATASET(ADD(RMF.MONITOR3.&SYSNAME..DATASET6)) /* newly added */
DATASET(ADD(RMF.MONITOR3.&SYSNAME..DATASET7)) /* newly added */
DATASET(ADD(RMF.MONITOR3.&SYSNAME..DATASET8)) /* newly added */
“Add” allows RMF to use the specified data sets to record sampled data.
Step 3. Restarted RMF Monitor III instances to enable the configuration:
-RO *ALL,F RMF,STOP III
ERB803I III: MONITOR III DATA SET SUPPORT TERMINATED
ERB102I III: TERMINATED
…..
-RO *ALL,F RMF.RMF,START III,MEMBER(03)
ERB115I START RMFGAT MONITOR III SESSION III
ERB105I III: DATA GATHERER ACTIVE
ERB100I III: ACTIVE
…..
We checked current configuration of Monitor III with its DISPLAY command:
-F RMF,D III
……..
DATASET(SWITCH) -- MEMBER
DATASET(ADD(RMF.MONITOR3.JA0.DATASET9)) -- MEMBER
DATASET(ADD(RMF.MONITOR3.JA0.DATASET8)) -- MEMBER
DATASET(ADD(RMF.MONITOR3.JA0.DATASET7)) -- MEMBER
DATASET(ADD(RMF.MONITOR3.JA0.DATASET6)) -- MEMBER
DATASET(ADD(RMF.MONITOR3.JA0.DATASET5)) -- MEMBER
DATASET(ADD(RMF.MONITOR3.JA0.DATASET4)) -- MEMBER
DATASET(ADD(RMF.MONITOR3.JA0.DATASET3)) -- MEMBER
DATASET(ADD(RMF.MONITOR3.JA0.DATASET2)) -- MEMBER
DATASET(ADD(RMF.MONITOR3.JA0.DATASET1)) -- MEMBER
DATASET(WHOLD(128)) -- MEMBER
CYCLE(1000) -- MEMBER
MEMBER(03) -- COMMAND
Step 4. Checked the VSAM data set usage in
RMF -> 3.Monitor III -> DI
Here’s the screenshot of DI after several days:
We can see that the 8 VSAM datasets are now in use for system JA0 with about 5 days’ worth of records. This now allows us to monitor resources from Monitor III for longer trends.
If you actively use RMF Mon III for real time analysis, and you have the DASD storage available to allow RMF Mon III to continuously record data for a longer period of time, you may find it worthwhile to continue using its familiar interface to gain a little more insight by extending its capabilities.
Please contact us if you have any questions or comments related to our RMF configuration.
Authors:
Jing Wen Chen(bjchenjw@cn.ibm.com)
Kieron Hinds(kdhinds@us.ibm.com)
Lora Milczewski(loran@us.ibm.com)
Yu Mei Dai(dyubj@cn.ibm.com)
Zhao Yu Wang(wangzyu@cn.ibm.com)