By Oneil Richardson

In this blog I'll cover a recent issue I found when assisting a customer to set up short historical collection for their Omegamon XE for DB2 Performance z/OS agents. We found that even though historical collection had been enabled in the TEMS it was not possible to select historical data for the agent to be displayed in the TEP workspaces, nor was it possible to see real-time data. After some further digging it was found that unique to the z/OS KD5 agent are some historical collection parameters that also need to be set to match those at the TEMS for historical collection to work and for the TEP to be able to correctly display real-time data. This blog covers the investigative steps and settings that were changed to correctly enable short term historical collection for this agent.

Initially the problem was reported as a TEPS issue; the customer was not able to see real time data from the agent. However when they tried to view 'real-time + last X hours' this option was inactive and if the option 'last X days' was selected they could see all historical data minus the last 24 hours. (in both cases 'X' is a selectable value) An SQL query directly to the warehouse tables showed that the required data was present. After confirming the customer had not customized the default workspace views I checked if the short term historical collection had been correctly enabled for the correct attributes. The customer had the following configuration...

Collection Location: TEMS

Collection Interval: 5 mins

Warehouse Interval 5 mins

I checked the default location for the short-term historical collection binary at the TEMS ($CANDLEHOME/TEMS/*), but was unable to see the corresponding binary (KDP_DB2MSG) relating to the attribute group, so I suspected this was the reason the workspaces relying on historical data were not populated. However, as data was being warehoused this lead me to believe that the binary must exist at the TEMA agent.

This was backed up by the following output in the TEPS logs once additional tracing (see bottom for settings) had been enabled to view the query SQL and resulting rows found...

KfwServices: Fri Apr 29 16:54:21 MSK 2016: TEMS(388695): SELECT DB2MSG.WRITETIME, DB2MSG.ORIGINNODE, DB2MSG.QW0197ID, DB2MSG.DB2ID,

DB2MSG.QW0197TX, DB2MSG.TIMESTAMP, DB2MSG.QW0197AL FROM KDP.DB2MSG AT('MYTEMS:CMS' ) HISTORY() WHERE (

(DB2MSG.WRITETIME >= ?) AND

(DB2MSG.WRITETIME <= ?) ) AND (SYSTEM.PARMA('NODELIST','MYDB2_AGT1',18) )

KfwServices: Fri Apr 29 16:54:21 MSK 2016: TEMS(388695): Values: '1160418165421000' '1160419165421000'

...

..

KfwServices: Fri Apr 29 16:54:21 MSK 2016: TEMS(388695): Rows returned: 0

Checking this theory out it was found that the binary did indeed exist at the TEMA agent. After discussion on this issue further with the z/OS DB2 agent team the following information was discovered...

- The z/OS DB2 agent is installed and configured using PARMGEN

- At the time this agent was installed the following PARMGEN settings were entered

KD5_PD_HISTCOLL_DATA_IN_TEMS_STC N

KD5_PD_HISTCOLL_DATA_IN_AGT_STC Y

This was in effect setting up historical collection at the TEMA agent and contradicting the historical collection settings that were enabled at the TEMS (where the short term history location was configured to be on the TEMS). It was this configuration that was causing the unexpected behaviour at the TEPS, as the query was expecting to find the short term history at the TEMS when it instead it existed at the TEMA. To resolve the problem, the customer decided to modify the z/OS DB2 TEMA collection settings to match the historical collection settings at the TEMS

KD5_PD_HISTCOLL_DATA_IN_TEMS_STC Y

KD5_PD_HISTCOLL_DATA_IN_AGT_STC N

After restarting the agent, the customer was able to see collection take place at the TEMS and real time data display correctly in the TEP. At this point it was noted that the 'Realtime + X hours' view was still inactive.This was confirmed to be working as designed based on the type of display chart the workspace was using...

The following information from p.214 of the 6.3 TEP User guide states: Take these steps to broaden the time range of data beyond the current data samplings. Procedure 1. Open the workspace containing the chart, table, or relational table-based topology view where you want to see historical data. 2. Click Time Span in the view's toolbar. 3. Select a time frame: Real time plus Last _ Hours (enabled for bar, plot, and area charts only), Last _ Hours (or Days, Weeks, or Months, if the data is warehoused), or Custom. (In this case the workspace in question was using a line graph, so the option was not selectable)

General Information for enabling historical collection at the TEMS versus enabling at the TEMA...

The performance implications for the data gathering step in large scale environments with several hundred agents connected to one TEMS are: - Large chunk of data warehousing from one box vs. much smaller chunk of data warehousing from multiple boxes. - All agents write to one file for each attribute groups. Slow workspace response time for shot-term historical data. - Demands large disk storage, CPU and memory usage of the TEMS. - Drastically lowers the number of agents that can be managed by the TEMS - Loss of access to the short term historical data during a failover to a secondary TEMS.

TEPS Trace Settings used:

RAS1=ERROR (unit:ctdatabus in,er)(unit:ctpub in,er)(unit:ctsql in,er)(unit:ctattribute detail)