MQ

A first look at MQ Resource Usage statistics handling using Java

MarkBluemel |Mar 1 2017 Updated

Context

There is a new mechanism in MQ Version 9 for making resource usage information available - a tree of Topics to which the information is published.

This blog introduces the mechanism, while the detailed "documentation" for it is in a sample program (in C) - amqsrua.

Recently I was contacted with a query on how to access the data in Java, specifically for the "CPU/QMgrSummary" topic - some of the "default" assumptions about how to interpret data in PCF response messages don't apply in this context.

As this area was new to me, I did some rapid scanning of the C sample, wrote some simple Java programs to explore the topic tree and put together a basic sample.

I really need to do a more detailed examination of this mechanism and write a comprehensive sample, equivalent to the C sample, but in the meantime here's a summary of what I found and what I produced.

NOTE: This is probably based on incomplete understanding, so I reserve the right to come back and edit this post or produce new and improved posts on the subject :-)

The Crucial Point

The most important thing to understand here is that the resource usage statistics topics are dynamically defined.

The topics are structured into classes (for example "CPU", "DISK") within which there are types (for example "CPU" currently has types called "SystemSummary" and "QMgrSummary").

Within a class and type there will be various usage statistics published, for example the "CPU/QMgrSummary" topic currently has statistics such as "User CPU", "System CPU", "RAM used".

Note my use of "currently" above - this can change over time. So we do not want to hard code much...

The Topics are organised into a self-describing form, under a hierarchy based at "$SYS/MQ/INFO/QMGR/%s/Monitor/METADATA/", where "%s" is the Queue Manager name.

A Quick Look At The Metadata

A good way to get an initial understanding of the hierarchy, apart from by perusing the C sample program, is by running it in debug mode (use the "-d 1" option).

Here's a small part of the output of such a run showing the data relating to "CPU/QMgrSummary" :-

Subscribing to topic: $SYS/MQ/INFO/QMGR/QM_T1/Monitor/METADATA/CLASSES
0000        CPU Platform central processing units
0001       DISK Platform persistent data stores
0002    STATMQI API usage statistics
0003      STATQ API per-queue usage statistics
Subscribing to topic: $SYS/MQ/INFO/QMGR/QM_T1/Monitor/METADATA/CPU/TYPES
0000 0000 SystemSummary CPU performance - platform wide
0000 0001 QMgrSummary CPU performance - running queue manager
...
Subscribing to topic: $SYS/MQ/INFO/QMGR/QM_T1/Monitor/METADATA/CPU/QMgrSummary
0000 0001 0000 10000 User CPU time - percentage estimate for queue manager
0000 0001 0001 10000 System CPU time - percentage estimate for queue manager
0000 0001 0002 1048576 RAM total bytes - estimate for queue manager

We see from the "CLASSES" topic that the "CPU" class is class 0, DISK is class 1 etc...

From the "CPU/TYPES" topic we see that "CPU/SystemSummary" is class 0, type 0 while "CPU/QMgrSummary" is class 0, type 1.

Moving on to "CPU/QMgrSummary", this is where the actual statistics are defined - for each statistic we have five pieces of information :-

1. The class identifier - because these are CPU statistics, the class is 0.
2. The type identifier - because these are "CPU/QMgrSummary" statistics the type is 1.
3. An identifier for the statistic - User CPU time has identifier 0, System CPU time has identifier 1, and so on.
4. The datatype of the statistic. All statistics are presented as 32-bit or 64-bit integers (PCF type MQCFIN or MQCFIN64) but may need to be interpreted in various ways.
   In the current case User and System CPU time have datatype 10000 (MQIAMO_MONITOR_PERCENT) which means they will be a value from 0 - 10000, which needs dividing by 100 to give a percentage figure to 2 decimal places, while RAM has datatype 1048576 (MQIAMO_MONITOR_MB) which means it represents a figure in megabytes.
5. A description for the statistic. (Note: There is internationalization available  - this example has taken the default hierarchy, but there are locale-specific versions - for now, see the C sample for details).

Messages published to a topic of this type (defining statistics for a class and type) will contain (not necessarily in this order) :-

1. A parameter (MQCFIN) for the class (MQIAMO_MONITOR_CLASS).
2. A parameter (MQCFIN) for the type (MQIAMO_MONITOR_TYPE).
3. A parameter group (MQCFGR) for each statistic (MQGACF_MONITOR_ELEMENT).
   Within the group will be
   1. A parameter (MQCFIN) containing the identifier for the parameter (MQIAMO_MONITOR_ELEMENT).
   2. A parameter (MQCFIN) containing the datatype for the parameter (MQIAMO_MONITOR_DATATYPE).
   3. A parameter (MQCFST) containing the description of the parameter (MQCAMO_MONITOR_DESC).

We can save this information away and then use it to interpret messages actually published to the statistic topic.

Here's a possible data structure for storing statistic definitions for a specific class and topic :-

  private static class StatisticDefinition {

    int dataType;
    String description;
  }

  private static Map<Integer, StatisticDefinition> statisticDefinitionsByParameterId = new HashMap<>();

And here's an example of how to load such a data structure :-

  private static void loadMetaData(Session session) throws JMSException, IOException, EOFException, MQDataException {
    Topic metaDataTopic = session.createTopic("topic://$SYS/MQ/INFO/QMGR/QM_T1/Monitor/METADATA/CPU/QMgrSummary");
    MessageConsumer metaDataConsumer = session.createConsumer(metaDataTopic);

    BytesMessage bytesMessage = (BytesMessage) metaDataConsumer.receive(15000); // in ms or 15 seconds
    byte[] bytesreceived = new byte[(int) bytesMessage.getBodyLength()];
    bytesMessage.readBytes(bytesreceived);

    // convert to MQMessage then to PCFMessage
    MQMessage mqMsg = new MQMessage();
    mqMsg.write(bytesreceived);
    mqMsg.encoding = bytesMessage.getIntProperty("JMS_IBM_Encoding");
    mqMsg.format = bytesMessage.getStringProperty("JMS_IBM_Format");
    mqMsg.seek(0);

    PCFMessage pcfMsg = new PCFMessage(mqMsg);

    @SuppressWarnings("unchecked")
    Enumeration<PCFParameter> params = pcfMsg.getParameters();
    while (params.hasMoreElements()) {
      PCFParameter param = params.nextElement();
      switch (param.getParameter()) {
        case MQConstants.MQGACF_MONITOR_ELEMENT : {
          // monitor element entries are a description of a statistic
          // defined in a group of parameters
          @SuppressWarnings("unchecked")
          Enumeration<PCFParameter> groupParams = ((MQCFGR) param).getParameters();
          int id = -1;
          String description = null;
          int dataType = -1;
          while (groupParams.hasMoreElements()) {
            PCFParameter groupParam = groupParams.nextElement();
            switch (groupParam.getParameter()) {
              case MQConstants.MQIAMO_MONITOR_ELEMENT :
                // this defines the integer which will identify a parameter
                // as relating to this statistic
                id = (Integer) groupParam.getValue();
                break;
              case MQConstants.MQIAMO_MONITOR_DATATYPE :
                // statistics are (always?) passed as 32- or 64-bit integers
                // but may represent a variety of things (such as percentages to 2 decimal places)
                // the datatype defines how to interpret them
                dataType = (Integer) groupParam.getValue();
                break;
              case MQConstants.MQCAMO_MONITOR_DESC :
                // something like "User CPU time - percentage estimate for queue manager"
                description = groupParam.getStringValue();
                break;
            }

          }
          if ((id != -1) && (description != null) && (dataType != -1)) {
            StatisticDefinition sd = new StatisticDefinition();
            sd.dataType = dataType;
            sd.description = description;
            statisticDefinitionsByParameterId.put(id, sd);
          }
          break;
        }

   ...
        default :
          // ignore others
      }
    }
  }

Resource Usage Messages

The messages published to statistics topics (for example "$SYS/MQ/INFO/QMGR/QM_T1/Monitor/CPU/QMgrSummary") will contain (not necessarily in this order) :-

1. A parameter (MQCFST) containing the Queue Manager name (MQCA_Q_MGR_NAME).
2. A parameter (MQCFIN) for the class (MQIAMO_MONITOR_CLASS).
3. A parameter (MQCFIN) for the type (MQIAMO_MONITOR_TYPE).
4. A parameter (MQCFIN64) for the interval since the last set of statistics (MQIAMO64_MONITOR_INTERVAL) in milliseconds.
5. Parameters (MQCFIN or MQCFIN64) for the statistics, identified by the MQIAMO_MONITOR_ELEMENT values defined in the meta data.

Here's some sample code to read and process such statistics, assuming the previous data structure has been established:

  private static void processStatistics(Session session) throws JMSException, IOException, EOFException, MQDataException {
    Destination statisticsDestination = session.createTopic("topic://$SYS/MQ/INFO/QMGR/QM_T1/Monitor/CPU/QMgrSummary");
    MessageConsumer statisticsConsumer = session.createConsumer(statisticsDestination);
    for (int count = 0; count < 50; count++) {
      BytesMessage bytesMessage = (BytesMessage) statisticsConsumer.receive(15000); // in ms or 15 seconds
      byte[] bytesreceived = new byte[(int) bytesMessage.getBodyLength()];
      bytesMessage.readBytes(bytesreceived);

      // convert to MQMessage then to PCFMessage
      MQMessage mqMsg = new MQMessage();
      mqMsg.write(bytesreceived);
      mqMsg.encoding = bytesMessage.getIntProperty("JMS_IBM_Encoding");
      mqMsg.format = bytesMessage.getStringProperty("JMS_IBM_Format");
      mqMsg.seek(0);

      PCFMessage pcfMsg = new PCFMessage(mqMsg);

      @SuppressWarnings("unchecked")
      Enumeration<PCFParameter> allParameters = pcfMsg.getParameters();
      while (allParameters.hasMoreElements()) {
        PCFParameter currentParameter = allParameters.nextElement();
        switch (currentParameter.getParameter()) {
          case MQConstants.MQCA_Q_MGR_NAME :
            System.out.format("CPU statistics for %s put at %s - %s%n", currentParameter.getStringValue().trim(), bytesMessage.getStringProperty("JMS_IBM_PutDate"),
                bytesMessage.getStringProperty("JMS_IBM_PutTime"));
            break;
          case MQConstants.MQIAMO_MONITOR_CLASS :
          case MQConstants.MQIAMO_MONITOR_TYPE :
            // Should perhaps check that it's as expected (defined by the meta-data)?
            // But I haven't for now...
            break;
          case MQConstants.MQIAMO64_MONITOR_INTERVAL :
            // Monitor interval, i.e. time since last publish, is in milliseconds
            long interval = ((MQCFIN64) currentParameter).getLongValue();
            int millis = (int) ((interval % 1000000L) / 1000);
            int seconds = (int) (interval / 1000000L);
            int minutes = seconds / 60;
            seconds %= 60;
            int hours = minutes / 60;
            minutes %= 60;
            System.out.format("Interval %d hours, %d minutes, %d seconds, %d milliseconds%n", hours, minutes, seconds, millis);
            break;
          default :
            // if it's not one of those above and it's a (long) integer,
            // then it's likely to be a statistic...
            switch (currentParameter.getType()) {
              case (MQConstants.MQCFT_INTEGER) : {
                MQCFIN statistic = (MQCFIN) currentParameter;
                int parameterId = statistic.getParameter();
                long parameterValue = statistic.getIntValue();
                printParameter(parameterId, parameterValue);
                break;
              }
              case (MQConstants.MQCFT_INTEGER64) : {
                MQCFIN64 statistic = (MQCFIN64) currentParameter;
                int parameterId = statistic.getParameter();
                long parameterValue = statistic.getLongValue();
                printParameter(parameterId, parameterValue);
                break;
              }
              default : //Ignore the rest
            }
        }
      }
    }
  }

  /*
   * this is mainly about the "datatype" and converting (long) integer values
   */
  private static void printParameter(int parameterId, long parameterValue) {
    StatisticDefinition sd = statisticDefinitionsByParameterId.get(parameterId);
    if (sd != null) {
      // If we recognised the id, we can display it!
      String result = null;
      switch (sd.dataType) {
        case MQConstants.MQIAMO_MONITOR_UNIT :
        case MQConstants.MQIAMO_MONITOR_DELTA :
          result = String.format("%s: %d", sd.description, parameterValue);
          break;
        case MQConstants.MQIAMO_MONITOR_HUNDREDTHS :
          result = String.format("%s: %2f", sd.description, ((double) parameterValue) / 100);
          break;
        case MQConstants.MQIAMO_MONITOR_PERCENT :
          result = String.format("%s: %2f%%", sd.description, ((double) parameterValue) / 100);
          break;
        case MQConstants.MQIAMO_MONITOR_KB :
          result = String.format("%s: %dKB", sd.description, parameterValue);
          break;
        case MQConstants.MQIAMO_MONITOR_MB :
          result = String.format("%s: %dMB", sd.description, parameterValue);
          break;
        case MQConstants.MQIAMO_MONITOR_MICROSEC :
          result = String.format("%s: %d uSec", sd.description, parameterValue);
          break;
      }
      System.out.println(result);
    }
  }