Two major pieces of ISAM run on an application server instance running within the appliance. Something I am often asked is how does an ISAM administrator get meaningful information about the resource consumption of this application server while under load. In ISAM 9.0.4.0 a monitoring capability was added to the runtime for this very purpose. This capability is enabled via advanced tuning and can prove essential when debugging issues seemingly caused by load.

Before going into any further detail I will point out considerations which need to be made before making use of this monitoring:

• A slight performance decrease in the runtimes operation speed (Less than 5%).
• These APIs are exposed via REST on the same endpoint as runtime traffic in an unauthenticated manner. If authentication is necessary a Reverse Proxy will be needed to protect the monitoring resources.

Multiple REST APIs are used to expose a suite of data about the runtime’s operation. These all reside under the context root /monitor. If the root resource /monitor is requested, it will return an object of the different resources which can be viewed. These include:

• JVM Details
• Thread Pool details
• Runtime database connection pool details
• Config database connection pool details
• Servlet monitoring details

All of the payloads are in JSON format. Its recommended to make a basic client to poll these endpoints and to log them to your preferred log aggregation service (Eg QRadar).

Enabling the Monitoring

To enable the monitor the advanced tuning parameter runtime_profile.enable.monitor must be set to the value true. After deploying this change a runtime restart will occur. You can check if the monitoring application has started by looking for the line:

I SRVE0242I: [monitor] [/monitor] [monitor]: Initialization successful.

In the messages.log of the runtime.

 

Monitoring Client

Below is a very basic monitoring client, which will request all of the monitoring data, add them to a single JSON payload and log the details to either STDOUT or into a file:

#!/bin/python3
import argparse
import requests
import time
import datetime
import pprint
import sys

import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)


def file_output(pth, entry):
    with open(pth, "a") as f:
        f.write(str(entry))
        f.write("\n")

def stdout_output(pth, entry):
    print(entry)

def pretty_stdout_output(pth, entry):
    pp = pprint.PrettyPrinter(indent=2, width=240)
    pp.pprint(entry)

def main():
    print(__file__)
    parser = argparse.ArgumentParser(prog=__file__)
    parser.add_argument('-runtime_uri', required=True, help='The runtime URI to use, Eg "https://192.168.1.1:444/jct/" this value will be used as the base to the request to "/monitor"')
    parser.add_argument('-interval', default=5, required=False, help='Duration to wait between polls.')
    parser.add_argument('-output', required=False, help='Output destination file. If missing STDOUT is used')
    parser.add_argument('-outputfmt', required=False, help='Output format. supported values: json. Default: json')
    parser.add_argument('-nopretty', action="store_true", required=False, help='Do not pretty print output to STDOUT. Ignored if -output is provided')


    args = parser.parse_args()
    cfg=vars(args)
    base = "{0}/monitor".format(cfg['runtime_uri'])
    
    output = pretty_stdout_output

    if cfg['nopretty']:
        output = stdout_output


    if cfg['output'] != None:
        output = file_output

    interval = int(cfg['interval'])

    print(f"polling {base} with interval {interval}", file=sys.stderr)

    while(True):

        req = requests.get(base, verify=False)
        assert(req.status_code == 200)
        json = req.json()
        keys = req.json().keys()

        entry = {}

        for a in keys:

            suffix = []
            if isinstance(json[a],str):
                suffix.append(json[a])
            else:
                for b in json[a]:
                    suffix.append(b)

            for c in suffix:

                url = f"{base}{c}"
                req = requests.get(url, verify=False)
                if(req.status_code != 200):
                    print(f"Error reching {url}", file=sys.stderr)
                    continue



                entry[c] = dict(req.json())
                entry['time'] = int(time.time())
                entry['fmt-time'] = str(datetime.datetime.utcnow())

        output(cfg['output'], entry)
        time.sleep(interval)


if __name__ == "__main__":
    main()

This code does depend on the library ‘requests’. Install it with the call pip install --user requests.

Interpreting the Output

When using the monitor feature, output like the following may be encountered:

{"error":"MBean with name WebSphere:type=ServletStats,name=IBM RTSS Runtime Server.AuthzServiceSOAP was not available \/ null"}

This is an indicator that the component you’re trying to view has not yet been initialized – for example, if a JDBC connection to the runtime database hasn’t been opened yet, then the monitoring for it will not yet have been initialised.

Example output from the client above is. Comments have been added to some of the keys:

{
  "/config": {
    "ConnectionHandleCount": 0,
    "CreateCount": 840,
    "Datasource": "config",
    "DestroyCount": 840,
    "FreeConnectionCount": 0,
    "ManagedConnectionCount": 0, // Number of pooled connections currently held
    "WaitTime": 0 // If this is a positive number, it indicates that at some point the connection pool hit max size. 
  },
  "/hvdb": {
    "ConnectionHandleCount": 0,
    "CreateCount": 338,
    "Datasource": "hvdb",
    "DestroyCount": 338,
    "FreeConnectionCount": 0,
    "ManagedConnectionCount": 0, // Number of pooled connections currently held
    "WaitTime": 0 // If this is a positive number, it indicates that at some point the connection pool hit max size. 
  },
  "/jvm": {
    "FreeMemory": 1884920424,
    "GcCount": 216,
    "GcTime": 1192,
    "Heap": 2147483648,
    "ProcessCPU": 147.38741951653265,
    "UpTime": 193267873,
    "UsedMemory": 262563224
  },
  "/servlet/IBM%20FIM%20Runtime%20Server/com.tivoli.am.fim.sts.jaxws.providers.WST13TrustServiceProvider": {
    "AppName": "IBM FIM Runtime Server", // This covers all requests to the STS via WS-trust 1.3
    "Description": "Report Servlet Stats for specified Servlet and application.",
    "RequestCount": 422,
    "RequestCountDetails": {
      "currentValue": 422,
      "description": "This shows number of requests to a servlet",
      "reading": {
        "count": 422,
        "timestamp": 1528038016029,
        "unit": "ns"
      },
      "unit": "ns"
    },
    "ResponseTime": 97607.9644549763,
    "ResponseTimeDetails": {
      "count": 422,
      "description": "Average Response Time for servlet",
      "maximumValue": 1640281,
      "mean": 97607.9644549763,
      "minimumValue": 42879,
      "reading": {
        "count": 422,
        "maximumValue": 1640281,
        "mean": 97607.9644549763,
        "minimumValue": 42879,
        "standardDeviation": 75163.73316909462,
        "timestamp": 1528038016029,
        "total": 41190561,
        "unit": "ns",
        "variance": 5649586783.914855
      },
      "standardDeviation": 75179.81414633089,
      "total": 41190561,
      "unit": "ns",
      "variance": 5650641414.489173
    },
    "ServletName": "com.tivoli.am.fim.sts.jaxws.providers.WST13TrustServiceProvider"
  },
  "/servlet/IBM%20FIM%20Runtime%20Server/com.tivoli.am.fim.war.runtime.liberty.LibertyRuntimeServlet": {
    "AppName": "IBM FIM Runtime Server", // This coveres all requests to endpoints under the /sps context root. 
    "Description": "Report Servlet Stats for specified Servlet and application.",
    "RequestCount": 4,
    "RequestCountDetails": {
      "currentValue": 4,
      "description": "This shows number of requests to a servlet",
      "reading": {
        "count": 4,
        "timestamp": 1528038016001,
        "unit": "ns"
      },
      "unit": "ns"
    },
    "ResponseTime": 738137528.75,
    "ResponseTimeDetails": {
      "count": 4,
      "description": "Average Response Time for servlet",
      "maximumValue": 1722931217,
      "mean": 738137528.75,
      "minimumValue": 155063405,
      "reading": {
        "count": 4,
        "maximumValue": 1722931217,
        "mean": 738137528.75,
        "minimumValue": 155063405,
        "standardDeviation": 567087299.3296814,
        "timestamp": 1528038016001,
        "total": 2952550115,
        "unit": "ns",
        "variance": 321588005061031600
      },
      "standardDeviation": 567087299.3296814,
      "total": 2952550115,
      "unit": "ns",
      "variance": 321588005061031600
    },
    "ServletName": "com.tivoli.am.fim.war.runtime.liberty.LibertyRuntimeServlet"
  },
  "/threads": {
    "ActiveThreads": 3,
    "PoolSize": 8
  },
  "fmt-time": "2018-06-03 15:00:16.073372",
  "time": 1528038016
}

Note: Count statistics, such as ConnectionHandleCount, can be negative as counters are not always synchronized. Synchronizing counters incurs a greater performance cost, so is not performed in regular operation.

For more details on the monitoring. See the following Knowledge Center documents:

• JVM Stats
• Servlet Stats
• Thread pool Stats
• JDBC Connection pool stats

#ISAM