How To: Java and JMX Observability with IBM SevOne & Rapid Network Automation

One of our great features in IBM SevOne is our open data platform - whether it's using one of our builtin plugins to retrieve data, monitoring your hybrid-cloud or pushing data in via our REST API, we offer a multitude of ways to get data into and out of the system. The ability to push or pull in data from any system, not just networks, and then baseline it, alert on it and report on it with Data Insight, makes SevOne a key part of your infrastructure monitoring stack.

Whilst it was not widely known, we had a builtin JMX plugin which allowed users to monitor Java based systems. However the ongoing maintenance of this plugin was becoming problematic and it had fallen behind newer industry standards, for this reason we chose to deprecate this plugin as part of our IBM SevOne 7.0 release.

HOWEVER, we know JMX monitoring is important to some of our users and we want to open it up to even more parts of the Java ecosystem, so we have put together a flexible monitoring stack that will allow users to continue to benefit from SevOne and JMX monitoring and demonstrates the flexibility of our open data platform. Enter stage left - JMX Exporter and Rapid Network Automation!

The Aim

The aim here is to enable JMX monitoring and support vendor specific protocols such as remoting+http (JBoss / Wildfly). To do this we need a way to bridge from Java to SevOne - for this we will use Prometheus JMX Exporter, and then to get the data from JMX Exporter into SevOne, we'll use Rapid Network Automation.

The Proposed Monitoring Stack

Prometheus JMX Exporter

Prometheus JMX Exporter provides a way for us to bridge between the Java domain, collect the JMX metrics and publish them in to a standard format for consumption over HTTP(s).

There are several benefits from using this translation layer -

1. It translates between Java specific JMX and HTTP, opening up metrics to a wider range of consumers
2. It is also extensible, meaning you can add in vendor specific plugins like JBoss to enable access to otherwise closed systems
3. It has an active community behind it, contributing templates for popular Java apps such as Kafka, Cassandra, Spark, Wildfly and others

The installation and configuration of JMX Exporter is covered in more detail here - https://github.com/prometheus/jmx_exporter/tree/release-1.0.1/docs. For this article I downloaded and ran the JMX Exporter .jar file manually, however I recommend you run this under a daemon such as systemd to handle restarts etc.

Apache Tomcat Monitoring

To monitor Apache Tomcat, we can use simple RMI based monitoring, this can also be coupled with a template to get your started from the JMX Exporter example configuration collection here.

The launch command I used was - 

java -jar jmx_prometheus_httpserver-1.0.1.jar 12345 config.yaml

..and my config.yaml -

startDelaySeconds: 0
hostPort: 127.0.0.1:8007
username: 
password: 
ssl: false
lowercaseOutputLabelNames: true
lowercaseOutputName: true
whitelistObjectNames: ["java.lang:type=OperatingSystem", "Catalina:*"]
blacklistObjectNames: []
rules:
  - pattern: 'Catalina<type=Server><>serverInfo: (.+)'
    name: tomcat_serverinfo
    value: 1
    labels:
      serverInfo: "$1"
    type: COUNTER
  - pattern: 'Catalina<type=GlobalRequestProcessor, name=\"(\w+-\w+)-(\d+)\"><>(\w+):'
    name: tomcat_$3_total
    labels:
      port: "$2"
      protocol: "$1"
    help: Tomcat global $3
    type: COUNTER
  - pattern: 'Catalina<j2eeType=Servlet, WebModule=//([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), name=([-a-zA-Z0-9+/$%~_-|!.]*), J2EEApplication=none, J2EEServer=none><>(requestCount|processingTime|errorCount):'
    name: tomcat_servlet_$3_total
    labels:
      module: "$1"
      servlet: "$2"
    help: Tomcat servlet $3 total
    type: COUNTER
  - pattern: 'Catalina<type=ThreadPool, name="(\w+-\w+)-(\d+)"><>(currentThreadCount|currentThreadsBusy|keepAliveCount|connectionCount|acceptCount|acceptorThreadCount|pollerThreadCount|maxThreads|minSpareThreads):'
    name: tomcat_threadpool_$3
    labels:
      port: "$2"
      protocol: "$1"
    help: Tomcat threadpool $3
    type: GAUGE
  - pattern: 'Catalina<type=Manager, host=([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), context=([-a-zA-Z0-9+/$%~_-|!.]*)><>(processingTime|sessionCounter|rejectedSessions|expiredSessions):'
    name: tomcat_session_$3_total
    labels:
      context: "$2"
      host: "$1"
    help: Tomcat session $3 total
    type: COUNTER   

Wildfly / JBoss Monitoring

To monitor Wildfly, we need to add in the JBoss specific monitoring extention - this is packaged as jboss-client.jar and can be found on the host where you have Wildfly/JBoss installed. Copy this .jar file to where you have installed JMX Exporter. An example Wildfly configuration file can be found here.

The launch command I used this time was a little different as we need to include the jboss-client.jar file - 

java -cp jmx_prometheus_httpserver-1.0.1.jar:jboss-client.jar io.prometheus.jmx.WebServer 12345 config.yaml

..and my config.yaml -

startDelaySeconds: 0
username: <your username>
password: <your password>
ssl: false
jmxUrl: service:jmx:remote+http://<IP of wildfly host>:9990
lowercaseOutputName: true
lowercaseOutputLabelNames: true
whitelistObjectNames: 
 # Whitelist objects to be collected, for performance reason
 # see https://github.com/prometheus/jmx_exporter/issues/246#issuecomment-367573931
 # Each object in the rules below has to be added to whitelistObjectNames too !
 # note that rules use regex (like "foo.*", whereas the whitelist use globbing expressions (like "foo*")
 - "jboss.as:subsystem=messaging-activemq,server=*,jms-queue=*"
 - "jboss.as:subsystem=messaging-activemq,server=*,jms-topic=*"
 - "jboss.as:subsystem=datasources,data-source=*,statistics=*"
 - "jboss.as:subsystem=datasources,xa-data-source=*,statistics=*"
 - "jboss.as:subsystem=transactions*"
 - "jboss.as:subsystem=undertow,server=*,http-listener=*"
 - "jboss.as:subsystem=undertow,server=*,https-listener=*"
 - "java.lang:*"
rules:
  - pattern: "^jboss.as<subsystem=messaging-activemq, server=.+, jms-(queue|topic)=(.+)><>(.+):"
    attrNameSnakeCase: true
    name: wildfly_messaging_$3
    labels:
      $1: $2

  - pattern: "^jboss.as<subsystem=datasources, (?:xa-)*data-source=(.+), statistics=(.+)><>(.+):"
    attrNameSnakeCase: true
    name: wildfly_datasource_$2_$3
    labels:
      name: $1

  - pattern: "^jboss.as<subsystem=transactions><>number_of_(.+):"
    attrNameSnakeCase: true
    name: wildfly_transaction_$1

  - pattern: "^jboss.as<subsystem=undertow, server=(.+), (http[s]?-listener)=(.+)><>(bytes_.+|error_count|processing_time|request_count):"
    attrNameSnakeCase: true
    name: wildfly_undertow_$4
    labels:
      server: $1
      listener: $3

To test that JMX Exporter is working, you can simply open a browser and head to http://<ip of the host with JMX exporter>:12345/metrics/ - you should now see metrics from your Tomcat, Wildfly or JBoss system similar to below - 

Example of metrics seen by browsing to the JMX Exporter web interface

Rapid Network Automation

Once we have JMX Exporter up and running, the next step is to set up a Rapid Network Automation workflow to grab the data and send it to SevOne. The good news is that we have done the hard work for you and created an example workflow to get you started. This can be downloaded here.

Once you have downloaded the workflow, it can be imported in to Rapid Network Automation following the guide here - https://www.ibm.com/docs/en/rapid-network-auto/1.1.x?topic=workflows-importing-exporting

You'll need to edit the workflow and set up authentication on the SevOne blocks to point to your own SevOne system.

Rapid Network Automation showing the JMX Exporter to IBM SevOne Workflow

Finally, you'll need to schedule the JMX workflow in Rapid Network Automation, setting the IP, hostname and port to match your own server where JMX Exporter is running - 

Setting the schedule and parameters in the JMX Exporter workflow

...and that is it! Within the next 5 to 10 minutes you'll have new JMX metrics appearing in SevOne for you to baseline, alert and report on!

Summary

JMX Metrics via JMX Exporter and Rapid Network Automation in SevOne

In summary, what we've outlined here is an example set of steps to help you get up and running with JMX Exporter, Rapid Network Automation and SevOne for Java JMX monitoring. Both the JMX Exporter configuration files and the Rapid Network Automation workflows are completely customisable and extensible - please do have a poke around and customise this for your specific environment.

If you want to learn more about Rapid Network Automation or SevOne then please do get in touch! If you want to go deeper in to the application stack and understand individual calls and traces then I highly recommend you also check out our 'sister' product, IBM Instana Observability.

Header image coffee bean background by Fatih Mehmet YILDIZ on Unsplash

#TechnicalBlog
#ImplementationTips