WebSphere Application Server & Liberty

After upgrading to WAS 9.0.5.7, or installing an iFix for WAS 8.5.5.16 and later 8.5.5.x servers, you will see a new `metrics.ear` file in your `<WAS_HOME>/installableApps` directory.  By installing the `metrics.ear` Prometheus will be able to scrape metrics from your entire WAS cell.  We've also published a Grafana dashboard - ready to use with metrics scraped from your WAS servers.  While you can easily customize a dashboard to include the metrics you care about, the sample WAS dashboard at grafana.com gives you a good place to start.

Getting Started

# my global config

global:
  scrape_interval: 30s # Set the scrape interval to every 30 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# A scrape configuration containing exactly one endpoint to scrape:
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'was-nd'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
     scrape_interval: 15s
    static_configs:
    - targets: ['<WAS_hostname>:<WAS_port>']

- job_name: 'was-nd'
    scheme: `https`
    basic_auth:
      username: someUser
      password: somePassword
    static_configs:
    - targets: ['<WAS_hostname>:<WAS_port>']

The quick start example, found in the previously mentioned repository, already has a Prometheus YAML file configured to work with the bundled WAS Docker container. However, if you want to run a standalone Prometheus instance for your own WAS instance, you can do so, by running the following Prometheus Docker image and specifying your own Prometheus YAML configuration file, with the target to be scraped set to your own WAS instance’s hostname and port, that has the `metrics.ear` application installed.

docker run -d --name prometheus_standalone -p 9090:9090 -v <PATH_TO_PROMETHEUS_YAML>:/etc/prometheus/prometheus.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml

Visualizing your Data with Grafana

Now that you have Prometheus set up and running you can use the provided web client at http://localhost:9090 to submit PromQL queries to the Prometheus time-series database to retrieve and display metric data. However, this facility for retrieving the metric data is rudimentary. The metrics data can only be presented as a table listing or line graph. Although you can have multiple queries in multiple visualizations for presenting multiple metrics of interest, they only exist for your current session. It's possible to save and share the URL that dynamically changes with the contents of the page, but that is rather inefficient and insecure. Of course, this Prometheus web client was never intended to be your final step in your metrics monitoring solution. Prometheus is often paired with Grafana which is another open source monitoring solution that is used to illustrate and visualize your metrics time-series data. Grafana provides a much more feature rich platform withconfiguration elements for security, alerting and implementing dynamic variables.  Grafana also has a long list of visualizations you can use. These visualizations can then be conveniently saved into a dashboard which can be shared and/or hosted on the Grafana server and secured through user-based access roles.

We have provided a Grafana dashboard that visualizes metrics from the following components: CPU, Memory Heap, Servlets, EJBs, Connection Pool, SIB, Sessions, Threadpool, Garbage Collection, and other JVM statistics. The dashboard can be downloaded from the Grafana community dashboards site.Depending on your deployment topology and PMI settings it can be the case that your `/metrics` endpoint may be inundated with large amounts of metrics, which may affect performance. The Grafana dashboard page also includes a wsadmin script that will modify the servers in your cell to enable the exact PMI stats needed for this Grafana dashboard. If you will be modifying the dashboard or creating your own dashboard, ensure that you enable the specific metrics that you want. We will cover performance tuning in the subsequent section.

If you chose not to deploy the quick-start demo, you will need to deploy Grafana manually.  For example, using Docker:

docker run -d --name grafana_standalone -p 3000:3000 grafana/grafana

The below steps will outline how to configure Grafana to pull data from Prometheus:

1. Go to the Configuration > Data Sources > Add data source:


2. Choose the Prometheus data source: 


3. Specify the target location for the Prometheus server. We will need to specify `host.docker.internal`:


Metrics Scraping Performance Considerations

The thought of hundreds or thousands or tens of thousands of metrics being processed at once may produce a shiver up your spine. Why? Because performance. Being able to capture metrics is great, but it needs to be fast. Performance was a priority when developing this feature and it does perform well under load, but there are steps that can be taken to produce the best results. Poorly performing nodes in your WAS ND cell can affect the ability of the metrics application to collect data. If a node is close to maximum CPU usage or has run out of memory, the entire metrics processing performance is impacted. Since metrics results need to be aggregated from all nodes before being presented, one thrashing node will impact the performance of the entire metrics data collection. The resultant long response time from the `/metrics` endpoint has a cascading effect. The Prometheus server by default has a 10 second scrape timeout. When an attempt to scrape a target `/metrics` endpoint times out, the scrape fails and no data is recorded for that interval. The result is empty time ranges in your Grafana visualizations.

To ensure you get good performance from your metrics application, check that all nodes have adequate CPU and memory resources for the workloads they are processing. Failing that, or if such configurations are out of your control, you can increase the value of the `scrape_timeout` in your Prometheus configuration file to better tolerate long metrics data collections.

The metrics application also comes with two configurable properties that relate to caching. The metrics application performs two main functions. Gathering and reporting metric data and keeping up to date with the server topology so that it can properly retrieve the metrics from all servers.  Each of these two functions can incur significant overhead if executed everytime the `/metrics` endpoint is hit. As a result the data from these processes are cached to avoid such performance overhead. The two properties are:

- com.ibm.ws.pmi.prometheus.resultCacheInterval (5s default)