What do IBM customers' deployments of "WebSphere" look like?  Some of our customers run WAS traditional and are running workloads in VMs. Others are enthusiastic users of Liberty -- our cloud optimized runtime for containers.  Many customers are leveraging public or private clouds, and many are doing so with Docker containers running on Kubernetes, perhaps doing so using IBM Cloud Private (ICP) as their platform.  If you've been using "WebSphere" for more than a few years, your own situation may be "all of the above".  

So how do you deal with your logs and metrics?

WebSphere Application Server v9.0.5 provides new opportunities to unite your log and metrics infrastructure.  You can aggregate your logs from across your unmanaged servers, VMs and Kubernetes containers into one place for analysis.  Dashboards give you the ability to spot problems across any of your deployments, analyze traffic from your servers, even analyze your servers' audit records.  All of this comes together in ICP, which comes with a built in Elastic Stack (Elasticsearch, Logstash, Kibana, Filebeat - https://www.elastic.co/ ) log analysis solution and a built in Prometheus (https://prometheus.io/) / Grafana (https://grafana.com/) metrics viewing and alerting solution.

Making logs work with log analysis tools

Liberty JSON logging provides the ability to write out logs, trace, FFDC, access logs, and audit events in a convenient JSON format -- one event per line of output.  This capability, built into the Liberty kernel, makes Liberty work very well with log analysis tools that consume your logs.  We provide, in the Liberty Cloud Pak, a set of Kibana and Grafana dashboards you can use for problem determination (leveraging the log, trace and FFDC events), traffic analysis (leveraging the access log events), and audit analysis (using the audit events).

Example JSON log output from Liberty:

{
    "type": "liberty_message",
 
    "host": "97d853b79f8e",
    "ibm_userDir": "/opt/ibm/wlp/usr/",
    "ibm_serverName": "defaultServer",
 
    "ibm_datetime": "2018-07-19T12: 20: 23.892+0000",
    "ibm_sequence": "1532002823892_0000000000001",
    "ibm_threadId": "00000001",
    "module": "com.ibm.ws.kernel.launch.internal.FrameworkManager",
    "ibm_messageId": "CWWKE0001I",
    "loglevel": "AUDIT",
    "message": "CWWKE0001I: The server defaultServer has been launched."
}

Traditional WAS also provides JSON logging capability.  This capability is provided by the logViewer command line tool that works with high performance extensible logging (HPEL).  As HPEL only contains log and trace data, traditional WAS JSON is limited to those event types.  Like Liberty, the logViewer outputs one line of JSON per event.  The traditional WAS Cloud Pak provides a Kibana dashboard you can use for problem determination.

Example JSON log output from WAS:

{ 
    "type": "was_message", 
 
    "ibm_cellName": "cell1", 
    "ibm_nodeName": "node1", 
    "ibm_serverName": "server1", 
 
    "host": "myhost.ibm.com", 
    "ibm_datetime": "2013-10-29T15: 59: 46.692", 
    "ibm_sequence": "1443849723473_000001",
    "ibm_className": "SessionContextRegistryImpl", 
    "ibm_methodName": "getSessionContext", 
    "ibm_threadId": "00000027", 
    "module": "com.ibm.ws.session.WASSessionCore", 
    "ibm_messageId": "SESN0176I", 
    "loglevel": "INFO", 
    "message": "SESN0176I: A new session context will be created for application key default_host/tradelite"
}

Notice that Liberty JSON contains fields to identify the host, user directory and server, which are the three fields needed to uniquely identify a Liberty server.  For the same purpose, traditional WAS JSON contains fields to identify the cell, node and server.  When running in a container in ICP, these fields are augmented with the Kubernetes namespace, pod and container name so you can easily tell where your records come from.  All other JSON field names are the same for Liberty and traditional WAS.

Liberty and traditional WAS JSON records can be told apart by their "type" field values ("liberty_message" vs. "was_message" in the case of message events).  All JSON events also include an "ibm_sequence" field, which can be used for sorting.  This is a useful field since the ibm_datetime (which represents the time when the event was created) only has millisecond precision.

Setting up your servers to send logs to ICP's Elastic Stack

Whether you decide to run with traditional WAS or Liberty, in ICP or outside ICP, you can send your server's logs to ICP's Elastic Stack.  With this mobility, you can migrate your apps from traditional WAS to Liberty, or start hosting them in ICP, without changing how you view or analyze your server logs.  That said, what you need to do to set up your servers depends on whether you are running traditional WAS or Liberty and where your servers are running.  

Docker containers in ICP

If you're running Liberty or traditional WAS in containers in ICP your logs go to ICP's Elastic Stack automatically.  Under the covers ICP runs a Filebeat container in a daemonset to collect those logs.  Better yet, if you build your Docker image from one of the official IBM images available at Docker Hub (http://hub.docker.com) your server logs will be emitted using the JSON formats discussed above by default and you can use the Kibana dashboards provided with the Cloud Paks to help with problem determination.  

Liberty and Traditional WAS Docker Images        

WebSphere Liberty Docker image - https://hub.docker.com/_/websphere-liberty
Open Liberty Docker image - https://hub.docker.com/_/open-liberty
WebSphere Liberty UBI Docker image - https://hub.docker.com/r/ibmcom/websphere-liberty/
Open Liberty UBI Docker image - https://hub.docker.com/r/openliberty/open-liberty
Traditional WebSphere Docker image - https://hub.docker.com/r/ibmcom/websphere-traditional

Everywhere else

Logs from WAS traditional or Liberty running anywhere with network connectivity to your ICP Kubernetes cluster can also be sent securely to ICP's Elastic Stack.  Note that these instructions (Sending WebSphere logs to an IBM Cloud Private cluster - https://www.ibm.com/support/knowledgecenter/en/SSTF9X/was-logging-icp.html ) are from the ICP VM Quickstarter documentation, but apply equally to any WAS traditional or Liberty server.

Use Kibana dashboards to explore your logs

One of the key advantages to centralizing your logs is to enable log analysis.  To that end, the Liberty and WAS traditional cloud paks provide Kibana dashboards to help you navigate your data.  Because these dashboards support logs from both Kubernetes and non-Kubernetes based servers, you can freely migrate your applications or move them to the cloud without needing to relearn how to analyze your logs.

In ICP you can get to Kibana from the left navigation by clicking Platform > Logging.  If you haven't done so already, you can import dashboards from your cloud pak using these instructions (https://www.ibm.com/support/knowledgecenter/en/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/twlp_icp_json_logging.html).  For Kibana dashboards to import cleanly the fields they reference need to already be present in the Elasticsearch index, so be sure to send your first logs to ICP's Elastic Stack before importing.

Once your dashboards are imported, you can see them by clicking on `Dashboard` in Kibana's left navigation.

Access any of the dashboards just by clicking on them.  

Problems dashboard

Traffic dashboard

Audit dashboard

Setting up your servers to send metrics to ICP's Prometheus/Grafana

Liberty containers in ICP

Metrics from Docker containers running Liberty are automatically collected by ICP.  ICP automatically deploys a Prometheus agent, in a daemonset, that polls Liberty's /metrics endpoint and forwards data to ICP's Prometheus.  

To take advantage of this automatic metric collection build your Docker image from one of the official Liberty images.

Liberty Docker Images        

WebSphere Liberty Docker image - https://hub.docker.com/_/websphere-liberty
Open Liberty Docker image - https://hub.docker.com/_/open-liberty
WebSphere Liberty UBI Docker image - https://hub.docker.com/r/ibmcom/websphere-liberty/
Open Liberty UBI Docker image - https://hub.docker.com/r/openliberty/open-liberty

In your Dockerfile, add the following two lines to ensure your Docker build sets up the required Liberty features (monitor-1.0, mpMetrics-1.1) for you.  More info about ARGs you can add to your Liberty Docker build can be found here (https://github.com/wasdev/ci.docker#enterprise-functionality)

ARG MP_MONITORING=true

RUN configure.sh

Use Grafana dashboards to explore your metrics

Once you have your metrics flowing to ICP's Prometheus data store you can begin analyzing them.  The Liberty cloud paks provide a Grafana dashboard to make it easy to view Liberty's metrics.

From the ICP console, click on Platform > Monitoring to get to Grafana.  If you haven't done so already, you can import the Grafana dashboard from your cloud pak using these instructions (https://www.ibm.com/support/knowledgecenter/en/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/twlp_icp_monitor.html).  

Servlet views in Liberty Grafana Dashboard

Session views in Liberty Grafana Dashboard

Application Navigator brings it all together

WAS 9.0.5 provides a tool you can deploy to ICP called the Application Navigator.  The Application Navigator gives an application-centric view of your Kubernetes and non-Kubernetes resources, such as your Liberty and traditional WAS servers.  

When using the Application Navigator, simply click on the action menu for the server whose logs (from Liberty or traditional WAS) or metrics (from Liberty containers) you want to view.  Doing so will take you directly to the Kibana or Grafana dashboard, with filtering in place to show you the logs/metrics for the server you selected.  

Conclusion

Your deployments may be a mix of Liberty and traditional WAS, running in Kubernetes or in separate collectives / cells.  A few things to remember:

• Consolidating your logs into ICP's Elastic Stack provides a convenient way to troubleshoot problems and performance issues
• Build your Docker images from the official Liberty or traditional WAS images to gather logs and metrics automatically
• The Liberty and traditional WAS cloud paks come with ready to use Kibana and Grafana dashboards
• Application Navigator shows you all parts of your application and provides actions to jump to Kibana and Grafana views of specific servers