WebSphere Application Server & Liberty

WebSphere Automation "How To" Series #7 : How to setup Instana to send alerts to WebSphere Automation

Previous blogs in this WebSphere Automation "How To" series :
WebSphere Automation "How To" Series #1 : How to get WebSphere Automation UI 
WebSphere Automation "How To" Series #2 : How to specify user roles and permissions 
WebSphere Automation "How To" Series #3 : How to configure WebSphere Automation with an Enterprise LDAP 
WebSphere Automation "How To" Series #4 : How to register WebSphere Application Server traditional servers using configuretWasUsageMetering.py script 
WebSphere Automation "How To" Series #5 : How to register WebSphere Liberty servers
WebSphere Automation "How To" Series #6 : How to configure email server and email addresses for notifications 


This post will focus on how to setup Instana to send alerts to WebSphere Automation.

This post assumes that you have an existing installation of Instana with access priviledges to the Instana administrative console. For more information on setting up and managing Instana, you can read the Instana documentation at https://www.ibm.com/docs/en/obi/current?topic=setting-up-managing-instana.

For this example, we have an existing installation of Instana where our user is a member of the Owner group. After logging into the Instana administrative console, we can navigate using the left pane to More -> Agents. The following is a screenshot showing this left pane.


After navigating to this Agents page, you can click the button 'Installing Instana Agents' to navigate to the subsequent page with Instana agent installation instructions based on the platform or operating system which your WebSphere servers are running. The following is a screenshot showing the Agents page on which the 'Installing Instana Agents' button is located.


For this example, we'll use an on-prem Linux VM with a tWAS v9.0.5.10 server. On the 'Installing Instana Agents' page, we click on the OS -> Linux, which displays the various options and modes for installing on Linux. By clicking on the Technology drop down list, the user can see the following possible options :
- Automatic Installation (One-liner)
- Packages (DEB, RPM)
- Archive (tar.gz)
- AWS Elastic Computing (EC2)
- Google Compute Engine (GCE)

The following is a screenshot showing this Technology drop down list.


For this example, we chose to use the 'Automatic Installation (One-liner)' from the Technology drop down list. In addition to the Technology drop down list, there are three radio buttons which the user should utilize to select their choices for Agent mode, Agent runtime and Installation mode. If the user toggles the radio buttons, the one-liner command is updated accordingly. We chose the following for the radio buttons :
- Agent mode : Dynamic
- Agent runtime : Eclipse OpenJ9 11
- Installation mode : Silent

Those choices resulted in the following one-liner script.
#!/bin/bash
curl -o setup_agent.sh https://setup.instana.io/agent && chmod 700 ./setup_agent.sh && sudo ./setup_agent.sh -a XXXXXXX -d XXXXXXX -t dynamic -e YYYYYY.ibm.com:1444 -j -y

The following is a screenshot showing the 'Installing Instana Agents' page with the options selected.


Using the browser, you can copy this script to your on-prem system and execute the script. On Linux, the agent is installed under /opt/instana/agent. By default, the agent will be displayed in 'Undefined Zone' under Instana's Infrastructure. To display your server under a user defined zone, you can edit the /opt/instana/agent/etc/instana/configuration.yaml file to uncomment the following default lines while specifying your desired availability-zone.

# Hardware & Zone
#com.instana.plugin.generic.hardware:
# enabled: true # disabled by default
# availability-zone: 'Datacenter A / Rack 42'

The following is a screenshot showing the Instana Infrastructure Map, which shows that servers are organized into zones.


You can start and stop the instana agent by utilizing the start or stop commands under /opt/instana/agent/bin directory. For more detailed information on Instana agents, you can view the Instana documentation at https://www.ibm.com/docs/en/obi/current?topic=instana-host-agent.

Now that we have installed and started the Instana agent for our on-prem host, we can view real-time monitored WebSphere Application Server metrics, such as heap memory, when our WebSphere Application Server is running. There are a number of ways in the Instana administrative console to navigate to that information, but one is to click on the JVM using the Infrastructure Map and then click the 'Open Dashboard' button. The following is a screenshot showing this.


Using default WebSphere Application Server genericJvmArguments settings, you will encounter the following message in Instana administration console when attempting to view the real-time JVM metrics.
This JVM has class sharing enabled. The host agent cannot attach to this Java Virtual Machine. Please refer to our documentation to disable the class sharing, and restart the JVM so that the agent can retry attaching. Specifically the command-line switches -Xshareclasses:none or if available -Xshareclasses:enableBCI might be needed. Until updated, neither traces nor metrics will be collected for this JVM.

The following is a screenshot showing the monitoring issue message.


To allow Instana monitoring  of WebSphere Application Server for this example, we have added genericJvmArguments "-Xshareclasses:enableBCI" to our WebSphere Application Server and restarted our server. The following is a screenshot showing the WebSphere Application Server JVM in Instana successfully after setting the genericJvmArguments.


Now that we have a running Instana agent and a running WebSphere Application Server, it is time to perform the Instana configuration which allows Instana to send alerts to WebSphere Automation.

First, we have to create a PID custom payload. Using the Instana administrative console, you can access the custom payload by navigating to 'Settings -> Team Settings -> Events & Alerts -> Custom Payload'. If a key for PID does not already exist, then you should click the 'Add Row' button , enter the following values and click 'Save' :
- Key: PID
- Value type: Dynamic
- Value : Internal -> Process ID

The following is a screenshot showing the Custom Payload page.


Second, we have to create an alert channel. Using the Instana administrative console, you can create an alert channel by navigating to 'Settings -> Team Settings -> Events & Alerts -> Alert Channels'. Click 'Add Alert Channel' to expand the alert channel options. The following is a screenshot showing the Alert Channels page with Alert Channel options.


We'll select the 'Generic Webhook' alert channel. On the 'Create Generic Webhook Alert Channel' page, you will enter the following values : Name, Webhook URL & Custom HTTP Request Headers. For the Name field, select your unique webhook name. For the Webhook URL field, you can get the Webhook URL for your WebSphere Automation installation with the following oc command :
oc get route cpd -o jsonpath=https://{.spec.host}/websphereauto/health/webhooks/instana/memleak

For Custom HTTP Request Headers, you'll first click 'Add Header' before setting the Key and Value information.

The first header information to add is as follows :
- Key : Content-Type
- Value : application/json

The second header information to add is as follows :
- Key : Authorization
- Value : Bearer <TOKEN>
where the <TOKEN> value can be retrieved with the following oc command :
oc serviceaccounts get-token wsa-health-webhooks-apis-sa

The following is a screenshot showing the 'Create Generic Webhook Alert Channel' page :


Third, we have to create an alert. Using the Instana administrative console, you can create an alert by navigating to 'Settings -> Team Settings -> Events & Alerts -> Alerts'. Click 'New Alert' to create the alert. When creating the new alert, you'll be prompted to do the following :
- In the Name field, type a name for the alert.
- In the Events menu, select 'Alert on Event(s)'.
- In the Events section, click 'Add Event'.
- In the Add Events window, select 'Memory Leak'. You can filter the list by using a keyword (e.g. leak) to find the entry. Click 'Add 1 Event'.
- In the Scope menu, select either 'All available entities' or 'Selected Entities Only (Dynamic Focus Query)'.
- Under Alerting, click the 'Add Alert Channel', select the alert channel previously created and click 'Add 1 Channel'.
- Nothing needs to be added for the optional Custom Payload section.
- Click 'Create'

The following are two screenshots showing the 'Create New Alert' page :


When using the built-in Instana 'Memory Leak' event, the rules for the memory leak algorithm to trigger an event are as follows :
- keep data points with beforeGC/afterGC/gcInv where gcInv != 0
- 60 min sliding window evaluated every 5 min
- aggregate data points into 5 min buckets
- ensure first and last buckets are > 30 min apart
- ensure there are more than 4 buckets
- ensure afterGC value in last bucket is the max compared to other buckets
- ensure that in more than 75% of buckets the afterGC value sets a new max that has grown from prev max by at least 15 MB/hr
- ensure that rate of growth of afterGC between first and last buckets is more than 15 MB/hr


You can find more IBM Docs related to WebSphere Automation at https://www.ibm.com/docs/en/ws-automation.
#websphere#automation#WSA #WebSphereAutomation​​​​​