WebSphere Automation "How To" Series #9 : How to trigger a memory leak health investigation when used heap is over 80 percent

View Only

WebSphere Automation "How To" Series #9 : How to trigger a memory leak health investigation when used heap is over 80 percent

By Brian Hanczaryk posted Thu May 05, 2022 06:18 AM

Like

Following the steps detailed in WebSphere Automation "How To" Series #4 : How to register WebSphere Application Server traditional servers using configuretWasUsageMetering.py script, we've registered one WebSphere Application Server traditional server on a RHEL 8 VM. The following is a screenshot of the WebSphere Automation UI after registering the server.

Following the steps detailed in WebSphere Automation "How To" Series #7 : How to setup Instana to send alerts to WebSphere Automation, we've installed and started the Instana agent on the RHEL 8 VM where the server is running. We've edited the /opt/instana/agent/etc/instana/configuration.yaml file to set the availability-zone to BPH. The following is the Instana Infrastructure Map screenshot showing the server under BPH zone.

Following the steps detailed in WebSphere Automation "How To" Series #8 : How to setup secure access to Linux or UNIX servers, we've setup secure access to our RHEL 8 VM.

Now that we've performed the necessary steps, we are ready to set Instana configuration to allow us to trigger a memory leak health investigation when used heap is over 80 percent. Following the steps detailed in WebSphere Automation "How To" Series #7 : How to setup Instana to send alerts to WebSphere Automation, we've configured our Custom Payload and our Alert Channel as those are exactly the same configuration steps as using the built-in Instana Memory Leak event versus setting up a custom event to trigger when used heap is over 80 percent.

Here is a screenshot showing the Instana Custom Payload page.

Here is a screenshot showing the Instana Alert Channel page where the user should add a new alert channel with type generic webhook.

Here is a screenshot showing the Instana Create Generic Webhook Alert Channel page.

With the Custom Payload and Alert Channel created, the next step is to create a new custom event to trigger when the used heap is over 80 percent. In Instana UI, navigate to Settings -> Team Settings -> Events & Alerts -> Events, where the user can create a new custom event by clicking the 'New Event' button. The following screenshot shows the Events page.

The following are the values we used to create a new custom event to trigger when the used heap is over 80 percent for any server in the BPH zone.

Name: BPHZone_Heap_Over_80
Description : BPHZone_Heap_Over_80
Issue Severity : Warning
Grace Period : 10 s
Source : Built-in metrics
Entity type : JVM
Metric : Memory > Used percentage
Time Window : 10 s
Aggregration : avg
Operator : >
Percentage : 80
Apply on (required) : Selected Entities Only (Dynamic Focus Query)
Dynamic Focus Query : entity.zone:"BPH"

The following are the screenshots showing the custom event information being entered in the Instana UI.

After creating the custom event, we will create the alert. In Instana UI, navigate to Settings -> Team Settings -> Events & Alerts -> Alerts. On the Alerts page, click 'New Alert'. The following is a screenshot showing the Alerts page.

The following are the values we used to create an alert for the previously created custom event for any server in the BPH zone.

Name : Heap80_bph
Events : Alert on Event(s)
Added Event : BPHZone_Heap_Over_80
Apply on (required) : Selected Entities Only (Dynamic Focus Query)
Dynamic Focus Query : entity.zone:"BPH"
Added Alert Channel : bph410Webhook_5-4

The following are the screenshots showing the alert information being entered in the Instana UI.

Now we have completed our configuration and are ready to drive high heap against our running server. Prior to driving high heap against our server, the WebSphere Automation UI shows that no memory leak health investigations has taken place as shown by the following screenshot.

For this example, we are using a simple application https://github.com/hanczaryk/MLApp to add 1MB objects to a vector. This MLApp has been built and deployed on our WebSphere Application Server traditional server. Prior to utilizing MLApp application to add objects to a vector, the Instana UI screenshot shows the following Heap Memory graph for our server.

Now using a shell script as shown in the README for https://github.com/hanczaryk/MLApp to add 325 MB, we've increased the used heap on our server to be over 80 percent of the heap. The following two Instana UI screenshots show the dramatic increase for our server's heap memory and that 2 issues were detected. One of the issues was a built-in event for high Garbage Collection Activity and the other was for our custom event BPHZone_Heap_Over_80.

When the alert was triggered for our custom event, our alert channel was notified which starts the health investigation. The following WebSphere Automation UI screenshot shows that a health investigation has completed.

In the WebSphere Automation UI, if we click on the investigation, then more detailed information about the investigation will appear in the right panel. The user will observe the following information :

When the issue was detected
Investigation ID
Problem type
Status
Server
Host
Cell
Why investigation started
Analysis summary
Link to download all the files associated with the investigation
History of investigation steps

The following are two WebSphere Automation UI screenshots showing this information for the above investigation.

You can find more IBM Docs related to WebSphere Automation at https://www.ibm.com/docs/en/ws-automation and more IBM Docs related to Instana at https://www.ibm.com/docs/en/obi.
#websphere#automation#WSA#WebSphereAutomation

0 comments

43 views

IBM Application Runtimes Community

Come for answers, stay for best practices. All we're missing is you.

WebSphere Application Server & Liberty