DataPower

How to use Web Services Management Agent subscriptions

By Hermanni Pernaa posted Thu August 26, 2021 08:30 AM

  
IBM DataPower Web Services Management Agent provides excellent out-of-the-box capabilities for monitoring your transactions and integrating with external monitoring and log management platforms. Although the DataPower documentation describes the steps to enable the functionality and provides samples for the different types of subscriptions there aren't that many how-to guides available.The purpose of this blog post is to show how to configure the Web Services Management Agent, create a pull subscription and integrate the agent with a commercial monitoring product. 

Step 1: Enabling the Web Services Management Endpoint

The Web Services Management endpoint setting can be found from default domain under XML Management Interface. Verify that the check box is selected and then click Cancel or if not selected click the check box and then apply and save the config.

Enabling the Web Services Management endpoint 

Step 2: Enable domain Web Services Management Agent

The second step is to enable the domain specific management agent. Change to the domain that contains the services that you want to monitor and navigate to the Web Services Management Agent settings.


Change the state to enabled and define the configuration parameters that are suitable for your needs. For demonstration purposes I'll set the capturing of all messages on and buffering mode (which is by the way deprecated) to buffer so that the records are stored even if my example subscription should time out. Then apply and save the configuration changes.
 
Enabling the Web Services Management Agent

Step 3: Configure your proxy service

The next step is to configure the service specific parameters. An example for a Multi-Protocol Gateway is shown in the picture below. We'll capture all messages for demonstration purposes (this comes with a performance penalty).

Configuring service specific params

All the steps from one to three are straight from IBM Knowledge Center and can be easily defined using the basic documentation. So let's go to the bits that are not directly available in the documentation....

Step 4: Verify/define your monitoring endpoint

This chapter describes the steps that you'll need to perform in order to ensure that your monitoring service has an endpoint which can be used to receive the events that the DataPower monitoring agent emits. Obviously the steps vary depending on the monitoring product that you are using but as an example I'll show the configurations related to Splunk Enterprise V8.

We'll be using a Splunk HTTP Event Collector (HEC) for receiving the events so lets create a new data input. First navigate to Settings and select Data Inputs.

Splunk settings
Create a new HTTP Event Collector by clicking Add New.

Creating a new HTTP Event Collector

Next, fill in the basic information. Example is shown below.

Configuring Splunk HEC - step 1
Next fill in the settings for sourcetype and indexes. These configurations are very relevant so fill them carefully. Lets select the sourcetype as _json because even though the DataPower events come in as SOAP, we'll modify them on the fly to match the JSON format expected by the Splunk endpoint. After filling in the correct values, review and save the configuration.

Splunk HEC sourcetype
After creating a new HEC input you'll have to verify that all the tokens are enabled and what are the port number and SSL settings of the HEC endpoint. From the HTTP Event Collector page open Global Settings and verify the values according the picture below.

Verify HEC endpoint settings

Step 5: Create a subscription

Now that we have the connection details related to the monitoring endpoint we can create a subscription for consuming the events emitted by the DataPower. The "problem" that we first have to address is the fact that the Web Services Management Agent push subscription isn't strictly fire-and-forget even though it doesn't require acknowledgements following any specification. The DataPower agent expects to receive a SOAP response (any kind of SOAP response is ok, SOAP fault is not ok) and if your monitoring endpoint support SOAP request-response pattern you are good to go. Unfortunately, because I am using Splunk HEC endpoint to consume the monitoring events, the monitoring events and related responses are expected to be JSON messages so I have to modify the messages somewhere in the middle. There are multiple ways to achieve this, for example creating a custom input for Splunk, but because this blog post is about DataPower lets use a simple XML Firewall to do the trick. 

Assuming that creating a new XML Firewall is a familiar process I am not going to include any screen captures of the steps. Please see the export of the XML FW that is included as an attachment to the end of this blog post. To summarize the processing policy follows the steps presented in the picture below.
 
XML Firewall processing policy
In addition to setting up the proxy and the processing flow presented above we also need to add a client TLS profile and a basic-auth policy, because, as we saw in the previous step, HEC endpoint has SSL enabled and the authentication tokens enabled. To perform the needed configurations first navigate to the user agent, that is connected to the newly created XML Firewall, settings. First we'll add a TLS policy that matches to HEC endpoint URI(s). 

User Agent TLS policy Next we'll add a basic authentication policy for which we'll need the HEC token from the Splunk configuration. The token can be found from the HEC inventory page, an example is shown below.

HEC token for the DataPower Agent
The token can then be used to create the authentication policy as shown below. URL matching expression can be set to match Splunk HEC (*/services/collector/*) and username to Splunk. Then we'll need to create a new password alias that contains the authentication token.  The end result is shown in the third picture below.

Create a basic auth policy
Create a password alias
Completed basic authentication policyAll the configurations are now ready and we can create the actual subscription using DataPower XML Management interface. Below is an example push subscription request that has been modified from the ready-made samples. The request schema follows the Web Services Eventing specification which can be found for example from https://www.w3.org/Submission/WS-Eventing/.

<?xml version="1.0" encoding="UTF-8"?>
<env:Envelope
    xmlns:wsman="http://schemas.xmlsoap.org/ws/2005/02/management"
    xmlns:wse="http://schemas.xmlsoap.org/ws/2004/08/eventing"
    xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing"
    xmlns:env="http://www.w3.org/2003/05/soap-envelope"
    xmlns:test="http://myexample.com/wsm-test">
  <env:Header>
    <wsa:To>/wsman?ResourceURI=(wsman:datapower.com/resources/2005/07/domain)</wsa:To>
    <wsa:ReplyTo>
      <wsa:Address env:mustUnderstand="true">
        http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous
      </wsa:Address>
    </wsa:ReplyTo>
    <wsa:Action env:mustUnderstand="true">
      http://schemas.xmlsoap.org/ws/2004/08/eventing/Subscribe
    </wsa:Action>
    <wsman:MaxEnvelopeSize env:mustUnderstand="true">51200</wsman:MaxEnvelopeSize>
    <wsa:MessageID>xsltmark:sub-push-req-001</wsa:MessageID>
    <wsman:OperationTimeout>PT60.000S</wsman:OperationTimeout>
    <wsman:System>wsman:datapower.com/resources/2005/07/ws-management</wsman:System>
    <wsman:SelectorSet Name="dp-domain">
      <wsman:Selector Name="domain">wsm-test</wsman:Selector>
    </wsman:SelectorSet>
  </env:Header>
  <env:Body>
    ​<wse:Subscribe>
      <wse:Delivery Mode="http://schemas.xmlsoap.org/ws/2005/02/management/Events">
        <wse:NotifyTo>
          <wsa:Address>http://127.0.0.1:2048</wsa:Address>
          <wsa:ReferenceProperties>
            <test:Subscription>splunk-subscription</test:Subscription>
          </wsa:ReferenceProperties>
        </wse:NotifyTo>
        <wsman:MaxElements>20</wsman:MaxElements>
        <wsman:MaxTime>PT60S</wsman:MaxTime>
      </wse:Delivery>
      <wse:Expires>PT10M</wse:Expires>
    </wse:Subscribe>
  </env:Body>
</env:Envelope>


The relevant fields for the subscription in this case are Subscribe/Deliver/NotifyTo/Address and the Subscribe/Expires that tells how long the subscription is active (ISO 8601 format, in my case 10 minutes). For the delivery address, if your monitoring platform supports SOAP messaging you can just fill in the event sink address, but in my case I'll use a localhost address that matches the newly created XML Firewall that handles the message transformations. Once the request is sent to the DataPower XML Management interface we should get back an ok response that is described below.

$ curl -d @create-push-subscription.xml -H "Content-Type: text/xml" -u admin:xxxxx -k https://localhost:5550/service/mgmt/3.0
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3050    0  1300  100  1750   3834   5162 --:--:-- --:--:-- --:--:--  9023

<?xml version="1.0" encoding="UTF-8"?>
<env:Envelope xmlns:wsman="http://schemas.xmlsoap.org/ws/2005/02/management" xmlns:wse="http://schemas.xmlsoap.org/ws/2004/08/eventing" xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:env="http://www.w3.org/2003/05/soap-envelope" xmlns:test="http://myexample.com/wsm-test">
  <env:Header xmlns:wsen="http://schemas.xmlsoap.org/ws/2004/09/enumeration">
    <wsa:To>http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</wsa:To> 
    <wsa:Action>http://schemas.xmlsoap.org/ws/2004/08/eventing/SubscribeResponse</wsa:Action> 
    <wsa:RelatesTo>xsltmark:sub-push-req-001</wsa:RelatesTo>
  </env:Header>
  <env:Body xmlns:wsen="http://schemas.xmlsoap.org/ws/2004/09/enumeration"> 
    <wse:SubscribeResponse>
      <wse:SubscriptionManager>
        <wsa:Address>/wsman?ResourceURI=(wsman:datapower.com/resources/2005/07/ws-gateway)</wsa:Address>
        <wsa:ReferenceProperties>
          <dpt:Domain xmlns:dpt="http://www.datapower.com/schemas/transactions">wsm-test</dpt:Domain>
          <wse:Identifier>uuid:2728e77d-817c-42c2-a261-b239e99a9a3a</wse:Identifier> 
        </wsa:ReferenceProperties>
      </wse:SubscriptionManager> 
      <wse:Expires>PT5M</wse:Expires>
      <wsen:EnumerationContext>uuid:2728e77d-817c-42c2-a261-b239e99a9a3a</wsen:EnumerationContext>
    </wse:SubscribeResponse>
  </env:Body>
</env:Envelope>​

The response should contain an identifier that shows that the subscription was created successfully. The subscription can also be verified for example from the WebGUI as shown below.
WSM agent subscribers

Step 5: Test the subscription

Now that the subscription is active we can now run a test and see if the monitoring event is successfully consumed by Splunk. Lets send a simple JSON to the example Multi-Protocol Gateway using cURL. 
$ curl -d '{"request": {"text": "This is my request"}}' -H "Content-Type: application/json" http://localhost:8080/mytestservice
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   100    0    57  100    43    263    199 --:--:-- --:--:-- --:--:--   465
{"response":{"text":"This is a simple response message"}}


As the response indicates DataPower has successfully processed the request message and responded accordingly. Lets check the if we have a corresponding event in Splunk. Running a search against integration index shows an event that contains the automatically extracted details related to the transaction in question. 

Splunk search - 15 min window integration index

Since we defined the sourcetype as JSON and transformed the SOAP message emitted by the DataPower to match the Splunk HEC input schema all fields get automatically "parsed" and can be used as search attributes. Below is an example of a Splunk table view that makes it possible to process and visualize the transaction data even further.
Custom search with automatically parsed fieldsHope this post gives you a bit more details on the Web Services Management agent and some new ideas on how to benefit from it.

As promised, here is an export of the proxy service that was used to transform the monitoring events between DataPower and Splunk HEC,
wsm-agent-proxy.zip

Permalink