Hi Shanmuga,
You really need to upgrade to v10, even if using a v5 compatible gateway, as v5 has been out of support now for a couple years now. Having said that, this was a common v5 issue back in the day. A colleague of mine pulled out a file he'd saved with something I'd provided to a customer years ago and quite frankly I'd forgotten about, but it gives a lot of very useful information. I'll cut and paste with some modifications to assist you:
The analytics records for the wsm agent are written as XML, and these records are provided by the wsm push subscriber to a stylesheeet local:///isp/wsm-post-analytics.xsl and combined into "bulk" records of up to 100 records where each record in that batch is converted to a JSON equivalent. These bulk records are sent to the analytics server. When an individual XML record is greater than the wsm agent's xmlmanager's parser limits, the error you are encountering is produced. This record becomes a "poison" record that would prevent subsequent records from being sent to analytics as a failure for any record will then cause the batch to be retried (and fail again due to this record) until the wsm buffer is full at which time no further records are sent to analytics until the domain is reloaded. This would be the case where the wsm agent status would be similar to the following:
<?xml version="1.0" encoding="UTF-8"?>
<Status>
<WSMAgentStatus>
<SpoolerCount>1</SpoolerCount>
<PolledSpoolers>1</PolledSpoolers>
<RecordsSeen>72113</RecordsSeen>
<RecordsLost>175859</RecordsLost>
<PendingRecords>0</PendingRecords>
<CompleteRecords>1</CompleteRecords>
<MemoryUsedKB>80899</MemoryUsedKB>
</WSMAgentStatus>
</Status>
The memory used is greater than the 64000KB and a significant number of records are being lost as there is no buffer available to store them. The push-subscribers hidden feature in API Connect uses a non-configurable XML manager called _wsm_ which has a document size limit of 16 MB in DataPower. It is not possible to configure this XML manager; however, it is possible to have the push-subscribers use a different XML manager in lieu of the standard _wsm_ used. To do this, use the following steps:
1. Create an XML manager which you wish to use for this purpose. Configure this XML manager on the XML parser tab from the GUI to have an XML bytes scanned / XML max node size of a desired value. Note that both the response and request payloads will be captured when payload logging is configured, the payload will be converted to an XML representation (JSONx), and additional information such as headers will be captured. All of this will make the actual size of the analytics data captured much larger than the size of an individual JSON payload. Set the XML External Reference Handling to "Allow" on the same tab. This is a requirement for the XML manager used in this case.
2. Modify the configuration of the wsm-agent using the CLI in the API Connect application domain (it is not possible to do it from the Legacy or Carbon GUI) to add the new XML manager (xml-manager) to the push-subscribers section. It may also be worth increasing the wsm-agent's max-memory setting, but otherwise without changing any of the other settings. For example, the new wsm-agent configuration would look something like the following when viewing the config:///domainname.cfg file for the domain:
wsm-agent
admin-state enabled
max-records 6000
max-memory 128000
capture-mode faults
buffer-mode discard
no mediation-enforcement-metrics
max-payload-size 0
push-subscribers
context "x2020"
url "http://127.26.10.243:2444/x2020/v1/events/_bulk"
push-processor local:///isp/wsm-post-analytics.xsl
damping-interval 1000
max-damping-limit 60000
max-push-records 0
xml-manager my-xml-manager <============== This is the ONLY thing different from what is in the existing domain.cfg file!!!!
exit
push-interval 100
push-priority normal
exit
Note that in the push subscriber url configuration, ensure the host IP is what is currently configured in the config:///domainname.cfg file. As noted above, it should be identical other than the xml manager property. After this, the wsm-agent will use the my-xml-manager (or whatever name you would decide to use) XML manager instead of the default _wsm_ XML manager.
At the time I wrote this many DataPower releases ago, I wrote: "although the XML manager configuration itself can be added to a gateway extension to persist the XML manager configuration across gateway deployments and upgrades, the update to the push subscriber would need to be done every time the gateway is removed and added." With the current DataPower releases, I'm not sure that is true, so you may attempt to test and verify this yourself. If you were to export your wsm agent and look at the export.xml within that zip as I just did, you'll see something like
<WebServicesAgent name="default" intrinsic="true" xmlns:dp="http://www.datapower.com/schemas/management" xmlns:env="http://www.w3.org/2003/05/soap-envelope">
<mAdminState>enabled</mAdminState>
<MaxRecords>6000</MaxRecords>
<MaxMemoryKB>64000</MaxMemoryKB>
<CaptureMode>faults</CaptureMode>
<BufferMode>discard</BufferMode>
<MediationMetrics>off</MediationMetrics>
<MaxPayloadSizeKB>0</MaxPayloadSizeKB>
<PushSubscribers>
<Context>x2020</Context>
<URL>http://127.240.34.22:2444/x2020/v1/events/_bulk</URL>
<PushProcessor>local:///isp/wsm-post-analytics.xsl</PushProcessor>
<DampingInterval>1000</DampingInterval>
<MaxDampingLimit>60000</MaxDampingLimit>
<MaxPushRecords>0</MaxPushRecords>
<XMLManager/> <!-- PLACE YOUR XML MANAGER OBJECT NAME HERE!!! -->
</PushSubscribers>
<PushInterval>100</PushInterval>
<PushPriority>normal</PushPriority>
</WebServicesAgent>
I don't know if you have an existing v5 gateway extension or not. If not, then you can have a simple export that would contain your new XML Manager and your Web Services Management Agent where you've updated that empty element for the XML Manager name in the export.xml of that zip. That should suffice as a valid v5 Gateway Extension. If you already have an existing extension, then the XML for the new wsm XML Manager and wsm object will need to be added to the export.xml of your existing Gateway Extension. Then deploy or update the extension in CMC. Hopefully that will persist these changes via a Gateway Extension, but as I said, please test and verify as we don't do anything with v5 anymore. If it doesn't work, then you'll need to have a process to ensure this update to the wsm manager on each DataPower appliance since it would be done outside of the CMC Gateway Extension update process.
Best Regards,
Steve
------------------------------
Steve Linn
Senior Consulting I/T Specialist
IBM
------------------------------
Original Message:
Sent: Tue June 11, 2024 01:44 AM
From: Shanmuga Rajendra
Subject: How to Enable web-services-management-agent in v10
Hi Steve,
We are still using APIC V5 and seeing below error in pushing API event records from DP Gateway to analytics. Any idea how to fix below error please?
xslt | error | 06/11 11:11:14.014 | scheduled | xmlmgr | _wsm_ | | 0x80c00010 | Processing of 'local:///isp/wsm-post-analytics.xsl' stopped: Max node size exceeded |
wsm-agent | info | 06/11 11:11:09.101 | | wsm-agent | default | | 0x80e00670 | Subscriber x2020 read 262 records(262) |
xslt | error | 06/11 11:10:08.915 | scheduled | xmlmgr | _wsm_ | | 0x80c00010 | Processing of 'local:///isp/wsm-post-analytics.xsl' stopped: Max node size exceeded |
wsm-agent | info | 06/11 11:10:05.734 | | wsm-agent | default | | 0x80e00670 | Subscriber x2020 read 262 records(262) |
------------------------------
Shanmuga Rajendra
Datapower admin
Original Message:
Sent: Thu May 25, 2023 02:37 PM
From: Steve Linn
Subject: How to Enable web-services-management-agent in v10
Hi Santhoshkumar,
Version 5 and v10's v5 compatible gateways use the WSM Agent, generate a wsm data record (which is XML based) for each transaction where analytics is requested, and utilize an undocumented wsm push subscriber to take those wsm data records, convert them to a batch of JSON records and push them to the v5 analytics server. For the v10 API Gateway Service, wsm is not utilized, but there is a builtin analytics engine built into the service. You do not need to enable the wsm agent for the API Gateway.
Regards,
Steve
------------------------------
Steve Linn
Senior Consulting I/T Specialist
IBM
Original Message:
Sent: Mon May 15, 2023 04:34 PM
From: santhoshkumar surisetty
Subject: How to Enable web-services-management-agent in v10
could you please guide how to enable Web Services Management Agent for v10 as all objects are deployed to API Gateway object.
https://community.ibm.com/community/user/integration/blogs/hermanni-pernaa1/2021/08/23/how-to-use-web-services-management-agent-subscript?CommunityKey=b13f4693-16ee-422b-9d0b-f5262e25426f
------------------------------
santhoshkumar surisetty
------------------------------