DataPower is shutting down and running out of memory when we attempt to make 100,000 url-open calls to MQ. We are receiving a single SOAP payload through APIC that contains over 100,000 nested structures. Each of these nested structures needs to be extracted and sent to MQ as an individual message.
IBM has recommended processing the data in chunks, such as 0–49, 50–99, and so on.
Can anyone help explain how we can implement this? Or please suggest if there are any alternative approaches.
Original Message:
Sent: Wed May 21, 2025 12:59 AM
From: Kumar .
Subject: DataPower Restart & API Connect on High Volume XSLT
Hi Hermanni,
DataPower is out of memory throttle during the 100k message test. However, messages were sometimes pulled by ACE from MQ. In the middle of the pulling process, DataPower would restart. After all the messages were pulled, APIC expected a response from ACE.
At that point, the ACE logs showed the following error:
"An error occurred whilst performing a TCP/IP socket operation. The socket was closed before all expected data was received, as the end of the HTTP headers has not been found."
Could you please elaborate on how to implement the following suggestion?
"If you expect to receive a SOAP input containing more than 100k nested structures that you want to loop through, I would recommend splitting the URL open/send operations from a single XSLT into multiple processing actions."
------------------------------
Kumar
Original Message:
Sent: Tue May 20, 2025 09:32 AM
From: Hermanni Pernaa
Subject: DataPower Restart & API Connect on High Volume XSLT
Hi Kumar,
If you expect to get an input SOAP containing more than 100K nested structures that you want to loop through I would try to split the url-open sending from one single XSLT to multiple processing actions. Have you performed any analysis on the issue? Are you running out CPU, memory or TCP/IP sockects which causes the DataPower to restart?
------------------------------
Hermanni Pernaa
Solutions Architect
Digia Plc
Helsinki
Original Message:
Sent: Mon May 19, 2025 08:29 AM
From: Kumar .
Subject: DataPower Restart & API Connect on High Volume XSLT
Dear All,
We have to process a SOAP payload containing a large number of nested loops (up to 200K). The XSLT iterates through these loops and sends each one individually to MQ using dp:url-open. So it will be 200K dp:url-open calls to MQ.
This works fine for smaller payloads(4 or 5 nested loops) when we test from SoapUI , but when we test with large volumes (100K+) through an application, we encounter:
- Automatic DataPower restarts
- API Connect Gateway Peering Manager going down.
- The request is not even logging in analytics but error in the backend system is
"An error occurred whilst performing a TCP/IP socket operation. the socket was closed before all expected data was received, as the end of the http headers
have not been found."
Can anyone share your Inputs?
------------------------------
Kumar
------------------------------