Stuffing so much information in single SOAP message is an antipattern. It won't be scalable regardless of the approach taken. This is because of size limitation within any XML processing engine. As a stop gap solution, you can try to increase XML size in xml manager. Also try to switch to GatwayScript for url-open call. Gateway script uses async model of computation and might provide you expected result.
Original Message:
Sent: Tue May 27, 2025 03:49 AM
From: Kumar .
Subject: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration
Dears,
DataPower is shutting down and running out of memory when we attempt to make 100,000 url-open
calls to MQ. We are receiving a single SOAP payload through APIC that contains over 100,000 nested structures. Each of these nested structures needs to be extracted and sent to MQ as an individual message.
IBM has recommended processing the data in chunks, such as 0–49, 50–99, and so on.
Can anyone help explain how we can implement this? Or please suggest if there are any alternative approaches.
------------------------------
Kumar
Original Message:
Sent: Thu May 15, 2025 06:49 AM
From: Kumar .
Subject: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration
Hello @Ajitabh Sharma,
Were you saying that the SOAP message won't go to the invoke step until the entire for-each loop is completed?
In your case it should be 0 for success scenario and non-zero for failed scenarios. By checking it you will get to know why some of the messages are not getting into MQ.
I'm not saying that some messages aren't reaching the MQ. My point is: if there are 10 nested loops in the given for-each
path, the SOAP message should not be sent to the invoke
step until all 10 messages have been placed in the queue.
Currently, both actions (sending to MQ and calling invoke) seem to be happening in parallel. I'm not sure if this is the expected behavior, as I'm not the one monitoring the MQ or the endpoint configured in the invoke.
Could you help with sample retry mechanism for dp:url-open?
------------------------------
Kumar
Original Message:
Sent: Thu May 15, 2025 06:32 AM
From: Ajitabh Sharma
Subject: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration
The XSLT code is good and should work as expected. Now if you are facing issue with MQ, especially the fact that all messages don't get onboarded to it, you might want to check response code of each dp:url-open in XSLT. In your case it should be 0 for success scenario and non-zero for failed scenarios. By checking it you will get to know why some of the messages are not getting into MQ.
As a side note, using dpmq:// is deprecated and you should be using idgmq://. Also building a retry mechanism for failed dp:url-open call is suggested.
------------------------------
Ajitabh Sharma
Original Message:
Sent: Thu May 15, 2025 05:01 AM
From: Kumar .
Subject: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration
Hello @Ajitabh Sharma
Please find below the for-each
loop to put the messages in the queue. There will be many messages to be put in Queue once they'll are queued then it should sent <soap:Envelope>
to the invoke operation.
<xsl:for-each select="$base/business/data">
<xsl:variable name="msg">
<InsertMsg>
<Test>
<xsl:value-of select="$data"/>
</Test>
<xsl:apply-templates select="."/>
</InsertMsg>
</xsl:variable>
<!-- Send to MQ -->
<dp:url-open target="dpmq://QM/?RequestQueue=Queuename" response="ignore">
<xsl:copy-of select="$msg"/>
</dp:url-open>
</xsl:for-each>
<soap:Envelope
xmlns:soap="http://www.w3.org/2003/05/soap-envelope"
xmlns:pay="http://test.com">
<soap:Header/>
<soap:Body>
<pay:LastMsg>
<Test>
<xsl:value-of select="$data"/>
</Test>
</pay:LastMsg>
</soap:Body>
</soap:Envelope>
------------------------------
Kumar
Original Message:
Sent: Thu May 15, 2025 04:21 AM
From: Ajitabh Sharma
Subject: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration
Please share your code files so that I can recommend.
------------------------------
Ajitabh Sharma
Original Message:
Sent: Thu May 15, 2025 03:52 AM
From: Kumar .
Subject: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration
Dear @Ajitabh Sharma
I have a nested loop in my SOAP Input, and I'm placing each set from the nested loop as a separate entry in the MQ. Once all the loop iterations have been queued, I need to send another message to the invoke operation. This message I am constructing in the same xslt with some details. This message won't be the same as that I am putting in MQ.
However, the issue is that the invoke call is being triggered before all the messages are pushed to the queue. How can I ensure that the invoke call happens only after all messages have been successfully pushed to the queue?
Your support please.
------------------------------
Kumar
Original Message:
Sent: Tue May 13, 2025 12:37 AM
From: Ajitabh Sharma
Subject: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration
Most of the answers provided in this thread will work. To summarize:
- In case both your transactions (sending data to MQ and subsequent API call) needs to go together, you can have a loop and inside that loop you can pick one SOAP message and perform both actions using url-open call (one after another). You can evaluate responses as well and decide to rollback data from MQ depending upon failure in subsequent API call (if that is what is required).
- If you wish to keep track of original transaction data (all of them) then you got to store them somewhere so that you can replay them if required. As a result, you will be able to replay entire transaction in case that is required. But this would require integration of event streaming system with DataPower.
The transaction requirements, especially error handling is not very clear from the question. So, these are the two broadways in which you can handle the scenario.
------------------------------
Ajitabh Sharma
Original Message:
Sent: Tue May 06, 2025 05:26 PM
From: Kumar .
Subject: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration
Dears,
When a user sends a request, we will capture the entire SOAP message and store it in a variable. Then, using XSLT, we will extract each nested element from the SOAP body iteratively. In the first iteration, we will extract the first nested node, save it, and in the second iteration, we will extract the second nested node, save it as well. Each of these extracted messages will then be sent to a message queue (MQ) for processing later send the same message to invoke. How can we implement this process efficiently? Find the attached sample request.
Any references will be helpful.
------------------------------
Kumar
------------------------------