API Connect

API Connect

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only
Expand all | Collapse all

Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

  • 1.  Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted Tue May 06, 2025 05:27 PM
    Edited by Kumar . Wed May 07, 2025 01:37 AM

     Dears,

    When a user sends a request, we will capture the entire SOAP message and store it in a variable. Then, using XSLT, we will extract each nested element from the SOAP body iteratively. In the first iteration, we will extract the first nested node, save it, and in the second iteration, we will extract the second nested node, save it as well. Each of these extracted messages will then be sent to a message queue (MQ) for processing later send the same message to invoke. How can we implement this process efficiently? Find the attached sample request.

    Any references will be helpful. 



    ------------------------------
    Kumar
    ------------------------------

    Attachment(s)

    txt
    test-soap.txt   3 KB 1 version
    txt
    nested2.txt   2 KB 1 version
    txt
    nested1.txt   2 KB 1 version


  • 2.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted Wed May 07, 2025 01:37 AM

    Hello,

    Can anyone help with your Insights please?



    ------------------------------
    Kumar
    ------------------------------



  • 3.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted Wed May 07, 2025 02:59 AM

    I haven't used XSLT for about 5 years now, so I am interested in the topic.

    I imagine you can achieve this with standard identity template and the dp:url-open extension.

    I'm sure there are better ways to write this XSLT however, it should suffice.

    <?xml version="1.0" encoding="utf-8"?>
    <xsl:stylesheet version="1.0"
      xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
      xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"
      xmlns:svc="http://example.com/service"
      xmlns:dp="http://www.datapower.com/extensions"
      extension-element-prefixes="dp"
      exclude-result-prefixes="dp">
    
      <xsl:output method="xml" />
    
      <!-- Standard identity template -->
      <xsl:template match="@* | node()">
        <xsl:copy>
          <xsl:apply-templates select="@* | node()" />
        </xsl:copy>
      </xsl:template>
    
      <!-- Rewrite each svc:Transaction message to dp:url-open -->
      <xsl:template match="svc:Transaction">
        <dp:url-open target="idgmq://manager/URI?RequestQueue=requestQueue;queryParameters">
          <soapenv:Envelope>
            <xsl:apply-templates select="/soapenv:Envelope/soapenv:Header" />
            <soapenv:Body>
              <svc:MainRequest>
                <xsl:apply-templates select="//svc:MainRequest/svc:TotalAmount" />
                <xsl:apply-templates select="//svc:MainRequest/svc:TotalCount" />
                <xsl:apply-templates select="//svc:MainRequest/svc:PaymentCategory" />
                <svc:Transactions>
                  <svc:Transaction>
                    <xsl:apply-templates select="@* | node()" />
                  </svc:Transaction>
                </svc:Transactions>
                <xsl:apply-templates select="//svc:MainRequest/svc:AgencyCode" />
                <xsl:apply-templates select="//svc:MainRequest/svc:EffectiveDate" />
              </svc:MainRequest>
            </soapenv:Body>
          </soapenv:Envelope>
        </dp:url-open>
      </xsl:template>
    </xsl:stylesheet>


    ------------------------------
    Brendon Stephens
    ------------------------------



  • 4.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted Wed May 07, 2025 06:24 AM
    Edited by Kumar . Wed May 07, 2025 06:36 AM

    @Brendon Stephens    First of all, thank you for sharing the reference and guidance-it has been very helpful. However, I'm currently facing an issue where the message is not being pushed to the MQ as expected.

    To address this properly, I would like to share my code one-on-one. Could you please share your email address so I can provide the full context securely?



    ------------------------------
    Kumar
    ------------------------------



  • 5.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted Wed May 07, 2025 06:15 PM

    Keep the messages public, for the benefit of the community.



    ------------------------------
    Brendon Stephens
    ------------------------------



  • 6.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted Thu May 08, 2025 08:10 AM

    Hi Kumar,

    just wanted to add that in addition to the problem description can you please provide a bit more details on the use case? For example how should the messages be sent to MQ? Is the expected action a fire-and-forget type of thing or are we expecting a response message back from MQ? 



    ------------------------------
    Hermanni Pernaa
    Solutions Architect
    Digia Plc
    Helsinki
    ------------------------------



  • 7.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted Fri May 09, 2025 04:24 PM

    @Hermanni PernaaYes, we just need to put the message in the queue and not expecting any response, but I also need to send the same message to another API endpoint and here we would receive a message from the other endpoint. I was able to put the message in MQ using dp:url-open, but for the other endpoint, can I use another dp:url-open in the same XSLT to send the same message to the other backend?




    ------------------------------
    Kumar
    ------------------------------



  • 8.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted Fri May 09, 2025 09:06 PM
    Hi Kumar,

    You can use another dp-url but I would do this way... 

    Since I already sent message to mq, I will output the required message/request to next action which would be the invoke(your second call)  From here I don't need to handle the response it directly goes to the client and if we have to handle the response received we can do that aswel. 





  • 9.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted Sun May 11, 2025 02:26 AM

    Hello Pratap,

     I have a SOAP API where the incoming request contains multiple nested transaction loops. I need to split each of these transactions, generate individual SOAP messages, and send each message to an MQ queue. After sending the message to the queue, As I mentioned before, I also need to send the same message to another endpoint within the same session.  Still can I use Invoke in this case? I have stored the message  in a variable called soapMsg and could be able to place it in Queue. How to use the same to send it Invoke in the same session? 

    <dp:url-open target="dpmq://...RequestQueue=..." response="ignore">
      <xsl:copy-of select="$soapMsg"/>
    </dp:url-open>



    ------------------------------
    Kumar
    ------------------------------



  • 10.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted Sun May 11, 2025 09:52 PM
    Hi Kumar,

    Yes, you can use it Or anyways you are storing the message in variable so you can just output that to the invoke and you can proceed with the flow. 

    Case2 : if you want to wait for the response of previous request then you should use promisify from util library to make it easy






  • 11.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted Mon May 12, 2025 07:08 AM

    Hi Kumar,

    as Pratap stated you can definitely use an url-open call for sending the second message inside the iterative "loop", and that is most likely the easiest way to implement the "happy case" scenario. I think It pretty much comes down to the complexity of the error handling that is needed. Like what if one or several subsequent calls fail and some are processed successfully? Do you need to terminate the whole transaction or do you need to keep processing the batch all the way through or perhaps compensate the successful ones? Just trying to highlight that you can implement the logic with several different ways but you need to evaluate to whole use case and the requirements before selecting the most suitable option.  



    ------------------------------
    Hermanni Pernaa
    Solutions Architect
    Digia Plc
    Helsinki
    ------------------------------



  • 12.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted Tue May 13, 2025 12:37 AM

    Most of the answers provided in this thread will work. To summarize:

    1. In case both your transactions (sending data to MQ and subsequent API call) needs to go together, you can have a loop and inside that loop you can pick one SOAP message and perform both actions using url-open call (one after another). You can evaluate responses as well and decide to rollback data from MQ depending upon failure in subsequent API call (if that is what is required).
    2. If you wish to keep track of original transaction data (all of them) then you got to store them somewhere so that you can replay them if required. As a result, you will be able to replay entire transaction in case that is required. But this would require integration of event streaming system with DataPower. 

    The transaction requirements, especially error handling is not very clear from the question. So, these are the two broadways in which you can handle the scenario. 



    ------------------------------
    Ajitabh Sharma
    ------------------------------



  • 13.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted Tue May 13, 2025 03:56 PM

    Dears,

    Thank you everyone for sharing your Insights.



    ------------------------------
    Kumar
    ------------------------------



  • 14.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted Thu May 15, 2025 03:52 AM
    Edited by Kumar . Thu May 15, 2025 03:57 AM

    Dear @Ajitabh Sharma

    I have a nested loop in my SOAP Input, and I'm placing each set from the nested loop as a separate entry in the MQ. Once all the loop iterations have been queued, I need to send another message to the invoke operation. This message I am constructing in the same xslt with some details. This message won't be the same as that I am putting in MQ.

    However, the issue is that the invoke call is being triggered before all the messages are pushed to the queue. How can I ensure that the invoke call happens only after all messages have been successfully pushed to the queue?

    Your support please.



    ------------------------------
    Kumar
    ------------------------------



  • 15.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted Thu May 15, 2025 04:21 AM

    Please share your code files so that I can recommend.



    ------------------------------
    Ajitabh Sharma
    ------------------------------



  • 16.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted Thu May 15, 2025 05:01 AM
    Edited by Kumar . Thu May 15, 2025 05:12 AM

    Hello @Ajitabh Sharma

    Please find below the for-each loop to put the messages in the queue. There will be many messages  to be put in Queue once they'll are queued then it  should sent <soap:Envelope> to the invoke operation.

    <xsl:for-each select="$base/business/data">

    <xsl:variable name="msg">
                    <InsertMsg>
                        <Test>
                            <xsl:value-of select="$data"/>
                        </Test>
                        <xsl:apply-templates select="."/>
                    </InsertMsg>
                </xsl:variable>
                <!-- Send to MQ -->
                <dp:url-open target="dpmq://QM/?RequestQueue=Queuename" response="ignore">
                    <xsl:copy-of select="$msg"/>
                </dp:url-open>
            </xsl:for-each>
            <soap:Envelope
                xmlns:soap="http://www.w3.org/2003/05/soap-envelope"
                xmlns:pay="http://test.com">
                <soap:Header/>
                <soap:Body>
                    <pay:LastMsg>
                        <Test>
                            <xsl:value-of select="$data"/>
                        </Test>
                    </pay:LastMsg>
                </soap:Body>
            </soap:Envelope>



    ------------------------------
    Kumar
    ------------------------------



  • 17.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted Thu May 15, 2025 06:32 AM

    The XSLT code is good and should work as expected. Now if you are facing issue with MQ, especially the fact that all messages don't get onboarded to it, you might want to check response code of each dp:url-open in XSLT. In your case it should be 0 for success scenario and non-zero for failed scenarios. By checking it you will get to know why some of the messages are not getting into MQ. 

    As a side note, using dpmq:// is deprecated and you should be using idgmq://. Also building a retry mechanism for failed dp:url-open call is suggested.

     



    ------------------------------
    Ajitabh Sharma
    ------------------------------



  • 18.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted Thu May 15, 2025 06:49 AM

    Hello @Ajitabh Sharma,

    Were you saying that the SOAP message won't go to the invoke step until the entire for-each loop is completed?

    In your case it should be 0 for success scenario and non-zero for failed scenarios. By checking it you will get to know why some of the messages are not getting into MQ. 

    I'm not saying that some messages aren't reaching the MQ. My point is: if there are 10 nested loops in the given for-each path, the SOAP message should not be sent to the invoke step until all 10 messages have been placed in the queue.

    Currently, both actions (sending to MQ and calling invoke) seem to be happening in parallel. I'm not sure if this is the expected behavior, as I'm not the one monitoring the MQ or the endpoint configured in the invoke.

    Could you help with sample retry mechanism for dp:url-open?



    ------------------------------
    Kumar
    ------------------------------



  • 19.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted 26 days ago

    Dears,

    DataPower is shutting down and running out of memory when we attempt to make 100,000 url-open calls to MQ. We are receiving a single SOAP payload through APIC that contains over 100,000 nested structures. Each of these nested structures needs to be extracted and sent to MQ as an individual message.

    IBM has recommended processing the data in chunks, such as 0–49, 50–99, and so on.

    Can anyone help explain how we can implement this? Or please suggest if there are any alternative approaches.



    ------------------------------
    Kumar
    ------------------------------



  • 20.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted 25 days ago

    Stuffing so much information in single SOAP message is an antipattern. It won't be scalable regardless of the approach taken. This is because of size limitation within any XML processing engine. As a stop gap solution, you can try to increase XML size in xml manager. Also try to switch to GatwayScript for url-open call. Gateway script uses async model of computation and might provide you expected result. 



    ------------------------------
    Ajitabh Sharma
    ------------------------------



  • 21.  RE: Extract Nested SOAP Body Elements Using XSLT and Send to MQ in Iteration

    Posted 25 days ago

    Basis of XSLT provided, the url-open calls must all finish before further service invocations. 



    ------------------------------
    Ajitabh Sharma
    ------------------------------