DataPower

 View Only
  • 1.  Bulk capture of payloads using DPOD's payload capture

    Posted Fri October 22, 2021 02:55 PM
    Hello!

    Some of our developers wanted to try using DPOD (Datapower Operations Dashboard) to capture payloads for use in creating a comprehensive suite of CI/CD test cases.  We know we can capture the payloads for a longer period of time than 5 minutes, but the problem would be actually pulling down those payloads from DPOD.  Currently it appears that we can only download the payloads one at a time via the gui.  I don't see anything in the rest api documentation that would facilitate the bulk download of those payloads.  Is it possible to do this via DPOD?

    ------------------------------
    Jennifer Stipe
    ------------------------------


  • 2.  RE: Bulk capture of payloads using DPOD's payload capture

    Posted Mon October 25, 2021 06:09 PM
    Hi Jennifer, I hope you are doing well!

    I am not sure if DPOD can or cannot do this, but from the DataPower perspective, you could enable Probe and then download all the captured transactions at once. You can configure the Probe Settings (in the MPGW) to capture up to 250 transactions before rolling over. 

    1. Open the Probes and click in Export Capture:

    2. Expand the capture in your file system, part of the content of the selected file contains the transaction details, including incoming payload (under the "wrap" element):

    3. Alternatively, if you want to automate extraction of the incoming payloads you could build something like this:

    unzip ../service-mpgw.zip > /dev/null 2>&1
    unzip mpgw@service-mpgw.zip > /dev/null 2>&1
    for i in `find . -name '[0-9]' -type d`; do cd $i; gzip -d 0.xml.gz > /dev/null 2>&1; xpath -e '//transaction-input' 0.xml > /dev/null 2>&1 | grep wrap | gsed "s/&lt;/</g" | gsed "s/<wrap>//" | gsed "s,</wrap></transaction-input>,,"; cd ../..; done;
    <xml>Hi there</xml>
    <xml>Hi there</xml>
    <xml>Hi there</xml>
    <xml>Hi there</xml>
    <xml>Hi there</xml>
    <xml>Hi there</xml>
    <xml>Hi there</xml>
    <xml>Hi there</xml>
    <xml>Hi there</xml>
    <xml>Hi there</xml>
    <xml>Hi there</xml>


    ------------------------------
    Bruno R Neves
    Integration Specialist
    IBM
    Tampa FL
    ------------------------------



  • 3.  RE: Bulk capture of payloads using DPOD's payload capture

    Posted Tue October 26, 2021 06:02 PM
    Hi, there is no REST API for getting payloads from DPOD. I suggest you open an RFE so this is considered when prioritizing features for next versions.

    ------------------------------
    Amit Munwes
    ------------------------------



  • 4.  RE: Bulk capture of payloads using DPOD's payload capture

    Posted Fri October 29, 2021 02:47 PM
    There is another approach in the DPOD documentation that might work.  See the section "Payloads are fully collected in DPOD, but are still not displayed". 

    https://montier.atlassian.net/wiki/spaces/DPOD011000/pages/454066821/Payload+capturing+using+WS-M+does+not+provide+results" title="https://montier.atlassian.net/wiki/spaces/dpod011000/pages/454066821/payload+capturing+using+ws-m+does+not+provide+results" href="https://montier.atlassian.net/wiki/spaces/DPOD011000/pages/454066821/Payload+capturing+using+WS-M+does+not+provide+results" rel="noopener noreferrer" target="_blank" tabindex="-1">https://montier.atlassian.net/wiki/spaces/DPOD011000/pages/454066821/Payload+capturing+using+WS-M+does+not+provide+results

    That describes how you can access the DPOD data store through Elasticsearch.  I've been able to find multiple payloads using the "Structured Query[+]" tab.
    Search "wdp-wsm_i1" or another wdp-wsm index.  You can enter search criteria below.  For example:  must  wdpWsm.deviceName wildcard test* will return all the payloads captured from a device with "test" in the name. 

    There are 3 options for the Output Results.  CSV output will allow you to generate a download link to a file that will contain all the transaction data including request / response payloads.

    Hope this helps!



    ------------------------------
    Tyler Nelson
    Erie Insurance
    ------------------------------



  • 5.  RE: Bulk capture of payloads using DPOD's payload capture

    IBM Champion
    Posted Mon November 01, 2021 10:09 AM
    Hi,

    I am not an expert when it comes to DPOD but if it uses DataPower WSM agent subscription under the covers perhaps you can just subscribe to the same feed.

    --HP

    ------------------------------
    Hermanni Pernaa
    ------------------------------