API Connect

API Connect

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.


#API Connect
#Applicationintegration
#APIConnect
 View Only
Expand all | Collapse all

Unable to capture Raw Payload in APIC v10

  • 1.  Unable to capture Raw Payload in APIC v10

    Posted 10/15/24 02:20 PM
    HI Team,

    Use Case:
    In APIC I have to verify signature received in the Http Header, using sha256 Alg.
    The received signature is generated in PHP(fyr as below snippet)
    PHP: base64_encode(hash_hmac('sha256',$Data, $Key)); 

    The same in APIC is replicated with below code
    var data = `${signatureTimestamp}${data}`;
    var hashedResult = crypto.createHmac('sha256', encodedKey).update(data).digest('hex');
    var verifySignature = Buffer.from(hashedResult).toString('base64');

    with the above snippet we were able to generate the same signature to validate.

    Issue: this is working from the Postman tool only, when we test it from php App then the signatures are mismatched.

    test1: I took the payload from the analytics and did json unescape from an online tool since it is stringified. and this payload is able to generate the correct signature.

    test2: when we trigger the request from the app, it doesn't work.

    so our idea is to compare the received raw request.
    In APIC I am reading requests as Buffer and converting to Buffer.toString() which I am using for signature generation.

    we tried logging the payload - again it's stringified.
    we tried logging the buffer - but not showing the complete log data in datapower logs.
    we tried logging the hexa hmac -- obviously it's not matching.

    So how to capture the actual request received to compare.

    or how to do unescape json (tried JSON.parse() is not matching.









  • 2.  RE: Unable to capture Raw Payload in APIC v10

    Posted 10/15/24 02:22 PM
    Tried to write to a data power temp file using the fs module in APIC v10, this does not work.

    On Tue, 15 Oct 2024 at 23:49, pratap vadlapati <pratapvadlapati@gmail.com> wrote:
    HI Team,

    Use Case:
    In APIC I have to verify signature received in the Http Header, using sha256 Alg.
    The received signature is generated in PHP(fyr as below snippet)
    PHP: base64_encode(hash_hmac('sha256',$Data, $Key)); 

    The same in APIC is replicated with below code
    var data = `${signatureTimestamp}${data}`;
    var hashedResult = crypto.createHmac('sha256', encodedKey).update(data).digest('hex');
    var verifySignature = Buffer.from(hashedResult).toString('base64');

    with the above snippet we were able to generate the same signature to validate.

    Issue: this is working from the Postman tool only, when we test it from php App then the signatures are mismatched.

    test1: I took the payload from the analytics and did json unescape from an online tool since it is stringified. and this payload is able to generate the correct signature.

    test2: when we trigger the request from the app, it doesn't work.

    so our idea is to compare the received raw request.
    In APIC I am reading requests as Buffer and converting to Buffer.toString() which I am using for signature generation.

    we tried logging the payload - again it's stringified.
    we tried logging the buffer - but not showing the complete log data in datapower logs.
    we tried logging the hexa hmac -- obviously it's not matching.

    So how to capture the actual request received to compare.

    or how to do unescape json (tried JSON.parse() is not matching.












  • 3.  RE: Unable to capture Raw Payload in APIC v10

    Posted 10/16/24 02:15 PM
    Hi @Steve 
    Could you pls help us on this below issue.

    On Tue, 15 Oct 2024 at 11:49 PM, pratap vadlapati <pratapvadlapati@gmail.com> wrote:
    HI Team,

    Use Case:
    In APIC I have to verify signature received in the Http Header, using sha256 Alg.
    The received signature is generated in PHP(fyr as below snippet)
    PHP: base64_encode(hash_hmac('sha256',$Data, $Key)); 

    The same in APIC is replicated with below code
    var data = `${signatureTimestamp}${data}`;
    var hashedResult = crypto.createHmac('sha256', encodedKey).update(data).digest('hex');
    var verifySignature = Buffer.from(hashedResult).toString('base64');

    with the above snippet we were able to generate the same signature to validate.

    Issue: this is working from the Postman tool only, when we test it from php App then the signatures are mismatched.

    test1: I took the payload from the analytics and did json unescape from an online tool since it is stringified. and this payload is able to generate the correct signature.

    test2: when we trigger the request from the app, it doesn't work.

    so our idea is to compare the received raw request.
    In APIC I am reading requests as Buffer and converting to Buffer.toString() which I am using for signature generation.

    we tried logging the payload - again it's stringified.
    we tried logging the buffer - but not showing the complete log data in datapower logs.
    we tried logging the hexa hmac -- obviously it's not matching.

    So how to capture the actual request received to compare.

    or how to do unescape json (tried JSON.parse() is not matching.












  • 4.  RE: Unable to capture Raw Payload in APIC v10

    Posted 10/18/24 05:46 PM

    Hi Pratap,
    Would you by chance be doing a parse policy prior to your GatewayScript?  If so there is a known issue where the parse policy is updating request.body with the parsed payload, removing white space and thus changing your hash.  The workaround is to 
    1) ensure you have x-ibm-configuration.buffering set to true.  This will buffer your request but will then not require a parse policy to make request.body available to your code.
    2) Have your GatewayScript do the read of request.body and do the hash BEFORE your parse policy.  This should provide you a hash off of the original payload with all of the whitespace present.
    Best Regards,
    Steve



    ------------------------------
    Steve Linn
    Senior Consulting I/T Specialist
    IBM
    ------------------------------



  • 5.  RE: Unable to capture Raw Payload in APIC v10

    Posted 10/19/24 12:58 PM
    Hi, Steve,

    Appreciate your reply! :)

    This is the first gateway script action in the flow. 
    There is parse and map but later in the flow(fyi).

    context.request.body.readAsBuffer(function(error, buffer) {});
    And later buffer.toString() sent to hamc.update
     
    This is the function using in the gateway script to read the raw request, does it make the difference than(request.body)

    Do I have to make any changes?








  • 6.  RE: Unable to capture Raw Payload in APIC v10

    Posted 10/21/24 05:56 AM

    Hi Pratap,

    Your code context.request.body.readAsBuffer should read the request payload from the stream into a buffer.  Have you tried to do the update without using a toString() function on the buffer?  You have not mentioned what your request.body is, but the toString() function would do some changes to the data based upon converting that data to a utf-8 equivalent, and the update function can take the buffer as is.  As for your issue with the fs module, what type of issue are you having?
    Best Regards,
    Steve Linn



    ------------------------------
    Steve Linn
    Senior Consulting I/T Specialist
    IBM
    ------------------------------



  • 7.  RE: Unable to capture Raw Payload in APIC v10

    Posted 10/21/24 06:06 AM
    Hi Steve,

    I am receiving JSON(huge)as payload so when I read it from request.body, when I log I can see buffer. But I have to do concatenation with the rcvd payload. So that's the reason i am converting to string. Concatenation with timestamp and Later updating hmac.

    So I tried to capture the raw payload rcvd so that i can compare it. I used fs module to write to temp file but I don't see any files written in the data power 






  • 8.  RE: Unable to capture Raw Payload in APIC v10

    Posted 10/21/24 07:04 AM
    PS.
    I am using the below snippet to write the payload received into file from gatewayscript.

    var temp_file = fs.temporary();
           
            var emptybuffer = "";
            var options = {'file': temp_file,'data': json, 'TTL': 15};
            fs.writeFile(options, function(error) {  
                if(error) {    
                    console.error(error);  
                    } else {  
                        fs.appendFile(temp_file, emptybuffer, function(error) {  
                            if(error){      
                                console.error(error);    
                                } else {      
                                    console.log('appendFile success.');    
                                    }     });   } });





  • 9.  RE: Unable to capture Raw Payload in APIC v10

    Posted 10/21/24 07:20 AM

    Hi Pratap,

    var options = {'file': temp_file,'data': json, 'TTL': 15}; is not a very long TTL.  That file will be removed in 15 seconds after it is written.  I'd try a longer TTL or even 0 to keep the file around until you're ready to delete it.

    Best Regards,
    Steve Linn



    ------------------------------
    Steve Linn
    Senior Consulting I/T Specialist
    IBM
    ------------------------------



  • 10.  RE: Unable to capture Raw Payload in APIC v10

    Posted 10/21/24 07:16 AM

    Hi Pratap,
    For the fs.writeFile, I too was having issues with the writeFile signature that specified just the file name, and I'll have that looked into, but the other signature with options does work

    let fs = require('fs');
    let options = {file: 'temporary:///test.json', data: request, TTL: 600};
    //fs.writeFile("temporary:///test.json", request, function(error) {  ... this signature was failing to write a file, I suspect due to a TTL issue
    fs.writeFile(options, function(error) {
      if(error) {
        console.error(error);
      } else {
        console.log('writeFile success.');
      }
    });

    I suspect the failing signature must be assuming a very short TTL and removes the file shortly after writing it.  Using the options approach, I believe a TTL of 0 will keep the file there indefinitely.  Also the callback function is only to note errors or success, so I could have just as easily provided an empty function

    fs.writeFile(options, function(error) {});

    Assuming the issue is caused by UTF-8 encoding of the toString() function which is changing your payload, you'll need to use buffers only.  You indicate you're getting a large JSON payload, so are you simply taking the ending curly brace and replacing it with something like , "timestamp": "sometimestampvalue"}
    If so, you could create a Buffer.from that string,  and this is some code off the top of my head :-)  but something like

    let buffer1 = buffer.slice(0,buffer.length-1); // get a buffer with all but the last character
    let buffer2 = Buffer.from(', "timestamp": "sometimestampvalue"}');
    let finalBuffer = Buffer.concat([buffer1, buffer2]);

    of course test and verify :-)  You could have white space after the end curly brace so another approach would be to find the last curly brace in the buffer.

    Best Regards,
    Steve Linn



    ------------------------------
    Steve Linn
    Senior Consulting I/T Specialist
    IBM
    ------------------------------



  • 11.  RE: Unable to capture Raw Payload in APIC v10

    Posted 10/22/24 09:29 AM

    Hi Pratap,
    Just to close the open question for the fs.writeFile function, the problem we were both having with the signature that only included the file name and the callback function is user error on both of our parts.  I should have read the documentation for the fs module, at least the initial section, but instead clicked on the link to the writeFile function and tried the signature without the options specified, namely 

    fs.writeFile("temporary:///test.json", request, function(error) {

    See https://www.ibm.com/docs/en/datapower-gateway/10.5.0?topic=apis-fs-module which states that the TTL option has a default of 10 seconds and is only applicable for files in the temporary:/// directory.  That's why my file was disappearing before I could look for it.  In your case, you were specifying options, but 15 seconds for a temporary file is too short a time before the file is automatically deleted.  Anyway, no product issue with fs.writeFile().

    Best Regards,
    Steve Linn



    ------------------------------
    Steve Linn
    Senior Consulting I/T Specialist
    IBM
    ------------------------------



  • 12.  RE: Unable to capture Raw Payload in APIC v10

    Posted 10/30/24 05:13 AM
    Hi Steve,

    In the context to the mismatch of Hmac signature,
    Test case: Idea is to compare request payload from postman which is working and request payload from the Php app.

    As I mentioned before I am reading request as buffer and converted toString('hex').
    And setting that hex string in analytics log custom_data. 

    In this above way I tried to capture the payloads.
    Request from app is directly triggered from Php app and getting mismatched signature.

    The request for postman is taken from the Analytics request log , did json unescape online, tried invoking the same api and now the signature is matched.

    So I compared the two request hex strings and they both are identical.

    We are so confused that what could be the reason for mismatch of the signature from the Php app.

    I am completely lost. Could pls comment of any ideas to try or do I have to check anything else.

    IBM api connect v10






  • 13.  RE: Unable to capture Raw Payload in APIC v10

    Posted 10/30/24 12:13 PM

    Hi Pratap,
    In your code

    var data = `${signatureTimestamp}${data}`;

    First, looking at this afresh, you have some variable named data which is being used in the string template to initialize a variable of the same name?  What is the contents of "data" prior to this instruction and how did you initialize it?  context.get('request.body') ? context.request.body.readAsBuffer?  If one of these, do you have a parse policy prior to your GatewayScript policy? If so, I still believe the issue may be the known issue I mentioned earlier in this thread where request.body is getting modified with a parsed version of request.body, which loses the white space and would generate a different hash.  Also, can you add a console.error to log the value of the signatureTimestamp and data variables prior to this  instruction.  If not the parse issue I was referencing, it could also be if you used context.get('request.body') that your value is null as your request payload is streamed to the API which means that request.body is not in context when the assembly begins to execute.  To ensure that request.body is read from the stream and thus accessible to your GatewayScript policy, you can either do a parse policy, but that has the known issue, thus the other case would be to use context.request.body.readAsBuffer which would read from the stream.  You could also enable buffering in your API by specifying the x-ibm-configuration.buffering = true property which would thus make the context.get('request.body') work as the assembly would not start until the payload has been fully received and buffered. Finally, can you access the GatewayScript debugger, either via CLI or the remote debugger?  If so, adding the following policy prior to your GatewayScript policy

          - set-variable:
              version: 2.0.0
              title: set-variable
              actions:
                - set: policy.gatewayscript.enableDebugger
                  value: true
                  type: boolean

    and then adding a debugger; statement where you want your code to pause as its first breakpoint, you can see exactly what are in your variables.  For how to use the debugger, see https://www.ibm.com/docs/en/datapower-gateway/10.5.0?topic=gatewayscript-debuggers

    Bottom line, it could be a number of things, but hopefully the above will give you some things to try.  If you're still having issues, I'd suggest opening a PMR to get some support help where you can share your API, sample request payload, and expected results versus actual results so we can take a closer look at the issue.

    Best Regards,
    Steve Linn



    ------------------------------
    Steve Linn
    Senior Consulting I/T Specialist
    IBM
    ------------------------------



  • 14.  RE: Unable to capture Raw Payload in APIC v10

    Posted 11/05/24 08:31 AM
    HI Steve,

    PMR is in progress submitting the required information.

    To my curiosity I am checking to enable debugging.

    Steps:
    1.  Enabled remote debugger in default domain.
    2.  Configured set variable as you suggested before the gateway script action.
    3.  written debug statements wherever required. (debugger;)
    I tried sending traffic on the api and the request is on hold (sure the debugger is hit)

    I went to the web ui and searched for Debug action status and i can see the session id and the file url.

    I don't have access to cli.
    I have a debugger IP and Port config.

    Now how can i debug the file. 
    I have chekd the documentation, and I can find only cli commands. unfortunately i don't have access to it.










  • 15.  RE: Unable to capture Raw Payload in APIC v10

    Posted 11/05/24 09:22 AM
    Edited by Steve Linn 11/05/24 09:24 AM

    Hi Pratap,
    For remote debugging, you need to enter into your chrome browser the url chrome://inspect Check/select Discover network targets where you would specify your appliance IP and the remote debugger port, by default 9229.  I'm not sure the frequency of its checks, but eventually (took me over 30 seconds) your paused GatewayScript should be shown and you'd click on the inspect link for that transaction. 
    I don't see that the DataPower documentation has specifics about the chrome debugger but I believe the chrome documentation should have some details.  In my case I clicked on inspect to get the following:
    The right pane has the buttons for step over, step into, continue, etc.  It will also show variables and their values below it.  You can set breakpoints by clicking in the gutter to the left of the source code line number.  The bottom pane you can also enter variable names and it will show the values there too.
    Best Regards,
    Steve Linn



    ------------------------------
    Steve Linn
    Senior Consulting I/T Specialist
    IBM
    ------------------------------



  • 16.  RE: Unable to capture Raw Payload in APIC v10

    Posted 10/31/24 10:10 AM
    P.s  I have tried below, still it doesn't match with the signature. This works with postman(restClient) and does not work from the app.

    1. Enabled x-ibm config buffering to true.
    2. Read request from request.body
    var requestBody = context.get('request.body');
    var strPayload = requestBody.toString('utf8');
     var digestData = `${signatureTimestamp}${strPayload}`



    On Sat, 19 Oct 2024 at 10:28 PM, pratap vadlapati <pratapvadlapati@gmail.com> wrote:
    Hi, Steve,

    Appreciate your reply! :)

    This is the first gateway script action in the flow. 
    There is parse and map but later in the flow(fyi).

    context.request.body.readAsBuffer(function(error, buffer) {});
    And later buffer.toString() sent to hamc.update
     
    This is the function using in the gateway script to read the raw request, does it make the difference than(request.body)

    Do I have to make any changes?








  • 17.  RE: Unable to capture Raw Payload in APIC v10

    Posted 11/01/24 12:17 PM

    Hi Pratap,
    Ok, so the parse policy issue isn't biting you!  I don't know what is in your payload, but the only other thing I can think of is that your toString() is causing some changes to your data based upon the utf-8 encoding.  I'd try to keep everything in a buffer object, for example, 

    context.request.body.readAsBuffer(function(error, buffer) {
      if (error) {
        // handle error
      } else {
        let newBufferArray = [];
        newBufferArray.push(new Buffer(signatureTimestamp));
        newBufferArray.push(buffer);
        var digestData = Buffer.concat(newBufferArray);
        // now do your hash with the digestData which is a buffer, not a string
      }
    });

    The effect is the same as you'd done before, concatenating your timestamp with the buffer, but there is not any utf-8 encoding that could be taking place to your buffer data, thus the data being used by hamc.update should be unchanged.  This function should be able to take a buffer instead of a string.  Looking at https://www.ibm.com/docs/en/datapower-gateway/10.5.0?topic=apis-crypto-module#crypto__hmac.update " If no encoding is provided and the input is a string, an encoding of 'utf8' is enforced. If data is a Buffer or Buffers, input encoding is ignored." So hopefully your issue is an encoding issue.

    If that doesn't work, I'm out of ideas, I'd suggest you open a PMR.

    Best Regards,
    Steve Linn



    ------------------------------
    Steve Linn
    Senior Consulting I/T Specialist
    IBM
    ------------------------------



  • 18.  RE: Unable to capture Raw Payload in APIC v10

    Posted 11/01/24 12:34 PM
    Hi Steve,

    Thanks for the support and expertise you shared with me.

    Just fyi:

    The payload is json.
    At present, I am reading it as buffer and timestamp is also made to buffer and it is Concatenated.

    Var Timestamp = 'sometimeStamp';
    var payload  = BufferJson;

    Var digestdata='${Timestamp}${BufferJson}';

    Now digestdata sent to hmac update

    This doesn't solve the problem.

    So I have raised PMR , curious to what I have just missed.

    Thanks.





  • 19.  RE: Unable to capture Raw Payload in APIC v10

    Posted 11/01/24 03:01 PM

    Hi Pratap,
    Looking at your code in this new post  ...

    variable (assume you meant var not Var) Timestamp is a string
    var payload I'll assume is a Buffer datatype
    variable (again I assume you meant var not Var) digestData ... assume you mean this is a string template that uses back quote characters, ie `${Timestamp}${BufferJson}` not the single quotes you have above.  If your code really has single quote, your string for digestData is simply what is between the single quotes, ie, '${Timestamp}${BufferJson}' and not a string where the value of the Timestamp variable is concatenated to the value of the BufferJson variable.  I see in your thread post a few weeks ago you had used back quote characters properly var data = `${signatureTimestamp}${data}`; so hopefully this is just a typo in your current post.  I believe a string template this will do an implicit toString() function on your buffer and will produce a string, not a buffer.  As long as the encoding doesn't change anything in your data it should work, but that is the only thing I can think of is that the utf-8 encoding is changing your data ever so slightly. Still JSON by default is supposed to be utf-8 encoded, so I don't know why that would make a difference with JSON data.  Let's see what the support ticket reveals.

    Good luck!
    Steve Linn



    ------------------------------
    Steve Linn
    Senior Consulting I/T Specialist
    IBM
    ------------------------------