Original Message:
Sent: Tue November 05, 2024 08:30 AM
From: pratap vadlapati
Subject: Unable to capture Raw Payload in APIC v10
HI Steve,
PMR is in progress submitting the required information.
To my curiosity I am checking to enable debugging.
Steps:
1. Enabled remote debugger in default domain.
2. Configured set variable as you suggested before the gateway script action.
3. written debug statements wherever required. (debugger;)
I tried sending traffic on the api and the request is on hold (sure the debugger is hit)
I went to the web ui and searched for Debug action status and i can see the session id and the file url.
I don't have access to cli.
I have a debugger IP and Port config.
Now how can i debug the file.
I have chekd the documentation, and I can find only cli commands. unfortunately i don't have access to it.
Original Message:
Sent: 10/30/2024 12:13:00 PM
From: Steve Linn
Subject: RE: Unable to capture Raw Payload in APIC v10
Hi Pratap,
In your code
var data = `${signatureTimestamp}${data}`;
First, looking at this afresh, you have some variable named data which is being used in the string template to initialize a variable of the same name? What is the contents of "data" prior to this instruction and how did you initialize it? context.get('request.body') ? context.request.body.readAsBuffer? If one of these, do you have a parse policy prior to your GatewayScript policy? If so, I still believe the issue may be the known issue I mentioned earlier in this thread where request.body is getting modified with a parsed version of request.body, which loses the white space and would generate a different hash. Also, can you add a console.error to log the value of the signatureTimestamp and data variables prior to this instruction. If not the parse issue I was referencing, it could also be if you used context.get('request.body') that your value is null as your request payload is streamed to the API which means that request.body is not in context when the assembly begins to execute. To ensure that request.body is read from the stream and thus accessible to your GatewayScript policy, you can either do a parse policy, but that has the known issue, thus the other case would be to use context.request.body.readAsBuffer which would read from the stream. You could also enable buffering in your API by specifying the x-ibm-configuration.buffering = true property which would thus make the context.get('request.body') work as the assembly would not start until the payload has been fully received and buffered. Finally, can you access the GatewayScript debugger, either via CLI or the remote debugger? If so, adding the following policy prior to your GatewayScript policy
- set-variable:
version: 2.0.0
title: set-variable
actions:
- set: policy.gatewayscript.enableDebugger
value: true
type: boolean
and then adding a debugger; statement where you want your code to pause as its first breakpoint, you can see exactly what are in your variables. For how to use the debugger, see https://www.ibm.com/docs/en/datapower-gateway/10.5.0?topic=gatewayscript-debuggers
Bottom line, it could be a number of things, but hopefully the above will give you some things to try. If you're still having issues, I'd suggest opening a PMR to get some support help where you can share your API, sample request payload, and expected results versus actual results so we can take a closer look at the issue.
Best Regards,
Steve Linn
------------------------------
Steve Linn
Senior Consulting I/T Specialist
IBM
------------------------------
Original Message:
Sent: Wed October 30, 2024 05:12 AM
From: pratap vadlapati
Subject: Unable to capture Raw Payload in APIC v10
Hi Steve,
In the context to the mismatch of Hmac signature,
Test case: Idea is to compare request payload from postman which is working and request payload from the Php app.
As I mentioned before I am reading request as buffer and converted toString('hex').
And setting that hex string in analytics log custom_data.
In this above way I tried to capture the payloads.
Request from app is directly triggered from Php app and getting mismatched signature.
The request for postman is taken from the Analytics request log , did json unescape online, tried invoking the same api and now the signature is matched.
So I compared the two request hex strings and they both are identical.
We are so confused that what could be the reason for mismatch of the signature from the Php app.
I am completely lost. Could pls comment of any ideas to try or do I have to check anything else.
IBM api connect v10
Original Message:
Sent: 10/22/2024 9:29:00 AM
From: Steve Linn
Subject: RE: Unable to capture Raw Payload in APIC v10
Hi Pratap,
Just to close the open question for the fs.writeFile function, the problem we were both having with the signature that only included the file name and the callback function is user error on both of our parts. I should have read the documentation for the fs module, at least the initial section, but instead clicked on the link to the writeFile function and tried the signature without the options specified, namely
fs.writeFile("temporary:///test.json", request, function(error) {
See https://www.ibm.com/docs/en/datapower-gateway/10.5.0?topic=apis-fs-module which states that the TTL option has a default of 10 seconds and is only applicable for files in the temporary:/// directory. That's why my file was disappearing before I could look for it. In your case, you were specifying options, but 15 seconds for a temporary file is too short a time before the file is automatically deleted. Anyway, no product issue with fs.writeFile().
Best Regards,
Steve Linn
------------------------------
Steve Linn
Senior Consulting I/T Specialist
IBM
Original Message:
Sent: Mon October 21, 2024 07:16 AM
From: Steve Linn
Subject: Unable to capture Raw Payload in APIC v10
Hi Pratap,
For the fs.writeFile, I too was having issues with the writeFile signature that specified just the file name, and I'll have that looked into, but the other signature with options does work
let fs = require('fs');let options = {file: 'temporary:///test.json', data: request, TTL: 600};//fs.writeFile("temporary:///test.json", request, function(error) { ... this signature was failing to write a file, I suspect due to a TTL issuefs.writeFile(options, function(error) { if(error) { console.error(error); } else { console.log('writeFile success.'); }});
I suspect the failing signature must be assuming a very short TTL and removes the file shortly after writing it. Using the options approach, I believe a TTL of 0 will keep the file there indefinitely. Also the callback function is only to note errors or success, so I could have just as easily provided an empty function
fs.writeFile(options, function(error) {});
Assuming the issue is caused by UTF-8 encoding of the toString() function which is changing your payload, you'll need to use buffers only. You indicate you're getting a large JSON payload, so are you simply taking the ending curly brace and replacing it with something like , "timestamp": "sometimestampvalue"}
If so, you could create a Buffer.from that string, and this is some code off the top of my head :-) but something like
let buffer1 = buffer.slice(0,buffer.length-1); // get a buffer with all but the last characterlet buffer2 = Buffer.from(', "timestamp": "sometimestampvalue"}');let finalBuffer = Buffer.concat([buffer1, buffer2]);
of course test and verify :-) You could have white space after the end curly brace so another approach would be to find the last curly brace in the buffer.
Best Regards,
Steve Linn
------------------------------
Steve Linn
Senior Consulting I/T Specialist
IBM
Original Message:
Sent: Mon October 21, 2024 06:05 AM
From: pratap vadlapati
Subject: Unable to capture Raw Payload in APIC v10
Hi Steve,
I am receiving JSON(huge)as payload so when I read it from request.body, when I log I can see buffer. But I have to do concatenation with the rcvd payload. So that's the reason i am converting to string. Concatenation with timestamp and Later updating hmac.
So I tried to capture the raw payload rcvd so that i can compare it. I used fs module to write to temp file but I don't see any files written in the data power
Original Message:
Sent: 10/21/2024 5:56:00 AM
From: Steve Linn
Subject: RE: Unable to capture Raw Payload in APIC v10
Hi Pratap,
Your code context.request.body.readAsBuffer should read the request payload from the stream into a buffer. Have you tried to do the update without using a toString() function on the buffer? You have not mentioned what your request.body is, but the toString() function would do some changes to the data based upon converting that data to a utf-8 equivalent, and the update function can take the buffer as is. As for your issue with the fs module, what type of issue are you having?
Best Regards,
Steve Linn
------------------------------
Steve Linn
Senior Consulting I/T Specialist
IBM
Original Message:
Sent: Sat October 19, 2024 12:58 PM
From: pratap vadlapati
Subject: Unable to capture Raw Payload in APIC v10
Hi, Steve,
Appreciate your reply! :)
This is the first gateway script action in the flow.
There is parse and map but later in the flow(fyi).
context.request.body.readAsBuffer(function(error, buffer) {});
And later buffer.toString() sent to hamc.update
This is the function using in the gateway script to read the raw request, does it make the difference than(request.body)
Do I have to make any changes?
Original Message:
Sent: 10/18/2024 5:46:00 PM
From: Steve Linn
Subject: RE: Unable to capture Raw Payload in APIC v10
Hi Pratap,
Would you by chance be doing a parse policy prior to your GatewayScript? If so there is a known issue where the parse policy is updating request.body with the parsed payload, removing white space and thus changing your hash. The workaround is to
1) ensure you have x-ibm-configuration.buffering set to true. This will buffer your request but will then not require a parse policy to make request.body available to your code.
2) Have your GatewayScript do the read of request.body and do the hash BEFORE your parse policy. This should provide you a hash off of the original payload with all of the whitespace present.
Best Regards,
Steve
------------------------------
Steve Linn
Senior Consulting I/T Specialist
IBM
Original Message:
Sent: Wed October 16, 2024 02:15 PM
From: pratap vadlapati
Subject: Unable to capture Raw Payload in APIC v10
Hi @Steve
Could you pls help us on this below issue.
HI Team,
Use Case:
In APIC I have to verify signature received in the Http Header, using sha256 Alg.
The received signature is generated in PHP(fyr as below snippet)
PHP: base64_encode(hash_hmac('sha256',$Data, $Key));
The same in APIC is replicated with below code
var data = `${signatureTimestamp}${data}`;var hashedResult = crypto.createHmac('sha256', encodedKey).update(data).digest('hex');var verifySignature = Buffer.from(hashedResult).toString('base64');
with the above snippet we were able to generate the same signature to validate.
Issue: this is working from the Postman tool only, when we test it from php App then the signatures are mismatched.
test1: I took the payload from the analytics and did json unescape from an online tool since it is stringified. and this payload is able to generate the correct signature.
test2: when we trigger the request from the app, it doesn't work.
so our idea is to compare the received raw request.
In APIC I am reading requests as Buffer and converting to Buffer.toString() which I am using for signature generation.
we tried logging the payload - again it's stringified.
we tried logging the buffer - but not showing the complete log data in datapower logs.
we tried logging the hexa hmac -- obviously it's not matching.
So how to capture the actual request received to compare.
or how to do unescape json (tried JSON.parse() is not matching.