First of all thanks for dedicating your time on this issue. I appreciate your efforts and patience.
In FW version 10.5.0.6, I am getting same issue in datapower.js file. As you mentioned, Distributed Variable settings have to be up and referencing a valid Gateway Peering configuration. Right now I will suggest my team to use MQ or any external chache to achieve this or setting system-variable is also working as of now. After 4-5 months we have to migrate to cloud then we have to use FW version 10.6.0.0. Then going forward we can implement using gateway peering if their no of appliance setting supports.
Original Message:
Sent: Thu September 11, 2025 11:04 PM
From: Joseph Morgan
Subject: Rewrite json value in a file to be accessed in future transactions
Shubhda,
I spent a while today going through FW versions from 10.5.0.6 forward, and, it appears using distrbuted variables won't be possible before 10.6.0.0 firmware. This is the first version I found where another critical configuration, the "Distributed Variable Settings" becomes available.
The process is further complicated by the fact that your Gateway Peering must be unique in reference to Local Port and Monitor Port across the entire appliance, and your Distributed Variable settings have to be up and referencing a valid Gateway Peering configuration.
I setup a simple rule on a service having two GWScript actions. The first writes the variable, the second reads it. It logs everything and it works fine. Here are my two JS files:
// SetDistributedVariable.jsvar distMD = require("distributed-metadata");distMD.setVariable('var://dist/MyDistributedContext/MyToken', "abc", handleError);function handleError(error, response) { if (error) { console.error('setVariable error = ' + error); } else { console.log('setVariable result = ' + response); }}
// GetDistributedVariable.jsvar distMD = require("distributed-metadata");distMD.getVariable('var://dist/MyDistributedContext/MyToken', handleError);function handleError(error, response) { if (error) { console.error("Error reading distributed variable: %s", error); } else { console.debug("Distributed Variable Value is: %s", response); }}
Hope this helps.
------------------------------
Joseph Morgan
CEO - Independent
Original Message:
Sent: Thu September 11, 2025 12:15 PM
From: Shubhda Shambhavi
Subject: Rewrite json value in a file to be accessed in future transactions
Hi Joseph,
It is just a lot of concepts which is hard to digest at a time.
Gateway peering is also new concept for me. Now not sure how distributed variable is associated with Gateway peering?
Without gateway peering can't we use Distributed variables?
I will read more about it and would try.
------------------------------
Shubhda Shambhavi
Original Message:
Sent: Thu September 11, 2025 11:52 AM
From: Joseph Morgan
Subject: Rewrite json value in a file to be accessed in future transactions
Shubhda,
Ah, yes, regarding the "datapower.js" file, I missed that is the main process.
I really don't know what I'm missing there. Maybe try requiring "apim", as it was originally developed for the API Connect but later came down for availability in later FW. I will setup a test on my 10.5.0.6, and see if I get the same error.
As for the number of appliances, it is recommended by IBM you have an odd number of appliances. That is 1, 3, 5, etc. The reason they don't officially support 2 is because of the HA algorithm they use, which appears to be an (N + 1) / 2 without epsilon for quorum control voting. That is, the HA algorithm they use is somewhat primitive. They could add epsilon or a ranked clerical system to solve it, but, it is what it is.
I have run tests in 10.6.0.5 systems and it works. However, it worked because the 2 appliances remained running. We didn't test a failure scenario.
As an additional note on this, I think what's so strange about their GW Peering algorithm is more related to probes. If one follows IBM recommendations to never turn on probes in production and there is no other need for gateway peering, it follows one could run 2 appliances in production. But, as we all know, real transactions can differ vastly from test transactions. So, the conclusion might be to run test/redacted production transactions in, say, pre-production or UAT with probes on. This would mean then, if strictly following IBM recommendations, one might need 3 appliances in pre-production/UAT (or just one, but that wouldn't be pre-production would it??), whereas only having 2 in production. I wish IBM would solve this issue and minimally add epsilon to their algorithm.
------------------------------
Joseph Morgan
CEO - Independent
Original Message:
Sent: Thu September 11, 2025 01:38 AM
From: Shubhda Shambhavi
Subject: Rewrite json value in a file to be accessed in future transactions
Hi Joseph,
The answer to your doubt for
mpgw (MPGW_GetAccessToken_TTS): GatewayScript processing Error 'Error: Unable to open the script module file 'distributed-metadata' In file 'gatewayscript:///datapower.js' line:311, stack:Error: Unable to open the script module file 'distributed-metadata' at Runtime.getModuleFilename (<anonymous>) at ModuleContext.<anonymous> (gatewayscript:///datapower.js:311:36) at Object.<anonymous> (local:///OMSRLNA/TTS/GetAccessToken_TTS_Response.js:8:21) at Script.execute (gatewayscript:///datapower.js:158:24) at Object.<anonymous> (gatewayscript:///datapower.js:585:55)'
how the GatewayScript processing error noted above relates to the 2 JS's you posted. It was complaining about line 311, and neither JS is remotely that long.
> If you see this path gatewayscript:///datapower.js:311:36
it is some default file present in DataPower box. I cannot find this in my File Management.
Option 1. For Distributed Variables : do we need to do some additional settings in our DataPower box?
For number of appliance, I will cross check in my team. You meant to say if we have two appliance then we can't achieve Gateway-peering?
------------------------------
Shubhda Shambhavi
Original Message:
Sent: Wed September 10, 2025 10:40 AM
From: Joseph Morgan
Subject: Rewrite json value in a file to be accessed in future transactions
Interesting. OK. First, I hope you can refresh your API Key and Secret.
I'm not sure how the GatewayScript processing error noted above relates to the 2 JS's you posted. It was complaining about line 311, and neither JS is remotely that long. I may be missing something.
I would highly recommend using one of three things for this:
- Distributed Variables - Looking at the code you attached, I'm not clear on why it is getting the error. Maybe I'm missing something here as well, and I don't have a 10.5.0.4 appliance on which to test.
- External cache: eXtreme Scale, Redis, something like that. eXtreme Scale could be a problem if you don't already have WebSphere ND to launch it from. Redis, on the other hand, is lightweight and easy to deploy stand-alone or in containers. Of course, one could maybe whip up a quick python/java server to act as the cache and drop it into Docker! (No, don't do that, your security team would flip out completely and making it handle load could be a killer).
- Database (I'm squinching a bit on this one): Like eXtreme Scale, if you don't already have one, it likely isn't worth the work to use it and it will add the most latency. However, it will work.
Keep in mind, DataPower was never meant to be stateful. We do now have the distributed variables, which is ideal for this sort of solution. With that said, you have to evaluate how many tokens you'll have to track at a given time, how quickly that 2nd transaction comes, and how you'd handle the situation if you lose one.
Another thing to keep in mind (though I can prove it isn't technically correct). IBM recommends a quorum (3 appliances) for using gateway peering. I think it is just a way of selling more licenses, but, they do say they don't support the use of 2, even though it will work. No one will argue that having 3 appliances is better than having 2, but having 5 is better than having 3, and having 100 is better than having 50. But, let's get real!
------------------------------
Joseph Morgan
CEO - Independent
Original Message:
Sent: Wed September 10, 2025 09:49 AM
From: Shubhda Shambhavi
Subject: Rewrite json value in a file to be accessed in future transactions
This raises questions about your implementation and environment:
- How many appliances serve these transactions? : 2 appliance, right now we have implemented in On-premise. But we have container based pods as well in cloud. So if we do new deployment there the pod get restarted.
- How long will this value need to persist beyond the first transaction? time would be short. It should be valid only for next transaction.
- How many transactions beyond the first transaction will this value be needed?: next consecutive transaction
- What is the value used for?:- Requirement- i. We have to make call to Auth URL by passing refresh_token in query parameter. It will return access token and refresh_token the body. Access token will be valid for 3 hours. To cache the response I am doing document caching in xml manager for TTL 3hours. Now the challenge is in the next transaction when we are calling Auth URL we have to pass refresh_token in query params that we have received in response of previous transaction. That's why we have to save this refresh token somewhere.
------------------------------
Shubhda Shambhavi
Original Message:
Sent: Wed September 10, 2025 09:05 AM
From: Joseph Morgan
Subject: Rewrite json value in a file to be accessed in future transactions
If you are using global variables and a domain restarts, it shouldn't cause an issue. If the appliance restarts, It will be gone. When using distributed variables (see note below), you don't lose it unless every appliance in the gateway peering go down at once or in such quick sequence that the peering can't recover shared state.
This raises questions about your implementation and environment:
- How many appliances serve these transactions? (I'm hoping more than one)
- How long will this value need to persist beyond the first transaction? (I was thinking time between transactions would be short)
- How many transactions beyond the first transaction will this value be needed?
- What is the value used for?
I'm asking these questions because it just became clear:
- Your appliances are in containers (due to the mention of restarting pods).
- I'm guessing you're trying to handle some kind of persistent state between transactions, like an expiration on an auth token of some kind (MPGW_GetAccessToken_TTS).
Using global variables is not recommended for this. Though it may work in development on one appliance, it will not work across appliances if your production is more than one appliance. Distributed variables is still recommended.
I do see you had some trouble in the code for what I'm guessing to be the require statement. It is difficult to know the problem without you sharing code.
------------------------------
Joseph Morgan
CEO - Independent
Original Message:
Sent: Wed September 10, 2025 05:55 AM
From: Shubhda Shambhavi
Subject: Rewrite json value in a file to be accessed in future transactions
I have one doubt. If restart of Pod or domain happens then this sytem variable will get reset or how it will respond?
------------------------------
Shubhda Shambhavi
Original Message:
Sent: Tue September 09, 2025 12:39 PM
From: Joseph Morgan
Subject: Rewrite json value in a file to be accessed in future transactions
Change "service-metadata" to "system-metadata".
var systemMD = require('system-metadata');// Then...systemMD.MyGlobalContext.MyToken = value;// ORsystemMD.setVariable('var://system/MyGlobalContext/MyToken', value);
For distributed variables, you'll need to use the "distributed-metadata" module:
var distributedMD = require('distributed-metadata');// Then...distributedMD.MyDistrubutedContext.MyToken = value;// ORdistributedMD.setVariable('var://dist/MyDistrubutedContext/MyToken', value);
------------------------------
Joseph Morgan
CEO - Independent
Original Message:
Sent: Tue September 09, 2025 11:50 AM
From: Shubhda Shambhavi
Subject: Rewrite json value in a file to be accessed in future transactions
How can we set global variable in GatewayScript?
I tried setting like this but it was giving error :
var sm = require("service-metadata");
var systemVar = sm.setVar('var://system/MyGlobalContext/MyToken','abc');
Error:
GatewayScript processing Error 'Error: Incorrect variable name: 'var://system/MyGlobalContext/MyToken' In file 'gatewayscript:///modules/service-metadata.js' line:99, stack:Error: Incorrect variable name: 'var://system/MyGlobalContext/MyToken' at getServiceVarProp (gatewayscript:///modules/service-metadata.js:99:15) at Object.setVar (gatewayscript:///modules/service-metadata.js:130:16) at Object.<anonymous> (local:///OMSRLNA/TTS/GetAccessToken_TTS_Response.js:28:21)'
------------------------------
Shubhda Shambhavi
Original Message:
Sent: Tue September 09, 2025 10:41 AM
From: Joseph Morgan
Subject: Rewrite json value in a file to be accessed in future transactions
And how will you use the file from the next transaction if your appliances are load-balanced? Are you forcing affinity back to the same appliance?
I would take a different approach, if you can.
If you're up to date on firmware versions, look up the use of distributed variables. Though this requires a gateway-peering setup, this might be your best solution.
You could also consider something like Redis or extreme scale. These would make it more scalable, but adds more latency than you might want.
A third, but not fully recommended option, would be to use global variables (I can hear the gasps and screaming now). The problem with global variables isn't the quite-wrong assessment about not using globals (as API connect seems to have no problems using them). It is more about transaction affinity. Like your first solution, it is a big problem if the transaction doesn't come to the same appliance.
------------------------------
Joseph Morgan
CEO - Independent
Original Message:
Sent: Tue September 09, 2025 12:22 AM
From: Shubhda Shambhavi
Subject: Rewrite json value in a file to be accessed in future transactions
Is there a way to write json value in a file and store it in DataPower local:// directory. In the next transaction, read the value from the same file to make a call and store the response json value in same file by rewriting to be accessed in next transaction?
I was trying using Gatewayscript but my script was failing.
In context variable values are alive for one transaction but I have to keep the value alive for next transaction. How this can be achieved?
------------------------------
Shubhda Shambhavi
------------------------------