DataPower

DataPower

Join this online group to communicate across IBM product users and experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only
Expand all | Collapse all

Rewrite json value in a file to be accessed in future transactions

  • 1.  Rewrite json value in a file to be accessed in future transactions

    Posted Tue September 09, 2025 09:50 AM

    Is there a way to write json value in a file and store it in DataPower local:// directory. In the next transaction, read the value from the same file to make a call and store the response json value in same file by rewriting to be accessed in next transaction?

    I was trying using Gatewayscript but my script was failing.

    In context variable values are alive for one transaction but I have to keep the value alive for next transaction. How this can be achieved?



    ------------------------------
    Shubhda Shambhavi
    ------------------------------


  • 2.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Tue September 09, 2025 10:42 AM

    And how will you use the file from the next transaction if your appliances are load-balanced?  Are you forcing affinity back to the same appliance?

    I would take a different approach, if you can.

    If you're up to date on firmware versions, look up the use of distributed variables.  Though this requires a gateway-peering setup, this might be your best solution.

    You could also consider something like Redis or extreme scale.  These would make it more scalable, but adds more latency than you might want.

    A third, but not fully recommended option, would be to use global variables (I can hear the gasps and screaming now).  The problem with global variables isn't the quite-wrong assessment about not using globals (as API connect seems to have no problems using them).  It is more about transaction affinity.  Like your first solution, it is a big problem if the transaction doesn't come to the same appliance.



    ------------------------------
    Joseph Morgan
    CEO - Independent
    ------------------------------



  • 3.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Tue September 09, 2025 11:07 AM

    This sounds like a good use case for distributed vars.

    https://www.ibm.com/docs/en/datapower-gateway/10.6.0?topic=administration-distributed-variables



    ------------------------------
    Ivan Heninger
    ------------------------------



  • 4.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Tue September 09, 2025 11:58 AM

    We are using Firmware version : IDG.10.5.0.4

    Can we implement Gatewaypeering there?



    ------------------------------
    Shubhda Shambhavi
    ------------------------------



  • 5.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Tue September 09, 2025 12:32 PM

    I believe so.  Though I don't have a 10.5.0.4 FW machine available, the IBM docs list it in the 10.5.x documentation.



    ------------------------------
    Joseph Morgan
    CEO - Independent
    ------------------------------



  • 6.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Tue September 09, 2025 11:51 AM

    How can we set global variable in GatewayScript?

    I tried setting like this but it was giving error :

    var sm = require("service-metadata");
    var systemVar = sm.setVar('var://system/MyGlobalContext/MyToken','abc');
    Error:
    GatewayScript processing Error 'Error: Incorrect variable name: 'var://system/MyGlobalContext/MyToken' In file 'gatewayscript:///modules/service-metadata.js' line:99, stack:Error: Incorrect variable name: 'var://system/MyGlobalContext/MyToken' at getServiceVarProp (gatewayscript:///modules/service-metadata.js:99:15) at Object.setVar (gatewayscript:///modules/service-metadata.js:130:16) at Object.<anonymous> (local:///OMSRLNA/TTS/GetAccessToken_TTS_Response.js:28:21)'


    ------------------------------
    Shubhda Shambhavi
    ------------------------------



  • 7.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Tue September 09, 2025 12:40 PM

    Change "service-metadata" to "system-metadata".

    var systemMD = require('system-metadata');
    
    // Then...
    systemMD.MyGlobalContext.MyToken = value;
    
    // OR
    systemMD.setVariable('var://system/MyGlobalContext/MyToken', value);
    

    For distributed variables, you'll need to use the "distributed-metadata" module:

    var distributedMD = require('distributed-metadata');
    
    // Then...
    distributedMD.MyDistrubutedContext.MyToken = value;
    
    // OR
    distributedMD.setVariable('var://dist/MyDistrubutedContext/MyToken', value);
    


    ------------------------------
    Joseph Morgan
    CEO - Independent
    ------------------------------



  • 8.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Wed September 10, 2025 04:04 AM

    For "system-metadata":

    Thanks a lot. I am able to set it, but the issue is I am not able to rewrite the same variable to update it with Refresh token.

    For "distributed-metadata"

    I am getting below error:

    mpgw (MPGW_GetAccessToken_TTS)GatewayScript processing Error 'Error: Unable to open the script module file 'distributed-metadata' In file 'gatewayscript:///datapower.js' line:311, stack:Error: Unable to open the script module file 'distributed-metadata' at Runtime.getModuleFilename (<anonymous>) at ModuleContext.<anonymous> (gatewayscript:///datapower.js:311:36) at Object.<anonymous> (local:///OMSRLNA/TTS/GetAccessToken_TTS_Response.js:8:21) at Script.execute (gatewayscript:///datapower.js:158:24) at Object.<anonymous> (gatewayscript:///datapower.js:585:55)'



    ------------------------------
    Shubhda Shambhavi
    ------------------------------



  • 9.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Wed September 10, 2025 05:39 AM

    Hi Joseph,

    I was doing something wrong. Your logic worked for me, for rewriting as well. Thanks a lot. I was struggling since two weeks to achieve this.

    var systemMD = require('system-metadata');
    
    // Then...
    systemMD.MyGlobalContext.MyToken = value;
    
    // OR
    systemMD.setVariable('var://system/MyGlobalContext/MyToken', value);


    ------------------------------
    Shubhda Shambhavi
    ------------------------------



  • 10.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Wed September 10, 2025 05:56 AM

    I have one doubt. If restart of Pod or domain happens then this sytem variable will get reset or how it will respond?



    ------------------------------
    Shubhda Shambhavi
    ------------------------------



  • 11.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Wed September 10, 2025 09:06 AM
    Edited by Joseph Morgan Wed September 10, 2025 09:06 AM

    If you are using global variables and a domain restarts, it shouldn't cause an issue.  If the appliance restarts, It will be gone. When using distributed variables (see note below), you don't lose it unless every appliance in the gateway peering go down at once or in such quick sequence that the peering can't recover shared state.

    This raises questions about your implementation and environment:

    1. How many appliances serve these transactions?  (I'm hoping more than one)
    2. How long will this value need to persist beyond the first transaction? (I was thinking time between transactions would be short)
    3. How many transactions beyond the first transaction will this value be needed?
    4. What is the value used for?

    I'm asking these questions because it just became clear:

    1. Your appliances are in containers (due to the mention of restarting pods).
    2. I'm guessing you're trying to handle some kind of persistent state between transactions, like an expiration on an auth token of some kind (MPGW_GetAccessToken_TTS).

    Using global variables is not recommended for this.  Though it may work in development on one appliance, it will not work across appliances if your production is more than one appliance.  Distributed variables is still recommended.   

    I do see you had some trouble in the code for what I'm guessing to be the require statement.  It is difficult to know the problem without you sharing code.



    ------------------------------
    Joseph Morgan
    CEO - Independent
    ------------------------------



  • 12.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Wed September 10, 2025 09:49 AM

    This raises questions about your implementation and environment:

    1. How many appliances serve these transactions?  : 2 appliance, right now we have implemented in On-premise. But we have container based pods as well in cloud. So if we do new deployment there the pod get restarted.
    2. How long will this value need to persist beyond the first transaction? time would be short. It should be valid only for next transaction.
    3. How many transactions beyond the first transaction will this value be needed?: next consecutive transaction
    4. What is the value used for?:- Requirement- i. We have to make call to Auth URL by passing refresh_token in query parameter. It will return access token and refresh_token the body.  Access token will be valid for 3 hours. To cache the response I am doing document caching in xml manager for TTL 3hours. Now the challenge is in the next transaction when we are calling Auth URL we have to pass refresh_token in query params that we have received in response of previous transaction. That's why we have to save this refresh token somewhere.


    ------------------------------
    Shubhda Shambhavi
    ------------------------------



  • 13.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Wed September 10, 2025 10:40 AM

    Interesting.  OK.  First, I hope you can refresh your API Key and Secret.  

    I'm not sure how the GatewayScript processing error noted above relates to the 2 JS's you posted.  It was complaining about line 311, and neither JS is remotely that long.  I may be missing something.

    I would highly recommend using one of three things for this:

    • Distributed Variables - Looking at the code you attached, I'm not clear on why it is getting the error.  Maybe I'm missing something here as well, and I don't have a 10.5.0.4 appliance on which to test.
    • External cache:  eXtreme Scale, Redis, something like that.   eXtreme Scale could be a problem if you don't already have WebSphere ND to launch it from.  Redis, on the other hand, is lightweight and easy to deploy stand-alone or in containers.   Of course, one could maybe whip up a quick python/java server to act as the cache and drop it into Docker! (No, don't do that, your security team would flip out completely and making it handle load could be a killer).  
    • Database (I'm squinching a bit on this one):  Like eXtreme Scale, if you don't already have one, it likely isn't worth the work to use it and it will add the most latency.  However, it will work.

    Keep in mind, DataPower was never meant to be stateful.  We do now have the distributed variables, which is ideal for this sort of solution.   With that said, you have to evaluate how many tokens you'll have to track at a given time, how quickly that 2nd transaction comes, and how you'd handle the situation if you lose one.   

    Another thing to keep in mind (though I can prove it isn't technically correct).  IBM recommends a quorum (3 appliances) for using gateway peering.  I think it is just a way of selling more licenses, but, they do say they don't support the use of 2, even though it will work.   No one will argue that having 3 appliances is better than having 2, but having 5 is better than having 3, and having 100 is better than having 50.  But, let's get real!



    ------------------------------
    Joseph Morgan
    CEO - Independent
    ------------------------------



  • 14.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Wed September 10, 2025 11:09 AM
    On the quorum topic: The background here (afaik) is that they want to be able to do majority voting. In the case of three appliances (or servers or whatever) you can set up the rule that, in the event of a network outage causing a disconnect, whoever has the majority of instances has the official version of the truth and anybody that doesn't will have to adopt that version for better or for worse upon reconnection. Now, with only two instances this is hard to impossible to decide. Technically, for that reason 100 instances are only a little less problematic - 99 would work well, though.

    Gerd, retired tech sales rep for WDP and other things




  • 15.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Wed September 10, 2025 11:14 AM
    On state: Just an idea, without trying to check feasibility. If you have MQ (or some other messaging implementation) you might be able to insert state info into a queue shared between the appliances and find a clever way to retrieve this info on whatever surviving appliance if the original one has gone AWOL.

    Gerd, retired tech sales for WDP and other beauties, including MQ.




  • 16.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Thu September 11, 2025 01:40 AM

    Yes, we are using IBM MQ in project. I can ask to approach that way.

    Thanks for your suggestion.



    ------------------------------
    Shubhda Shambhavi
    ------------------------------



  • 17.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Thu September 11, 2025 01:38 AM

    Hi Joseph,

    The answer to your doubt for

    mpgw (MPGW_GetAccessToken_TTS)GatewayScript processing Error 'Error: Unable to open the script module file 'distributed-metadata' In file 'gatewayscript:///datapower.js' line:311, stack:Error: Unable to open the script module file 'distributed-metadata' at Runtime.getModuleFilename (<anonymous>) at ModuleContext.<anonymous> (gatewayscript:///datapower.js:311:36) at Object.<anonymous> (local:///OMSRLNA/TTS/GetAccessToken_TTS_Response.js:8:21) at Script.execute (gatewayscript:///datapower.js:158:24) at Object.<anonymous> (gatewayscript:///datapower.js:585:55)'

    how the GatewayScript processing error noted above relates to the 2 JS's you posted.  It was complaining about line 311, and neither JS is remotely that long.  

    > If you see this path gatewayscript:///datapower.js:311:36 

    it is some default file present in DataPower box. I cannot find this in my File Management.

    Option 1. For Distributed Variables : do we need to do some additional settings in our DataPower box?

    For number of appliance, I will cross check in my team. You meant to say if we have two appliance then we can't achieve Gateway-peering?



    ------------------------------
    Shubhda Shambhavi
    ------------------------------



  • 18.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Thu September 11, 2025 11:53 AM

    Shubhda, 

    Ah, yes, regarding the "datapower.js" file, I missed that is the main process.  

    I really don't know what I'm missing there.  Maybe try requiring "apim", as it was originally developed for the API Connect but later came down for availability in later FW.  I will setup a test on my 10.5.0.6, and see if I get the same error.  

    As for the number of appliances, it is recommended by IBM you have an odd number of appliances.  That is 1, 3, 5, etc.  The reason they don't officially support 2 is because of the HA algorithm they use, which appears to be an (N + 1) / 2 without epsilon for quorum control voting.  That is, the HA algorithm they use is somewhat primitive.  They could add epsilon or a ranked clerical system to solve it, but, it is what it is.

    I have run tests in 10.6.0.5 systems and it works.  However, it worked because the 2 appliances remained running.  We didn't test a failure scenario.

    As an additional note on this, I think what's so strange about their GW Peering algorithm is more related to probes.  If one follows IBM recommendations to never turn on probes in production and there is no other need for gateway peering, it follows one could run 2 appliances in production.  But, as we all know, real transactions can differ vastly from test transactions.  So, the conclusion might be to run test/redacted production transactions in, say, pre-production or UAT with probes on.   This would mean then, if strictly following IBM recommendations, one might need 3 appliances in pre-production/UAT (or just one, but that wouldn't be pre-production would it??), whereas only having 2 in production.   I wish IBM would solve this issue and minimally add epsilon to their algorithm.



    ------------------------------
    Joseph Morgan
    CEO - Independent
    ------------------------------



  • 19.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Thu September 11, 2025 12:15 PM

    Hi Joseph,

    It is just a lot of concepts which is hard to digest at a time.

    Gateway peering is also new concept for me. Now not sure how distributed variable is associated with Gateway peering? 

    Without gateway peering can't we use Distributed variables?

    I will read more about it and would try.



    ------------------------------
    Shubhda Shambhavi
    ------------------------------



  • 20.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Thu September 11, 2025 12:40 PM

    Distributed variables require Gateway Peering because it sets up back-channels for distributing those values to other appliances in a cluster.  That is, when using distributed variables, they are "copied" to the other members of the peering group. 

    Without gateway peering, you cannot use distributed variables.

    You did enable Gateway Peering, yes?   



    ------------------------------
    Joseph Morgan
    CEO - Independent
    ------------------------------



  • 21.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Thu September 11, 2025 03:24 PM

    Shubhda,

    I setup a test on one machine, and I had the code incorrect.  Here is a correction:

    var distMD = require("distributed-metadata");
    
    distMD.setVariable('var://dist/MyDistrubutedContext/MyToken', "abc", handleError);
    
    function handleError(error, response) {
        // Handle error in here
    }
    


    ------------------------------
    Joseph Morgan
    CEO - Independent
    ------------------------------



  • 22.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Thu September 11, 2025 11:04 PM

    Shubhda,

    I spent a while today going through FW versions from 10.5.0.6 forward, and, it appears using distrbuted variables won't be possible before 10.6.0.0 firmware.  This is the first version I found where another critical configuration, the "Distributed Variable Settings" becomes available.

    The process is further complicated by the fact that your Gateway Peering must be unique in reference to Local Port and Monitor Port across the entire appliance, and your Distributed Variable settings have to be up and referencing a valid Gateway Peering configuration.   

    I setup a simple rule on a service having two GWScript actions.  The first writes the variable, the second reads it.  It logs everything and it works fine.   Here are my two JS files:

    // SetDistributedVariable.js
    var distMD = require("distributed-metadata");
    
    distMD.setVariable('var://dist/MyDistributedContext/MyToken', "abc", handleError);
    
    function handleError(error, response) {
        if (error) {
            console.error('setVariable error = ' + error);
        } 
        else {
            console.log('setVariable result = ' + response);
        }
    }
    
    

    // GetDistributedVariable.js
    var distMD = require("distributed-metadata");
    
    distMD.getVariable('var://dist/MyDistributedContext/MyToken', handleError);
    
    function handleError(error, response) {
        if (error) {
            console.error("Error reading distributed variable: %s", error);
        }
        else {
            console.debug("Distributed Variable Value is: %s", response);
        }
    }
    

    Hope this helps.



    ------------------------------
    Joseph Morgan
    CEO - Independent
    ------------------------------



  • 23.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Fri September 12, 2025 03:24 AM

    Hi Joseph,

    First of all thanks for dedicating your time on this issue. I appreciate your efforts and patience.

    In FW version 10.5.0.6, I am getting same issue in datapower.js file. As you mentioned, Distributed Variable settings have to be up and referencing a valid Gateway Peering configuration.   Right now I will suggest my team to use MQ or any external chache to achieve this or setting system-variable is also working as of now. After 4-5 months we have to migrate to cloud then we have to use FW version 10.6.0.0. Then going forward we can implement using gateway peering if their no of appliance setting supports.

    Thank you for explaining each points in detail.



    ------------------------------
    Shubhda Shambhavi
    ------------------------------



  • 24.  RE: Rewrite json value in a file to be accessed in future transactions

    Posted Wed September 10, 2025 09:56 AM

    I have attached the rquest and response js files.



    ------------------------------
    Shubhda Shambhavi
    ------------------------------

    Attachment(s)

    js
    request.js   2 KB 1 version
    js
    response.js   1 KB 1 version