z/OS Connect Enterprise Edition

Expand all | Collapse all

ZOS Connect and JWT Token invocation of External Api

  • 1.  ZOS Connect and JWT Token invocation of External Api

    Posted Tue April 02, 2024 05:00 AM
    Edited by Demelza Farrer Tue April 02, 2024 09:09 AM

    I am looking for a practical working of a working server configuration for the following scenario.

    I need to invoke an Api from Zos connect. What they asked me to invoke this as below.

    We need to invoke it using JWT and asked me to generate one locally on Mainframe. When i checked IBM support documentation i see we need to setup 

    You can call an API by sending a JSON Web Token (JWT) that is locally generated by the IBM z/OS Connect server. The generated JWT contains a subject claim that is the z/OS application asserted user ID passed from the communication stub.

    What Api team is asking is you use a certificate to create the JWT and then give us the Public Key for that certificate so they can install on their servers where the Api is hosed which should authenticate the bearer token method from zos connect.

    Anyone has this working using a Certificate.. Can they provide details like.

    Can i generate the JWT locally using the Zos connect certificate? or CICS certificate that i use to connect? Or i still use the Link user id but give the certificate that we have secured the CICS region or Zos connect cert to the Api team? They want he JWT that is signed by this cert so they can accept this when invoking their API. They will install the public key of this cert on their servers which should be able to check and allow the call to be successful. Any experience in this kind of call would like to know the details.

    Thanks

    Varma.



    ------------------------------
    Varma Nadim
    ------------------------------



  • 2.  RE: ZOS Connect and JWT Token invocation of External Api

    Posted Mon April 15, 2024 04:29 PM

    Hi Varma

    The documentation on building a JWT in the z/OS Connect Server here https://www.ibm.com/docs/en/zos-connect/zosconnect/3.0?topic=jwt-configuring-locally-generated includes the references to a keystore and key within it. This is the private key that is used to sign the JWT and the certificate associated with this key-pair would be the one that is sent to the API team so that they can verify the JWTs that are sent from the z/OS Connect Server.

    Andrew



    ------------------------------
    Andrew Smithson
    Software Engineer
    IBM
    Winchester
    01962 817190
    ------------------------------



  • 3.  RE: ZOS Connect and JWT Token invocation of External Api

    Posted Fri May 03, 2024 03:05 PM

    Hi Andrew,

    Thank you but we followed al the documentation IBM provided and also spent time with RACF and ZOS connect Admin still we are not sure why it is not working.

    Please understand calling a JWT OAUTH server and getting JWT and using it to invoke works fine but we need JWT locally generated.

    Appreciate if you can provide details of server.xml JWT configuration you may have used to test this would help us greatly.

    Please let me know if you can join us on Monday morning 9:00 EST time when i have lined up IBM SME to come help but he even said he never tried JWT locally generated. WE need this to work otherwise major project will be in danger i may need to stop working with Zos connect and see alternatives.

    Please let me know i will include you in the meeting on Monday.



    ------------------------------
    Varma Nadim
    ------------------------------



  • 4.  RE: ZOS Connect and JWT Token invocation of External Api

    Posted Sun May 05, 2024 09:01 AM

    Andrew,

    We made progress after opening the Un authenticated user access which is definitely a security hole but since we are doing POC we opened it up and error we are getting now is this.

    [5/5/24 8:12:13:251 EDT] 0000009b Authenticatio 3   zosconnect_endpointConnection > authenticationConfigRef: [jwtConfig]
    [5/5/24 8:12:13:251 EDT] 0000009b Authenticatio 3   cachedAvailbleAuthenticationConfigs: {jwtConfig=com.ibm.zosconnect.endpoint.connection.internal.AuthTokenLocalImpl@9ef25548}
    [5/5/24 8:12:13:251 EDT] 0000009b Authenticatio 3   cachedUnAvailableAuthenticationConfigs: {}
    [5/5/24 8:12:13:251 EDT] 0000009b Authenticatio 3   start handle cached config com.ibm.zosconnect.endpoint.connection.internal.AuthTokenLocalImpl@9ef25548, id jwtConfig
    [5/5/24 8:12:13:256 EDT] 0000009b Authenticatio 3   Not able to handle id jwtConfig, object com.ibm.zosconnect.endpoint.connection.internal.AuthTokenLocalImpl@9ef25548 in cachedAuthenticationConfigs
    [5/5/24 8:12:13:256 EDT] 0000009b Authenticatio 3   Merge conflictAuthenticationConfigs after AuthenticationConfig check, conflictAuthenticationConfigs: {}
    [5/5/24 8:12:13:256 EDT] 0000009b Authenticatio 3   Merge conflictAuthenticationConfigs after AuthenticationConfig check, cachedAvailbleAuthenticationConfigs: {}
    [5/5/24 8:12:13:256 EDT] 0000009b ARRequestImpl 3   TimeTokenGetStart timestamp set to [0, -33, 12, 66, 90, 126, -50, 49, -74, 0, 0, 0, 8, 0, 0, 4]
    [5/5/24 8:12:13:276 EDT] 0000009b IssuerUtils   3   issuer url is:test
    [5/5/24 8:12:13:307 EDT] 0000009b JwtData       3   hs256 Signing key type is shared secret
    [5/5/24 8:12:13:313 EDT] 0000009b JwtTokenExcep E   CWWKS6016E: The signing key that is required by the signature algorithm [HS256] is not available. Verify that the signature algorithm and the jwkEnabled [false] are configured properly. 
    [5/5/24 8:12:13:338 EDT] 0000009b IncidentImpl  I   FFDC1015I: An FFDC Incident has been created: "CWWKS6016E: The signing key that is required by the signature algorithm [HS256] is not available. Verify that the signature algorithm and the jwkEnabled [false] are configured properly.  com.ibm.ws.security.jwt.internal.TokenImpl 34" at ffdc_24.05.05_08.12.13.0.log
    [5/5/24 8:12:13:341 EDT] 0000009b AuthTokenLoca E   BAQR1008E: An error occurred when the z/OS Connect server attempted to generate a JWT. Error:CWWKS6016E: The signing key that is required by the signature algorithm [HS256] is not available. Verify that the signature algorithm and the jwkEnabled [false] are configured properly. 
    [5/5/24 8:12:13:353 EDT] 0000009b ARRequestImpl 3   TimeTokenGetFinish timestamp set to [0, -33, 12, 66, 90, -106, 83, -7, 68, 0, 0, 0, 8, 0, 0, 4]
    [5/5/24 8:12:13:358 EDT] 0000009b ApiRequesterI 3   Number of in-flight requests for API requester "Eligibility-Demographics-Service-Web-API-Internal_v1" is now decremented to "0".
    [5/5/24 8:12:13:365 EDT] 0000009b HttpException 3   Set HTTP servlet response code to 500
    [5/5/24 8:12:13:367 EDT] 0000009b ARResponseImp >  setPayload Entry  

    How do i make the signature algorithm HS256 make available so we can generate this properly any pointers are appreciated...



    ------------------------------
    Varma Nadim
    ------------------------------



  • 5.  RE: ZOS Connect and JWT Token invocation of External Api

    Posted Tue May 07, 2024 12:46 PM

    Varma

    Apologies for not responding sooner, it was a long weekend here in the UK.

    Looking at the error message it would appear that in the JWTBuilder your configuration has HS256 as the algorithm to use for the JWT signature. As this is a symmetric algorithm the key needs to be specified in the sharedKey parameter in the JWTBuilder. ref https://www.ibm.com/docs/en/was-liberty/zos?topic=configuration-jwtbuilder

    Hope this helps.

    Andrew



    ------------------------------
    Andrew Smithson
    Software Engineer
    IBM
    Winchester
    01962 817190
    ------------------------------



  • 6.  RE: ZOS Connect and JWT Token invocation of External Api

    Posted Tue May 07, 2024 05:52 PM

    Thank you for the information. We have a certificate with Private Key and we added that to a new name to test this JWT stuff not to confuse with others we have.

    RACF obtained the certificate and then added that to the Key Ring for Liberty on Mainframe. you can check details below. 

        <jwtBuilder id="jwtBuilder"                  
            scope="scope1"                  
            audiences="0ec35b7c1af840d9bab3ed0a5572c129"      
            jti="true"                  
            expiry="1h"      
            issuer="test"      
            signatureAlgorithm="HS256"                  
            keyStoreRef="defaultKeyStore"                  
            keyAlias="ZCHT01TS.JWT01"
            sharedKey="ZCHT01TS.JWT01"/> 

              <keyStore id="defaultKeyStore"    
                   location="safkeyring:///Keyring.LIBERTY"
                   password="XXXXXXXX" type="JCERACFKS"
                   fileBased="false" readOnly="true" />

    Error in trace 

    [5/7/24 15:32:17:937 EDT] 00000034 JwtData    3  hs256 Signing key type is shared secret

    [5/7/24 15:32:17:996 EDT] 00000034 JwtUtils   3  JSON String ={"typ":"JWT","alg":"HS256"}

    [5/7/24 15:32:17:998 EDT] 00000034 JwtUtils   3  Value [JWT] is not a valid JSON array: Unexpected character 'J' on line 1, column 14

    [5/7/24 15:32:17:998 EDT] 00000034 JwtUtils   3  Value [JWT] is not a valid JSON object: Unexpected character 'J' on line 1, column 1

    [5/7/24 15:32:17:998 EDT] 00000034 JwtUtils   3  Value [HS256] is not a valid JSON array: Unexpected character 'H' on line 1, column 14

    [5/7/24 15:32:17:998 EDT] 00000034 JwtUtils   3  Value [HS256] is not a valid JSON object: Unexpected character 'H' on line 1, column 1

    [5/7/24 15:32:18:015 EDT] 00000034 ARRequestImpl 3  TimeTokenGetFinish timestamp set to [0, -33, 15, 40, 114, -12, -8, 27, 22, 0, 0, 0, 8, 120, 0, 4]

    [5/7/24 15:32:18:159 EDT] 00000034 RestClientImp 3  Adding Accept header application/json

    [5/7/24 15:32:18:159 EDT] 00000034 RestClientImp 3  Request header key parameters are :[authorization, content-type, accept]

    [5/7/24 15:32:18:159 EDT] 00000034 RestClientImp 3  No request cookies



    ------------------------------
    Varma Nadim
    ------------------------------



  • 7.  RE: ZOS Connect and JWT Token invocation of External Api

    Posted Wed May 08, 2024 05:57 AM

    Hi Varma

    If you want to use a certificate for generating the signature of the JWT then you will need to specify the RS256 algorithm which uses asymmetric keys for generating the signature. If you want to use HS256 then you will need to specify the sharedKey parameter on the JWTBuilder as that algorithm uses a symmetric key.

    There is some useful information about the differences between the two algorithms here -> RS256 vs HS256 JWT signing algorithms

    Andrew



    ------------------------------
    Andrew Smithson
    Software Engineer
    IBM
    Winchester
    01962 817190
    ------------------------------



  • 8.  RE: ZOS Connect and JWT Token invocation of External Api

    Posted Thu May 09, 2024 09:04 AM

    Thank you. I think we made big progress on this issue today.

    I have encountered another issue this is JAVA Heap issue .  Why are we seeing this Java heap issue do i need to reachout to ZOS team to allocate more Java heap to resolve the issue or Zos connect admin to allocate more Java heap space.  Please point to any documentation that can help move forward this issue. This even created ffdc incident ...

    [5/9/24 8:54:31:732 EDT] 00000063 ApplicationEr E   SRVE0777E: Exception thrown by application class 'com.ibm.cicsts.transform.dataTransformImpl.DataPage.<init>:100'
    java.lang.OutOfMemoryError: Java heap space
    at com.ibm.cicsts.transform.dataTransformImpl.DataPage.<init>(DataPage.java:100)
    at com.ibm.cicsts.transform.dataTransformImpl.ProcessingState.addNewDataPage(ProcessingState.java:350)
    at com.ibm.cicsts.transform.dataTransformImpl.ProcessingState.addTopLevelDataPage(ProcessingState.java:321)
    at com.ibm.cicsts.transform.dataTransformImpl.ProcessingState.<init>(ProcessingState.java:162)
    at com.ibm.cicsts.transform.dataTransform.XMLToData.transform(XMLToData.java:250)
    at com.ibm.cics.wlp.json.JsonUtils.convertJSONToData(JsonUtils.java:607)
    at com.ibm.cics.wlp.json.JsonUtils.convertJSONToData(JsonUtils.java:563)
    at com.ibm.zosconnect.apirequester.internal.xform.SharedDataXformImpl.getBytes(Unknown Source)
    at com.ibm.zosconnect.apirequester.internal.ARInvokeHandler.reverseTransform(Unknown Source)
    at com.ibm.zosconnect.apirequester.internal.ARInvokeHandler.handleResponseData(Unknown Source)
    at com.ibm.zosconnect.apirequester.internal.ARInvokeHandler.handle(Unknown Source)
    at com.ibm.zosconnect.apirequester.internal.ApiRequesterManagerImpl.invoke(Unknown Source)
    at com.ibm.zosconnect.apirequester.internal.proxy.ApiRequesterManagerProxyImpl$1.run(Unknown Source)
    at com.ibm.zosconnect.apirequester.internal.proxy.ApiRequesterManagerProxyImpl$1.run(Unknown Source)
    at com.ibm.zosconnect.apirequester.internal.proxy.InterceptorExecutor.execute(Unknown Source)
    at com.ibm.zosconnect.apirequester.internal.proxy.InterceptorExecutor.executeRequester(Unknown Source)
    at com.ibm.zosconnect.apirequester.internal.proxy.ApiRequesterManagerProxyImpl.invoke(Unknown Source)
    at com.ibm.zosconnect.apirequester.internal.web.ApiRequesterServlet.handleInvokeApiRequester(Unknown Source)
    at com.ibm.zosconnect.apirequester.internal.web.ApiRequesterServlet.doPut(Unknown Source)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
    at com.ibm.zosconnect.apirequester.internal.web.ApiRequesterServlet.service(Unknown Source)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1260)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:748)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:445)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:197)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:100)
    at com.ibm.ws.app.manager.wab.internal.OsgiDirectoryProtectionFilter.doFilter(OsgiDirectoryProtectionFilter.java:92)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:203)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:93)
    at com.ibm.zosconnect.apirequester.internal.web.ApiRequesterWebFilter$1.run(Unknown Source)
    at com.ibm.zosconnect.apirequester.internal.web.ApiRequesterWebFilter$1.run(Unknown Source)
    at java.security.AccessController.doPrivileged(AccessController.java:529)
    at javax.security.auth.Subject.doAs(Subject.java:488)
    at com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:125)
    at com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:92)
    at com.ibm.zosconnect.apirequester.internal.web.ApiRequesterWebFilter.processDoFilterWithSubject(Unknown Source)
    at com.ibm.zosconnect.apirequester.internal.web.ApiRequesterWebFilter.doFilter(Unknown Source)
    at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:203)
    at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:93)
    at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:1068)
    at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1260)
    at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:5080)
    at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.handleRequest(DynamicVirtualHost.java:318)
    at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1038)
    at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.run(DynamicVirtualHost.java:283)
    at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:1248)
    at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.wrapHandlerAndExecute(HttpDispatcherLink.java:470)
    at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:429)
    at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:569)
    at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:503)
    at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:363)
    at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.ready(HttpInboundLink.java:330)
    at com.ibm.ws.channel.ssl.internal.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1131)
    at com.ibm.ws.channel.ssl.internal.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:680)
    at com.ibm.ws.channel.ssl.internal.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1826)
    at com.ibm.ws.tcpchannel.internal.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:140)
    at com.ibm.io.async.AbstractAsyncFuture$WorkCallback.run(AbstractAsyncFuture.java:386)
    at com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:247)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(Thread.java:785)


    ------------------------------
    Varma Nadim
    ------------------------------



  • 9.  RE: ZOS Connect and JWT Token invocation of External Api

    Posted Mon May 13, 2024 04:36 AM

    Hi Verma

    The documentation here provides information on monitoring and tuning the Java Heap for z/OS Connect. Threads, Java heap, MEMLIMIT, REGION

    Ibm remove preview
    Threads, Java heap, MEMLIMIT, REGION
    To ensure your IBM z/OS Connect server performs optimally, you need to carefully configure the Java heap, MEMLIMIT, and REGION values, and continually monitor your server while your workload is running.
    View this on Ibm >

    Thanks

    Andrew



    ------------------------------
    Andrew Smithson
    Software Engineer
    IBM
    Winchester
    01962 817190
    ------------------------------



  • 10.  RE: ZOS Connect and JWT Token invocation of External Api

    Posted Tue May 21, 2024 06:39 PM

    Thanks Andrew. 

    Suddenly after we IPLED the LPAR we are receiving the following issue now.... Not sure what changed from RACF admin we have implemented the following.

    1. RDEFINE SURROGAT *.BAQTOKEN UACC(NONE)
    2. PERMIT *.BAQTOKEN CLASS(SURROGAT) ID(WSGUEST) ACCESS(READ

    Not sure what it is complaining about and what Operation is not allowed... In fact the RACF admin defined this as WARNING and he said basically everyone has READ access to the profile because of that definition.

    ABCUSER is the User trying to execute the Transaction which invokes the API. Not able to really get any further after spending lot of time with my RACF admin...

    [5/21/24 16:38:06:292 EDT] 00000039 ARRequestImpl 3   Stored query string 
                                     

    [5/21/24 16:38:06:295 EDT] 00000039 ZosConnectSaf 3   Link ID WSGUEST does not have READ access to profile ABCUSER.BAQTOKEN, errno:139, errno2:154,665,647, errnoMsg:EDC5139I Operation not permitted.
    [5/21/24 16:38:06:295 EDT] 00000039 ZosConnectSaf 3   CallerSubject is null



    ------------------------------
    Varma Nadim
    ------------------------------



  • 11.  RE: ZOS Connect and JWT Token invocation of External Api

    Posted Tue May 21, 2024 06:57 PM

    Thank you.

    We are back to square one not sure why !!!

    RACF admin says it has to be outside RACF and if that is the case it must be on Zos connect or liberty side but the trace is not helping us to find the root cause..  Earlier we resolved the issue with

    1. RDEFINE SURROGAT *.BAQTOKEN UACC(NONE) – Resource has been put in RACF WARNING MODE
    2. PERMIT *.BAQTOKEN CLASS(SURROGAT) ID(WSGUEST) ACCESS(READ)
    3. setr refresh raclist(SURROGAT)
    4. Run zSecure Carla AUDIT Event Report against ID WSGUEST to check for any violations - Use Current Date SMF Data as the source input.

    WE received an error earlier but that is changed the message a bit the following is the error before we defined the Surrogate profile.

    [5/2/24 9:17:53:988 EDT] 00000039 ZosConnectSaf 3   Link ID WSGUEST does not have READ access to profile NVN9899.BAQTOKEN, errno:143, errno2:154,665,167, errnoMsg:EDC5143I No such process.

    [5/2/24 9:17:53:988 EDT] 00000039 ZosConnectSaf 3   CallerSubject is null

    Now this is completely different at least from the get go as we have not changed anything other than the ARA to invoke just a simple API as earlier API had tons of data we asked them to give a simple API and when we try to execute that we are getting this issue below…

    Suddenly after we IPL the LPAR we are receiving the following issue now.... Not sure what changed

    Not sure what it is complaining about and what Operation is not allowed... In fact the RACF admin defined this as WARNING and he said basically everyone has READ access to the profile because of that definition.

    ABCUSER is the User trying to execute the Transaction which invokes the API. Not able to really get any further after spending lot of time with my RACF admin...

    [5/21/24 16:38:06:292 EDT] 00000039 ARRequestImpl 3   Stored query string 
                                     

    [5/21/24 16:38:06:295 EDT] 00000039 ZosConnectSaf 3   Link ID WSGUEST does not have READ access to profile NVN9899.BAQTOKEN, errno:139, errno2:154,665,647, errnoMsg:EDC5139I Operation not permitted.
    [5/21/24 16:38:06:295 EDT] 00000039 ZosConnectSaf 3   CallerSubject is null



    ------------------------------
    Varma Nadim
    ------------------------------



  • 12.  RE: ZOS Connect and JWT Token invocation of External Api

    Posted Tue May 28, 2024 06:15 PM

    After checking few other options finally, we have asked our ZOS team to see if they need to do the following commands after IPL.

    Sure enough once they issued them I see the Api is working fine.

    I am surprised IBM document never said to issue these after every IPL.


    extattr +p /usr/lib/java_runtime/libifaedjreg64.so

    extattr +p <your library file path>

    Again we are doing this for the first time and we found the issue and cause the hard.

    WE are still trying to test few things more on this.

    At this time I moved past the permission issue.



    ------------------------------
    Varma Nadim
    ------------------------------