Original Message:
Sent: Tue May 21, 2024 06:57 PM
From: Varma Nadim
Subject: ZOS Connect and JWT Token invocation of External Api
Thank you.
We are back to square one not sure why !!!
RACF admin says it has to be outside RACF and if that is the case it must be on Zos connect or liberty side but the trace is not helping us to find the root cause.. Earlier we resolved the issue with
- RDEFINE SURROGAT *.BAQTOKEN UACC(NONE) – Resource has been put in RACF WARNING MODE
- PERMIT *.BAQTOKEN CLASS(SURROGAT) ID(WSGUEST) ACCESS(READ)
- setr refresh raclist(SURROGAT)
- Run zSecure Carla AUDIT Event Report against ID WSGUEST to check for any violations - Use Current Date SMF Data as the source input.
WE received an error earlier but that is changed the message a bit the following is the error before we defined the Surrogate profile.
[5/2/24 9:17:53:988 EDT] 00000039 ZosConnectSaf 3 Link ID WSGUEST does not have READ access to profile NVN9899.BAQTOKEN, errno:143, errno2:154,665,167, errnoMsg:EDC5143I No such process.
[5/2/24 9:17:53:988 EDT] 00000039 ZosConnectSaf 3 CallerSubject is null
Now this is completely different at least from the get go as we have not changed anything other than the ARA to invoke just a simple API as earlier API had tons of data we asked them to give a simple API and when we try to execute that we are getting this issue below…
Suddenly after we IPL the LPAR we are receiving the following issue now.... Not sure what changed
Not sure what it is complaining about and what Operation is not allowed... In fact the RACF admin defined this as WARNING and he said basically everyone has READ access to the profile because of that definition.
ABCUSER is the User trying to execute the Transaction which invokes the API. Not able to really get any further after spending lot of time with my RACF admin...
[5/21/24 16:38:06:292 EDT] 00000039 ARRequestImpl 3 Stored query string
[5/21/24 16:38:06:295 EDT] 00000039 ZosConnectSaf 3 Link ID WSGUEST does not have READ access to profile NVN9899.BAQTOKEN, errno:139, errno2:154,665,647, errnoMsg:EDC5139I Operation not permitted.
[5/21/24 16:38:06:295 EDT] 00000039 ZosConnectSaf 3 CallerSubject is null
------------------------------
Varma Nadim
Original Message:
Sent: Mon May 13, 2024 04:35 AM
From: Andrew Smithson
Subject: ZOS Connect and JWT Token invocation of External Api
Hi Verma
The documentation here provides information on monitoring and tuning the Java Heap for z/OS Connect. Threads, Java heap, MEMLIMIT, REGION
Ibm | remove preview |
| Threads, Java heap, MEMLIMIT, REGION | To ensure your IBM z/OS Connect server performs optimally, you need to carefully configure the Java heap, MEMLIMIT, and REGION values, and continually monitor your server while your workload is running. | View this on Ibm > |
|
|
Thanks
Andrew
------------------------------
Andrew Smithson
Software Engineer
IBM
Winchester
01962 817190
Original Message:
Sent: Thu May 09, 2024 09:04 AM
From: Varma Nadim
Subject: ZOS Connect and JWT Token invocation of External Api
Thank you. I think we made big progress on this issue today.
I have encountered another issue this is JAVA Heap issue . Why are we seeing this Java heap issue do i need to reachout to ZOS team to allocate more Java heap to resolve the issue or Zos connect admin to allocate more Java heap space. Please point to any documentation that can help move forward this issue. This even created ffdc incident ...
[5/9/24 8:54:31:732 EDT] 00000063 ApplicationEr E SRVE0777E: Exception thrown by application class 'com.ibm.cicsts.transform.dataTransformImpl.DataPage.<init>:100'
java.lang.OutOfMemoryError: Java heap space
at com.ibm.cicsts.transform.dataTransformImpl.DataPage.<init>(DataPage.java:100)
at com.ibm.cicsts.transform.dataTransformImpl.ProcessingState.addNewDataPage(ProcessingState.java:350)
at com.ibm.cicsts.transform.dataTransformImpl.ProcessingState.addTopLevelDataPage(ProcessingState.java:321)
at com.ibm.cicsts.transform.dataTransformImpl.ProcessingState.<init>(ProcessingState.java:162)
at com.ibm.cicsts.transform.dataTransform.XMLToData.transform(XMLToData.java:250)
at com.ibm.cics.wlp.json.JsonUtils.convertJSONToData(JsonUtils.java:607)
at com.ibm.cics.wlp.json.JsonUtils.convertJSONToData(JsonUtils.java:563)
at com.ibm.zosconnect.apirequester.internal.xform.SharedDataXformImpl.getBytes(Unknown Source)
at com.ibm.zosconnect.apirequester.internal.ARInvokeHandler.reverseTransform(Unknown Source)
at com.ibm.zosconnect.apirequester.internal.ARInvokeHandler.handleResponseData(Unknown Source)
at com.ibm.zosconnect.apirequester.internal.ARInvokeHandler.handle(Unknown Source)
at com.ibm.zosconnect.apirequester.internal.ApiRequesterManagerImpl.invoke(Unknown Source)
at com.ibm.zosconnect.apirequester.internal.proxy.ApiRequesterManagerProxyImpl$1.run(Unknown Source)
at com.ibm.zosconnect.apirequester.internal.proxy.ApiRequesterManagerProxyImpl$1.run(Unknown Source)
at com.ibm.zosconnect.apirequester.internal.proxy.InterceptorExecutor.execute(Unknown Source)
at com.ibm.zosconnect.apirequester.internal.proxy.InterceptorExecutor.executeRequester(Unknown Source)
at com.ibm.zosconnect.apirequester.internal.proxy.ApiRequesterManagerProxyImpl.invoke(Unknown Source)
at com.ibm.zosconnect.apirequester.internal.web.ApiRequesterServlet.handleInvokeApiRequester(Unknown Source)
at com.ibm.zosconnect.apirequester.internal.web.ApiRequesterServlet.doPut(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at com.ibm.zosconnect.apirequester.internal.web.ApiRequesterServlet.service(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1260)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:748)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:445)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:197)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:100)
at com.ibm.ws.app.manager.wab.internal.OsgiDirectoryProtectionFilter.doFilter(OsgiDirectoryProtectionFilter.java:92)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:203)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:93)
at com.ibm.zosconnect.apirequester.internal.web.ApiRequesterWebFilter$1.run(Unknown Source)
at com.ibm.zosconnect.apirequester.internal.web.ApiRequesterWebFilter$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(AccessController.java:529)
at javax.security.auth.Subject.doAs(Subject.java:488)
at com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:125)
at com.ibm.websphere.security.auth.WSSubject.doAs(WSSubject.java:92)
at com.ibm.zosconnect.apirequester.internal.web.ApiRequesterWebFilter.processDoFilterWithSubject(Unknown Source)
at com.ibm.zosconnect.apirequester.internal.web.ApiRequesterWebFilter.doFilter(Unknown Source)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:203)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:93)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:1068)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1260)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:5080)
at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.handleRequest(DynamicVirtualHost.java:318)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1038)
at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.run(DynamicVirtualHost.java:283)
at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:1248)
at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.wrapHandlerAndExecute(HttpDispatcherLink.java:470)
at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:429)
at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:569)
at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:503)
at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:363)
at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.ready(HttpInboundLink.java:330)
at com.ibm.ws.channel.ssl.internal.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1131)
at com.ibm.ws.channel.ssl.internal.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:680)
at com.ibm.ws.channel.ssl.internal.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1826)
at com.ibm.ws.tcpchannel.internal.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:140)
at com.ibm.io.async.AbstractAsyncFuture$WorkCallback.run(AbstractAsyncFuture.java:386)
at com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:247)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:785)
------------------------------
Varma Nadim
Original Message:
Sent: Wed May 08, 2024 05:56 AM
From: Andrew Smithson
Subject: ZOS Connect and JWT Token invocation of External Api
Hi Varma
If you want to use a certificate for generating the signature of the JWT then you will need to specify the RS256 algorithm which uses asymmetric keys for generating the signature. If you want to use HS256 then you will need to specify the sharedKey parameter on the JWTBuilder as that algorithm uses a symmetric key.
There is some useful information about the differences between the two algorithms here -> RS256 vs HS256 JWT signing algorithms
Andrew
------------------------------
Andrew Smithson
Software Engineer
IBM
Winchester
01962 817190
Original Message:
Sent: Tue May 07, 2024 05:51 PM
From: Varma Nadim
Subject: ZOS Connect and JWT Token invocation of External Api
Thank you for the information. We have a certificate with Private Key and we added that to a new name to test this JWT stuff not to confuse with others we have.
RACF obtained the certificate and then added that to the Key Ring for Liberty on Mainframe. you can check details below.

<jwtBuilder id="jwtBuilder"
scope="scope1"
audiences="0ec35b7c1af840d9bab3ed0a5572c129"
jti="true"
expiry="1h"
issuer="test"
signatureAlgorithm="HS256"
keyStoreRef="defaultKeyStore"
keyAlias="ZCHT01TS.JWT01"
sharedKey="ZCHT01TS.JWT01"/>
<keyStore id="defaultKeyStore"
location="safkeyring:///Keyring.LIBERTY"
password="XXXXXXXX" type="JCERACFKS"
fileBased="false" readOnly="true" />
Error in trace
[5/7/24 15:32:17:937 EDT] 00000034 JwtData 3 hs256 Signing key type is shared secret
[5/7/24 15:32:17:996 EDT] 00000034 JwtUtils 3 JSON String ={"typ":"JWT","alg":"HS256"}
[5/7/24 15:32:17:998 EDT] 00000034 JwtUtils 3 Value [JWT] is not a valid JSON array: Unexpected character 'J' on line 1, column 14
[5/7/24 15:32:17:998 EDT] 00000034 JwtUtils 3 Value [JWT] is not a valid JSON object: Unexpected character 'J' on line 1, column 1
[5/7/24 15:32:17:998 EDT] 00000034 JwtUtils 3 Value [HS256] is not a valid JSON array: Unexpected character 'H' on line 1, column 14
[5/7/24 15:32:17:998 EDT] 00000034 JwtUtils 3 Value [HS256] is not a valid JSON object: Unexpected character 'H' on line 1, column 1
[5/7/24 15:32:18:015 EDT] 00000034 ARRequestImpl 3 TimeTokenGetFinish timestamp set to [0, -33, 15, 40, 114, -12, -8, 27, 22, 0, 0, 0, 8, 120, 0, 4]
[5/7/24 15:32:18:159 EDT] 00000034 RestClientImp 3 Adding Accept header application/json
[5/7/24 15:32:18:159 EDT] 00000034 RestClientImp 3 Request header key parameters are :[authorization, content-type, accept]
[5/7/24 15:32:18:159 EDT] 00000034 RestClientImp 3 No request cookies
------------------------------
Varma Nadim
Original Message:
Sent: Tue May 07, 2024 12:46 PM
From: Andrew Smithson
Subject: ZOS Connect and JWT Token invocation of External Api
Varma
Apologies for not responding sooner, it was a long weekend here in the UK.
Looking at the error message it would appear that in the JWTBuilder your configuration has HS256 as the algorithm to use for the JWT signature. As this is a symmetric algorithm the key needs to be specified in the sharedKey parameter in the JWTBuilder. ref https://www.ibm.com/docs/en/was-liberty/zos?topic=configuration-jwtbuilder
Hope this helps.
Andrew
------------------------------
Andrew Smithson
Software Engineer
IBM
Winchester
01962 817190
Original Message:
Sent: Sun May 05, 2024 09:00 AM
From: Varma Nadim
Subject: ZOS Connect and JWT Token invocation of External Api
Andrew,
We made progress after opening the Un authenticated user access which is definitely a security hole but since we are doing POC we opened it up and error we are getting now is this.
[5/5/24 8:12:13:251 EDT] 0000009b Authenticatio 3 zosconnect_endpointConnection > authenticationConfigRef: [jwtConfig]
[5/5/24 8:12:13:251 EDT] 0000009b Authenticatio 3 cachedAvailbleAuthenticationConfigs: {jwtConfig=com.ibm.zosconnect.endpoint.connection.internal.AuthTokenLocalImpl@9ef25548}
[5/5/24 8:12:13:251 EDT] 0000009b Authenticatio 3 cachedUnAvailableAuthenticationConfigs: {}
[5/5/24 8:12:13:251 EDT] 0000009b Authenticatio 3 start handle cached config com.ibm.zosconnect.endpoint.connection.internal.AuthTokenLocalImpl@9ef25548, id jwtConfig
[5/5/24 8:12:13:256 EDT] 0000009b Authenticatio 3 Not able to handle id jwtConfig, object com.ibm.zosconnect.endpoint.connection.internal.AuthTokenLocalImpl@9ef25548 in cachedAuthenticationConfigs
[5/5/24 8:12:13:256 EDT] 0000009b Authenticatio 3 Merge conflictAuthenticationConfigs after AuthenticationConfig check, conflictAuthenticationConfigs: {}
[5/5/24 8:12:13:256 EDT] 0000009b Authenticatio 3 Merge conflictAuthenticationConfigs after AuthenticationConfig check, cachedAvailbleAuthenticationConfigs: {}
[5/5/24 8:12:13:256 EDT] 0000009b ARRequestImpl 3 TimeTokenGetStart timestamp set to [0, -33, 12, 66, 90, 126, -50, 49, -74, 0, 0, 0, 8, 0, 0, 4]
[5/5/24 8:12:13:276 EDT] 0000009b IssuerUtils 3 issuer url is:test
[5/5/24 8:12:13:307 EDT] 0000009b JwtData 3 hs256 Signing key type is shared secret
[5/5/24 8:12:13:313 EDT] 0000009b JwtTokenExcep E CWWKS6016E: The signing key that is required by the signature algorithm [HS256] is not available. Verify that the signature algorithm and the jwkEnabled [false] are configured properly.
[5/5/24 8:12:13:338 EDT] 0000009b IncidentImpl I FFDC1015I: An FFDC Incident has been created: "CWWKS6016E: The signing key that is required by the signature algorithm [HS256] is not available. Verify that the signature algorithm and the jwkEnabled [false] are configured properly. com.ibm.ws.security.jwt.internal.TokenImpl 34" at ffdc_24.05.05_08.12.13.0.log
[5/5/24 8:12:13:341 EDT] 0000009b AuthTokenLoca E BAQR1008E: An error occurred when the z/OS Connect server attempted to generate a JWT. Error:CWWKS6016E: The signing key that is required by the signature algorithm [HS256] is not available. Verify that the signature algorithm and the jwkEnabled [false] are configured properly.
[5/5/24 8:12:13:353 EDT] 0000009b ARRequestImpl 3 TimeTokenGetFinish timestamp set to [0, -33, 12, 66, 90, -106, 83, -7, 68, 0, 0, 0, 8, 0, 0, 4]
[5/5/24 8:12:13:358 EDT] 0000009b ApiRequesterI 3 Number of in-flight requests for API requester "Eligibility-Demographics-Service-Web-API-Internal_v1" is now decremented to "0".
[5/5/24 8:12:13:365 EDT] 0000009b HttpException 3 Set HTTP servlet response code to 500
[5/5/24 8:12:13:367 EDT] 0000009b ARResponseImp > setPayload Entry
How do i make the signature algorithm HS256 make available so we can generate this properly any pointers are appreciated...
------------------------------
Varma Nadim
Original Message:
Sent: Mon April 15, 2024 04:28 PM
From: Andrew Smithson
Subject: ZOS Connect and JWT Token invocation of External Api
Hi Varma
The documentation on building a JWT in the z/OS Connect Server here https://www.ibm.com/docs/en/zos-connect/zosconnect/3.0?topic=jwt-configuring-locally-generated includes the references to a keystore and key within it. This is the private key that is used to sign the JWT and the certificate associated with this key-pair would be the one that is sent to the API team so that they can verify the JWTs that are sent from the z/OS Connect Server.
Andrew
------------------------------
Andrew Smithson
Software Engineer
IBM
Winchester
01962 817190
Original Message:
Sent: Fri March 29, 2024 11:53 AM
From: Varma Nadim
Subject: ZOS Connect and JWT Token invocation of External Api
I am looking for a practical working of a working server configuration for the following scenario.
I need to invoke an Api from Zos connect. What they asked me to invoke this as below.
We need to invoke it using JWT and asked me to generate one locally on Mainframe. When i checked IBM support documentation i see we need to setup
You can call an API by sending a JSON Web Token (JWT) that is locally generated by the IBM z/OS Connect server. The generated JWT contains a subject claim that is the z/OS application asserted user ID passed from the communication stub.
What Api team is asking is you use a certificate to create the JWT and then give us the Public Key for that certificate so they can install on their servers where the Api is hosed which should authenticate the bearer token method from zos connect.
Anyone has this working using a Certificate.. Can they provide details like.
Can i generate the JWT locally using the Zos connect certificate? or CICS certificate that i use to connect? Or i still use the Link user id but give the certificate that we have secured the CICS region or Zos connect cert to the Api team? They want he JWT that is signed by this cert so they can accept this when invoking their API. They will install the public key of this cert on their servers which should be able to check and allow the call to be successful. Any experience in this kind of call would like to know the details.
Thanks
Varma.
------------------------------
Varma Nadim
------------------------------