Hello!

We have a problem with cookie jar. In our pre-prod environment, Webseal does not sent JSESSIONID back to the Liberty backend, causing the Liberty to break its session

 Problem is that we have this working in test environment but failing in pre-prod-environment. So.. likely reason is some sort of misconfiguration or difference between environments, but we cannot locate the difference

 Here is our setup

•  ibmcom/verify-access:10.0.2.0
• Isva configured with Ansible – basically with same playbooks with only differences in inventory variables for which no real difference for the concerned parts
• Webseal configuration for affected part
• virtual host junction  
• managed-cookies-list: JSESSIONID, ANOTHER_JSESSIONID
• Identical configuration in both environment
• Environments running on cloud Kubernetes - Differences between test cluster and pre-prod:
• Pre-prod having 2 pods for Webseal and Isvaruntime
• Test cluster having self-signed certs,  SSL terminated on Webseal
• Pre-prod having Cloud providers Ingress with managed certificate. SSL terminated on providers HTTPS Loadbalancer

Investigations so far

• We have ruled out the Liberty backend being the problem by
• tracing the requests and responses in Webseal. JSESSIONID- cookie from Liberty is coming on response and is not sent back to Liberty on following requests
• Responses from Liberty backend are identical in both environments
• Liberty backend also having 2 pods, but session affinity works there, confirmed from logs (would not even matter because Liberty's have DB session persistence working without problems)
• The same backends are also used from old ISAM and with it, no problems, cookie jar working fine there
• PD sessions are staying the same
• Scaled down Webseal deployment to single instance, just in case ( Though we have not seen any issue with Webseal session affinity even with 2 pods)
• use-legacy-cookiejar-behaviour= true  (https://www.ibm.com/support/pages/apar/IV76653 )  - Does not fix the issue, but causes following error
• An error occurred fulfulling the current request to /sps/wssoi.
  This error was caused by an internal/unexpected error on the invoked protocol module leading to the exception displayed below.
  Please validate configuration of the executing protocol and environment.
  This is not a problem with the SPS.
• [preserve-cookie-names]  name= JSESSIONID  (or JSESSIONID*)  -  no effect, same error as above
• Bot use-legacy-cookiejar-behaviour = true and [preserve-cookie-names]  -  no effect, same error as above
• In our logs we can see following errors.  Could these be relevant to our problem?
• 0x38B9A41B webseald ERROR wns session WSRemoteCache.cpp 2231 0x7fb5f0171700 -- DPWNS1051E Addition or update of a session cache entry failed.
• 0x38B9A427 webseald ERROR wns session WSRemoteCache.cpp 3214 0x7fb5f0171700 -- DPWNS1063E An attempt to delete a session failed with error code 0x38a0a13e.

Any ideas where to look further?   Is there some specific debug flags we could turn on to find what is the issue

Best regards, 

Jan Lindstam

Hello!

We have a problem with cookie jar. In our pre-prod environment, Webseal does not sent JSESSIONID back to the Liberty backend, causing the Liberty to break its session

 Problem is that we have this working in test environment but failing in pre-prod-environment. So.. likely reason is some sort of misconfiguration or difference between environments, but we cannot locate the difference

 Here is our setup

•  ibmcom/verify-access:10.0.2.0
• Isva configured with Ansible – basically with same playbooks with only differences in inventory variables for which no real difference for the concerned parts
• Webseal configuration for affected part
• virtual host junction  
• managed-cookies-list: JSESSIONID, ANOTHER_JSESSIONID
• Identical configuration in both environment
• Environments running on cloud Kubernetes - Differences between test cluster and pre-prod:
• Pre-prod having 2 pods for Webseal and Isvaruntime
• Test cluster having self-signed certs,  SSL terminated on Webseal
• Pre-prod having Cloud providers Ingress with managed certificate. SSL terminated on providers HTTPS Loadbalancer

Investigations so far

• We have ruled out the Liberty backend being the problem by
• tracing the requests and responses in Webseal. JSESSIONID- cookie from Liberty is coming on response and is not sent back to Liberty on following requests
• Responses from Liberty backend are identical in both environments
• Liberty backend also having 2 pods, but session affinity works there, confirmed from logs (would not even matter because Liberty's have DB session persistence working without problems)
• The same backends are also used from old ISAM and with it, no problems, cookie jar working fine there
• PD sessions are staying the same
• Scaled down Webseal deployment to single instance, just in case ( Though we have not seen any issue with Webseal session affinity even with 2 pods)
• use-legacy-cookiejar-behaviour= true  (https://www.ibm.com/support/pages/apar/IV76653 )  - Does not fix the issue, but causes following error
• An error occurred fulfulling the current request to /sps/wssoi.
  This error was caused by an internal/unexpected error on the invoked protocol module leading to the exception displayed below.
  Please validate configuration of the executing protocol and environment.
  This is not a problem with the SPS.
• [preserve-cookie-names]  name= JSESSIONID  (or JSESSIONID*)  -  no effect, same error as above
• Bot use-legacy-cookiejar-behaviour = true and [preserve-cookie-names]  -  no effect, same error as above
• In our logs we can see following errors.  Could these be relevant to our problem?
• 0x38B9A41B webseald ERROR wns session WSRemoteCache.cpp 2231 0x7fb5f0171700 -- DPWNS1051E Addition or update of a session cache entry failed.
• 0x38B9A427 webseald ERROR wns session WSRemoteCache.cpp 3214 0x7fb5f0171700 -- DPWNS1063E An attempt to delete a session failed with error code 0x38a0a13e.

Any ideas where to look further?   Is there some specific debug flags we could turn on to find what is the issue

Best regards, 

Jan Lindstam

• from your analysis of the JSESSIONID being sent from Liberty to WebSEAL.  Are you seeing the JSESSIONID cookies being returned to the browser, or do they remain at WebSEAL. 
  
  • For 'authenticated' sessions (PD-S-Session-ID = 1_xxx) in theory the JSESSIONID never gets sent to the browser (cookie-jar kicking in)
  • For 'unauthenticated' sessions the 'create-unauth-sessions' must be set to true for the cookie-jar to store that JSESSIONID.  If this is the case you'll likely want to look at setting 'clear-cookie-jar-on-reauth = false'
    
    • 1./ PD-S-Session-ID = 0_xxx
    • 2./ timed out session 
    • 3./ sessions logged out
• How many WebSEAL (and potentially AAC/Fed runtime) pods are active in your pre-prod environment? And how active is the pre-prod environment used?
  • Reason I asked is in the situation that I was debugging, I ran into the worker-threads limits of the DSC. 
    • By default this is set to 60 worker threads
    • I had an environment with
      • some 30+ Reverse Proxies all connected to the DSC (using up at least 1 worker-thread for polling/updating session info)
      • 4 AAC/Fed runtimes (that were also configured to replicate session state into the DSC)  (so using up at least worker-thread each)
      • some custom logic (in back-end apps) reaching out to the LMI > dscadmin interface (so using up at least worker-thread each)
      • several 'concurrent user logins' per second
  • This actually meant that at certain times the 60 worker-threads were all used up, and things got into a funny state.
    • DPWNS1063E and DPWNS1051E were also observed
• In terms of debug statements, I would look at 'pdweb.wns' or 'pdweb.wns.session' trace points in WebSEAL as well. (that's where I learned what was happening with the cookie-jar behavior when trying to store this info in the DSC)

Hello!

We have a problem with cookie jar. In our pre-prod environment, Webseal does not sent JSESSIONID back to the Liberty backend, causing the Liberty to break its session

 Problem is that we have this working in test environment but failing in pre-prod-environment. So.. likely reason is some sort of misconfiguration or difference between environments, but we cannot locate the difference

 Here is our setup

•  ibmcom/verify-access:10.0.2.0
• Isva configured with Ansible – basically with same playbooks with only differences in inventory variables for which no real difference for the concerned parts
• Webseal configuration for affected part
• virtual host junction  
• managed-cookies-list: JSESSIONID, ANOTHER_JSESSIONID
• Identical configuration in both environment
• Environments running on cloud Kubernetes - Differences between test cluster and pre-prod:
• Pre-prod having 2 pods for Webseal and Isvaruntime
• Test cluster having self-signed certs,  SSL terminated on Webseal
• Pre-prod having Cloud providers Ingress with managed certificate. SSL terminated on providers HTTPS Loadbalancer

Investigations so far

• We have ruled out the Liberty backend being the problem by
• tracing the requests and responses in Webseal. JSESSIONID- cookie from Liberty is coming on response and is not sent back to Liberty on following requests
• Responses from Liberty backend are identical in both environments
• Liberty backend also having 2 pods, but session affinity works there, confirmed from logs (would not even matter because Liberty's have DB session persistence working without problems)
• The same backends are also used from old ISAM and with it, no problems, cookie jar working fine there
• PD sessions are staying the same
• Scaled down Webseal deployment to single instance, just in case ( Though we have not seen any issue with Webseal session affinity even with 2 pods)
• use-legacy-cookiejar-behaviour= true  (https://www.ibm.com/support/pages/apar/IV76653 )  - Does not fix the issue, but causes following error
• An error occurred fulfulling the current request to /sps/wssoi.
  This error was caused by an internal/unexpected error on the invoked protocol module leading to the exception displayed below.
  Please validate configuration of the executing protocol and environment.
  This is not a problem with the SPS.
• [preserve-cookie-names]  name= JSESSIONID  (or JSESSIONID*)  -  no effect, same error as above
• Bot use-legacy-cookiejar-behaviour = true and [preserve-cookie-names]  -  no effect, same error as above
• In our logs we can see following errors.  Could these be relevant to our problem?
• 0x38B9A41B webseald ERROR wns session WSRemoteCache.cpp 2231 0x7fb5f0171700 -- DPWNS1051E Addition or update of a session cache entry failed.
• 0x38B9A427 webseald ERROR wns session WSRemoteCache.cpp 3214 0x7fb5f0171700 -- DPWNS1063E An attempt to delete a session failed with error code 0x38a0a13e.

Any ideas where to look further?   Is there some specific debug flags we could turn on to find what is the issue

Best regards, 

Jan Lindstam

• We had DSC in use.  Tried removing it - did not change the outcome
• We run the traces on the working environment (dev) and in non-working environment (pre).  To our eyes, there were just following differences
  • dev was able to find the cookie from cookie-jar
    • WSCookieJar::prepareRequest Putting cookie [name: JSESSIONID, path: /XXXX, domain: ] into request.
      
  • Pre did enter and return from the WsCookieJar::prepareRequest and processClientRequest methods but seemed to be doing nothing
  • Pre uses PD-S-SESSION - cookie instead of PD-H-SESSION
  • In front of Pre-environment there are multiple frontend- proxies before the Cloud providers HTTPS proxy (handling CDN among other stuff).  We are seeing that requests to Webseal are coming from varying front-end proxy ip-addresses - even for this one users requests flow - the request for which the response should store the cookie is coming via different proxy than the next request which should utilize the stored cookie 
    • There is x-forwarded-for header which does  have consistent ip- header
• In pre-prod - just to rule out things - we switched  from external HTTPS lb to internal one,  the problem vanished. SSL is terminated in Webseal and there is just one client ip in the traces ( No CDN or front-end proxies)

• from your analysis of the JSESSIONID being sent from Liberty to WebSEAL.  Are you seeing the JSESSIONID cookies being returned to the browser, or do they remain at WebSEAL. 
  
  • For 'authenticated' sessions (PD-S-Session-ID = 1_xxx) in theory the JSESSIONID never gets sent to the browser (cookie-jar kicking in)
  • For 'unauthenticated' sessions the 'create-unauth-sessions' must be set to true for the cookie-jar to store that JSESSIONID.  If this is the case you'll likely want to look at setting 'clear-cookie-jar-on-reauth = false'
    
    • 1./ PD-S-Session-ID = 0_xxx
    • 2./ timed out session 
    • 3./ sessions logged out
• How many WebSEAL (and potentially AAC/Fed runtime) pods are active in your pre-prod environment? And how active is the pre-prod environment used?
  • Reason I asked is in the situation that I was debugging, I ran into the worker-threads limits of the DSC. 
    • By default this is set to 60 worker threads
    • I had an environment with
      • some 30+ Reverse Proxies all connected to the DSC (using up at least 1 worker-thread for polling/updating session info)
      • 4 AAC/Fed runtimes (that were also configured to replicate session state into the DSC)  (so using up at least worker-thread each)
      • some custom logic (in back-end apps) reaching out to the LMI > dscadmin interface (so using up at least worker-thread each)
      • several 'concurrent user logins' per second
  • This actually meant that at certain times the 60 worker-threads were all used up, and things got into a funny state.
    • DPWNS1063E and DPWNS1051E were also observed
• In terms of debug statements, I would look at 'pdweb.wns' or 'pdweb.wns.session' trace points in WebSEAL as well. (that's where I learned what was happening with the cookie-jar behavior when trying to store this info in the DSC)

Hello!

We have a problem with cookie jar. In our pre-prod environment, Webseal does not sent JSESSIONID back to the Liberty backend, causing the Liberty to break its session

 Problem is that we have this working in test environment but failing in pre-prod-environment. So.. likely reason is some sort of misconfiguration or difference between environments, but we cannot locate the difference

 Here is our setup

•  ibmcom/verify-access:10.0.2.0
• Isva configured with Ansible – basically with same playbooks with only differences in inventory variables for which no real difference for the concerned parts
• Webseal configuration for affected part
• virtual host junction  
• managed-cookies-list: JSESSIONID, ANOTHER_JSESSIONID
• Identical configuration in both environment
• Environments running on cloud Kubernetes - Differences between test cluster and pre-prod:
• Pre-prod having 2 pods for Webseal and Isvaruntime
• Test cluster having self-signed certs,  SSL terminated on Webseal
• Pre-prod having Cloud providers Ingress with managed certificate. SSL terminated on providers HTTPS Loadbalancer

Investigations so far

• We have ruled out the Liberty backend being the problem by
• tracing the requests and responses in Webseal. JSESSIONID- cookie from Liberty is coming on response and is not sent back to Liberty on following requests
• Responses from Liberty backend are identical in both environments
• Liberty backend also having 2 pods, but session affinity works there, confirmed from logs (would not even matter because Liberty's have DB session persistence working without problems)
• The same backends are also used from old ISAM and with it, no problems, cookie jar working fine there
• PD sessions are staying the same
• Scaled down Webseal deployment to single instance, just in case ( Though we have not seen any issue with Webseal session affinity even with 2 pods)
• use-legacy-cookiejar-behaviour= true  (https://www.ibm.com/support/pages/apar/IV76653 )  - Does not fix the issue, but causes following error
• An error occurred fulfulling the current request to /sps/wssoi.
  This error was caused by an internal/unexpected error on the invoked protocol module leading to the exception displayed below.
  Please validate configuration of the executing protocol and environment.
  This is not a problem with the SPS.
• [preserve-cookie-names]  name= JSESSIONID  (or JSESSIONID*)  -  no effect, same error as above
• Bot use-legacy-cookiejar-behaviour = true and [preserve-cookie-names]  -  no effect, same error as above
• In our logs we can see following errors.  Could these be relevant to our problem?
• 0x38B9A41B webseald ERROR wns session WSRemoteCache.cpp 2231 0x7fb5f0171700 -- DPWNS1051E Addition or update of a session cache entry failed.
• 0x38B9A427 webseald ERROR wns session WSRemoteCache.cpp 3214 0x7fb5f0171700 -- DPWNS1063E An attempt to delete a session failed with error code 0x38a0a13e.

Any ideas where to look further?   Is there some specific debug flags we could turn on to find what is the issue

Best regards, 

Jan Lindstam

• We had DSC in use.  Tried removing it - did not change the outcome
• We run the traces on the working environment (dev) and in non-working environment (pre).  To our eyes, there were just following differences
  • dev was able to find the cookie from cookie-jar
    • WSCookieJar::prepareRequest Putting cookie [name: JSESSIONID, path: /XXXX, domain: ] into request.
      
  • Pre did enter and return from the WsCookieJar::prepareRequest and processClientRequest methods but seemed to be doing nothing
  • Pre uses PD-S-SESSION - cookie instead of PD-H-SESSION
  • In front of Pre-environment there are multiple frontend- proxies before the Cloud providers HTTPS proxy (handling CDN among other stuff).  We are seeing that requests to Webseal are coming from varying front-end proxy ip-addresses - even for this one users requests flow - the request for which the response should store the cookie is coming via different proxy than the next request which should utilize the stored cookie 
    • There is x-forwarded-for header which does  have consistent ip- header
• In pre-prod - just to rule out things - we switched  from external HTTPS lb to internal one,  the problem vanished. SSL is terminated in Webseal and there is just one client ip in the traces ( No CDN or front-end proxies)

• from your analysis of the JSESSIONID being sent from Liberty to WebSEAL.  Are you seeing the JSESSIONID cookies being returned to the browser, or do they remain at WebSEAL. 
  
  • For 'authenticated' sessions (PD-S-Session-ID = 1_xxx) in theory the JSESSIONID never gets sent to the browser (cookie-jar kicking in)
  • For 'unauthenticated' sessions the 'create-unauth-sessions' must be set to true for the cookie-jar to store that JSESSIONID.  If this is the case you'll likely want to look at setting 'clear-cookie-jar-on-reauth = false'
    
    • 1./ PD-S-Session-ID = 0_xxx
    • 2./ timed out session 
    • 3./ sessions logged out
• How many WebSEAL (and potentially AAC/Fed runtime) pods are active in your pre-prod environment? And how active is the pre-prod environment used?
  • Reason I asked is in the situation that I was debugging, I ran into the worker-threads limits of the DSC. 
    • By default this is set to 60 worker threads
    • I had an environment with
      • some 30+ Reverse Proxies all connected to the DSC (using up at least 1 worker-thread for polling/updating session info)
      • 4 AAC/Fed runtimes (that were also configured to replicate session state into the DSC)  (so using up at least worker-thread each)
      • some custom logic (in back-end apps) reaching out to the LMI > dscadmin interface (so using up at least worker-thread each)
      • several 'concurrent user logins' per second
  • This actually meant that at certain times the 60 worker-threads were all used up, and things got into a funny state.
    • DPWNS1063E and DPWNS1051E were also observed
• In terms of debug statements, I would look at 'pdweb.wns' or 'pdweb.wns.session' trace points in WebSEAL as well. (that's where I learned what was happening with the cookie-jar behavior when trying to store this info in the DSC)

Hello!

We have a problem with cookie jar. In our pre-prod environment, Webseal does not sent JSESSIONID back to the Liberty backend, causing the Liberty to break its session

 Problem is that we have this working in test environment but failing in pre-prod-environment. So.. likely reason is some sort of misconfiguration or difference between environments, but we cannot locate the difference

 Here is our setup

•  ibmcom/verify-access:10.0.2.0
• Isva configured with Ansible – basically with same playbooks with only differences in inventory variables for which no real difference for the concerned parts
• Webseal configuration for affected part
• virtual host junction  
• managed-cookies-list: JSESSIONID, ANOTHER_JSESSIONID
• Identical configuration in both environment
• Environments running on cloud Kubernetes - Differences between test cluster and pre-prod:
• Pre-prod having 2 pods for Webseal and Isvaruntime
• Test cluster having self-signed certs,  SSL terminated on Webseal
• Pre-prod having Cloud providers Ingress with managed certificate. SSL terminated on providers HTTPS Loadbalancer

Investigations so far

• We have ruled out the Liberty backend being the problem by
• tracing the requests and responses in Webseal. JSESSIONID- cookie from Liberty is coming on response and is not sent back to Liberty on following requests
• Responses from Liberty backend are identical in both environments
• Liberty backend also having 2 pods, but session affinity works there, confirmed from logs (would not even matter because Liberty's have DB session persistence working without problems)
• The same backends are also used from old ISAM and with it, no problems, cookie jar working fine there
• PD sessions are staying the same
• Scaled down Webseal deployment to single instance, just in case ( Though we have not seen any issue with Webseal session affinity even with 2 pods)
• use-legacy-cookiejar-behaviour= true  (https://www.ibm.com/support/pages/apar/IV76653 )  - Does not fix the issue, but causes following error
• An error occurred fulfulling the current request to /sps/wssoi.
  This error was caused by an internal/unexpected error on the invoked protocol module leading to the exception displayed below.
  Please validate configuration of the executing protocol and environment.
  This is not a problem with the SPS.
• [preserve-cookie-names]  name= JSESSIONID  (or JSESSIONID*)  -  no effect, same error as above
• Bot use-legacy-cookiejar-behaviour = true and [preserve-cookie-names]  -  no effect, same error as above
• In our logs we can see following errors.  Could these be relevant to our problem?
• 0x38B9A41B webseald ERROR wns session WSRemoteCache.cpp 2231 0x7fb5f0171700 -- DPWNS1051E Addition or update of a session cache entry failed.
• 0x38B9A427 webseald ERROR wns session WSRemoteCache.cpp 3214 0x7fb5f0171700 -- DPWNS1063E An attempt to delete a session failed with error code 0x38a0a13e.

Any ideas where to look further?   Is there some specific debug flags we could turn on to find what is the issue

Best regards, 

Jan Lindstam