IBM Security Verify

  • 1.  Problems with Cookie jar

    IBM Select
    Posted Fri October 22, 2021 05:12 AM

    Hello!

    We have a problem with cookie jar. In our pre-prod environment, Webseal does not sent JSESSIONID back to the Liberty backend, causing the Liberty to break its session

     Problem is that we have this working in test environment but failing in pre-prod-environment. So.. likely reason is some sort of misconfiguration or difference between environments, but we cannot locate the difference

     Here is our setup

    •  ibmcom/verify-access:10.0.2.0
    • Isva configured with Ansible – basically with same playbooks with only differences in inventory variables for which no real difference for the concerned parts
    • Webseal configuration for affected part
      • virtual host junction  
      • managed-cookies-list: JSESSIONID, ANOTHER_JSESSIONID
      • Identical configuration in both environment
    • Environments running on cloud Kubernetes - Differences between test cluster and pre-prod:
      • Pre-prod having 2 pods for Webseal and Isvaruntime
      • Test cluster having self-signed certs,  SSL terminated on Webseal
      • Pre-prod having Cloud providers Ingress with managed certificate. SSL terminated on providers HTTPS Loadbalancer

     

    Investigations so far

    • We have ruled out the Liberty backend being the problem by
      • tracing the requests and responses in Webseal. JSESSIONID- cookie from Liberty is coming on response and is not sent back to Liberty on following requests
      • Responses from Liberty backend are identical in both environments
      • Liberty backend also having 2 pods, but session affinity works there, confirmed from logs (would not even matter because Liberty's have DB session persistence working without problems)
      • The same backends are also used from old ISAM and with it, no problems, cookie jar working fine there
    • PD sessions are staying the same
    • Scaled down Webseal deployment to single instance, just in case ( Though we have not seen any issue with Webseal session affinity even with 2 pods)
    • use-legacy-cookiejar-behaviour= true  (https://www.ibm.com/support/pages/apar/IV76653 )  - Does not fix the issue, but causes following error
      • An error occurred fulfulling the current request to /sps/wssoi.
        This error was caused by an internal/unexpected error on the invoked protocol module leading to the exception displayed below.
        Please validate configuration of the executing protocol and environment.
        This is not a problem with the SPS.
    • [preserve-cookie-names]  name= JSESSIONID  (or JSESSIONID*)  -  no effect, same error as above
    • Bot use-legacy-cookiejar-behaviour = true and [preserve-cookie-names]  -  no effect, same error as above
    • In our logs we can see following errors.  Could these be relevant to our problem?
      • 0x38B9A41B webseald ERROR wns session WSRemoteCache.cpp 2231 0x7fb5f0171700 -- DPWNS1051E Addition or update of a session cache entry failed.
      • 0x38B9A427 webseald ERROR wns session WSRemoteCache.cpp 3214 0x7fb5f0171700 -- DPWNS1063E An attempt to delete a session failed with error code 0x38a0a13e.

     

    Any ideas where to look further?   Is there some specific debug flags we could turn on to find what is the issue

     

    Best regards, 

    Jan Lindstam



    ------------------------------
    Jan Lindstam
    ------------------------------


  • 2.  RE: Problems with Cookie jar

    Posted Fri October 22, 2021 06:07 AM
    Hello Jan,

    It looks like you've done a good level of debug here.

    Are you seeing this all the time in the pre-prod environment - or is it intermittent issue?

    It's just a guess but the error: "DPWNS1051E Addition or update of a session cache entry failed." might indicate that the addition of the cookie to the cookie jar (which is, I think, part of the session data) is failing for some reason.  I could suspect that the session is somehow being lost but you already said that the session ID doesn't change when the issue is seen.

    Are you using Distributed Session Cache (DSC) or Redis cache for session sharing?

    I'm not sure what else to suggest except that you probably need to open a support call so they can help you with the additional tracing you might need to get to the bottom of this.  I'd be interested to know the solution.

    Jon.

    ------------------------------
    Jon Harry
    Consulting IT Security Specialist
    IBM
    ------------------------------



  • 3.  RE: Problems with Cookie jar

    Posted Fri October 22, 2021 06:29 AM
    I recently looked into an issue that had some similarities to what you are describing here.
    Although you are running this on a Kubernetes env, where I was debugging on Appliances.

    Couple of questions:
    • from your analysis of the JSESSIONID being sent from Liberty to WebSEAL.  Are you seeing the JSESSIONID cookies being returned to the browser, or do they remain at WebSEAL. 
      • For 'authenticated' sessions (PD-S-Session-ID = 1_xxx) in theory the JSESSIONID never gets sent to the browser (cookie-jar kicking in)
      • For 'unauthenticated' sessions the 'create-unauth-sessions' must be set to true for the cookie-jar to store that JSESSIONID.  If this is the case you'll likely want to look at setting 'clear-cookie-jar-on-reauth = false'
        • 1./ PD-S-Session-ID = 0_xxx
        • 2./ timed out session 
        • 3./ sessions logged out
    • How many WebSEAL (and potentially AAC/Fed runtime) pods are active in your pre-prod environment? And how active is the pre-prod environment used?
      • Reason I asked is in the situation that I was debugging, I ran into the worker-threads limits of the DSC. 
        • By default this is set to 60 worker threads
        • I had an environment with
          • some 30+ Reverse Proxies all connected to the DSC (using up at least 1 worker-thread for polling/updating session info)
          • 4 AAC/Fed runtimes (that were also configured to replicate session state into the DSC)  (so using up at least worker-thread each)
          • some custom logic (in back-end apps) reaching out to the LMI > dscadmin interface (so using up at least worker-thread each)
          • several 'concurrent user logins' per second
      • This actually meant that at certain times the 60 worker-threads were all used up, and things got into a funny state.
        • DPWNS1063E and DPWNS1051E were also observed
    • In terms of debug statements, I would look at 'pdweb.wns' or 'pdweb.wns.session' trace points in WebSEAL as well. (that's where I learned what was happening with the cookie-jar behavior when trying to store this info in the DSC)

    Hope this gets you started!

    Regards,

    Hans

    ------------------------------
    HANS VANDEWEGHE
    ------------------------------



  • 4.  RE: Problems with Cookie jar

    IBM Select
    Posted Tue October 26, 2021 04:05 AM
    Hi Jon and Hans,

    thank you for responses!

    We are currently still  using our pre-prod only within the project team, so the usage is really small - basically just two persons.

    Some further investigations
    • We had DSC in use.  Tried removing it - did not change the outcome
    • We run the traces on the working environment (dev) and in non-working environment (pre).  To our eyes, there were just following differences
      • dev was able to find the cookie from cookie-jar
        • WSCookieJar::prepareRequest Putting cookie [name: JSESSIONID, path: /XXXX, domain: ] into request.
      • Pre did enter and return from the WsCookieJar::prepareRequest and processClientRequest methods but seemed to be doing nothing
      • Pre uses PD-S-SESSION - cookie instead of PD-H-SESSION
      • In front of Pre-environment there are multiple frontend- proxies before the Cloud providers HTTPS proxy (handling CDN among other stuff).  We are seeing that requests to Webseal are coming from varying front-end proxy ip-addresses - even for this one users requests flow - the request for which the response should store the cookie is coming via different proxy than the next request which should utilize the stored cookie 
        • There is x-forwarded-for header which does  have consistent ip- header
    • In pre-prod - just to rule out things - we switched  from external HTTPS lb to internal one,  the problem vanished. SSL is terminated in Webseal and there is just one client ip in the traces ( No CDN or front-end proxies)

    We are suspecting that issue would be the changing frontend-proxy addresses, so the next thing to check is if client_identifier would help https://www.ibm.com/docs/en/sva/10.0.0?topic=overview-validation-client-identifier-session

    Best regards
    Jan

    ------------------------------
    Jan Lindstam
    ------------------------------



  • 5.  RE: Problems with Cookie jar

    Posted Tue October 26, 2021 05:34 AM
    Hi Jan,

    Unless you have client-identifier already set (and are going to unset it), I don't think it will help in this situation.  This parameter restricts the session rather than helping it.  The key index for the session remains the PD-*-SESSION-ID cookie.

    If you have multiple paths into your system, it could be that some of these paths are causing the session to be lost or changing the way that WebSEAL views the session.  You mention that your load-balancer terminates HTTPS session.  In that case, do you have the following set:

    web-http-port = 443
    web-http-protocol = https

    These parameters tell WebSEAL that even though traffic is being received over HTTP, it should be treated as though it came from HTTPS (because you have a terminating load balancer at the front end).

    Also, it might help to set:

    web-host-name = www.external-homename.com

    To the hostname that end users are connecting to.  Without this, WebSEAL can struggle to figure out the hostname that external users are connecting to.

    Finally, if none of the above helps, it might be worth testing with:

    use-same-session = yes

    This will cause WebSEAL to share sessions between HTTP and HTTPS and will use the PD-H-SESSION-ID cookie in both cases.  I'm not sure this is how you want things set up but if this works it's an interesting point for the debug.

    Jon.

    ------------------------------
    Jon Harry
    Consulting IT Security Specialist
    IBM
    ------------------------------



  • 6.  RE: Problems with Cookie jar

    IBM Select
    Posted Tue October 26, 2021 09:57 AM
    Thank you Jon, you are saviour!

    These did the trick.. we had these incorrectly set to port 80 and web-http-protocol not configured

    web-http-port = 443
    web-http-protocol = https

    Other parameters - web-host-name and use-same-session we did not set as it works without.  Client_identifier in our earlier test did not have relevance, as you suspected..

    Best regards,
    Jan

    ------------------------------
    Jan Lindstam
    ------------------------------