IBM Security Verify

 View Only
  • 1.  WebSeal closing TCP connection

    Posted Thu November 03, 2022 02:06 AM

    Hi,

    I already have a ticket with the support but as they have little to no experience with ISVA, severe issues with the English language and a nonexistent ability to troubleshoot properly, I'm desperate to try my luck here.

    We have an application that is calling one of our webseals with three requests on top of each other, two GET's followed by a POST.
    The two GET's gets through just fine, but the POST call is being closed by webseal and I can't figure out why.

    If we add a 3-second delay before the POST call it works as expected (probably because a new TCP connection has been established here).

    We have a BigIP VIP in front of the webseals (this is where we can see that the connection is being closed).

    Here are our current settings:
    client-connect-timeout = 120
    persistent-con-timeout = 20
    intra-connection-timeout = 60
    connection-request-limit = 100

    Any idea why ISVA/webseal chooses to close the TCP connection?



    ------------------------------
    Jonatan Wålegård
    ------------------------------


  • 2.  RE: WebSeal closing TCP connection

    Posted Thu November 03, 2022 03:54 AM
    Hello Jonatan,

    I remember a while ago when ISAM 9.0.7 was released, there was a (new) default setting that introduced somewhat similar behavior when a (persistent connectionà request followed a prior logout request (pmkslogout, but also EAI type of logouts) (using the same persistent connection port).

    I don't have a lot of experience with the networking aspect/handling of persistent connections on more recent versions of the ISVA product, so not aware of potential other gotcha's related to connection handling. (the 9.0.7 case is described here , might be worth trying strict-ssl-sessions = false regardles whether or not your prior requests are logout requests).

    Are there configurable options on the front VIP (provided this is on a non-prod environment), to see if you can find what makes WebSEAL trip over the connection (re-use)?
    • disable persistent connections (disable the keep-alive) 
    • using a different SSL/TLS version (e.g TLS 1.2 vs TLS 1.3)
    I understand it doesn't really give you an answer to your question, but might help to narrow down where things fail.


    Usually when it's a combination of persistent connections/TLS problems, tracing the right components can also help to narrow down behavior to workable findings that can be further investigated.

    From your analysis of the 2 GET requests, followed by a (failing POST) request, I think you've already done the folliwing. 
    But in any case, my go-to approach here would be a combination of:
    • TCP traces on the appliance : which can be filtered with tcp.flags.reset==1 to see where (unexpected) connection closures take place, and importantly will also show the matching client IP address and port (the ephemeral port from the BigIP)
    • using that ephemeral port, you can correlate this to requests in a pdweb.snoop trace (which shows port/ip addresses) : can sometimes be useful in rather busy environments, if you need to match certain requests (from a request.log resulting in GSK_ * type errors in the msg__webseald. log)
    • Use indicators of errors (GSK_* type errors) in msg__webseald.log to search a GSKIT trace
    • GSKIT traces (steps) produce a binary output file, typically investigated by an L3 Support team (ISVA or GSKIT L3 Support) via appropriate tooling.  However you can run a linux strings command against the output files, to print out anything readable.

    I appreciate this does not solve your problem, but it might help to narrow down things further.

    Best regards,
    Hans

    ------------------------------
    HANS VANDEWEGHE
    ------------------------------



  • 3.  RE: WebSeal closing TCP connection

    Posted Thu November 03, 2022 06:26 AM
    @HANS VANDEWEGHE That is a great start thanks! I will start trying those things you mentioned.

    Appreciate it!

    As for the trace logs, we already took network trace from Big IP VIP and there was the proof of ISVA closing the connection.
    I also took net trace on ISVA but the data from and to the VIP seems to be encrypted.. Don't know how to turn it off..

    I checked the request and pdweb.debug log already, and the request wasn't showing there, basically non-existent, so I'm assuming it's being cut off at an earlier stage.​

    ------------------------------
    Jonatan Wålegård
    ------------------------------



  • 4.  RE: WebSeal closing TCP connection

    Posted Thu November 03, 2022 06:54 AM
    Indeed, the request log and pdweb.debug tracing are handled/written out at a later stage in the handling of the request (in the worker-thread cycle).
    The pdweb.snoop trace might actually still show some entries related to the TCP ports being no longer available.  (the pdweb.snoop has it's trace points fairly early on in the request handling stage of the worker-thread)

    It won't show the payload/request (and indeed the encrypted VIP traffic also won't show it), but that's not necessarily of interest here anyway.   I don't think your issue is a content issue, it indeed sounds more related to connection reuse being cut off early.   

    Are you seeing any GSKIT related errors in the msg__webseald  log at all?

    Perhaps a silly question, but did this issue arise recently (like after an upgrade)?   
    Or is it just a complete new implementation, or behavior not really noticed before?

    BTW, This Support Webinar also discussed in some detail how to use pdweb.snoop tracing for analysis of remote connectivity problems. (although more focussed on the back-end junction side of things, but the idea is the same for front-end connections)

    ------------------------------
    HANS VANDEWEGHE
    ------------------------------



  • 5.  RE: WebSeal closing TCP connection

    Posted Mon November 07, 2022 04:18 AM

    Just a small update.

    Snoop shows nothing, and webseal log shows no error.

    It is production environment so I can't really change any settings yet.



    ------------------------------
    Jonatan Wålegård
    ------------------------------



  • 6.  RE: WebSeal closing TCP connection

    Posted Tue November 08, 2022 03:31 AM
    Sounds like a tricky one to troubleshoot, indeed.
    I take it the pdweb.snoop shows the first 2 incoming request/outgoing responses? 
    And the 3rd request (POST) isn't showing up?

    Not the actual POST request, but also nothing being logged in terms:
    • thread(xyz)
    • remote ip_address:port 

    For example, if this was your 2nd GET Request (Receiving xyz bytes in thread(14) from 192.168.42.1:64179)
    597       2022-11-07-15:40:22.732-06:00I----- thread(14) trace.pdweb.snoop.client:1 /build/isam/src/i4w/pdwebrte/webcore/amw_snoop.cpp:164: 
    598       ----------------------------------------
    599       Thread 140593929713408; fd 256; local 192.168.42.202:443; remote 192.168.42.1:64179
    600       Receiving 193 bytes
    601       0x0000   4745 5420 2f74 6573 7434 2048 5454 502f        GET./test4.HTTP/​


    Once that's being processed (Sending xyz bytes in thread(14) to 192.168.42.1:64179):

    2022-11-07-15:40:22.732-06:00I----- thread(14) trace.pdweb.snoop.client:1 /build/isam/src/i4w/pdwebrte/webcore/amw_snoop.cpp:190: 
    618       ----------------------------------------
    619       Thread 140593929713408; fd 256; local 192.168.42.202:443; remote 192.168.42.1:64179
    620       Sending 1823 bytes
    621       0x0000   4854 5450 2f31 2e31 2034 3034 204e 6f74        HTTP/1.1.404.Not


    You're not seeing any log lines with that thread(14) or remote 192.168.42.1:64179 string?  

    Are you using TLS between BigIP and WebSEAL?  
    What version of TLS is being used there?

    I did see there is 1 APAR related to persistent connections, getting fixed in the upcoming version of ISVA 10.0.5.  Unfortunately not a lot of detail available on the issue addressed by the APAR (it does appear to indicate a connection close header being sent - which probably isn't matching your findings).
    https://www.ibm.com/support/pages/apar/IJ41346
    If by any chance you're also able to reproduce this in a lower env, it might be worth contacting IBM Support to see if they have a fixpack for IJ41346 to see if it makes any difference.




    ------------------------------
    HANS VANDEWEGHE
    ------------------------------



  • 7.  RE: WebSeal closing TCP connection

    Posted Wed November 23, 2022 09:50 AM
    @HANS VANDEWEGHE Just wanted to let you know that strict-ssl-sessions = false actually did the trick even though we weren't using pkmslogout. It is up to support to update the article if they haven't already done it. Thanks

    ------------------------------
    Jonatan Wålegård
    ------------------------------



  • 8.  RE: WebSeal closing TCP connection

    Posted Wed November 23, 2022 10:05 AM
    Thanks for the follow-up here Jonatan.  Glad to hear your issue got resolved. 
    It indeed looks like a setting that we need to keep in mind for TLS sessions getting unexpected interrupts.

    ------------------------------
    HANS VANDEWEGHE
    ------------------------------



  • 9.  RE: WebSeal closing TCP connection

    Posted Thu November 03, 2022 09:36 AM
    Hi Jonatan,

    Is the POST request sending a webseal form to webseal or a form to backend service? If it is webseal form then I would check that webseal conf doesn't have anything odd configured which would prevent handling the POST form. I remember seeing case where GET was accepted but POST looked like terminated. Sadly I can't remember the details.

    Regards,
    Antti Merihaara

    ------------------------------
    Antti Merihaara
    ------------------------------