WebSeal closing TCP connection

5. RE: WebSeal closing TCP connection

Like

Jonatan Wålegård

Posted 11/07/22 04:18 AM

Just a small update.

Snoop shows nothing, and webseal log shows no error.

It is production environment so I can't really change any settings yet.

------------------------------
Jonatan Wålegård
------------------------------

Original Message

6. RE: WebSeal closing TCP connection

Like

HANS VANDEWEGHE

Posted 11/08/22 03:31 AM

Sounds like a tricky one to troubleshoot, indeed.
I take it the pdweb.snoop shows the first 2 incoming request/outgoing responses?
And the 3rd request (POST) isn't showing up?

Not the actual POST request, but also nothing being logged in terms:

thread(xyz)
remote ip_address:port

For example, if this was your 2nd GET Request (Receiving xyz bytes in thread(14) from 192.168.42.1:64179)

597       2022-11-07-15:40:22.732-06:00I----- thread(14) trace.pdweb.snoop.client:1 /build/isam/src/i4w/pdwebrte/webcore/amw_snoop.cpp:164: 
598       ----------------------------------------
599       Thread 140593929713408; fd 256; local 192.168.42.202:443; remote 192.168.42.1:64179
600       Receiving 193 bytes
601       0x0000   4745 5420 2f74 6573 7434 2048 5454 502f        GET./test4.HTTP/

Once that's being processed (Sending xyz bytes in thread(14) to 192.168.42.1:64179):

2022-11-07-15:40:22.732-06:00I----- thread(14) trace.pdweb.snoop.client:1 /build/isam/src/i4w/pdwebrte/webcore/amw_snoop.cpp:190: 
618       ----------------------------------------
619       Thread 140593929713408; fd 256; local 192.168.42.202:443; remote 192.168.42.1:64179
620       Sending 1823 bytes
621       0x0000   4854 5450 2f31 2e31 2034 3034 204e 6f74        HTTP/1.1.404.Not

You're not seeing any log lines with that thread(14) or remote 192.168.42.1:64179 string?

Are you using TLS between BigIP and WebSEAL?
What version of TLS is being used there?

I did see there is 1 APAR related to persistent connections, getting fixed in the upcoming version of ISVA 10.0.5. Unfortunately not a lot of detail available on the issue addressed by the APAR (it does appear to indicate a connection close header being sent - which probably isn't matching your findings).
https://www.ibm.com/support/pages/apar/IJ41346
If by any chance you're also able to reproduce this in a lower env, it might be worth contacting IBM Support to see if they have a fixpack for IJ41346 to see if it makes any difference.

------------------------------
HANS VANDEWEGHE
------------------------------

Original Message

7. RE: WebSeal closing TCP connection

Like

Jonatan Wålegård

Posted 11/23/22 09:50 AM

@HANS VANDEWEGHE Just wanted to let you know that strict-ssl-sessions = false actually did the trick even though we weren't using pkmslogout. It is up to support to update the article if they haven't already done it. Thanks

------------------------------
Jonatan Wålegård
------------------------------

Original Message

Original Message:
Sent: Tue November 08, 2022 03:31 AM
From: HANS VANDEWEGHE
Subject: WebSeal closing TCP connection

Sounds like a tricky one to troubleshoot, indeed.
I take it the pdweb.snoop shows the first 2 incoming request/outgoing responses?
And the 3rd request (POST) isn't showing up?

Not the actual POST request, but also nothing being logged in terms:

thread(xyz)
remote ip_address:port

For example, if this was your 2nd GET Request (Receiving xyz bytes in thread(14) from 192.168.42.1:64179)

597       2022-11-07-15:40:22.732-06:00I----- thread(14) trace.pdweb.snoop.client:1 /build/isam/src/i4w/pdwebrte/webcore/amw_snoop.cpp:164: 598       ----------------------------------------599       Thread 140593929713408; fd 256; local 192.168.42.202:443; remote 192.168.42.1:64179600       Receiving 193 bytes601       0x0000   4745 5420 2f74 6573 7434 2048 5454 502f        GET./test4.HTTP/

Once that's being processed (Sending xyz bytes in thread(14) to 192.168.42.1:64179):

2022-11-07-15:40:22.732-06:00I----- thread(14) trace.pdweb.snoop.client:1 /build/isam/src/i4w/pdwebrte/webcore/amw_snoop.cpp:190: 618       ----------------------------------------619       Thread 140593929713408; fd 256; local 192.168.42.202:443; remote 192.168.42.1:64179620       Sending 1823 bytes621       0x0000   4854 5450 2f31 2e31 2034 3034 204e 6f74        HTTP/1.1.404.Not

You're not seeing any log lines with that thread(14) or remote 192.168.42.1:64179 string?

Are you using TLS between BigIP and WebSEAL?
What version of TLS is being used there?

I did see there is 1 APAR related to persistent connections, getting fixed in the upcoming version of ISVA 10.0.5. Unfortunately not a lot of detail available on the issue addressed by the APAR (it does appear to indicate a connection close header being sent - which probably isn't matching your findings).
https://www.ibm.com/support/pages/apar/IJ41346
If by any chance you're also able to reproduce this in a lower env, it might be worth contacting IBM Support to see if they have a fixpack for IJ41346 to see if it makes any difference.

------------------------------
HANS VANDEWEGHE
------------------------------

Original Message:
Sent: Mon November 07, 2022 04:17 AM
From: Jonatan Wålegård
Subject: WebSeal closing TCP connection

Just a small update.

Snoop shows nothing, and webseal log shows no error.

It is production environment so I can't really change any settings yet.

------------------------------
Jonatan Wålegård

Original Message:
Sent: Thu November 03, 2022 06:53 AM
From: HANS VANDEWEGHE
Subject: WebSeal closing TCP connection

Indeed, the request log and pdweb.debug tracing are handled/written out at a later stage in the handling of the request (in the worker-thread cycle).
The pdweb.snoop trace might actually still show some entries related to the TCP ports being no longer available. (the pdweb.snoop has it's trace points fairly early on in the request handling stage of the worker-thread)

It won't show the payload/request (and indeed the encrypted VIP traffic also won't show it), but that's not necessarily of interest here anyway. I don't think your issue is a content issue, it indeed sounds more related to connection reuse being cut off early.

Are you seeing any GSKIT related errors in the msg__webseald log at all?

Perhaps a silly question, but did this issue arise recently (like after an upgrade)?
Or is it just a complete new implementation, or behavior not really noticed before?

BTW, This Support Webinar also discussed in some detail how to use pdweb.snoop tracing for analysis of remote connectivity problems. (although more focussed on the back-end junction side of things, but the idea is the same for front-end connections)

------------------------------
HANS VANDEWEGHE

Original Message:
Sent: Thu November 03, 2022 06:26 AM
From: Jonatan Wålegård
Subject: WebSeal closing TCP connection

@HANS VANDEWEGHE That is a great start thanks! I will start trying those things you mentioned.

Appreciate it!

As for the trace logs, we already took network trace from Big IP VIP and there was the proof of ISVA closing the connection.
I also took net trace on ISVA but the data from and to the VIP seems to be encrypted.. Don't know how to turn it off..

I checked the request and pdweb.debug log already, and the request wasn't showing there, basically non-existent, so I'm assuming it's being cut off at an earlier stage.

------------------------------
Jonatan Wålegård

Original Message:
Sent: Thu November 03, 2022 03:53 AM
From: HANS VANDEWEGHE
Subject: WebSeal closing TCP connection

Hello Jonatan,

I remember a while ago when ISAM 9.0.7 was released, there was a (new) default setting that introduced somewhat similar behavior when a (persistent connectionà request followed a prior logout request (pmkslogout, but also EAI type of logouts) (using the same persistent connection port).

I don't have a lot of experience with the networking aspect/handling of persistent connections on more recent versions of the ISVA product, so not aware of potential other gotcha's related to connection handling. (the 9.0.7 case is described here , might be worth trying strict-ssl-sessions = false regardles whether or not your prior requests are logout requests).

Are there configurable options on the front VIP (provided this is on a non-prod environment), to see if you can find what makes WebSEAL trip over the connection (re-use)?

disable persistent connections (disable the keep-alive)
using a different SSL/TLS version (e.g TLS 1.2 vs TLS 1.3)

I understand it doesn't really give you an answer to your question, but might help to narrow down where things fail.

Usually when it's a combination of persistent connections/TLS problems, tracing the right components can also help to narrow down behavior to workable findings that can be further investigated.

From your analysis of the 2 GET requests, followed by a (failing POST) request, I think you've already done the folliwing.
But in any case, my go-to approach here would be a combination of:

TCP traces on the appliance : which can be filtered with tcp.flags.reset==1 to see where (unexpected) connection closures take place, and importantly will also show the matching client IP address and port (the ephemeral port from the BigIP)
using that ephemeral port, you can correlate this to requests in a pdweb.snoop trace (which shows port/ip addresses) : can sometimes be useful in rather busy environments, if you need to match certain requests (from a request.log resulting in GSK_ * type errors in the msg__webseald. log)
Use indicators of errors (GSK_* type errors) in msg__webseald.log to search a GSKIT trace
GSKIT traces (steps) produce a binary output file, typically investigated by an L3 Support team (ISVA or GSKIT L3 Support) via appropriate tooling. However you can run a linux strings command against the output files, to print out anything readable.

I appreciate this does not solve your problem, but it might help to narrow down things further.

Best regards,
Hans

------------------------------
HANS VANDEWEGHE

8. RE: WebSeal closing TCP connection

Like

HANS VANDEWEGHE

Posted 11/23/22 10:05 AM

Thanks for the follow-up here Jonatan. Glad to hear your issue got resolved.
It indeed looks like a setting that we need to keep in mind for TLS sessions getting unexpected interrupts.

------------------------------
HANS VANDEWEGHE
------------------------------

Original Message

Original Message:
Sent: Wed November 23, 2022 09:49 AM
From: Jonatan Wålegård
Subject: WebSeal closing TCP connection

@HANS VANDEWEGHE Just wanted to let you know that strict-ssl-sessions = false actually did the trick even though we weren't using pkmslogout. It is up to support to update the article if they haven't already done it. Thanks

------------------------------
Jonatan Wålegård
------------------------------

Original Message:
Sent: Tue November 08, 2022 03:31 AM
From: HANS VANDEWEGHE
Subject: WebSeal closing TCP connection

Sounds like a tricky one to troubleshoot, indeed.
I take it the pdweb.snoop shows the first 2 incoming request/outgoing responses?
And the 3rd request (POST) isn't showing up?

Not the actual POST request, but also nothing being logged in terms:

thread(xyz)
remote ip_address:port

For example, if this was your 2nd GET Request (Receiving xyz bytes in thread(14) from 192.168.42.1:64179)

597       2022-11-07-15:40:22.732-06:00I----- thread(14) trace.pdweb.snoop.client:1 /build/isam/src/i4w/pdwebrte/webcore/amw_snoop.cpp:164: 598       ----------------------------------------599       Thread 140593929713408; fd 256; local 192.168.42.202:443; remote 192.168.42.1:64179600       Receiving 193 bytes601       0x0000   4745 5420 2f74 6573 7434 2048 5454 502f        GET./test4.HTTP/

Once that's being processed (Sending xyz bytes in thread(14) to 192.168.42.1:64179):

2022-11-07-15:40:22.732-06:00I----- thread(14) trace.pdweb.snoop.client:1 /build/isam/src/i4w/pdwebrte/webcore/amw_snoop.cpp:190: 618       ----------------------------------------619       Thread 140593929713408; fd 256; local 192.168.42.202:443; remote 192.168.42.1:64179620       Sending 1823 bytes621       0x0000   4854 5450 2f31 2e31 2034 3034 204e 6f74        HTTP/1.1.404.Not

You're not seeing any log lines with that thread(14) or remote 192.168.42.1:64179 string?

Are you using TLS between BigIP and WebSEAL?
What version of TLS is being used there?

I did see there is 1 APAR related to persistent connections, getting fixed in the upcoming version of ISVA 10.0.5. Unfortunately not a lot of detail available on the issue addressed by the APAR (it does appear to indicate a connection close header being sent - which probably isn't matching your findings).
https://www.ibm.com/support/pages/apar/IJ41346
If by any chance you're also able to reproduce this in a lower env, it might be worth contacting IBM Support to see if they have a fixpack for IJ41346 to see if it makes any difference.

------------------------------
HANS VANDEWEGHE

Original Message:
Sent: Mon November 07, 2022 04:17 AM
From: Jonatan Wålegård
Subject: WebSeal closing TCP connection

Just a small update.

Snoop shows nothing, and webseal log shows no error.

It is production environment so I can't really change any settings yet.

------------------------------
Jonatan Wålegård

Original Message:
Sent: Thu November 03, 2022 06:53 AM
From: HANS VANDEWEGHE
Subject: WebSeal closing TCP connection

Indeed, the request log and pdweb.debug tracing are handled/written out at a later stage in the handling of the request (in the worker-thread cycle).
The pdweb.snoop trace might actually still show some entries related to the TCP ports being no longer available. (the pdweb.snoop has it's trace points fairly early on in the request handling stage of the worker-thread)

It won't show the payload/request (and indeed the encrypted VIP traffic also won't show it), but that's not necessarily of interest here anyway. I don't think your issue is a content issue, it indeed sounds more related to connection reuse being cut off early.

Are you seeing any GSKIT related errors in the msg__webseald log at all?

Perhaps a silly question, but did this issue arise recently (like after an upgrade)?
Or is it just a complete new implementation, or behavior not really noticed before?

BTW, This Support Webinar also discussed in some detail how to use pdweb.snoop tracing for analysis of remote connectivity problems. (although more focussed on the back-end junction side of things, but the idea is the same for front-end connections)

------------------------------
HANS VANDEWEGHE

Original Message:
Sent: Thu November 03, 2022 06:26 AM
From: Jonatan Wålegård
Subject: WebSeal closing TCP connection

@HANS VANDEWEGHE That is a great start thanks! I will start trying those things you mentioned.

Appreciate it!

As for the trace logs, we already took network trace from Big IP VIP and there was the proof of ISVA closing the connection.
I also took net trace on ISVA but the data from and to the VIP seems to be encrypted.. Don't know how to turn it off..

I checked the request and pdweb.debug log already, and the request wasn't showing there, basically non-existent, so I'm assuming it's being cut off at an earlier stage.

------------------------------
Jonatan Wålegård

Original Message:
Sent: Thu November 03, 2022 03:53 AM
From: HANS VANDEWEGHE
Subject: WebSeal closing TCP connection

Hello Jonatan,

I remember a while ago when ISAM 9.0.7 was released, there was a (new) default setting that introduced somewhat similar behavior when a (persistent connectionà request followed a prior logout request (pmkslogout, but also EAI type of logouts) (using the same persistent connection port).

I don't have a lot of experience with the networking aspect/handling of persistent connections on more recent versions of the ISVA product, so not aware of potential other gotcha's related to connection handling. (the 9.0.7 case is described here , might be worth trying strict-ssl-sessions = false regardles whether or not your prior requests are logout requests).

Are there configurable options on the front VIP (provided this is on a non-prod environment), to see if you can find what makes WebSEAL trip over the connection (re-use)?

disable persistent connections (disable the keep-alive)
using a different SSL/TLS version (e.g TLS 1.2 vs TLS 1.3)

I understand it doesn't really give you an answer to your question, but might help to narrow down where things fail.

Usually when it's a combination of persistent connections/TLS problems, tracing the right components can also help to narrow down behavior to workable findings that can be further investigated.

From your analysis of the 2 GET requests, followed by a (failing POST) request, I think you've already done the folliwing.
But in any case, my go-to approach here would be a combination of:

TCP traces on the appliance : which can be filtered with tcp.flags.reset==1 to see where (unexpected) connection closures take place, and importantly will also show the matching client IP address and port (the ephemeral port from the BigIP)
using that ephemeral port, you can correlate this to requests in a pdweb.snoop trace (which shows port/ip addresses) : can sometimes be useful in rather busy environments, if you need to match certain requests (from a request.log resulting in GSK_ * type errors in the msg__webseald. log)
Use indicators of errors (GSK_* type errors) in msg__webseald.log to search a GSKIT trace
GSKIT traces (steps) produce a binary output file, typically investigated by an L3 Support team (ISVA or GSKIT L3 Support) via appropriate tooling. However you can run a linux strings command against the output files, to print out anything readable.

I appreciate this does not solve your problem, but it might help to narrow down things further.

Best regards,
Hans

------------------------------
HANS VANDEWEGHE

IBM Verify

IBM Verify