What happened to DPWWA2025W/DPWWA2026W msg ?

View Only

Expand all | Collapse all

What happened to DPWWA2025W/DPWWA2026W msg ?

1. What happened to DPWWA2025W/DPWWA2026W msg ?

0 Like
IBM Champion

Sylvain Gilbert
Posted Wed March 20, 2019 09:57 AM

Reply
Hello

We made this observation about 1 year ago and then forget to investigate it further but this week the same behavior was made in our Production environment.

It seems that beginning with ISAM Virtual Appliance (V8 or V9, we are running 9) that the below messages logged normally in webseald server log file are no longer reported:

DPWWA2025W    IBM Security Access Manager WebSEAL has lost contact with junction server: %s (0x38cf07e9)
DPWWA2026W    IBM Security Access Manager WebSEAL has regained contact with junction (%s) server: %s (0x38cf07ea)

We have had an incident this week because we could not detect that a junction state changed from "up" to "down". This is because we cannot trace in our logs the message DPWWA2025W. The root cause of the incident of course is a human manual error but it all comes down to the delay it took us to realize the problem and fix it. The lack of the usual messages contributed to worsen the incident duration.

We did see another message more detailed "DPWAD0411E  The TCP/IP host information could not be determined from the server hostname: someservername. Ensure that the server hostname is correct and that the domain name server is functioning correctly" appearing in the webseal server logs which we could opt to track also in the future but I can image other cause than this particular one for a junction to go down.

Before I open a support case, just want to know if there was any conscious effort not to log those 2 events anymore?

Thanks in advanced for your comments/responses.

Sylvain Gilbert, p. ing., Conseiller Technique Infrastructures Securité/Security Infrastructures Technical Advisor

Intact Corporation Financière | 1935 des Cascades, Saint-Hyacinthe, QC J2S 8K9

(T) 855.646.8228, x 86667 | (M) 450.223.9537

sylvain.gilbert@intact.net | www.intactfc.com

2. RE: What happened to DPWWA2025W/DPWWA2026W msg ?

Like

Scott Exton

Posted Thu March 21, 2019 02:14 AM

Sylvain,

I can't explain what is happening in your environment, but there has definitely not been a conscious decision to remove those messages from the WebSEAL log file. You should still be notified when a junction is unavailable, and if this is no longer happening I would most likely consider this a defect.

Thanks.

Scott A. Exton
Senior Software Engineer
Chief Programmer - IBM Security Access Manager
IBM Master Inventor


Phone: 61-7-5552-4008 E-mail: scotte@au1.ibm.com	L11 & L7 Seabank Southport, QLD 4215 Australia

Original Message

3. RE: What happened to DPWWA2025W/DPWWA2026W msg ?

Like

Nick Lloyd

Posted Fri March 22, 2019 01:39 PM

Hi Sylvain,

The messages have not been removed. This is from my 9.0.6.0 lab:

2019-03-11-08:32:26.895-05:00I----- 0x38CF07E9 webseald WARNING wwa jct RemoteJunction.cpp 2148 0x7f5528045700 -- DPWWA2025W IBM Security Access Manager WebSEAL has lost contact with junction (/jct) server: 192.168.61.200
2019-03-11-08:37:26.002-05:00I----- 0x38CF07EA webseald WARNING wwa jct RemoteJunction.cpp 2108 0x7f5528045700 -- DPWWA2026W IBM Security Access Manager WebSEAL has regained contact with junction (/jct) server: 192.168.61.200

However, I was sort of able to recreate this. I think there might be a timing issue.

What version is in use?

------------------------------
Nick
ISAM Level II Support
------------------------------

Original Message

Original Message:
Sent: 03-21-2019 02:14 AM
From: Scott Exton
Subject: What happened to DPWWA2025W/DPWWA2026W msg ?

Sylvain,

I can't explain what is happening in your environment, but there has definitely not been a conscious decision to remove those messages from the WebSEAL log file. You should still be notified when a junction is unavailable, and if this is no longer happening I would most likely consider this a defect.

Thanks.

Scott A. Exton
Senior Software Engineer
Chief Programmer - IBM Security Access Manager
IBM Master Inventor


Phone: 61-7-5552-4008 E-mail: scotte@au1.ibm.com	L11 & L7 Seabank Southport, QLD 4215 Australia

Original Message------

Hello

We made this observation about 1 year ago and then forget to investigate it further but this week the same behavior was made in our Production environment.

It seems that beginning with ISAM Virtual Appliance (V8 or V9, we are running 9) that the below messages logged normally in webseald server log file are no longer reported:

DPWWA2025W IBM Security Access Manager WebSEAL has lost contact with junction server: %s (0x38cf07e9)
DPWWA2026W IBM Security Access Manager WebSEAL has regained contact with junction (%s) server: %s (0x38cf07ea)

We have had an incident this week because we could not detect that a junction state changed from "up" to "down". This is because we cannot trace in our logs the message DPWWA2025W. The root cause of the incident of course is a human manual error but it all comes down to the delay it took us to realize the problem and fix it. The lack of the usual messages contributed to worsen the incident duration.

We did see another message more detailed "DPWAD0411E The TCP/IP host information could not be determined from the server hostname: someservername. Ensure that the server hostname is correct and that the domain name server is functioning correctly" appearing in the webseal server logs which we could opt to track also in the future but I can image other cause than this particular one for a junction to go down.

Before I open a support case, just want to know if there was any conscious effort not to log those 2 events anymore?

Thanks in advanced for your comments/responses.

Sylvain Gilbert, p. ing., Conseiller Technique Infrastructures Securité/Security Infrastructures Technical Advisor

Intact Corporation Financière | 1935 des Cascades, Saint-Hyacinthe, QC J2S 8K9

(T) 855.646.8228, x 86667 | (M) 450.223.9537

sylvain.gilbert@intact.net | www.intactfc.com

4. RE: What happened to DPWWA2025W/DPWWA2026W msg ?

0 Like
IBM Champion

Sylvain Gilbert
Posted Fri March 22, 2019 01:56 PM

Reply
Hi Nick

After consulting the group, I have opened a support case.

After the initial opening of the case, I was able to identify that those expected messages are indeed reported when letting WebSEAL do the usual logging in its local default file (default config).

This is something we don't do in our environment because instead we direct all logs in near real time to our SIEM solutions via rsyslog. No logs are maintained on our Appliances.

So already now we have a distinction that this is probably not an issue of ISAM V7 vs ISAM V9 but more vs local file logging vs remote syslog logging.

I have yet to perform a packet trace to determine if those 2 syslog messages are at least leaving the Appliance or not at all.

Sylvain Gilbert, p. ing., Conseiller Technique Infrastructures Securité/Security Infrastructures Technical Advisor

Intact Corporation Financière | 1935 des Cascades, Saint-Hyacinthe, QC J2S 8K9

(T) 855.646.8228, x 86667 | (M) 450.223.9537

sylvain.gilbert@intact.net | www.intactfc.com

Original Message
5. RE: What happened to DPWWA2025W/DPWWA2026W msg ?

0 Like
Nick Lloyd
Posted Fri March 22, 2019 02:17 PM

Reply
The message should be sent with syslog. Here is from my syslog setup (9.0.6.0):

ar 17 09:25:31 isam9060.level2.org ISAM9060[2020] 2019-03-17-09:25:18.304-05:00I----- 0x38CF07E9 webseald WARNING wwa jct RemoteJunction.cpp 2148 0x7fe9780cc700 -- DPWWA2025W IBM Security Access Manager WebSEAL has lost contact with junction (ISAMLMI) server: 192.168.61.190

What is the Case number? I can recreate this (at least one scenario) and would like to move this to the Case. Thanks.

------------------------------
Nick
ISAM Level II Support
------------------------------

Original Message

6. RE: What happened to DPWWA2025W/DPWWA2026W msg ?

Like

IBM Champion

Sylvain Gilbert

Posted Sat April 13, 2019 11:34 AM

Nevermind this thread. I found out that I was not waiting long enough for the junction ping timeout interval (default of 5 min) before looking for this particular message. What tricked me, is that I was using the "/isam" junction to the AAC (rest-end points) in my test setup to reproduce the issue. The same Web Reverse Proxy instance was also having a "back-end" STS connectity configured with the same AAC/Liberty instance. When I forced my AAC/Liberty to go down, the Web Reverse Proxy instance was reporting very rapidely that it lost its connection to the back-end "STS" in its logs, whereas the DPWWA2025W event came at a late timer, much later relatively speaking it the logs, which had been already rotated and archived elsewhere.
When I am looking in our enterprise data lake, I can see all references to DPWWA2025W events properly reported for the last few months.

Learned lessons: Also go to our Enterprise Data Lake UI and not search in the middle temporary fast rorating log storage.

Closed as user error.

------------------------------
Sylvain Gilbert
------------------------------

Original Message

Original Message:
Sent: Thu March 21, 2019 02:14 AM
From: Scott Exton
Subject: What happened to DPWWA2025W/DPWWA2026W msg ?

Sylvain,

I can't explain what is happening in your environment, but there has definitely not been a conscious decision to remove those messages from the WebSEAL log file. You should still be notified when a junction is unavailable, and if this is no longer happening I would most likely consider this a defect.

Thanks.

Scott A. Exton
Senior Software Engineer
Chief Programmer - IBM Security Access Manager
IBM Master Inventor


Phone: 61-7-5552-4008 E-mail: scotte@au1.ibm.com	L11 & L7 Seabank Southport, QLD 4215 Australia

Original Message------

Hello

We made this observation about 1 year ago and then forget to investigate it further but this week the same behavior was made in our Production environment.

It seems that beginning with ISAM Virtual Appliance (V8 or V9, we are running 9) that the below messages logged normally in webseald server log file are no longer reported:

DPWWA2025W IBM Security Access Manager WebSEAL has lost contact with junction server: %s (0x38cf07e9)
DPWWA2026W IBM Security Access Manager WebSEAL has regained contact with junction (%s) server: %s (0x38cf07ea)

We have had an incident this week because we could not detect that a junction state changed from "up" to "down". This is because we cannot trace in our logs the message DPWWA2025W. The root cause of the incident of course is a human manual error but it all comes down to the delay it took us to realize the problem and fix it. The lack of the usual messages contributed to worsen the incident duration.

We did see another message more detailed "DPWAD0411E The TCP/IP host information could not be determined from the server hostname: someservername. Ensure that the server hostname is correct and that the domain name server is functioning correctly" appearing in the webseal server logs which we could opt to track also in the future but I can image other cause than this particular one for a junction to go down.

Before I open a support case, just want to know if there was any conscious effort not to log those 2 events anymore?

Thanks in advanced for your comments/responses.

Sylvain Gilbert, p. ing., Conseiller Technique Infrastructures Securité/Security Infrastructures Technical Advisor

Intact Corporation Financière | 1935 des Cascades, Saint-Hyacinthe, QC J2S 8K9

(T) 855.646.8228, x 86667 | (M) 450.223.9537

sylvain.gilbert@intact.net | www.intactfc.com

IBM Security Verify