Db2

I am seeing a strange issue in DMC . 

Everything works normal after i did setup the DMC console.

But when i do an restart of the DMC console , i see all the databases configured using an SSL port turns into RED . Then if i go into each and every monitoring DB i configured , hit a TEST CONNECTION button , then all the databases turns GREEN again . They everything works normal .

If some issues in keystore created , then nothing should work. But it is breaking only if the DMC is restarted . Anyone seen this issue or any recommendations would be of great help . 

These disconnected issues i am seeing only when the DMC console is restarted . Post restart i am seeing the following messages in DMC log :

[8/2/21 15:42:54:961 EDT] 00000257 com.ibm.ws.channel.ssl.internal.SSLHandshakeErrorTracker E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: no cipher suites in common

In DB2 dialog i am seeing this error

2021-08-02-00.04.04.991482-240 I58313283E579 LEVEL: Error
PID : 23430 TID : 140489449793280 PROC : db2sysc 0
INSTANCE: dmcinst1 NODE : 000 DB : REPODBNP
APPHDL : 0-13403
HOSTNAME: qc1dsm04.tchtest.org
EDUID : 92 EDUNAME: db2agent () 0
FUNCTION: DB2 UDB, common communication, sqlcctcpinit, probe:949
MESSAGE : ZRC=0x00000036=54

DATA #1 : String, 59 bytes
SSL socket setup failed. Client IP address and port number:
DATA #2 : String, 12 bytes
10.102.13.45
DATA #3 : signed integer, 4 bytes
36518


------------------------------
Saravanan Santhanam
------------------------------

#Db2

Hi Saravanan, I was seeing something very similar on 3.1.4 and 3.1.5. I only have a small test environment with two monitored databases, both using SSL connections. After a DMC server restart only one of the two databases would show red in the console until I did a test connection then both would be OK.

I saw the message below in one of the log files (sorry, can't remember which one) repeating every five minutes until after I had done test connections on the the database failing to connect.

Exception java.io.IOException: Error opening socket to server clsadanan05.clsad.ibmcls/10.22.11.198 on port 50,211 with message: Keystore was tampered with, or password was incorrect. ERRORCODE=-4499, SQLSTATE=08001

The keystore file had not been tampered with! I figured this might be a contention issue so I tried generating a different keystore file for each database. To my surprise it fixed the issue!

My DMC project is currently on hold. I was intending to follow this issue up when the project restarts.


------------------------------
Colin Chapman
------------------------------

Original Message:
Sent: Mon August 02, 2021 03:47 PM
From: Saravanan Santhanam
Subject: DMC - CWWKO0801E - DMC restart SSL issues

I am seeing a strange issue in DMC . 

Everything works normal after i did setup the DMC console.

But when i do an restart of the DMC console , i see all the databases configured using an SSL port turns into RED . Then if i go into each and every monitoring DB i configured , hit a TEST CONNECTION button , then all the databases turns GREEN again . They everything works normal .

If some issues in keystore created , then nothing should work. But it is breaking only if the DMC is restarted . Anyone seen this issue or any recommendations would be of great help . 

These disconnected issues i am seeing only when the DMC console is restarted . Post restart i am seeing the following messages in DMC log :

[8/2/21 15:42:54:961 EDT] 00000257 com.ibm.ws.channel.ssl.internal.SSLHandshakeErrorTracker E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: no cipher suites in common

In DB2 dialog i am seeing this error

2021-08-02-00.04.04.991482-240 I58313283E579 LEVEL: Error
PID : 23430 TID : 140489449793280 PROC : db2sysc 0
INSTANCE: dmcinst1 NODE : 000 DB : REPODBNP
APPHDL : 0-13403
HOSTNAME: qc1dsm04.tchtest.org
EDUID : 92 EDUNAME: db2agent () 0
FUNCTION: DB2 UDB, common communication, sqlcctcpinit, probe:949
MESSAGE : ZRC=0x00000036=54

DATA #1 : String, 59 bytes
SSL socket setup failed. Client IP address and port number:
DATA #2 : String, 12 bytes
10.102.13.45
DATA #3 : signed integer, 4 bytes
36518


------------------------------
Saravanan Santhanam
------------------------------

#Db2

Thank you so much Colin . 

"The keystore file had not been tampered with! I figured this might be a contention issue so I tried generating a different keystore file for each database. To my surprise it fixed the issue!"

It would be great if you could put in the command you used both on the DMC server as well for the monitored DBs server , so will cross check once . 




------------------------------
Saravanan Santhanam
------------------------------

Original Message:
Sent: Tue August 03, 2021 03:35 AM
From: Colin Chapman
Subject: DMC - CWWKO0801E - DMC restart SSL issues

Hi Saravanan, I was seeing something very similar on 3.1.4 and 3.1.5. I only have a small test environment with two monitored databases, both using SSL connections. After a DMC server restart only one of the two databases would show red in the console until I did a test connection then both would be OK.

I saw the message below in one of the log files (sorry, can't remember which one) repeating every five minutes until after I had done test connections on the the database failing to connect.

Exception java.io.IOException: Error opening socket to server clsadanan05.clsad.ibmcls/10.22.11.198 on port 50,211 with message: Keystore was tampered with, or password was incorrect. ERRORCODE=-4499, SQLSTATE=08001

The keystore file had not been tampered with! I figured this might be a contention issue so I tried generating a different keystore file for each database. To my surprise it fixed the issue!

My DMC project is currently on hold. I was intending to follow this issue up when the project restarts.


------------------------------
Colin Chapman

Original Message:
Sent: Mon August 02, 2021 03:47 PM
From: Saravanan Santhanam
Subject: DMC - CWWKO0801E - DMC restart SSL issues

I am seeing a strange issue in DMC . 

Everything works normal after i did setup the DMC console.

But when i do an restart of the DMC console , i see all the databases configured using an SSL port turns into RED . Then if i go into each and every monitoring DB i configured , hit a TEST CONNECTION button , then all the databases turns GREEN again . They everything works normal .

If some issues in keystore created , then nothing should work. But it is breaking only if the DMC is restarted . Anyone seen this issue or any recommendations would be of great help . 

These disconnected issues i am seeing only when the DMC console is restarted . Post restart i am seeing the following messages in DMC log :

[8/2/21 15:42:54:961 EDT] 00000257 com.ibm.ws.channel.ssl.internal.SSLHandshakeErrorTracker E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: no cipher suites in common

In DB2 dialog i am seeing this error

2021-08-02-00.04.04.991482-240 I58313283E579 LEVEL: Error
PID : 23430 TID : 140489449793280 PROC : db2sysc 0
INSTANCE: dmcinst1 NODE : 000 DB : REPODBNP
APPHDL : 0-13403
HOSTNAME: qc1dsm04.tchtest.org
EDUID : 92 EDUNAME: db2agent () 0
FUNCTION: DB2 UDB, common communication, sqlcctcpinit, probe:949
MESSAGE : ZRC=0x00000036=54

DATA #1 : String, 59 bytes
SSL socket setup failed. Client IP address and port number:
DATA #2 : String, 12 bytes
10.102.13.45
DATA #3 : signed integer, 4 bytes
36518


------------------------------
Saravanan Santhanam
------------------------------

#Db2

Nothing special really Saravanan. Rather than importing the arm files for each of my instances into the same jks files I created a separate keystore file for each instance.  Not an ideal solution but it seemed to work.

/opt/IBM/DMC/ibm-datamgmtconsole/java/jre/bin/keytool -import -storepass ******** -trustcacerts -noprompt -alias prdaur -file /repos/dba/SSL/clsadanan05_prdaur.arm -keystore ~/security/dmc.prdaur.jks

/opt/IBM/DMC/ibm-datamgmtconsole/java/jre/bin/keytool -import -storepass ******** -trustcacerts -noprompt -alias prdaus -file /repos/dba/SSL/clsadanan05_prdaus.arm -keystore ~/security/dmc.prdaus.jks

------------------------------
Colin Chapman
------------------------------

Original Message:
Sent: Tue August 03, 2021 09:46 AM
From: Saravanan S
Subject: DMC - CWWKO0801E - DMC restart SSL issues

Thank you so much Colin . 

"The keystore file had not been tampered with! I figured this might be a contention issue so I tried generating a different keystore file for each database. To my surprise it fixed the issue!"

It would be great if you could put in the command you used both on the DMC server as well for the monitored DBs server , so will cross check once . 




------------------------------
Saravanan Santhanam

Original Message:
Sent: Tue August 03, 2021 03:35 AM
From: Colin Chapman
Subject: DMC - CWWKO0801E - DMC restart SSL issues

Hi Saravanan, I was seeing something very similar on 3.1.4 and 3.1.5. I only have a small test environment with two monitored databases, both using SSL connections. After a DMC server restart only one of the two databases would show red in the console until I did a test connection then both would be OK.

I saw the message below in one of the log files (sorry, can't remember which one) repeating every five minutes until after I had done test connections on the the database failing to connect.

Exception java.io.IOException: Error opening socket to server clsadanan05.clsad.ibmcls/10.22.11.198 on port 50,211 with message: Keystore was tampered with, or password was incorrect. ERRORCODE=-4499, SQLSTATE=08001

The keystore file had not been tampered with! I figured this might be a contention issue so I tried generating a different keystore file for each database. To my surprise it fixed the issue!

My DMC project is currently on hold. I was intending to follow this issue up when the project restarts.


------------------------------
Colin Chapman

Original Message:
Sent: Mon August 02, 2021 03:47 PM
From: Saravanan Santhanam
Subject: DMC - CWWKO0801E - DMC restart SSL issues

I am seeing a strange issue in DMC . 

Everything works normal after i did setup the DMC console.

But when i do an restart of the DMC console , i see all the databases configured using an SSL port turns into RED . Then if i go into each and every monitoring DB i configured , hit a TEST CONNECTION button , then all the databases turns GREEN again . They everything works normal .

If some issues in keystore created , then nothing should work. But it is breaking only if the DMC is restarted . Anyone seen this issue or any recommendations would be of great help . 

These disconnected issues i am seeing only when the DMC console is restarted . Post restart i am seeing the following messages in DMC log :

[8/2/21 15:42:54:961 EDT] 00000257 com.ibm.ws.channel.ssl.internal.SSLHandshakeErrorTracker E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: no cipher suites in common

In DB2 dialog i am seeing this error

2021-08-02-00.04.04.991482-240 I58313283E579 LEVEL: Error
PID : 23430 TID : 140489449793280 PROC : db2sysc 0
INSTANCE: dmcinst1 NODE : 000 DB : REPODBNP
APPHDL : 0-13403
HOSTNAME: qc1dsm04.tchtest.org
EDUID : 92 EDUNAME: db2agent () 0
FUNCTION: DB2 UDB, common communication, sqlcctcpinit, probe:949
MESSAGE : ZRC=0x00000036=54

DATA #1 : String, 59 bytes
SSL socket setup failed. Client IP address and port number:
DATA #2 : String, 12 bytes
10.102.13.45
DATA #3 : signed integer, 4 bytes
36518


------------------------------
Saravanan Santhanam
------------------------------

#Db2

Thank you so much Colin . 

Tried the workaround you suggested , but not having luck . Still seeing the same issue . 

Already opened a case with IBM , but they are suggesting to open a separate ticket with IBM under DB2 category . The one case i raised was for DMC . 

Will let you know if i hear any update .

------------------------------
Saravanan S
------------------------------

Original Message:
Sent: Tue August 03, 2021 12:05 PM
From: Colin Chapman
Subject: DMC - CWWKO0801E - DMC restart SSL issues

Nothing special really Saravanan. Rather than importing the arm files for each of my instances into the same jks files I created a separate keystore file for each instance.  Not an ideal solution but it seemed to work.

/opt/IBM/DMC/ibm-datamgmtconsole/java/jre/bin/keytool -import -storepass ******** -trustcacerts -noprompt -alias prdaur -file /repos/dba/SSL/clsadanan05_prdaur.arm -keystore ~/security/dmc.prdaur.jks

/opt/IBM/DMC/ibm-datamgmtconsole/java/jre/bin/keytool -import -storepass ******** -trustcacerts -noprompt -alias prdaus -file /repos/dba/SSL/clsadanan05_prdaus.arm -keystore ~/security/dmc.prdaus.jks

------------------------------
Colin Chapman

Original Message:
Sent: Tue August 03, 2021 09:46 AM
From: Saravanan S
Subject: DMC - CWWKO0801E - DMC restart SSL issues

Thank you so much Colin . 

"The keystore file had not been tampered with! I figured this might be a contention issue so I tried generating a different keystore file for each database. To my surprise it fixed the issue!"

It would be great if you could put in the command you used both on the DMC server as well for the monitored DBs server , so will cross check once . 




------------------------------
Saravanan Santhanam

Original Message:
Sent: Tue August 03, 2021 03:35 AM
From: Colin Chapman
Subject: DMC - CWWKO0801E - DMC restart SSL issues

Hi Saravanan, I was seeing something very similar on 3.1.4 and 3.1.5. I only have a small test environment with two monitored databases, both using SSL connections. After a DMC server restart only one of the two databases would show red in the console until I did a test connection then both would be OK.

I saw the message below in one of the log files (sorry, can't remember which one) repeating every five minutes until after I had done test connections on the the database failing to connect.

Exception java.io.IOException: Error opening socket to server clsadanan05.clsad.ibmcls/10.22.11.198 on port 50,211 with message: Keystore was tampered with, or password was incorrect. ERRORCODE=-4499, SQLSTATE=08001

The keystore file had not been tampered with! I figured this might be a contention issue so I tried generating a different keystore file for each database. To my surprise it fixed the issue!

My DMC project is currently on hold. I was intending to follow this issue up when the project restarts.


------------------------------
Colin Chapman

Original Message:
Sent: Mon August 02, 2021 03:47 PM
From: Saravanan Santhanam
Subject: DMC - CWWKO0801E - DMC restart SSL issues

I am seeing a strange issue in DMC . 

Everything works normal after i did setup the DMC console.

But when i do an restart of the DMC console , i see all the databases configured using an SSL port turns into RED . Then if i go into each and every monitoring DB i configured , hit a TEST CONNECTION button , then all the databases turns GREEN again . They everything works normal .

If some issues in keystore created , then nothing should work. But it is breaking only if the DMC is restarted . Anyone seen this issue or any recommendations would be of great help . 

These disconnected issues i am seeing only when the DMC console is restarted . Post restart i am seeing the following messages in DMC log :

[8/2/21 15:42:54:961 EDT] 00000257 com.ibm.ws.channel.ssl.internal.SSLHandshakeErrorTracker E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: no cipher suites in common

In DB2 dialog i am seeing this error

2021-08-02-00.04.04.991482-240 I58313283E579 LEVEL: Error
PID : 23430 TID : 140489449793280 PROC : db2sysc 0
INSTANCE: dmcinst1 NODE : 000 DB : REPODBNP
APPHDL : 0-13403
HOSTNAME: qc1dsm04.tchtest.org
EDUID : 92 EDUNAME: db2agent () 0
FUNCTION: DB2 UDB, common communication, sqlcctcpinit, probe:949
MESSAGE : ZRC=0x00000036=54

DATA #1 : String, 59 bytes
SSL socket setup failed. Client IP address and port number:
DATA #2 : String, 12 bytes
10.102.13.45
DATA #3 : signed integer, 4 bytes
36518


------------------------------
Saravanan Santhanam
------------------------------

#Db2

Hi Saravanan,

Thanks for reporting the issue and we will reproduce and fix this issue in upcoming release.  Please use the workaround you mentioned for now.

------------------------------
Jason Sizto
------------------------------

Original Message:
Sent: Mon August 02, 2021 03:47 PM
From: Saravanan S
Subject: DMC - CWWKO0801E - DMC restart SSL issues

I am seeing a strange issue in DMC . 

Everything works normal after i did setup the DMC console.

But when i do an restart of the DMC console , i see all the databases configured using an SSL port turns into RED . Then if i go into each and every monitoring DB i configured , hit a TEST CONNECTION button , then all the databases turns GREEN again . They everything works normal .

If some issues in keystore created , then nothing should work. But it is breaking only if the DMC is restarted . Anyone seen this issue or any recommendations would be of great help . 

These disconnected issues i am seeing only when the DMC console is restarted . Post restart i am seeing the following messages in DMC log :

[8/2/21 15:42:54:961 EDT] 00000257 com.ibm.ws.channel.ssl.internal.SSLHandshakeErrorTracker E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: no cipher suites in common

In DB2 dialog i am seeing this error

2021-08-02-00.04.04.991482-240 I58313283E579 LEVEL: Error
PID : 23430 TID : 140489449793280 PROC : db2sysc 0
INSTANCE: dmcinst1 NODE : 000 DB : REPODBNP
APPHDL : 0-13403
HOSTNAME: qc1dsm04.tchtest.org
EDUID : 92 EDUNAME: db2agent () 0
FUNCTION: DB2 UDB, common communication, sqlcctcpinit, probe:949
MESSAGE : ZRC=0x00000036=54

DATA #1 : String, 59 bytes
SSL socket setup failed. Client IP address and port number:
DATA #2 : String, 12 bytes
10.102.13.45
DATA #3 : signed integer, 4 bytes
36518


------------------------------
Saravanan Santhanam
------------------------------

#Db2

Thank you Jason .

------------------------------
Saravanan S
------------------------------

Original Message:
Sent: Tue August 03, 2021 02:14 PM
From: Jason Sizto
Subject: DMC - CWWKO0801E - DMC restart SSL issues

Hi Saravanan,

Thanks for reporting the issue and we will reproduce and fix this issue in upcoming release.  Please use the workaround you mentioned for now.

------------------------------
Jason Sizto

Original Message:
Sent: Mon August 02, 2021 03:47 PM
From: Saravanan S
Subject: DMC - CWWKO0801E - DMC restart SSL issues

I am seeing a strange issue in DMC . 

Everything works normal after i did setup the DMC console.

But when i do an restart of the DMC console , i see all the databases configured using an SSL port turns into RED . Then if i go into each and every monitoring DB i configured , hit a TEST CONNECTION button , then all the databases turns GREEN again . They everything works normal .

If some issues in keystore created , then nothing should work. But it is breaking only if the DMC is restarted . Anyone seen this issue or any recommendations would be of great help . 

These disconnected issues i am seeing only when the DMC console is restarted . Post restart i am seeing the following messages in DMC log :

[8/2/21 15:42:54:961 EDT] 00000257 com.ibm.ws.channel.ssl.internal.SSLHandshakeErrorTracker E CWWKO0801E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLHandshakeException: no cipher suites in common

In DB2 dialog i am seeing this error

2021-08-02-00.04.04.991482-240 I58313283E579 LEVEL: Error
PID : 23430 TID : 140489449793280 PROC : db2sysc 0
INSTANCE: dmcinst1 NODE : 000 DB : REPODBNP
APPHDL : 0-13403
HOSTNAME: qc1dsm04.tchtest.org
EDUID : 92 EDUNAME: db2agent () 0
FUNCTION: DB2 UDB, common communication, sqlcctcpinit, probe:949
MESSAGE : ZRC=0x00000036=54

DATA #1 : String, 59 bytes
SSL socket setup failed. Client IP address and port number:
DATA #2 : String, 12 bytes
10.102.13.45
DATA #3 : signed integer, 4 bytes
36518


------------------------------
Saravanan Santhanam
------------------------------

#Db2