We have updated some of our certificates used in AMS from self-signed to digital. All but two servers that use these certs have been migrated. We have multiple queues using the new digital certificates with no issues, but there is one and only one queue (used for failover) that will not allow messages from 'le' user to process through with security policy in place and end up in DLQ. The queue that is getting errors when messages try to hit is CCSDS.FAILOVER.P.Q:. There are currently two queues that use the same cert in security policy and this queue is the only one that has issues. The other queue is LEBO.SDS.STORE.REQ.P.Q. Here is the error I am seeing in MQ logs when messages try to go to failover queue:
----- amqrccca.c : 969 --------------------------------------------------------
09/21/2021 05:01:25 AM - Process(47475.21)
User(le) Program(java)
Host(
Server1) Installation(Installation1)
VRMF(9.2.0.0) QMgr(Server1)
Time(2021-09-21T10:01:25.269Z)
ArithInsert1(60)
CommentInsert1(
CN=LEO_APP_PROD)
AMQ9021E: An error occured during the certificate import for the following DN:
CN=LEO_APP_PROD, result: 60
EXPLANATION:
The distinguished name is not present in the keystore or invalid.
ACTION:
Consult the GSKit appendix in the Information Center for the explanation of the
GSKit reason code and take corrective action. If the problem persists, contact
your IBM service representative.
----- smqodida.c : 3450 -------------------------------------------------------
09/21/2021 05:01:25 AM - Process(47475.21)
User(le) Program(java)
Host(
Server1) Installation(Installation1)
VRMF(9.2.0.0) QMgr(Server1)
Time(2021-09-21T10:01:25.270Z)
ArithInsert1(60)
CommentInsert1(
CN=LEO_APP_PROD)
AMQ9069E: The IBM MQ security policy interceptor failed to validate a
certificate.
EXPLANATION:
The IBM MQ security policy interceptor could not validate a certificate when
attempting to protect or unprotect a message. The cryptographic provider failed
to verify the certificate with a distinguished name of 'CN=LEO_APP_PROD'. The
validation may have failed for this certificate, or another certificate in the
validation chain. The return code from the cryptographic provider was 60.
ACTION:
Ensure that all certificates required to validate the identify of valid message
signers are available.
*********************************************
AMS Security Policies on Server1:
Policy Details:
Policy name:
CCSDS.FAILOVER.P.QQuality of protection: PRIVACY
Signature algorithm: SHA256
Encryption algorithm: AES256
Signer DNs: -
Recipient DNs:
CN=PCI_APP_PROD_2020
CN=LEO_APP_PRODCN=SDS_APP_PROD
Key reuse count: 0
Toleration: 1
Policy Details:
Policy name:
LEBO.SDS.STORE.REQ.P.QQuality of protection: PRIVACY
Signature algorithm: SHA256
Encryption algorithm: AES256
Signer DNs: -
Recipient DNs:
CN=PCI_APP_PROD_2020
CN=LEO_APP_PRODKey reuse count: 0
Toleration: 1
Server1 is still using self-signed certificate:
[mqm@Server1 2020]$ runmqakm -cert -list -db key.kdb -stashed
Certificates found
* default, - personal, ! trusted, # secret key
! was_app_prod_2020
! eu_app_prod
!
leo_app_prod <=added extracted digital cert from Server2
! sds_app_prod
- pci_app_prod_2020 <=self-signed cert
[mqm@Server1 2020]$
Cert partial details:
[mqm@Server1 2020]$ runmqakm -cert -details -label leo_app_prod -db key.kdb -stashed
Label : leo_app_prod
Key Size : 2048
Version : X509 V3
Serial : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Issuer : "CN=LEINTERNAL CA SUB1,DC=leinternal,DC=com"
Subject : CN=LEO_APP_PROD
******************************************
Server2 has been migrated to digital.
[mqm@Server2 .mqs]$ runmqakm -cert -list -db key.kdb -stashedCertificates found
* default, - personal, ! trusted, # secret key
! pci_app_prod_2020
!
"CN=LEINTERNAL CA SUB1, DC=leinternal, DC=com" !
"CN=LEINTERNAL CA, DC=leinternal, DC=com" ! sds_app_prod
-
leo_app_prod [mqm@Server2 .mqs]$
Cert partial details:
[mqm@Server2 .mqs]$ runmqakm -cert -details -label leo_app_prod -db key.kdb -stashed
Label : leo_app_prod
Key Size : 2048
Version : X509 V3
Serial : XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Issuer : "CN=LEINTERNAL CA SUB1,DC=leinternal,DC=com"
Subject : CN=LEO_APP_PROD
Opened a case to get assistance when we initially did this transition work - L2 stated that putting the CA and CA SUB1 on Server1 was not needed. All certs listed in key.kdb that do not have 2020 on end are digital certs. All queues (but the one queue) that has a security policy in place are not having issues with 'le' user and messages are processing using those new certs. So all seemed fine until one message fails and tries to go to failover queue, can't so ends up in DLQ. Those messages that end up in DLQ cannot reprocess, so we have to manually process them.
Opened another case for this particular issue and L2 is stating that we need to add the CA and CA SUB1 certs to key.kdb on Server1. To me that don't make sense as why are all the other queues covered by these policies working with the exception of this one queue? Anyone have some insight on this or run into this same issue?
------------------------------
Gwen Buzynski
------------------------------