I recently had a customer NIM environment start exhibiting failures for all NIM operations. We had implemented SSL for NIMSH on all clients to improve security and prevent LPM operations across systems from breaking NIM control due to CPU ID's.

We found out the hard way that the NIM master generates custom SSL certificates in /ssl_nimsh that are only valid for one year. At the end of that time, every LPAR rejected NIM operations. That included our weekly mksysb.

The error messages are completely obfuscated, so we could not tell that the certificate had expired. The message on the NIM master was generic:

# nim -o lslpp NIMCLIENT  0042-001 nim: processing error encountered on "master":     0042-006 m_lslpp: (From_Master) connect Error 0          0042-404 nconn: Error connecting to SSL object.          0042-406 nconn: Error verifying SSL object after connection.  405 nconn: Error with certificate at depth .

and the client log had little information:

  Thu Jul 11 13:21:11 2024        success: we got 1st write query is 0  Thu Jul 11 13:21:11 2024        success: we got 2nd write local id is 00XXXXXX00  Thu Jul 11 13:21:11 2024        success: we got 3rd write remote id is 00XXXXXX00  Thu Jul 11 13:21:11 2024        success: we got 4th write command is /usr/lpp/bos.sysmgt/nim/methods/c_nimpush "/usr/lpp/bos.sysmgt/nim/methods/c_ckspot" "-l" "-ast_applied=3" "-ast_committed=5" "-aplatform=yes" "-aname=NIMCLIENT "-alocation=/usr"  Thu Jul 11 13:21:11 2024        passing OpenSSL setting of 1  Thu Jul 11 13:21:11 2024        set symbol table  Thu Jul 11 13:21:11 2024         cert filename discovered: /ssl_nimsh/certs/MASTER.0  Thu Jul 11 13:21:11 2024        seed_prng  Thu Jul 11 13:21:11 2024        Loading certificates..  Thu Jul 11 13:21:11 2024        Negotiating SSL_CTX object using SSLv23_method  Thu Jul 11 13:21:11 2024        Loading private key file..  Thu Jul 11 13:21:11 2024        create BIO  Thu Jul 11 13:21:11 2024        Error accepting SSL connection  Thu Jul 11 13:21:11 2024        -Error: peer certificate: application verification failure  Thu Jul 11 13:21:11 2024        Error checking SSL object after connection

After opening a ticket with IBM support, it was determined the certificates had expired. We would need to disable SSL on the NIM master, delete the old certificates, and enable SSL again. This also meant that every client LPAR had to be manually touched to deploy the new certificates. That meant disabling SSL, fixing CPUID conflicts, and enabling SSL again.

What a mess.

IBM has not documented how to use an external CA authority with NIM, nor how to generate long term certificates. That's unfortunate since no one wants to redo the entire NIM environment every year.

During testing it was discovered that the NIM clients would REWRITE the SSL libraries in /usr/lib! IBM support says that should be OK, but I can't recommend doing 'nimclient -c' on a system with a running database which depends on the AIX SSL library (ie: IRIS/EPIC). It sounds like a potential race condition waiting to happen. Check your "genkld | grep ssl" and look for ssl.so files for potential conflicts.

There is an alternate command which can fetch updated certificates from the NIM master without touching the SSL libraries. I've examined the file dates on "/usr/lib/*ssl*" to confirm before and after the command.

nimclient -o get_cert -a master_name=MASTER

This downloads the latest certificate and stages it. The next call to NIMSH will reload the updated certificate as long as the master host name doesn't change. This has promise, and I'm looking at adding a weekly cron job to do that pull in case we have to regenerate the NIM master certificate again.

This isn't documented anywhere, so I thought I'd post for other NIM and LPM customers to find.

Feature requests for IBM:

 - Document how to use SSL certificates from an external CA

 - Extend the duration of the default certificates, or add a command line parameter to 'nimconfig -c' where we can specify a duration

 - Have NIMSH automatically fetch the latest certificates intermittently, instead of relying on a user created cron job

 - Convert from NIMSH to SSH. Really guys, it isn't hard to stand up a dedicated SSHD server separate from the user one with a different port and dedicated config with static keys for NIM. I've been tempted to do this and symlink rsh to ssh.

 - Never touch SSL libraries. NIM doesn't use them exclusively and blindly rewriting them while applications are online sounds like a problem waiting to happen. Alternatively make NIM use statically compiled binaries with built in SSL. At least add a warning to the documentation that SSL libraries will be modified and that could impact applications.

 - Improve the errors messages due to certificate expiration.

------------------------------
========================
Russell Adams
https://adamssystems.nl/
========================
------------------------------

Thanks for sharing Russell. This is really helpful!!

I recall that during initial setup of SSL for the NIM server, you could extend the life of the SSL certs by modifying the sample config files and then running nimconfig -c. This was often overlooked (undocumented). e.g.

root@nim1 /usr/samples/nim/ssl # grep default_days *

SSL_client.cnf:default_days     = 365

SSL_root.cnf:default_days     = 365

SSL_server.cnf:default_days     = 365

However, the solution you've shared here is far, far better.

I note that the command, nimclient -o get_cert, is briefly mentioned here, https://www.ibm.com/docs/en/aix/7.3?topic=authentication-enabling-cryptographic-from-command-line, but in the context of communicating with an alternate NIM server.

Thanks again for this useful tip, I'll be sure to share it with others!

------------------------------
Chris Gibson

Original Message:
Sent: Mon August 19, 2024 12:12 PM
From: Russell Adams
Subject: NIM w/ SSL issues and potential impact to running applications

I recently had a customer NIM environment start exhibiting failures for all NIM operations. We had implemented SSL for NIMSH on all clients to improve security and prevent LPM operations across systems from breaking NIM control due to CPU ID's.

We found out the hard way that the NIM master generates custom SSL certificates in /ssl_nimsh that are only valid for one year. At the end of that time, every LPAR rejected NIM operations. That included our weekly mksysb.

The error messages are completely obfuscated, so we could not tell that the certificate had expired. The message on the NIM master was generic:

# nim -o lslpp NIMCLIENT  0042-001 nim: processing error encountered on "master":     0042-006 m_lslpp: (From_Master) connect Error 0          0042-404 nconn: Error connecting to SSL object.          0042-406 nconn: Error verifying SSL object after connection.  405 nconn: Error with certificate at depth .

and the client log had little information:

  Thu Jul 11 13:21:11 2024        success: we got 1st write query is 0  Thu Jul 11 13:21:11 2024        success: we got 2nd write local id is 00XXXXXX00  Thu Jul 11 13:21:11 2024        success: we got 3rd write remote id is 00XXXXXX00  Thu Jul 11 13:21:11 2024        success: we got 4th write command is /usr/lpp/bos.sysmgt/nim/methods/c_nimpush "/usr/lpp/bos.sysmgt/nim/methods/c_ckspot" "-l" "-ast_applied=3" "-ast_committed=5" "-aplatform=yes" "-aname=NIMCLIENT "-alocation=/usr"  Thu Jul 11 13:21:11 2024        passing OpenSSL setting of 1  Thu Jul 11 13:21:11 2024        set symbol table  Thu Jul 11 13:21:11 2024         cert filename discovered: /ssl_nimsh/certs/MASTER.0  Thu Jul 11 13:21:11 2024        seed_prng  Thu Jul 11 13:21:11 2024        Loading certificates..  Thu Jul 11 13:21:11 2024        Negotiating SSL_CTX object using SSLv23_method  Thu Jul 11 13:21:11 2024        Loading private key file..  Thu Jul 11 13:21:11 2024        create BIO  Thu Jul 11 13:21:11 2024        Error accepting SSL connection  Thu Jul 11 13:21:11 2024        -Error: peer certificate: application verification failure  Thu Jul 11 13:21:11 2024        Error checking SSL object after connection

After opening a ticket with IBM support, it was determined the certificates had expired. We would need to disable SSL on the NIM master, delete the old certificates, and enable SSL again. This also meant that every client LPAR had to be manually touched to deploy the new certificates. That meant disabling SSL, fixing CPUID conflicts, and enabling SSL again.

What a mess.

IBM has not documented how to use an external CA authority with NIM, nor how to generate long term certificates. That's unfortunate since no one wants to redo the entire NIM environment every year.

During testing it was discovered that the NIM clients would REWRITE the SSL libraries in /usr/lib! IBM support says that should be OK, but I can't recommend doing 'nimclient -c' on a system with a running database which depends on the AIX SSL library (ie: IRIS/EPIC). It sounds like a potential race condition waiting to happen. Check your "genkld | grep ssl" and look for ssl.so files for potential conflicts.

There is an alternate command which can fetch updated certificates from the NIM master without touching the SSL libraries. I've examined the file dates on "/usr/lib/*ssl*" to confirm before and after the command.

nimclient -o get_cert -a master_name=MASTER

This downloads the latest certificate and stages it. The next call to NIMSH will reload the updated certificate as long as the master host name doesn't change. This has promise, and I'm looking at adding a weekly cron job to do that pull in case we have to regenerate the NIM master certificate again.

This isn't documented anywhere, so I thought I'd post for other NIM and LPM customers to find.

Feature requests for IBM:

 - Document how to use SSL certificates from an external CA

 - Extend the duration of the default certificates, or add a command line parameter to 'nimconfig -c' where we can specify a duration

 - Have NIMSH automatically fetch the latest certificates intermittently, instead of relying on a user created cron job

 - Convert from NIMSH to SSH. Really guys, it isn't hard to stand up a dedicated SSHD server separate from the user one with a different port and dedicated config with static keys for NIM. I've been tempted to do this and symlink rsh to ssh.

 - Never touch SSL libraries. NIM doesn't use them exclusively and blindly rewriting them while applications are online sounds like a problem waiting to happen. Alternatively make NIM use statically compiled binaries with built in SSL. At least add a warning to the documentation that SSL libraries will be modified and that could impact applications.

 - Improve the errors messages due to certificate expiration.

------------------------------
========================
Russell Adams
https://adamssystems.nl/
========================
------------------------------

Thanks Russell and Chris for your posts on this topic. I will ensure it gets awareness in development and we will see where we can make improvements in future updates. It sounds like improved documentation would be a good start.

Thanks for sharing Russell. This is really helpful!!

I recall that during initial setup of SSL for the NIM server, you could extend the life of the SSL certs by modifying the sample config files and then running nimconfig -c. This was often overlooked (undocumented). e.g.

root@nim1 /usr/samples/nim/ssl # grep default_days *

SSL_client.cnf:default_days     = 365

SSL_root.cnf:default_days     = 365

SSL_server.cnf:default_days     = 365

However, the solution you've shared here is far, far better.

I note that the command, nimclient -o get_cert, is briefly mentioned here, https://www.ibm.com/docs/en/aix/7.3?topic=authentication-enabling-cryptographic-from-command-line, but in the context of communicating with an alternate NIM server.

Thanks again for this useful tip, I'll be sure to share it with others!

------------------------------
Chris Gibson

Original Message:
Sent: Mon August 19, 2024 12:12 PM
From: Russell Adams
Subject: NIM w/ SSL issues and potential impact to running applications

I recently had a customer NIM environment start exhibiting failures for all NIM operations. We had implemented SSL for NIMSH on all clients to improve security and prevent LPM operations across systems from breaking NIM control due to CPU ID's.

We found out the hard way that the NIM master generates custom SSL certificates in /ssl_nimsh that are only valid for one year. At the end of that time, every LPAR rejected NIM operations. That included our weekly mksysb.

The error messages are completely obfuscated, so we could not tell that the certificate had expired. The message on the NIM master was generic:

# nim -o lslpp NIMCLIENT  0042-001 nim: processing error encountered on "master":     0042-006 m_lslpp: (From_Master) connect Error 0          0042-404 nconn: Error connecting to SSL object.          0042-406 nconn: Error verifying SSL object after connection.  405 nconn: Error with certificate at depth .

and the client log had little information:

  Thu Jul 11 13:21:11 2024        success: we got 1st write query is 0  Thu Jul 11 13:21:11 2024        success: we got 2nd write local id is 00XXXXXX00  Thu Jul 11 13:21:11 2024        success: we got 3rd write remote id is 00XXXXXX00  Thu Jul 11 13:21:11 2024        success: we got 4th write command is /usr/lpp/bos.sysmgt/nim/methods/c_nimpush "/usr/lpp/bos.sysmgt/nim/methods/c_ckspot" "-l" "-ast_applied=3" "-ast_committed=5" "-aplatform=yes" "-aname=NIMCLIENT "-alocation=/usr"  Thu Jul 11 13:21:11 2024        passing OpenSSL setting of 1  Thu Jul 11 13:21:11 2024        set symbol table  Thu Jul 11 13:21:11 2024         cert filename discovered: /ssl_nimsh/certs/MASTER.0  Thu Jul 11 13:21:11 2024        seed_prng  Thu Jul 11 13:21:11 2024        Loading certificates..  Thu Jul 11 13:21:11 2024        Negotiating SSL_CTX object using SSLv23_method  Thu Jul 11 13:21:11 2024        Loading private key file..  Thu Jul 11 13:21:11 2024        create BIO  Thu Jul 11 13:21:11 2024        Error accepting SSL connection  Thu Jul 11 13:21:11 2024        -Error: peer certificate: application verification failure  Thu Jul 11 13:21:11 2024        Error checking SSL object after connection

After opening a ticket with IBM support, it was determined the certificates had expired. We would need to disable SSL on the NIM master, delete the old certificates, and enable SSL again. This also meant that every client LPAR had to be manually touched to deploy the new certificates. That meant disabling SSL, fixing CPUID conflicts, and enabling SSL again.

What a mess.

IBM has not documented how to use an external CA authority with NIM, nor how to generate long term certificates. That's unfortunate since no one wants to redo the entire NIM environment every year.

During testing it was discovered that the NIM clients would REWRITE the SSL libraries in /usr/lib! IBM support says that should be OK, but I can't recommend doing 'nimclient -c' on a system with a running database which depends on the AIX SSL library (ie: IRIS/EPIC). It sounds like a potential race condition waiting to happen. Check your "genkld | grep ssl" and look for ssl.so files for potential conflicts.

There is an alternate command which can fetch updated certificates from the NIM master without touching the SSL libraries. I've examined the file dates on "/usr/lib/*ssl*" to confirm before and after the command.

nimclient -o get_cert -a master_name=MASTER

This downloads the latest certificate and stages it. The next call to NIMSH will reload the updated certificate as long as the master host name doesn't change. This has promise, and I'm looking at adding a weekly cron job to do that pull in case we have to regenerate the NIM master certificate again.

This isn't documented anywhere, so I thought I'd post for other NIM and LPM customers to find.

Feature requests for IBM:

 - Document how to use SSL certificates from an external CA

 - Extend the duration of the default certificates, or add a command line parameter to 'nimconfig -c' where we can specify a duration

 - Have NIMSH automatically fetch the latest certificates intermittently, instead of relying on a user created cron job

 - Convert from NIMSH to SSH. Really guys, it isn't hard to stand up a dedicated SSHD server separate from the user one with a different port and dedicated config with static keys for NIM. I've been tempted to do this and symlink rsh to ssh.

 - Never touch SSL libraries. NIM doesn't use them exclusively and blindly rewriting them while applications are online sounds like a problem waiting to happen. Alternatively make NIM use statically compiled binaries with built in SSL. At least add a warning to the documentation that SSL libraries will be modified and that could impact applications.

 - Improve the errors messages due to certificate expiration.

------------------------------
========================
Russell Adams
https://adamssystems.nl/
========================
------------------------------

Thanks for sharing Russell. This is really helpful!!

I recall that during initial setup of SSL for the NIM server, you could extend the life of the SSL certs by modifying the sample config files and then running nimconfig -c. This was often overlooked (undocumented). e.g.

root@nim1 /usr/samples/nim/ssl # grep default_days *

SSL_client.cnf:default_days     = 365

SSL_root.cnf:default_days     = 365

SSL_server.cnf:default_days     = 365

However, the solution you've shared here is far, far better.

I note that the command, nimclient -o get_cert, is briefly mentioned here, https://www.ibm.com/docs/en/aix/7.3?topic=authentication-enabling-cryptographic-from-command-line, but in the context of communicating with an alternate NIM server.

Thanks again for this useful tip, I'll be sure to share it with others!

------------------------------
Chris Gibson

Original Message:
Sent: Mon August 19, 2024 12:12 PM
From: Russell Adams
Subject: NIM w/ SSL issues and potential impact to running applications

I recently had a customer NIM environment start exhibiting failures for all NIM operations. We had implemented SSL for NIMSH on all clients to improve security and prevent LPM operations across systems from breaking NIM control due to CPU ID's.

We found out the hard way that the NIM master generates custom SSL certificates in /ssl_nimsh that are only valid for one year. At the end of that time, every LPAR rejected NIM operations. That included our weekly mksysb.

The error messages are completely obfuscated, so we could not tell that the certificate had expired. The message on the NIM master was generic:

# nim -o lslpp NIMCLIENT  0042-001 nim: processing error encountered on "master":     0042-006 m_lslpp: (From_Master) connect Error 0          0042-404 nconn: Error connecting to SSL object.          0042-406 nconn: Error verifying SSL object after connection.  405 nconn: Error with certificate at depth .

and the client log had little information:

  Thu Jul 11 13:21:11 2024        success: we got 1st write query is 0  Thu Jul 11 13:21:11 2024        success: we got 2nd write local id is 00XXXXXX00  Thu Jul 11 13:21:11 2024        success: we got 3rd write remote id is 00XXXXXX00  Thu Jul 11 13:21:11 2024        success: we got 4th write command is /usr/lpp/bos.sysmgt/nim/methods/c_nimpush "/usr/lpp/bos.sysmgt/nim/methods/c_ckspot" "-l" "-ast_applied=3" "-ast_committed=5" "-aplatform=yes" "-aname=NIMCLIENT "-alocation=/usr"  Thu Jul 11 13:21:11 2024        passing OpenSSL setting of 1  Thu Jul 11 13:21:11 2024        set symbol table  Thu Jul 11 13:21:11 2024         cert filename discovered: /ssl_nimsh/certs/MASTER.0  Thu Jul 11 13:21:11 2024        seed_prng  Thu Jul 11 13:21:11 2024        Loading certificates..  Thu Jul 11 13:21:11 2024        Negotiating SSL_CTX object using SSLv23_method  Thu Jul 11 13:21:11 2024        Loading private key file..  Thu Jul 11 13:21:11 2024        create BIO  Thu Jul 11 13:21:11 2024        Error accepting SSL connection  Thu Jul 11 13:21:11 2024        -Error: peer certificate: application verification failure  Thu Jul 11 13:21:11 2024        Error checking SSL object after connection

After opening a ticket with IBM support, it was determined the certificates had expired. We would need to disable SSL on the NIM master, delete the old certificates, and enable SSL again. This also meant that every client LPAR had to be manually touched to deploy the new certificates. That meant disabling SSL, fixing CPUID conflicts, and enabling SSL again.

What a mess.

IBM has not documented how to use an external CA authority with NIM, nor how to generate long term certificates. That's unfortunate since no one wants to redo the entire NIM environment every year.

During testing it was discovered that the NIM clients would REWRITE the SSL libraries in /usr/lib! IBM support says that should be OK, but I can't recommend doing 'nimclient -c' on a system with a running database which depends on the AIX SSL library (ie: IRIS/EPIC). It sounds like a potential race condition waiting to happen. Check your "genkld | grep ssl" and look for ssl.so files for potential conflicts.

There is an alternate command which can fetch updated certificates from the NIM master without touching the SSL libraries. I've examined the file dates on "/usr/lib/*ssl*" to confirm before and after the command.

nimclient -o get_cert -a master_name=MASTER

This downloads the latest certificate and stages it. The next call to NIMSH will reload the updated certificate as long as the master host name doesn't change. This has promise, and I'm looking at adding a weekly cron job to do that pull in case we have to regenerate the NIM master certificate again.

This isn't documented anywhere, so I thought I'd post for other NIM and LPM customers to find.

Feature requests for IBM:

 - Document how to use SSL certificates from an external CA

 - Extend the duration of the default certificates, or add a command line parameter to 'nimconfig -c' where we can specify a duration

 - Have NIMSH automatically fetch the latest certificates intermittently, instead of relying on a user created cron job

 - Convert from NIMSH to SSH. Really guys, it isn't hard to stand up a dedicated SSHD server separate from the user one with a different port and dedicated config with static keys for NIM. I've been tempted to do this and symlink rsh to ssh.

 - Never touch SSL libraries. NIM doesn't use them exclusively and blindly rewriting them while applications are online sounds like a problem waiting to happen. Alternatively make NIM use statically compiled binaries with built in SSL. At least add a warning to the documentation that SSL libraries will be modified and that could impact applications.

 - Improve the errors messages due to certificate expiration.

------------------------------
========================
Russell Adams
https://adamssystems.nl/
========================
------------------------------

Thanks for sharing Russell. This is really helpful!!

I recall that during initial setup of SSL for the NIM server, you could extend the life of the SSL certs by modifying the sample config files and then running nimconfig -c. This was often overlooked (undocumented). e.g.

root@nim1 /usr/samples/nim/ssl # grep default_days *

SSL_client.cnf:default_days     = 365

SSL_root.cnf:default_days     = 365

SSL_server.cnf:default_days     = 365

However, the solution you've shared here is far, far better.

I note that the command, nimclient -o get_cert, is briefly mentioned here, https://www.ibm.com/docs/en/aix/7.3?topic=authentication-enabling-cryptographic-from-command-line, but in the context of communicating with an alternate NIM server.

Thanks again for this useful tip, I'll be sure to share it with others!

------------------------------
Chris Gibson

Original Message:
Sent: Mon August 19, 2024 12:12 PM
From: Russell Adams
Subject: NIM w/ SSL issues and potential impact to running applications

I recently had a customer NIM environment start exhibiting failures for all NIM operations. We had implemented SSL for NIMSH on all clients to improve security and prevent LPM operations across systems from breaking NIM control due to CPU ID's.

We found out the hard way that the NIM master generates custom SSL certificates in /ssl_nimsh that are only valid for one year. At the end of that time, every LPAR rejected NIM operations. That included our weekly mksysb.

The error messages are completely obfuscated, so we could not tell that the certificate had expired. The message on the NIM master was generic:

# nim -o lslpp NIMCLIENT  0042-001 nim: processing error encountered on "master":     0042-006 m_lslpp: (From_Master) connect Error 0          0042-404 nconn: Error connecting to SSL object.          0042-406 nconn: Error verifying SSL object after connection.  405 nconn: Error with certificate at depth .

and the client log had little information:

  Thu Jul 11 13:21:11 2024        success: we got 1st write query is 0  Thu Jul 11 13:21:11 2024        success: we got 2nd write local id is 00XXXXXX00  Thu Jul 11 13:21:11 2024        success: we got 3rd write remote id is 00XXXXXX00  Thu Jul 11 13:21:11 2024        success: we got 4th write command is /usr/lpp/bos.sysmgt/nim/methods/c_nimpush "/usr/lpp/bos.sysmgt/nim/methods/c_ckspot" "-l" "-ast_applied=3" "-ast_committed=5" "-aplatform=yes" "-aname=NIMCLIENT "-alocation=/usr"  Thu Jul 11 13:21:11 2024        passing OpenSSL setting of 1  Thu Jul 11 13:21:11 2024        set symbol table  Thu Jul 11 13:21:11 2024         cert filename discovered: /ssl_nimsh/certs/MASTER.0  Thu Jul 11 13:21:11 2024        seed_prng  Thu Jul 11 13:21:11 2024        Loading certificates..  Thu Jul 11 13:21:11 2024        Negotiating SSL_CTX object using SSLv23_method  Thu Jul 11 13:21:11 2024        Loading private key file..  Thu Jul 11 13:21:11 2024        create BIO  Thu Jul 11 13:21:11 2024        Error accepting SSL connection  Thu Jul 11 13:21:11 2024        -Error: peer certificate: application verification failure  Thu Jul 11 13:21:11 2024        Error checking SSL object after connection

After opening a ticket with IBM support, it was determined the certificates had expired. We would need to disable SSL on the NIM master, delete the old certificates, and enable SSL again. This also meant that every client LPAR had to be manually touched to deploy the new certificates. That meant disabling SSL, fixing CPUID conflicts, and enabling SSL again.

What a mess.

IBM has not documented how to use an external CA authority with NIM, nor how to generate long term certificates. That's unfortunate since no one wants to redo the entire NIM environment every year.

During testing it was discovered that the NIM clients would REWRITE the SSL libraries in /usr/lib! IBM support says that should be OK, but I can't recommend doing 'nimclient -c' on a system with a running database which depends on the AIX SSL library (ie: IRIS/EPIC). It sounds like a potential race condition waiting to happen. Check your "genkld | grep ssl" and look for ssl.so files for potential conflicts.

There is an alternate command which can fetch updated certificates from the NIM master without touching the SSL libraries. I've examined the file dates on "/usr/lib/*ssl*" to confirm before and after the command.

nimclient -o get_cert -a master_name=MASTER

This downloads the latest certificate and stages it. The next call to NIMSH will reload the updated certificate as long as the master host name doesn't change. This has promise, and I'm looking at adding a weekly cron job to do that pull in case we have to regenerate the NIM master certificate again.

This isn't documented anywhere, so I thought I'd post for other NIM and LPM customers to find.

Feature requests for IBM:

 - Document how to use SSL certificates from an external CA

 - Extend the duration of the default certificates, or add a command line parameter to 'nimconfig -c' where we can specify a duration

 - Have NIMSH automatically fetch the latest certificates intermittently, instead of relying on a user created cron job

 - Convert from NIMSH to SSH. Really guys, it isn't hard to stand up a dedicated SSHD server separate from the user one with a different port and dedicated config with static keys for NIM. I've been tempted to do this and symlink rsh to ssh.

 - Never touch SSL libraries. NIM doesn't use them exclusively and blindly rewriting them while applications are online sounds like a problem waiting to happen. Alternatively make NIM use statically compiled binaries with built in SSL. At least add a warning to the documentation that SSL libraries will be modified and that could impact applications.

 - Improve the errors messages due to certificate expiration.

------------------------------
========================
Russell Adams
https://adamssystems.nl/
========================
------------------------------

Regarding displaying "certificate has expired" Already handled in 73F, Will consider enable this feature "Extend the duration of the default certificates, or add a command line parameter to 'nimconfig -c' where we can specify a duration" subsequent release

Thanks for sharing Russell. This is really helpful!!

I recall that during initial setup of SSL for the NIM server, you could extend the life of the SSL certs by modifying the sample config files and then running nimconfig -c. This was often overlooked (undocumented). e.g.

root@nim1 /usr/samples/nim/ssl # grep default_days *

SSL_client.cnf:default_days     = 365

SSL_root.cnf:default_days     = 365

SSL_server.cnf:default_days     = 365

However, the solution you've shared here is far, far better.

I note that the command, nimclient -o get_cert, is briefly mentioned here, https://www.ibm.com/docs/en/aix/7.3?topic=authentication-enabling-cryptographic-from-command-line, but in the context of communicating with an alternate NIM server.

Thanks again for this useful tip, I'll be sure to share it with others!

------------------------------
Chris Gibson

Original Message:
Sent: Mon August 19, 2024 12:12 PM
From: Russell Adams
Subject: NIM w/ SSL issues and potential impact to running applications

I recently had a customer NIM environment start exhibiting failures for all NIM operations. We had implemented SSL for NIMSH on all clients to improve security and prevent LPM operations across systems from breaking NIM control due to CPU ID's.

We found out the hard way that the NIM master generates custom SSL certificates in /ssl_nimsh that are only valid for one year. At the end of that time, every LPAR rejected NIM operations. That included our weekly mksysb.

The error messages are completely obfuscated, so we could not tell that the certificate had expired. The message on the NIM master was generic:

# nim -o lslpp NIMCLIENT  0042-001 nim: processing error encountered on "master":     0042-006 m_lslpp: (From_Master) connect Error 0          0042-404 nconn: Error connecting to SSL object.          0042-406 nconn: Error verifying SSL object after connection.  405 nconn: Error with certificate at depth .

and the client log had little information:

  Thu Jul 11 13:21:11 2024        success: we got 1st write query is 0  Thu Jul 11 13:21:11 2024        success: we got 2nd write local id is 00XXXXXX00  Thu Jul 11 13:21:11 2024        success: we got 3rd write remote id is 00XXXXXX00  Thu Jul 11 13:21:11 2024        success: we got 4th write command is /usr/lpp/bos.sysmgt/nim/methods/c_nimpush "/usr/lpp/bos.sysmgt/nim/methods/c_ckspot" "-l" "-ast_applied=3" "-ast_committed=5" "-aplatform=yes" "-aname=NIMCLIENT "-alocation=/usr"  Thu Jul 11 13:21:11 2024        passing OpenSSL setting of 1  Thu Jul 11 13:21:11 2024        set symbol table  Thu Jul 11 13:21:11 2024         cert filename discovered: /ssl_nimsh/certs/MASTER.0  Thu Jul 11 13:21:11 2024        seed_prng  Thu Jul 11 13:21:11 2024        Loading certificates..  Thu Jul 11 13:21:11 2024        Negotiating SSL_CTX object using SSLv23_method  Thu Jul 11 13:21:11 2024        Loading private key file..  Thu Jul 11 13:21:11 2024        create BIO  Thu Jul 11 13:21:11 2024        Error accepting SSL connection  Thu Jul 11 13:21:11 2024        -Error: peer certificate: application verification failure  Thu Jul 11 13:21:11 2024        Error checking SSL object after connection

After opening a ticket with IBM support, it was determined the certificates had expired. We would need to disable SSL on the NIM master, delete the old certificates, and enable SSL again. This also meant that every client LPAR had to be manually touched to deploy the new certificates. That meant disabling SSL, fixing CPUID conflicts, and enabling SSL again.

What a mess.

IBM has not documented how to use an external CA authority with NIM, nor how to generate long term certificates. That's unfortunate since no one wants to redo the entire NIM environment every year.

During testing it was discovered that the NIM clients would REWRITE the SSL libraries in /usr/lib! IBM support says that should be OK, but I can't recommend doing 'nimclient -c' on a system with a running database which depends on the AIX SSL library (ie: IRIS/EPIC). It sounds like a potential race condition waiting to happen. Check your "genkld | grep ssl" and look for ssl.so files for potential conflicts.

There is an alternate command which can fetch updated certificates from the NIM master without touching the SSL libraries. I've examined the file dates on "/usr/lib/*ssl*" to confirm before and after the command.

nimclient -o get_cert -a master_name=MASTER

This downloads the latest certificate and stages it. The next call to NIMSH will reload the updated certificate as long as the master host name doesn't change. This has promise, and I'm looking at adding a weekly cron job to do that pull in case we have to regenerate the NIM master certificate again.

This isn't documented anywhere, so I thought I'd post for other NIM and LPM customers to find.

Feature requests for IBM:

 - Document how to use SSL certificates from an external CA

 - Extend the duration of the default certificates, or add a command line parameter to 'nimconfig -c' where we can specify a duration

 - Have NIMSH automatically fetch the latest certificates intermittently, instead of relying on a user created cron job

 - Convert from NIMSH to SSH. Really guys, it isn't hard to stand up a dedicated SSHD server separate from the user one with a different port and dedicated config with static keys for NIM. I've been tempted to do this and symlink rsh to ssh.

 - Never touch SSL libraries. NIM doesn't use them exclusively and blindly rewriting them while applications are online sounds like a problem waiting to happen. Alternatively make NIM use statically compiled binaries with built in SSL. At least add a warning to the documentation that SSL libraries will be modified and that could impact applications.

 - Improve the errors messages due to certificate expiration.

------------------------------
========================
Russell Adams
https://adamssystems.nl/
========================
------------------------------

I recently had a customer NIM environment start exhibiting failures for all NIM operations. We had implemented SSL for NIMSH on all clients to improve security and prevent LPM operations across systems from breaking NIM control due to CPU ID's.

We found out the hard way that the NIM master generates custom SSL certificates in /ssl_nimsh that are only valid for one year. At the end of that time, every LPAR rejected NIM operations. That included our weekly mksysb.

The error messages are completely obfuscated, so we could not tell that the certificate had expired. The message on the NIM master was generic:

# nim -o lslpp NIMCLIENT  0042-001 nim: processing error encountered on "master":     0042-006 m_lslpp: (From_Master) connect Error 0          0042-404 nconn: Error connecting to SSL object.          0042-406 nconn: Error verifying SSL object after connection.  405 nconn: Error with certificate at depth .

and the client log had little information:

  Thu Jul 11 13:21:11 2024        success: we got 1st write query is 0  Thu Jul 11 13:21:11 2024        success: we got 2nd write local id is 00XXXXXX00  Thu Jul 11 13:21:11 2024        success: we got 3rd write remote id is 00XXXXXX00  Thu Jul 11 13:21:11 2024        success: we got 4th write command is /usr/lpp/bos.sysmgt/nim/methods/c_nimpush "/usr/lpp/bos.sysmgt/nim/methods/c_ckspot" "-l" "-ast_applied=3" "-ast_committed=5" "-aplatform=yes" "-aname=NIMCLIENT "-alocation=/usr"  Thu Jul 11 13:21:11 2024        passing OpenSSL setting of 1  Thu Jul 11 13:21:11 2024        set symbol table  Thu Jul 11 13:21:11 2024         cert filename discovered: /ssl_nimsh/certs/MASTER.0  Thu Jul 11 13:21:11 2024        seed_prng  Thu Jul 11 13:21:11 2024        Loading certificates..  Thu Jul 11 13:21:11 2024        Negotiating SSL_CTX object using SSLv23_method  Thu Jul 11 13:21:11 2024        Loading private key file..  Thu Jul 11 13:21:11 2024        create BIO  Thu Jul 11 13:21:11 2024        Error accepting SSL connection  Thu Jul 11 13:21:11 2024        -Error: peer certificate: application verification failure  Thu Jul 11 13:21:11 2024        Error checking SSL object after connection

After opening a ticket with IBM support, it was determined the certificates had expired. We would need to disable SSL on the NIM master, delete the old certificates, and enable SSL again. This also meant that every client LPAR had to be manually touched to deploy the new certificates. That meant disabling SSL, fixing CPUID conflicts, and enabling SSL again.

What a mess.

IBM has not documented how to use an external CA authority with NIM, nor how to generate long term certificates. That's unfortunate since no one wants to redo the entire NIM environment every year.

During testing it was discovered that the NIM clients would REWRITE the SSL libraries in /usr/lib! IBM support says that should be OK, but I can't recommend doing 'nimclient -c' on a system with a running database which depends on the AIX SSL library (ie: IRIS/EPIC). It sounds like a potential race condition waiting to happen. Check your "genkld | grep ssl" and look for ssl.so files for potential conflicts.

There is an alternate command which can fetch updated certificates from the NIM master without touching the SSL libraries. I've examined the file dates on "/usr/lib/*ssl*" to confirm before and after the command.

nimclient -o get_cert -a master_name=MASTER

This downloads the latest certificate and stages it. The next call to NIMSH will reload the updated certificate as long as the master host name doesn't change. This has promise, and I'm looking at adding a weekly cron job to do that pull in case we have to regenerate the NIM master certificate again.

This isn't documented anywhere, so I thought I'd post for other NIM and LPM customers to find.

Feature requests for IBM:

 - Document how to use SSL certificates from an external CA

 - Extend the duration of the default certificates, or add a command line parameter to 'nimconfig -c' where we can specify a duration

 - Have NIMSH automatically fetch the latest certificates intermittently, instead of relying on a user created cron job

 - Convert from NIMSH to SSH. Really guys, it isn't hard to stand up a dedicated SSHD server separate from the user one with a different port and dedicated config with static keys for NIM. I've been tempted to do this and symlink rsh to ssh.

 - Never touch SSL libraries. NIM doesn't use them exclusively and blindly rewriting them while applications are online sounds like a problem waiting to happen. Alternatively make NIM use statically compiled binaries with built in SSL. At least add a warning to the documentation that SSL libraries will be modified and that could impact applications.

 - Improve the errors messages due to certificate expiration.