On Tue, Aug 20, 2024 at 04:49:19PM +0000, Srikanth Thanneeru via IBM TechXchange Community wrote:
> 0042-535 nconn: The certificate has expired.
> 0042-406 nconn: Error verifying SSL object after connection.
> nconn: SSL shutdown failed.405 nconn: Error with certificate at depth .
the file. A command line option would certainly help. It would also be
duration.
doublecheck.
Thanks.
Principal Consultant Adams Systems Consultancy
Original Message:
Sent: 8/20/2024 12:49:00 PM
From: Srikanth Thanneeru
Subject: RE: NIM w/ SSL issues and potential impact to running applications
Regarding displaying "certificate has expired" Already handled in 73F, Will consider enable this feature "Extend the duration of the default certificates, or add a command line parameter to 'nimconfig -c' where we can specify a duration" subsequent release
nim -o lslpp aix-install-zep01-lp015
+ grep bos.rte
0042-001 nim: processing error encountered on "master":
0042-006 m_lslpp: (From_Master) connect Error 0
0042-535 nconn: The certificate has expired.
0042-406 nconn: Error verifying SSL object after connection.
nconn: SSL shutdown failed.405 nconn: Error with certificate at depth .
------------------------------
Srikanth Thanneeru
------------------------------
Original Message:
Sent: Tue August 20, 2024 02:12 AM
From: Chris Gibson
Subject: NIM w/ SSL issues and potential impact to running applications
Thanks for sharing Russell. This is really helpful!!
I recall that during initial setup of SSL for the NIM server, you could extend the life of the SSL certs by modifying the sample config files and then running nimconfig -c
. This was often overlooked (undocumented). e.g.
root@nim1 /usr/samples/nim/ssl # grep default_days *
SSL_client.cnf:default_days = 365
SSL_root.cnf:default_days = 365
SSL_server.cnf:default_days = 365
However, the solution you've shared here is far, far better.
I note that the command, nimclient -o get_cert
, is briefly mentioned here, https://www.ibm.com/docs/en/aix/7.3?topic=authentication-enabling-cryptographic-from-command-line, but in the context of communicating with an alternate NIM server.
Thanks again for this useful tip, I'll be sure to share it with others!
------------------------------
Chris Gibson
Original Message:
Sent: Mon August 19, 2024 12:12 PM
From: Russell Adams
Subject: NIM w/ SSL issues and potential impact to running applications
I recently had a customer NIM environment start exhibiting failures for all NIM operations. We had implemented SSL for NIMSH on all clients to improve security and prevent LPM operations across systems from breaking NIM control due to CPU ID's.
We found out the hard way that the NIM master generates custom SSL certificates in /ssl_nimsh that are only valid for one year. At the end of that time, every LPAR rejected NIM operations. That included our weekly mksysb.
The error messages are completely obfuscated, so we could not tell that the certificate had expired. The message on the NIM master was generic:
# nim -o lslpp NIMCLIENT 0042-001 nim: processing error encountered on "master": 0042-006 m_lslpp: (From_Master) connect Error 0 0042-404 nconn: Error connecting to SSL object. 0042-406 nconn: Error verifying SSL object after connection. 405 nconn: Error with certificate at depth .
and the client log had little information:
Thu Jul 11 13:21:11 2024 success: we got 1st write query is 0 Thu Jul 11 13:21:11 2024 success: we got 2nd write local id is 00XXXXXX00 Thu Jul 11 13:21:11 2024 success: we got 3rd write remote id is 00XXXXXX00 Thu Jul 11 13:21:11 2024 success: we got 4th write command is /usr/lpp/bos.sysmgt/nim/methods/c_nimpush "/usr/lpp/bos.sysmgt/nim/methods/c_ckspot" "-l" "-ast_applied=3" "-ast_committed=5" "-aplatform=yes" "-aname=NIMCLIENT "-alocation=/usr" Thu Jul 11 13:21:11 2024 passing OpenSSL setting of 1 Thu Jul 11 13:21:11 2024 set symbol table Thu Jul 11 13:21:11 2024 cert filename discovered: /ssl_nimsh/certs/MASTER.0 Thu Jul 11 13:21:11 2024 seed_prng Thu Jul 11 13:21:11 2024 Loading certificates.. Thu Jul 11 13:21:11 2024 Negotiating SSL_CTX object using SSLv23_method Thu Jul 11 13:21:11 2024 Loading private key file.. Thu Jul 11 13:21:11 2024 create BIO Thu Jul 11 13:21:11 2024 Error accepting SSL connection Thu Jul 11 13:21:11 2024 -Error: peer certificate: application verification failure Thu Jul 11 13:21:11 2024 Error checking SSL object after connection
After opening a ticket with IBM support, it was determined the certificates had expired. We would need to disable SSL on the NIM master, delete the old certificates, and enable SSL again. This also meant that every client LPAR had to be manually touched to deploy the new certificates. That meant disabling SSL, fixing CPUID conflicts, and enabling SSL again.
What a mess.
IBM has not documented how to use an external CA authority with NIM, nor how to generate long term certificates. That's unfortunate since no one wants to redo the entire NIM environment every year.
During testing it was discovered that the NIM clients would REWRITE the SSL libraries in /usr/lib! IBM support says that should be OK, but I can't recommend doing 'nimclient -c' on a system with a running database which depends on the AIX SSL library (ie: IRIS/EPIC). It sounds like a potential race condition waiting to happen. Check your "genkld | grep ssl" and look for ssl.so files for potential conflicts.
There is an alternate command which can fetch updated certificates from the NIM master without touching the SSL libraries. I've examined the file dates on "/usr/lib/*ssl*" to confirm before and after the command.
nimclient -o get_cert -a master_name=MASTER
This downloads the latest certificate and stages it. The next call to NIMSH will reload the updated certificate as long as the master host name doesn't change. This has promise, and I'm looking at adding a weekly cron job to do that pull in case we have to regenerate the NIM master certificate again.
This isn't documented anywhere, so I thought I'd post for other NIM and LPM customers to find.
Feature requests for IBM:
- Document how to use SSL certificates from an external CA
- Extend the duration of the default certificates, or add a command line parameter to 'nimconfig -c' where we can specify a duration
- Have NIMSH automatically fetch the latest certificates intermittently, instead of relying on a user created cron job
- Convert from NIMSH to SSH. Really guys, it isn't hard to stand up a dedicated SSHD server separate from the user one with a different port and dedicated config with static keys for NIM. I've been tempted to do this and symlink rsh to ssh.
- Never touch SSL libraries. NIM doesn't use them exclusively and blindly rewriting them while applications are online sounds like a problem waiting to happen. Alternatively make NIM use statically compiled binaries with built in SSL. At least add a warning to the documentation that SSL libraries will be modified and that could impact applications.
- Improve the errors messages due to certificate expiration.
------------------------------
========================
Russell Adams
https://adamssystems.nl/
========================
------------------------------