AIX

AIX

Connect with fellow AIX users and experts to gain knowledge, share insights, and solve problems.

 View Only
  • 1.  NIM w/ SSL issues and potential impact to running applications

    Posted Mon August 19, 2024 12:13 PM

    I recently had a customer NIM environment start exhibiting failures for all NIM operations. We had implemented SSL for NIMSH on all clients to improve security and prevent LPM operations across systems from breaking NIM control due to CPU ID's.

    We found out the hard way that the NIM master generates custom SSL certificates in /ssl_nimsh that are only valid for one year. At the end of that time, every LPAR rejected NIM operations. That included our weekly mksysb.

    The error messages are completely obfuscated, so we could not tell that the certificate had expired. The message on the NIM master was generic:

    # nim -o lslpp NIMCLIENT
      0042-001 nim: processing error encountered on "master":
         0042-006 m_lslpp: (From_Master) connect Error 0
              0042-404 nconn: Error connecting to SSL object.
              0042-406 nconn: Error verifying SSL object after connection.
      405 nconn: Error with certificate at depth .

    and the client log had little information:

      Thu Jul 11 13:21:11 2024        success: we got 1st write query is 0
      Thu Jul 11 13:21:11 2024        success: we got 2nd write local id is 00XXXXXX00
      Thu Jul 11 13:21:11 2024        success: we got 3rd write remote id is 00XXXXXX00
      Thu Jul 11 13:21:11 2024        success: we got 4th write command is /usr/lpp/bos.sysmgt/nim/methods/c_nimpush "/usr/lpp/bos.sysmgt/nim/methods/c_ckspot" "-l" "-ast_applied=3" "-ast_committed=5" "-aplatform=yes" "-aname=NIMCLIENT "-alocation=/usr"
      Thu Jul 11 13:21:11 2024        passing OpenSSL setting of 1
      Thu Jul 11 13:21:11 2024        set symbol table
      Thu Jul 11 13:21:11 2024         cert filename discovered: /ssl_nimsh/certs/MASTER.0
      Thu Jul 11 13:21:11 2024        seed_prng
      Thu Jul 11 13:21:11 2024        Loading certificates..
      Thu Jul 11 13:21:11 2024        Negotiating SSL_CTX object using SSLv23_method
      Thu Jul 11 13:21:11 2024        Loading private key file..
      Thu Jul 11 13:21:11 2024        create BIO
      Thu Jul 11 13:21:11 2024        Error accepting SSL connection
      Thu Jul 11 13:21:11 2024        -Error: peer certificate: application verification failure
      Thu Jul 11 13:21:11 2024        Error checking SSL object after connection
    

    After opening a ticket with IBM support, it was determined the certificates had expired. We would need to disable SSL on the NIM master, delete the old certificates, and enable SSL again. This also meant that every client LPAR had to be manually touched to deploy the new certificates. That meant disabling SSL, fixing CPUID conflicts, and enabling SSL again.

    What a mess.

    IBM has not documented how to use an external CA authority with NIM, nor how to generate long term certificates. That's unfortunate since no one wants to redo the entire NIM environment every year.

    During testing it was discovered that the NIM clients would REWRITE the SSL libraries in /usr/lib! IBM support says that should be OK, but I can't recommend doing 'nimclient -c' on a system with a running database which depends on the AIX SSL library (ie: IRIS/EPIC). It sounds like a potential race condition waiting to happen. Check your "genkld | grep ssl" and look for ssl.so files for potential conflicts.

    There is an alternate command which can fetch updated certificates from the NIM master without touching the SSL libraries. I've examined the file dates on "/usr/lib/*ssl*" to confirm before and after the command.

    nimclient -o get_cert -a master_name=MASTER

    This downloads the latest certificate and stages it. The next call to NIMSH will reload the updated certificate as long as the master host name doesn't change. This has promise, and I'm looking at adding a weekly cron job to do that pull in case we have to regenerate the NIM master certificate again.

    This isn't documented anywhere, so I thought I'd post for other NIM and LPM customers to find.

    Feature requests for IBM:

     - Document how to use SSL certificates from an external CA

     - Extend the duration of the default certificates, or add a command line parameter to 'nimconfig -c' where we can specify a duration

     - Have NIMSH automatically fetch the latest certificates intermittently, instead of relying on a user created cron job

     - Convert from NIMSH to SSH. Really guys, it isn't hard to stand up a dedicated SSHD server separate from the user one with a different port and dedicated config with static keys for NIM. I've been tempted to do this and symlink rsh to ssh.

     - Never touch SSL libraries. NIM doesn't use them exclusively and blindly rewriting them while applications are online sounds like a problem waiting to happen. Alternatively make NIM use statically compiled binaries with built in SSL. At least add a warning to the documentation that SSL libraries will be modified and that could impact applications.

     - Improve the errors messages due to certificate expiration.



    ------------------------------
    ========================
    Russell Adams
    https://adamssystems.nl/
    ========================
    ------------------------------


  • 2.  RE: NIM w/ SSL issues and potential impact to running applications

    Posted Tue August 20, 2024 02:13 AM
    Edited by Chris Gibson Tue August 20, 2024 04:05 AM

    Thanks for sharing Russell. This is really helpful!!

    I recall that during initial setup of SSL for the NIM server, you could extend the life of the SSL certs by modifying the sample config files and then running nimconfig -c. This was often overlooked (undocumented). e.g.

    root@nim1 /usr/samples/nim/ssl # grep default_days *

    SSL_client.cnf:default_days     = 365

    SSL_root.cnf:default_days     = 365

    SSL_server.cnf:default_days     = 365

    However, the solution you've shared here is far, far better.

    I note that the command, nimclient -o get_cert, is briefly mentioned here, https://www.ibm.com/docs/en/aix/7.3?topic=authentication-enabling-cryptographic-from-command-line, but in the context of communicating with an alternate NIM server.

    Thanks again for this useful tip, I'll be sure to share it with others!



    ------------------------------
    Chris Gibson
    ------------------------------



  • 3.  RE: NIM w/ SSL issues and potential impact to running applications

    Posted Tue August 20, 2024 11:18 AM

    Thanks Russell and Chris for your posts on this topic. I will ensure it gets awareness in development and we will see where we can make improvements in future updates. It sounds like improved documentation would be a good start.



    ------------------------------
    Carl Burnett
    DE, IBM Infrastructure, IBM Power
    ------------------------------



  • 4.  RE: NIM w/ SSL issues and potential impact to running applications

    Posted Tue August 20, 2024 07:43 PM

    Thanks Carl!



    ------------------------------
    Chris Gibson
    ------------------------------



  • 5.  RE: NIM w/ SSL issues and potential impact to running applications

    Posted Tue August 20, 2024 12:46 PM
    # pwd
    /usr/samples/nim/ssl
    # pwd
    /usr/samples/nim/ssl
    #  cat SSL_Makefile.mk | grep CERTFILES_EXPIRE
    CERTFILES_EXPIRE = -days 3650  =====> Default value 365 , you can change this value , so that we can increase certificate expiry days 
     
     
    client 
    #  date 1030010124
    Wed Oct 30 01:01:54 CDT 2024
     
    Master
    #  nim -o lslpp  idevp8-lp14  | grep bos.rte
      bos.rte                    7.3.3.0  COMMITTED  Base Operating System Runtime
      bos.rte.Dt                 7.3.0.0  COMMITTED  Desktop Integrator
      bos.rte.ILS                7.3.3.0  COMMITTED  International Language Support
      bos.rte.SRC                7.3.3.0  COMMITTED  System Resource Controller
      
    client 
    #  date 0930010132
    Thu Sep 30 01:01:14 CDT 2032
     
    Master
    #  nim -o lslpp  idevp8-lp14  | grep bos.rte
      bos.rte                    7.3.3.0  COMMITTED  Base Operating System Runtime
      bos.rte.Dt                 7.3.0.0  COMMITTED  Desktop Integrator
      bos.rte.ILS                7.3.3.0  COMMITTED  International Language Support
      bos.rte.SRC 
      
    client:
    #  date 0930010134
    Sat Sep 30 01:01:37 CDT 2034
     
    Master 
    #  nim -o lslpp  idevp8-lp14  | grep bos.rte
    0042-001 nim: processing error encountered on "master":
       0042-006 m_lslpp: (From_Master) connect Error 0
     
    nconn: SSL shutdown failed.405 nconn: Error with certificate at depth .
     
    Client : 
    #   date 0930010132
    Thu Sep 30 01:01:48 CDT 2032
    #
     
    Master 
    #  nim -o lslpp  idevp8-lp14  | grep bos.rte
      bos.rte                    7.3.3.0  COMMITTED  Base Operating System Runtime
      bos.rte.Dt                 7.3.0.0  COMMITTED  Desktop Integrator
      bos.rte.ILS                7.3.3.0  COMMITTED  International Language Support
      bos.rte.SRC                7.3.3.0  COMMITTED  System Resource Controller
      bos.rte.X11                7.3.0.0  COMMITTED  AIXwindows Device Support



    ------------------------------
    Srikanth Thanneeru
    ------------------------------



  • 6.  RE: NIM w/ SSL issues and potential impact to running applications

    Posted Tue August 20, 2024 12:49 PM

    Regarding displaying "certificate has expired" Already handled in 73F, Will consider enable this feature "Extend the duration of the default certificates, or add a command line parameter to 'nimconfig -c' where we can specify a duration" subsequent release

    nim -o lslpp aix-install-zep01-lp015
    + grep bos.rte
    0042-001 nim: processing error encountered on "master":
       0042-006 m_lslpp: (From_Master) connect Error 0
     
            0042-535 nconn: The certificate has expired.
            0042-406 nconn: Error verifying SSL object after connection.
    nconn: SSL shutdown failed.405 nconn: Error with certificate at depth .


    ------------------------------
    Srikanth Thanneeru
    ------------------------------



  • 7.  RE: NIM w/ SSL issues and potential impact to running applications

    Posted Tue August 20, 2024 02:19 PM
    On Tue, Aug 20, 2024 at 04:49:19PM +0000, Srikanth Thanneeru via IBM TechXchange Community wrote:
    > Regarding displaying "certificate has expired" Already handled in
    > 73F,
    >
    > nim -o lslpp aix-install-zep01-lp015
    > + grep bos.rte
    > 0042-001 nim: processing error encountered on "master":
    > 0042-006 m_lslpp: (From_Master) connect Error 0
    >
    > 0042-535 nconn: The certificate has expired.
    > 0042-406 nconn: Error verifying SSL object after connection.
    > nconn: SSL shutdown failed.405 nconn: Error with certificate at depth .

    That's pretty slick!

    > Will consider enable this feature "Extend the duration of the
    > default certificates, or add a command line parameter to 'nimconfig
    > -c' where we can specify a duration" subsequent release

    So in my support ticket, I was advised not to adjust the duration in
    the file. A command line option would certainly help. It would also be
    good to print a message with the expiration of the certificate being
    created, so an administrator may realize they need to change the
    duration.

    I know if I ran the nimconfig command, and I saw a message which told
    me the certificate was only valid for year, I would stop and
    doublecheck.

    Thanks.

    ------------------------------------------------------------------
    Russell Adams Russell.Adams@AdamsSystems.nl
    Principal Consultant Adams Systems Consultancy
    https://adamssystems.nl/




  • 8.  RE: NIM w/ SSL issues and potential impact to running applications

    Posted Tue August 20, 2024 02:56 PM

    Hello Russell,

    Thank you for your valuable posting.  I was just thinking about implementing NIM certificates and this saved me a lot of trouble.

    And I agree with your Feature requests for IBM, except (perhaps) with converting NIMSH.



    ------------------------------
    F. Alejandro Osso
    IntegraWare Informática
    ------------------------------