AIX

 View Only
Expand all | Collapse all

Obtaining security APARs, given the runaround and what a hassle

  • 1.  Obtaining security APARs, given the runaround and what a hassle

    Posted Mon April 17, 2023 12:19 PM

    I can't recall the last time I was so thoroughly disappointed with IBM. I'm opening a support ticket and copying it here because whoever composes this nonsense should be thoroughly embarrassed. I can only hope someone in IBM Security reads what they are publishing.



    Trying to download APAR for IJ36681 (nimsh vulnerability). I'm incredibly disappointed in IBM's security release procedure, and opening a support ticket after wasting over an hour just trying to get one fix. I have ten more to get.

    https://www.ibm.com/support/pages/apar/IJ36681

    The page says I can get the fix from subscription services, except that URL is a 404.

    Instead I goto the ASCII version of the security advisory.

    https://aix.software.ibm.com/aix/efixes/security/nimsh_advisory.asc

    Looks like for my goal AIX level of 7200-05-03, I need:

    https://aix.software.ibm.com/aix/efixes/security/nimsh_fix.tar

    In that file I need:

    7.2.5.3    IJ36681m3a.220324.epkg.Z  

    So now I need to verify the checksum:

    98cc59b5bb5947a7f8d29ee87742ac094117844cb5b309c2b5a5d2378b727687  IJ36681m3a.220324.epkg.Z

    openssl dgst -sha256 -verify [pubkey_file] -signature [ifix_file].sig [ifix_file]

    But first I must validate the ASCII announcement. There is an Advisory.asc.sig file in the tarball.

    What key do I verify with? From the bulletin:

    ftp://ftp.software.ibm.com/systems/power/AIX/systems_p_os_aix_security_pubkey.txt

    Except IBM is correctly ending it's support of unencrypted FTP, nor does my organization allow it. I can't get the key, and I wouldn't trust it if I could.

    Searching IBM's site for the file "systems_p_os_aix_security_pubkey.txt" has hits on all the bulletins, but not a place to download and confirm this is an authentic key.

    The other URLs to IBM SECURITY in the bulletin don't have a link to a key either.

    IBM Secure Engineering Web Portal
            http://www.ibm.com/security/secure-engineering/bulletins.html

    IBM Product Security Incident Response Blog
            https://www.ibm.com/blogs/psirt/

    So where exactly is a customer to find a key, given this is being distributed outside of Fix Central and the common AIX distribution methods?

    Or am I the only one verifying the download?



    ------------------------------
    ========================
    Russell Adams
    https://adamssystems.nl/
    ========================
    ------------------------------


  • 2.  RE: Obtaining security APARs, given the runaround and what a hassle

    Posted Mon April 17, 2023 05:13 PM

    Hi Russell,

    Thank you for the feedback, and I'm sorry for the difficulties faced.

    The AIX security bulletin public key should be included on AIX systems, starting with 7.2 TL5 SP3, in the /etc/security/certificates directory:

    /etc/security/certificates/AIX_PSIRT_pubkey.txt


    I've gone ahead and uploaded the public key used for bulletin and fix verification to the same directory as the AIX bulletins and fixes so that HTTPS may be used to pull these though. Additionally, I've created a new key to verify that public key. The locations for these are:

    https://aix.software.ibm.com/aix/efixes/security/systems_p_os_aix_security_pubkey.txt
    https://aix.software.ibm.com/aix/efixes/security/systems_p_os_aix_security_pubkey.txt.sig
    https://aix.software.ibm.com/aix/efixes/security/systems_p_os_aix_security_verify.txt

    The public key used for bulletin and fix verification may be verified with:

    > openssl dgst -sha256 -verify systems_p_os_aix_security_verify.txt -signature systems_p_os_aix_security_pubkey.txt.sig systems_p_os_aix_security_pubkey.txt

    The checksums for the public key used for bulletin and fix verification and the additional verification key are:

    > openssl dgst -sha256 systems_p_os_aix_security_pubkey.txt
    SHA256(systems_p_os_aix_security_pubkey.txt)= 98d1efb466c6946618b5111117a68b0cfe39b27e8718672896754faa81288d76

    > openssl dgst -sha256 systems_p_os_aix_security_verify.txt
    SHA256(systems_p_os_aix_security_verify.txt)= 88956e6a7c06613114b82ac913fd10a48fde090ad000249f67f704006e837572


    All of this information will be provided in future AIX/VIOS security bulletins.

    Our APAR template text was updated around mid-2022, so current and future APARs should link to the appropriate My Notifications page.



    ------------------------------
    Roy ST. JOHN
    ------------------------------



  • 3.  RE: Obtaining security APARs, given the runaround and what a hassle

    Posted Tue April 18, 2023 04:53 AM
    Edited by Russell Adams Tue April 18, 2023 08:09 AM

    On Mon, Apr 17, 2023 at 09:13:28PM +0000, Roy ST. JOHN via IBM Community wrote:
    > Thank you for the feedback, and I'm sorry for the difficulties
    > faced.

    Roy, thank you for your excellent response.

    Too bad support is still trying to tell me that the hash is right
    there, "why can't I just use the hash"? They don't get it should be
    signed. I'll escalate that later today.

    > The AIX security bulletin public key should be included on AIX systems, starting with 7.2 TL5 SP3, in the /etc/security/certificates directory:
    > /etc/security/certificates/AIX_PSIRT_pubkey.txt

    This is great! Where was this documented?

    I think this key should be used for everything because it's implicitly
    trusted as it was distributed with the OS!

    > I've gone ahead and uploaded the public key used for bulletin and
    > fix verification to the same directory as the AIX bulletins and
    > fixes so that HTTPS may be used to pull these though. Additionally,
    > I've created a new key to verify that public key. The locations for
    > these are:

    New non-ftp links help. However the keys and locations need to be
    better documented and publicized. It was very frustrating to visit
    multiple IBM Security pages and see no mention of any keys.

    Is it worth making more keys? I'd rather trust the one shipped with
    the OS, and perhaps you can post that with high visibility? Verifying
    a security bulletin with that OS trusted key means I can confirm the
    bulletin and my downloads natively on AIX.

    Other vendors will use PGP keys and give not just a key file, but the
    short and long fingerprints for verification. Many of those keys are
    mirrored on PGP key servers, so you can pull the fingerprint and
    verify across sources. I know the SSL keys aren't quite the same.

    I did a search for "redhat security pgp public key" and the very first
    hit is an entire page of PGP keys, their purposes, fingerprints and
    more.

    https://access.redhat.com/security/team/key/

    IBM should be outperforming them in communicating security
    information.

    Or perhaps Linux vendors need to advertise more since they publish
    100x CVE's and leak like a sieve. ;]

    > Our APAR template text was updated around mid-2022, so current and
    > future APARs should link to the appropriate My Notifications page.

    The "my notifications" logic seems very poor. Please consider linking
    to a permanent IBM page for the APAR or ifix instead.

    I went to my subscriptions repeatedly and couldn't find these patches.

    Thanks.


    ------------------------------------------------------------------
    Russell Adams Russell.Adams@AdamsSystems.nl
    Principal Consultant Adams Systems Consultancy
    https://adamssystems.nl/




  • 4.  RE: Obtaining security APARs, given the runaround and what a hassle

    Posted Tue April 18, 2023 08:17 AM

    Roy,

    I don't see /etc/security/certificates/AIX_PSIRT_pubkey.txt

    root@nim2:/etc/security/certificates=> oslevel -s
    7200-05-01-2038

    root@nim2:/etc/security/certificates=> ls -l
    total 40
    -rw-r-----    1 root     security        571 Jan 04 2021  certificate_61
    -rw-r-----    1 root     security        571 Jan 04 2021  certificate_610
    -rw-r-----    1 root     security        571 Jan 04 2021  certificate_71
    -rw-r-----    1 root     security        846 Jan 04 2021  certificate_72
    -r-xr-xr-x    1 root     system          933 Sep 23 2020  certificate_rsct_3.2
    drwxr-x---    2 root     security        256 Dec 09 2016  tnc

    Is that a 7.3 addition?



    ------------------------------
    ========================
    Russell Adams
    https://adamssystems.nl/
    ========================
    ------------------------------



  • 5.  RE: Obtaining security APARs, given the runaround and what a hassle

    Posted Tue April 18, 2023 02:56 PM

    Hi Russell,

    Thank you again for the continued feedback. Your points are much appreciated, and as you note, it is concerning from a security perspective that you are the first to raise concern regarding the public key only being accessible over ftp.

    The /etc/security/certificates/AIX_PSIRT_pubkey.txt file shipped starting with 7.2 TL5 SP3 (7200-05-03) and AIX bos.rte.security fileset level 7.2.5.100, so that is perhaps why it's not on your system. This public key is the same as the key available from the web.

    AIX can definitely do better to document the public key, usage, and verification though.

    The RedHat page you linked is a good reference point, and I should be able to work on a similar document to at least cover AIX's security bulletin and fix signing process, i.e.:

    - Location of the locally installed bulletin public key (starting with 7200-05-03)
    - Location of the mirrored public key, now accessible via https
    - Commands to verify the public key
    - Commands to use the public key to verify the security bulletin and security iFixes


    Your feedback about the APAR information is appreciated as well, and I will confer with the AIX development team to see what we may be able to do to improve the information relayed in the security vulnerability APAR text.



    ------------------------------
    Roy ST. JOHN
    ------------------------------



  • 6.  RE: Obtaining security APARs, given the runaround and what a hassle

    Posted Tue April 18, 2023 03:54 PM
    On Tue, Apr 18, 2023 at 06:55:39PM +0000, Roy ST. JOHN via IBM Community wrote:
    > Thank you again for the continued feedback. Your points are much
    > appreciated, and as you note, it is concerning from a security
    > perspective that you are the first to raise concern regarding the
    > public key only being accessible over ftp.

    Roy, your response has been fantastic. I really appreciate it.

    Support spent today trying to explain to me how to take a checksum.
    Later after escalating the ticket they were trying to tutor me on
    using Filezilla to get an FTP link. I'll be escalating again tomorrow.

    However the FTP URL for the key wasn't the only concern. If you check
    the HTML version of the bulletin, it lacks any reference to the key
    completely.

    I have a long history of confirming checksums and signatures in the
    OSS world. I was part of a group that found a remote shell hack in the
    00's of TCPDUMP after their site was compromised. Checksums are a
    minimum to confirm good downloads, but signatures are better.

    > The /etc/security/certificates/AIX_PSIRT_pubkey.txt file shipped
    > starting with 7.2 TL5 SP3 (7200-05-03) and AIX bos.rte.security
    > fileset level 7.2.5.100, so that is perhaps why it's not on your
    > system. This public key is the same as the key available from the
    > web.

    That explains it. I am planning to install 7200-05-03 right now, and
    trying to get all the security APARs that FLRT said it was missing.

    I did dump the SSL info and fingerprints of all the other certificates
    there for comparison. Those should likely be documented somewhere too.

    > AIX can definitely do better to document the public key, usage, and
    > verification though.

    Absolutely. It shouldn't take a CATE to track these things down.

    > Your feedback about the APAR information is appreciated as well, and
    > I will confer with the AIX development team to see what we may be
    > able to do to improve the information relayed in the security
    > vulnerability APAR text.

    Another point is that I spent the afternoon going to each CVE,
    downloading the tarball, examining the advisory to find the right
    fileset, and confirming all checksums and signatures. Yes, I'm aware
    checksum is redundant if you have a signature, but I did it just for
    the checkbox in the upgrade docs.

    Given I'm trying to stage these ifixes on our NIM server prior to the
    update, I can't easily compare filesets on a live system.

    What would be really useful would be a quarterly distribution of all
    AIX CVE's and fixes in a single signed announcement and tarball. I
    understand they are all ifixes, but they shouldn't have to be chased
    down individually.

    I didn't go straight to 7200-05-05 because our vendor recommended SP3,
    despite many of the fixes being included in SP4. Thus I have to get
    the security patches. I'm certain it's not an uncommon scenario.

    I wonder if SUMA can play a role here. "smit suma", "securely download
    all current security apars" would be awesome.

    Finally on my wishlist would be a feature where every LPP file
    downloaded from Fix Central included a .sig file so I can validate
    en-masse all of the files for a TL or SP in AIX. Today I parse the
    *.pd.sdd file (intended for Download Director?) after downloading via
    SFTP to confirm the checksums of each file, but a signature would be
    far better. Especially if I can use a key already shipped with the OS.

    Thanks.

    ------------------------------------------------------------------
    Russell Adams Russell.Adams@AdamsSystems.nl
    Principal Consultant Adams Systems Consultancy
    https://adamssystems.nl/




  • 7.  RE: Obtaining security APARs, given the runaround and what a hassle

    Posted Fri April 21, 2023 10:24 AM
    On Tue, Apr 18, 2023 at 07:54:24PM +0000, Russell Adams via IBM Community wrote:
    > On Tue, Apr 18, 2023 at 06:55:39PM +0000, Roy ST. JOHN via IBM Community wrote:
    > > Thank you again for the continued feedback. Your points are much
    > > appreciated, and as you note, it is concerning from a security
    > > perspective that you are the first to raise concern regarding the
    > > public key only being accessible over ftp.
    >
    > Roy, your response has been fantastic. I really appreciate it.

    Just have to give credit where credit is due. Roy has taken ownership
    of my ticket, and is diligently trying to address these security
    complaints and more.

    He's clearly an asset to IBM with a clear understanding of security
    issues, and I really appreciate his rapid response.

    I look forward to overall improvements in IBM's distribution of
    security updates under his guidance.

    ------------------------------------------------------------------
    Russell Adams Russell.Adams@AdamsSystems.nl
    Principal Consultant Adams Systems Consultancy
    https://adamssystems.nl/




  • 8.  RE: Obtaining security APARs, given the runaround and what a hassle

    Posted Tue April 18, 2023 04:55 AM
    On Mon, Apr 17, 2023 at 09:13:28PM +0000, Roy ST. JOHN via IBM Community wrote:
    > Thank you for the feedback, and I'm sorry for the difficulties faced.

    Again, thanks for the detailed reply.

    It occurred to me: am I the only one verifying the authenticity of the
    bulletins?

    It seems no one else raised the issue of the missing key, and support
    is clueless.

    Food for thought. Maybe the bulletin should emphasize that the
    signature needs to be checked prior to download and checksumming.


    ------------------------------------------------------------------
    Russell Adams Russell.Adams@AdamsSystems.nl
    Principal Consultant Adams Systems Consultancy
    https://adamssystems.nl/




  • 9.  RE: Obtaining security APARs, given the runaround and what a hassle

    Posted Thu June 08, 2023 04:04 AM

    No you are not.

    What I'm doing is mirroring "https://aix.software.ibm.com/aix/efixes/" to a NIM server using one shell script and the another script verifies advisory signatures and then it verifies .epkg.Z file checksums, before it tries to place the .epkg.Z file to appropriate directory structure. I have to manually exclude superseded .emgr.Z files. I'm storing this information is stored in a separate file..

    Parsing the advisories is not very straightforward as there some inconsistencies how versions are noted in the advisories, but IMHO I can improve my shell script.

    Then there are advisories related to OpenSSH, OpenSSL, Java, perl and python which basically say that go download new bff package/bff fileset from https://www-01.ibm.com/marketing/iwm/iwm/web/pickUrxNew.do 

    So if one likes to keep AIX as up to date as possible, one has to get updates from four to five different sources and then use two to three different ways of installing these updates.

    1. get Techology level and service packs from Fix Central
    2. get ifixes/efixes aka. .epkg.Z files from https://aix.software.ibm.com/aix/efixes/
    3. Separate installlp installable packages from https://www-01.ibm.com/marketing/iwm/iwm/web/pickUrxNew.do 
    4. Java 8 updates from Fix Central
    5. If there are RPM packages installed from IBM AIX toolbox, in order to update those using dnf, requires internet access or mirroring https://aix.software.ibm.com/aix/freeSoftware/aixtoolbox/ to a local server and setting up local repository

    So yes, IMHO this could and should be simpler.

    Br, Esa



    ------------------------------
    Esa Kärkkäinen
    ------------------------------



  • 10.  RE: Obtaining security APARs, given the runaround and what a hassle

    Posted Tue April 18, 2023 09:19 AM

    Life is very simple.
    I go to this url
    Index of /aix/efixes/security

    Ibm remove preview
    Index of /aix/efixes/security
    View this on Ibm >



    then based on date - down load tar file and read me file.
    look at my oslevel -s and match up my oslevel to e fix first - download tar file put it on server, untar the file  and install the e fix needed either for AIX 7.2 or AIX 7.1



    ------------------------------
    minesh patel
    ------------------------------



  • 11.  RE: Obtaining security APARs, given the runaround and what a hassle

    Posted Tue April 18, 2023 09:50 AM

    Where is that URL published?

    You still have to confirm the signatures on any advisory and tarball, and that key isn't well published either.



    ------------------------------
    ========================
    Russell Adams
    https://adamssystems.nl/
    ========================
    ------------------------------



  • 12.  RE: Obtaining security APARs, given the runaround and what a hassle

    Posted Tue April 18, 2023 10:01 AM

    https://www.ibm.com/support/pages/security-bulletin-aix-vulnerable-arbitrary-code-execution-due-libxml2-cve-2022-40303-and-cve-2022-40304

    The above URL give all the detail and has ssh key listed there that you can reference and match with what you get.



    ------------------------------
    minesh patel
    ------------------------------



  • 13.  RE: Obtaining security APARs, given the runaround and what a hassle

    Posted Tue April 18, 2023 10:24 AM
    On Tue, Apr 18, 2023 at 02:01:24PM +0000, minesh patel via IBM Community wrote:
    > https://www.ibm.com/support/pages/security-bulletin-aix-vulnerable-arbitrary-code-execution-due-libxml2-cve-2022-40303-and-cve-2022-40304
    >
    > The above URL give all the detail and has ssh key listed there that
    > you can reference and match with what you get.

    On that page is gives a link to a .sig file which is the signature for
    that bulletin, but where is the public key?

    That's the point everyone is missing: where is the KEY?

    I have to have a copy of IBM's public key that they sign security
    releases with so that I can verify the signature file for the page.

    Only Roy has understood this. I'm grateful he's making updates.

    ------------------------------------------------------------------
    Russell Adams Russell.Adams@AdamsSystems.nl
    Principal Consultant Adams Systems Consultancy
    https://adamssystems.nl/




  • 14.  RE: Obtaining security APARs, given the runaround and what a hassle

    Posted Mon June 05, 2023 02:42 PM

    IBM has opened an "ideas" for security updates. I would encourage everyone to help upvote improvements to LPP to address security issues.

    https://ibm-power-systems.ideas.ibm.com/ideas/AIX-I-687



    ------------------------------
    ========================
    Russell Adams
    https://adamssystems.nl/
    ========================
    ------------------------------



  • 15.  RE: Obtaining security APARs, given the runaround and what a hassle

    Posted Tue June 06, 2023 08:32 AM

    I agree with you 100%  This hack IBM uses really sucks and it is no wonder AIX is dying..  

    But to answer your question, no we are not verifying the key.. we actually had a home grown setup that would pull all the efixes down for what our servers needed but was also ha hack which required the info to be in the advisory file.. since patching to  72_TL5_FP6_2320  our process no longer works very well  as the file has not been upated and they also ship some thing as rpms and not efixes.. This process of providing security fixes need to be updated to the  work in the current world, it is no longer 1999..  This needs to work as well as Fix packs  and need to be built in a way it can be automated.



    ------------------------------
    Douglas Probst
    ------------------------------



  • 16.  RE: Obtaining security APARs, given the runaround and what a hassle

    Posted Tue June 06, 2023 08:36 AM
    On that topic, I wonder if SUMA validates downloads.

    ------------------------------------------------------------------
    Russell Adams Russell.Adams@AdamsSystems.nl
    Principal Consultant Adams Systems Consultancy
    https://adamssystems.nl/