On Tue, Apr 18, 2023 at 06:55:39PM +0000, Roy ST. JOHN via IBM Community wrote:
> Thank you again for the continued feedback. Your points are much
> appreciated, and as you note, it is concerning from a security
> perspective that you are the first to raise concern regarding the
> public key only being accessible over ftp.
Roy, your response has been fantastic. I really appreciate it.
Support spent today trying to explain to me how to take a checksum.
Later after escalating the ticket they were trying to tutor me on
using Filezilla to get an FTP link. I'll be escalating again tomorrow.
However the FTP URL for the key wasn't the only concern. If you check
the HTML version of the bulletin, it lacks any reference to the key
completely.
I have a long history of confirming checksums and signatures in the
OSS world. I was part of a group that found a remote shell hack in the
00's of TCPDUMP after their site was compromised. Checksums are a
minimum to confirm good downloads, but signatures are better.
> The /etc/security/certificates/AIX_PSIRT_pubkey.txt file shipped
> starting with 7.2 TL5 SP3 (7200-05-03) and AIX bos.rte.security
> fileset level 7.2.5.100, so that is perhaps why it's not on your
> system. This public key is the same as the key available from the
> web.
That explains it. I am planning to install 7200-05-03 right now, and
trying to get all the security APARs that FLRT said it was missing.
I did dump the SSL info and fingerprints of all the other certificates
there for comparison. Those should likely be documented somewhere too.
> AIX can definitely do better to document the public key, usage, and
> verification though.
Absolutely. It shouldn't take a CATE to track these things down.
> Your feedback about the APAR information is appreciated as well, and
> I will confer with the AIX development team to see what we may be
> able to do to improve the information relayed in the security
> vulnerability APAR text.
Another point is that I spent the afternoon going to each CVE,
downloading the tarball, examining the advisory to find the right
fileset, and confirming all checksums and signatures. Yes, I'm aware
checksum is redundant if you have a signature, but I did it just for
the checkbox in the upgrade docs.
Given I'm trying to stage these ifixes on our NIM server prior to the
update, I can't easily compare filesets on a live system.
What would be really useful would be a quarterly distribution of all
AIX CVE's and fixes in a single signed announcement and tarball. I
understand they are all ifixes, but they shouldn't have to be chased
down individually.
I didn't go straight to 7200-05-05 because our vendor recommended SP3,
despite many of the fixes being included in SP4. Thus I have to get
the security patches. I'm certain it's not an uncommon scenario.
I wonder if SUMA can play a role here. "smit suma", "securely download
all current security apars" would be awesome.
Finally on my wishlist would be a feature where every LPP file
downloaded from Fix Central included a .sig file so I can validate
en-masse all of the files for a TL or SP in AIX. Today I parse the
*.pd.sdd file (intended for Download Director?) after downloading via
SFTP to confirm the checksums of each file, but a signature would be
far better. Especially if I can use a key already shipped with the OS.
Thanks.
------------------------------------------------------------------
Russell Adams
Russell.Adams@AdamsSystems.nlPrincipal Consultant Adams Systems Consultancy
https://adamssystems.nl/
Original Message:
Sent: 4/18/2023 2:56:00 PM
From: Roy ST. JOHN
Subject: RE: Obtaining security APARs, given the runaround and what a hassle
Hi Russell,
Thank you again for the continued feedback. Your points are much appreciated, and as you note, it is concerning from a security perspective that you are the first to raise concern regarding the public key only being accessible over ftp.
The /etc/security/certificates/AIX_PSIRT_pubkey.txt file shipped starting with 7.2 TL5 SP3 (7200-05-03) and AIX bos.rte.security fileset level 7.2.5.100, so that is perhaps why it's not on your system. This public key is the same as the key available from the web.
AIX can definitely do better to document the public key, usage, and verification though.
The RedHat page you linked is a good reference point, and I should be able to work on a similar document to at least cover AIX's security bulletin and fix signing process, i.e.:
- Location of the locally installed bulletin public key (starting with 7200-05-03)
- Location of the mirrored public key, now accessible via https
- Commands to verify the public key
- Commands to use the public key to verify the security bulletin and security iFixes
Your feedback about the APAR information is appreciated as well, and I will confer with the AIX development team to see what we may be able to do to improve the information relayed in the security vulnerability APAR text.
------------------------------
Roy ST. JOHN
------------------------------
Original Message:
Sent: Tue April 18, 2023 04:52 AM
From: Russell Adams
Subject: Obtaining security APARs, given the runaround and what a hassle
On Mon, Apr 17, 2023 at 09:13:28PM +0000, Roy ST. JOHN via IBM Community wrote:
> Thank you for the feedback, and I'm sorry for the difficulties
> faced.
Roy, thank you for your excellent response.
Too bad support is still trying to tell me that the hash is right
there, "why can't I just use the hash"? They don't get it should be
signed. I'll escalate that later today.
> The AIX security bulletin public key should be included on AIX systems, starting with 7.2 TL5 SP3, in the /etc/security/certificates directory:
> /etc/security/certificates/AIX_PSIRT_pubkey.txt
This is great! Where was this documented?
I think this key should be used for everything because it's implicitly
trusted as it was distributed with the OS!
> I've gone ahead and uploaded the public key used for bulletin and
> fix verification to the same directory as the AIX bulletins and
> fixes so that HTTPS may be used to pull these though. Additionally,
> I've created a new key to verify that public key. The locations for
> these are:
New non-ftp links help. However the keys and locations need to be
better documented and publicized. It was very frustrating to visit
multiple IBM Security pages and see no mention of any keys.
Is it worth making more keys? I'd rather trust the one shipped with
the OS, and perhaps you can post that with high visibility? Verifying
a security bulletin with that OS trusted key means I can confirm the
bulletin and my downloads natively on AIX.
Other vendors will use PGP keys and give not just a key file, but the
short and long fingerprints for verification. Many of those keys are
mirrored on PGP key servers, so you can pull the fingerprint and
verify across sources. I know the SSL keys aren't quite the same.
I did a search for "redhat security pgp public key" and the very first
hit is an entire page of PGP keys, their purposes, fingerprints and
more.
https://access.redhat.com/security/team/key/
IBM should be outperforming them in communicating security
information.
Or perhaps Linux vendors need to advertise more since they publish
100x CVE's and leak like a sieve. ;]
> Our APAR template text was updated around mid-2022, so current and
> future APARs should link to the appropriate My Notifications page.
The "my notifications" logic seems very poor. Please consider linking
to a permanent IBM page for the APAR or ifix instead.
I went to my subscriptions repeatedly and couldn't find these patches.
Thanks.
------------------------------------------------------------------
Russell Adams Russell.Adams@AdamsSystems.nl
Principal Consultant Adams Systems Consultancy
https://adamssystems.nl/
Original Message:
Sent: 4/17/2023 4:10:00 PM
From: Roy ST. JOHN
Subject: RE: Obtaining security APARs, given the runaround and what a hassle
Hi Russell,
Thank you for the feedback, and I'm sorry for the difficulties faced.
The AIX security bulletin public key should be included on AIX systems, starting with 7.2 TL5 SP3, in the /etc/security/certificates directory:
/etc/security/certificates/AIX_PSIRT_pubkey.txt
I've gone ahead and uploaded the public key used for bulletin and fix verification to the same directory as the AIX bulletins and fixes so that HTTPS may be used to pull these though. Additionally, I've created a new key to verify that public key. The locations for these are:
https://aix.software.ibm.com/aix/efixes/security/systems_p_os_aix_security_pubkey.txt
https://aix.software.ibm.com/aix/efixes/security/systems_p_os_aix_security_pubkey.txt.sig
https://aix.software.ibm.com/aix/efixes/security/systems_p_os_aix_security_verify.txt
The public key used for bulletin and fix verification may be verified with:
> openssl dgst -sha256 -verify systems_p_os_aix_security_verify.txt -signature systems_p_os_aix_security_pubkey.txt.sig systems_p_os_aix_security_pubkey.txt
The checksums for the public key used for bulletin and fix verification and the additional verification key are:
> openssl dgst -sha256 systems_p_os_aix_security_pubkey.txt
SHA256(systems_p_os_aix_security_pubkey.txt)= 98d1efb466c6946618b5111117a68b0cfe39b27e8718672896754faa81288d76
> openssl dgst -sha256 systems_p_os_aix_security_verify.txt
SHA256(systems_p_os_aix_security_verify.txt)= 88956e6a7c06613114b82ac913fd10a48fde090ad000249f67f704006e837572
All of this information will be provided in future AIX/VIOS security bulletins.
Our APAR template text was updated around mid-2022, so current and future APARs should link to the appropriate My Notifications page.
------------------------------
Roy ST. JOHN
Original Message:
Sent: Mon April 17, 2023 12:19 PM
From: Russell Adams
Subject: Obtaining security APARs, given the runaround and what a hassle
I can't recall the last time I was so thoroughly disappointed with IBM. I'm opening a support ticket and copying it here because whoever composes this nonsense should be thoroughly embarrassed. I can only hope someone in IBM Security reads what they are publishing.
Trying to download APAR for IJ36681 (nimsh vulnerability). I'm incredibly disappointed in IBM's security release procedure, and opening a support ticket after wasting over an hour just trying to get one fix. I have ten more to get.
https://www.ibm.com/support/pages/apar/IJ36681
The page says I can get the fix from subscription services, except that URL is a 404.
Instead I goto the ASCII version of the security advisory.
https://aix.software.ibm.com/aix/efixes/security/nimsh_advisory.asc
Looks like for my goal AIX level of 7200-05-03, I need:
https://aix.software.ibm.com/aix/efixes/security/nimsh_fix.tar
In that file I need:
7.2.5.3 IJ36681m3a.220324.epkg.Z
So now I need to verify the checksum:
98cc59b5bb5947a7f8d29ee87742ac094117844cb5b309c2b5a5d2378b727687 IJ36681m3a.220324.epkg.Z
openssl dgst -sha256 -verify [pubkey_file] -signature [ifix_file].sig [ifix_file]
But first I must validate the ASCII announcement. There is an Advisory.asc.sig file in the tarball.
What key do I verify with? From the bulletin:
ftp://ftp.software.ibm.com/systems/power/AIX/systems_p_os_aix_security_pubkey.txt
Except IBM is correctly ending it's support of unencrypted FTP, nor does my organization allow it. I can't get the key, and I wouldn't trust it if I could.
Searching IBM's site for the file "systems_p_os_aix_security_pubkey.txt" has hits on all the bulletins, but not a place to download and confirm this is an authentic key.
The other URLs to IBM SECURITY in the bulletin don't have a link to a key either.
IBM Secure Engineering Web Portal
http://www.ibm.com/security/secure-engineering/bulletins.html
IBM Product Security Incident Response Blog
https://www.ibm.com/blogs/psirt/
So where exactly is a customer to find a key, given this is being distributed outside of Fix Central and the common AIX distribution methods?
Or am I the only one verifying the download?
------------------------------
========================
Russell Adams
https://adamssystems.nl/
========================
------------------------------
EDIT: Sorry Roy, not Ron! Fixed.