Testing AIX 7.3 TL03 new features and LKU part2

View Only

Testing AIX 7.3 TL03 new features and LKU part2

By Christian Sonnemans posted 5 days ago

Like

This blog is update and a sort of erratum on my last blog that I wore:

https://community.ibm.com/community/user/power/blogs/christian-sonnemans1/2025/01/30/testing-aix-73-tl03-new-features-and-lku

Extra blackout time explained:

First, I’d like to mention the differences between the blackout times measured between AIX 7.2 and AIX 7.3. I mentioned that for medium and rather small LPARS the blackout time can increase with a measured max. of 9 seconds.

For large memory systems however there should be an significant improvement in blackout time.

But now we can explain why the blackout time was increased.

Use case:

For those people who are wondering: "why bother 9 seconds?"

Imagine a heavy transaction system that services many ATM,s with hundreds / thousands transactions per second, and you have to pay a fee for every transaction you miss the income, because a third party is stand-in during the backout-time.

Extra blackout-time explained.

Thanks to excellent debugging work via a case I logged, we now can explain what was changed and also why I noticed a variation between LPARS on the same level (AIX 7.3).

First difference:

First difference between AIX 7.2 and AIX7.3 is the behavior of cpu reduction.

For the people who do not know what this feature is in LKU is will explain this.

This feature makes it possible to start up a surrogate LPAR despite there is not enough cpu available in the cpu pool (default cpu pool) or if the pool where the lpar belongs to.

Under AIX 7.2 this feature is only used when it is enabled into the liveupdate.data AND if there is a real shortage on cpu in the pool.

With AIX 7.3 the behavior is different and when it is enabled via the stanza in the liveupdate.data it always kicks in, even if there is no shortage of cpu in the pool.

This behavior gives a penalty of roughly an extra 6 seconds.

Example stanza with backout time enabled

general:

kext_check = yes

cpu_reduction = yes

Second Issue:

The second issue on AIX 7.3 TL03 was that a lot of new features and optimizations were implemented, and one of them is "shared memory dump optimization " this gives an extra blackout penalty of 1~3 seconds.

There is a plan to revise the dump optimization, but that will be done after other more necessary improvements will be made in upcoming SP’s.

By implementing the next SP and in the fall SP IBM expects further improvements that will mitigate those 3 seconds. And maybe those improvements win back even some more blackout time.

AIX IPSEC is now compatible with AIX LKU and investigation on this

I mentioned that I investigated this feature, but documentation on this was not (yet) very clear.

First test that I made were not so successful.

Due to leak of documentation and my first test I also mentioned that it probably was only developed for IPsec tunnels and not for IP-sec filtering.

This assumption is wrong!

LKU can handle both with this new stanza in the liveupdate.data ip-sec tunneling an ip-sec filtering.

For those who don’t know what ip sec ip-filtering is, I will explain shortly what it is:

IP-filtering can block and allow certain ip-adresses and address ranges with specific port numbers.

NOTE: This is not a state-full inspection firewall such as IP-tables. For more information please read this article of Chris Gibson:

https://techchannel.com/performance/filtering-ip-addresses-with-aix-ipsec/

How to use IP-sec Tunnels please read this article of Rajya Lakshmi Marathu

https://community.ibm.com/community/user/power/blogs/rajya-lakshmi-marathu/2024/11/17/create-ipsec-vpn-between-aix-linux-with-libreswan

Therefore I am delighted to explain how this feature works and that there are even more improvements in the near future as well.

How does this feature work for now with for IP-filtering:

With the AIX 7.3TL03 version the steps are now:

Step1: First requirement is that before this feature can be used during a LKU is that the ipsec_logd must be stopped.

This can be done with the following:

Stopping the ipsec_logd before we can do a live update with without unloading the ipsec_v4 device.

/usr/sbin/mkfilt -v 4 -g stop AND / OR /usr/sbin/mkfilt -v 6 -g for ipV6

NOTE:

We do not longer have to remove the device ipsec_v4 OR ipsec_v6

Step 2:

Edit the liveupdate.data general stanza with:

general:

kext_check = yes

cpu_reduction = no

ipsec_auto_migrate = yes

Step 3:

Run LKU with the extra general stanza.

Step 4:

After the LKU start the filers again and logging with:

/usr/sbin/mkfilt -v 4 -u -g start

Step 5:

Verification step (not required)

lsdev -l ipsec_v4

If not than:

/usr/sbin/mkdev -c ipsec -t 4

Check if filters exists:

# ls -l /dev/ipsec4_filt

crwx------ 1 root system 36, 0 Jan 24 08:37 /dev/ipsec4_filt

If not that recreate filters with genfilt see some example below:

usr/sbin/genfilt -n 2 -v 4 -a P -s 0.0.0.0 -m 0.0.0.0 -d 0.0.0.0 -M 0.0.0.0 -g n -c icmp -o eq -p 0 -O any -P 0 -r L -w I -l N -t 0 -i all -D echo_reply:permit:all

List filter list

/usr/sbin/lsfilt -O -v 4

So for now the only requirement is that ipsec_logd must be stopped, and the method for doing this is to unload the ip-filters and start it afterwards

/usr/sbin/mkfilt -v 4 -g stop

And

/usr/sbin/mkfilt -v 4 -u -g start

This compared with the the old < 7.3TL03 method:

Before LKU we the steps to be run:

-1 in the prep phase we now first unload the ip filters.

-2 then we remove the ipsec_v4 and ipsec_v6 devices.

Then run the LKU on the LPAR

-3 in the post phase we first have to recreate the ipsec_v4 and ipsec_v6 devices

-4 we load the ip filters again.

More good news on this that also the stop and start will automated during the LKU action in the near future. I will receive a efix when this is implemented and of course will test this.

In other words, we can remove the ip-filters in the prep phase if we use the stanza field in the liveupdate.data

For more information about LKU in combination with IP-sec Tunneling please read:

https://developer.ibm.com/blogs/awb-live-update-with-active-ipsec-configuration/

@Nayden Stoyanov(Ned): Thank you again for investigation all my findings (again) it is a real pleasure to work with you!

Endnote:

As you may noticed, I like discussions to approve the security for AIX Patch management AIX Live Update and other AIX security subjects.

IBM Power

Connect, learn, share, and engage with IBM Power.

Power