01/29/2025-09:09:09 The live update preview succeeded.
Non-interruptable live update operation begins in 10 seconds.
01/29/2025-09:09:19 Live AIX update is starting.
01/29/2025-09:09:29 Initializing live update on original LPAR.
01/29/2025-09:09:29 Validating original LPAR environment.
01/29/2025-09:09:29 Beginning live update operation on original LPAR.
01/29/2025-09:09:39 Requesting resources required for live update.
01/29/2025-09:10:59 Notifying applications of impending live update.
01/29/2025-09:10:59 Creating rootvg for boot of surrogate.
01/29/2025-09:10:59 Starting alt_disk_copy.
01/29/2025-09:13:19 Completed alt_disk_copy.
01/29/2025-09:13:19 Rootvg for the surrogate is ready.
01/29/2025-09:13:19 Starting the surrogate LPAR.
01/29/2025-09:13:19 Surrogate AIX boot started.
01/29/2025-09:14:29 Surrogate AIX reboot started.
01/29/2025-09:15:49 Surrogate LPAR AIX is running.
01/29/2025-09:15:49 Creating mirror of original LPAR's rootvg.
01/29/2025-09:16:39 Original rootvg mirror is active.
01/29/2025-09:16:39 Moving workload to surrogate LPAR.
01/29/2025-09:17:16 Blackout Time started.
01/29/2025-09:17:37 Blackout Time end.
01/29/2025-09:17:37 Workload is running on surrogate LPAR.
01/29/2025-09:17:37 Completing transfer of system resources from the original to the surrogate LPAR.
01/29/2025-09:18:17 Starting cleanup of the original LPAR.
01/29/2025-09:18:57 Shutting down the Original LPAR.
01/29/2025-09:19:07 Deleting the original LPAR.
01/29/2025-09:20:17 Live AIX update completed in 0h 10m 58s.
File /etc/inittab has been modified.
One or more of the files listed in /etc/check_config.files have changed.
See /var/adm/ras/config.diff for details.
Dynamically modifying system resources using LKU with HMC profile
Remember why is this feature one of the most wanted items?
Use case: Did you ever run into the situation than your LPAR was sized wrong, and you hit the limits of you maximum shared processing units and maximum virtual processing units?
Or for memory maximum memory settings?
The only way to solve this problem is now shutdown your LPAR, modify the profile of your LPAR with new maximum / minimum settings. And then start you LPAR again with this new (modified saved) profile settings.
This is of course very annoying, shutting down you application or database server for just applying a new profile.
Short recap see also verry detailed blog of Chris:
-1 turn off automatic profile sync on HMC.
-2 change the profile max virtual processing units and or virtual processing units.
And or change the max memory size, and or max CPU usage (maximum virtual processor / maximum shared processing unit).
-3 you can verify on the running lpar what are the current values:
Example for max Vritual CPUs:
lparstat -i |grep Max Virtual CPUs
Maximum Virtual CPUs :8
Change the profile via HMC see example below:
-3 save the profile (profile sync is still off now).
-4 in the liveupdate.data (nim resource or local on lpar) a new stanza field is introduced:
Under the stanza hmc: add the line clone_from_hmc_profile = yes
See example of complete liveupdate.data below:
general:
kext_check = yes
cpu_reduction = no
hmc:
lpar_id =
management_console = hmc_system
user = liveupdate
clone_from_hmc_profile = yes
# trace:
# trc_option = -anl -T 20M -L 40M
disks:
nhdisk = hdisk1
mhdisk = hdisk2
tohdisk =
tshdisk =
-5 run a normal LKU operation, initiated by NIM or manual geninstall -k or Ansible.
-6 After a successful LKU operation remove the clone_from_hmc_profile = yes from the liveupdate.data file local on LPAR or nim resource.
-7 enable the profile sync again after operation is finished.
-8 after the LU operation
You can verify again with lparstat -i if your max value is changed.
lparstat -i |grep Max Virtual CPUs
Maximum Virtual CPUs :10
Pitfall: Note if you use cpu_reduction in the liveupdate.data
cpu_reduction = yes
You will have to change this to NO before allowing changing the max processing units or virtual cpu’s!
Cpu reduction and clone_from_hmc_profile are exclusive! Only one of them can have value yes.
Note: a complete template for TL03 (Chris mentioned this also) can be find at:
/var/adm/ras/liveupdate/lvupdate.template
Live Library Update (LLU)
Both Chris and I tested this Tech Preview (for test environments only) of this very new feature of TL03.
It is no coincidence that we both noticed this a very useful enhancement.
And we both liked to test with it. Why is this an potential important enhancement?
Again for those who attended my session at the TechXchange last year, I explained in my speakers session, that we use four phases for making LU successful pre-stage prep-stage lu-stage and post-stage.
Short recap: pre-stage:
check prerequisites, right resources: but nothing is stopped.
Prep stage: stop necessary processes, devices (IPSEC ip-filtering), TE, but not vital functions for application.
LU stage: actual LU operation on our patient with a short freeze moment.
Post-stage: start processes stopped in the prep stage. But also doing aftercare such as restarting things that are still attached to the old kernel (genld -u)
Man page: genld: -u Lists only processes that have old versions of loaded objects.
Therefore this new technology LLU triggered me to dig into this! With this new technology it will be possible to do a scan of those processes and do a LLU operation Live Library Update for those processes.
One example for this is the sshd daemon, we now do a restart for those programs in the post stage.
See also part of the documentation:
The online documentation states that the Live Library Update (LLU) function “eliminates downtime for workloads when the AIX operating system is updated. The AIX updates include both the kernel and user-space updates. A kernel update requires the logical partition to be rebooted or a Live Kernel Update (LKU). The LLU function shifts the applications from using the old library to the updated new library without any downtime. A library is an entity that provides a set of variables and functions to be used by a program. A library can be an archive or a shared object file. On the AIX operating system, archives can contain both static object files and shared object files as members. In the LLU context, a library denotes a shared object file that is contained in an archive. The LLU function requires the library to be built as a split library. A library is called split (or LLU-capable) when the shared object file of the library is divided into two separate entities
Important to know: there are now only two libraries available, but very important ones that are LLU-capable:
libc.a and libpthreads.a.
Considering the first such library is at TL3, the first possible chance to really test LLU would be when a iFix or sp is released which updates one of those two libraries.
See also the tests of Chris, he fakes by copying the library first to another place and the copy it over the original. But actually the library stay the same.
System wide LLU enablement
From the documentation:
During an AIX operating system update operation, the system loader considers the following set of criteria to determines whether the process is LLU-capable or not:
The value of the llu_mode tunable parameter. The llu_mode tunable parameter is a system-wide tunable parameter that you can set by using the following commands:
raso -o llu_mode=0
The LLU function is disabled for all processes regardless of the LLU program attributes.
raso -o llu_mode=1
The LLU function is enabled for processes on which the LLU program attributes are enabled.
The value of the LLU program attribute. If the llu_mode tunable parameter is set to 1, the LLU function is enabled by default for all processes unless explicitly overridden by using the LLU program attribute.
The LLU function can be overridden by using one of the following commands or methods:
ld -o program main.o -lc /lib/crt0.o -b {llu|nollu}
The llu option enables the LLU function and the nollu option disables the LLU function.
ldedit -b {llu|nollu} program
The llu option enables the LLU function and the nollu option disables the LLU function.
LDR_CNTRL=LLU={yes|no}
The LDR_CNTRL environment variable takes precedence over the LLU program attribute.
raso -o llu_mode=2
Thanks Ned for explaining the following:
After "# raso -o llu_mode=1" is enabled, each binary must be restarted to pick up the LLU mode and be LLU capable.
You can do # raso -po llu_mode=1 so that the change persists across reboots.
# ldedit -b {llu|nollu} program , can by used to enable LLU for specific binaries , but it is not be required if llu_mode=1 and the binary is already restarted.
Future plan of IBM:
As for the split libraries, the current plan is to continue building AIX base libraries to handle common workloads like Oracle and DB2.
Running and testing it manually:
For now after set the system wide parameter: raso -po llu_mode=1
As explained above, you cannot really test the two available libraries, but you can use the method Chris describes:
A real test with LLU can be done when sp1 is available, and if both the libc.a and libpthreads.a. have new versions.
WARNING: this is still a tech preview, so do not use this feature on production systems!
See below the method Chris uses:
# mkdir -p /llu
# cp -p /usr/ccs/lib/libc.a /llu/
# ls -tlr /llu
total 31672
-r-xr-xr-x 1 bin bin 16208302 Dec 15 17:00 libc.a
llvupdate -P
llvupdate preview
No process requires a Live library Update operation.
cp -p -f /llu/libc.a /usr/ccs/lib/libc.a
llvupdate -P
llvupdate preview
An LLU-capable library is newer for process 1.
Library needs to be updated /usr/lib/libc.a(_shr.o)
Validating new module /usr/lib/libc.a(_shr.o)
An LLU-capable library is newer for process 4391388.
Library needs to be updated /usr/lib/libc.a(_shr_64.o)
Validating new module /usr/lib/libc.a(_shr_64.o)
An LLU-capable library is newer for process 4784408.
Library needs to be updated /usr/lib/libc.a(_shr_64.o)
An LLU-capable library is newer for process 5177620.
Library needs to be updated /usr/lib/libc.a(_shr.o)
...etc...
An LLU-capable library is newer for process 14025142.
Library needs to be updated /usr/lib/libc.a(_shr.o)
You can now run llupdate manually after this “fake update of the libc.a” and it will try to shift to the “new” updated library.
llvupdate -a -n2 -t60
this command does a complete scan and tries it 2 times with a timeout of 60 seconds.
Also Chris mentioned that he run into two “problems” one pfcdaemon and IBM.DRMd
The first one sounds very familiar for me, in the first blog post I made AIX “AIX Live Update Cookbook” I already mentioned this deamon. This pfcdeamon is a AIX flash cache demon. Remember the four phases, in the prep phase we now stop this daemon. Therefore I did not run into this problem.
For the IBM.DRMd IBM is already working on this.
Running testing it automatically after an LU operation:
If you like in the future to invoke a llupdate after a live update you can enable this by
First set the raso parameter and update the /etc/tunables so that it survives a reboot (or live update):
raso -po llu_mode=1
And than create in the lvupdate.template a new stanza entry such as:
llvupdate:
llu = yes
retries = 3
timeout = 30
The logging of this automatic run with 3 retries and timeout of 30 seconds can be found at the familiar please same place for the LU operation:
/var/adm/ras/liveupdate/logs
Conclusion: for us as AIX system administrators this new technology is again very useful and helps us to shorten downtime, and even more importantly not to restart crucial process after an LU operation.
I have to admit that I cannot wait to do a real llu with updated split libraries in SP1.
Therefore I like to encourage you to test with this and share your findings with IBM developers! Together we can make this technology WORK!
Extra note about wrong assumptions genld -u and llvupupdate -P we both made:
Also Chris and I made the same assumption that genld -u and llvupdate -P (preview) should have the same output, but llvupdate and genld different programs with different purposes see below the answer Chris got on his question, from development.
Question: What is the difference between llvupdate -P and genld -u? We were expecting genld to produce a similar report for processes running old libraries, but it does not.
Answer: llvupdate and genld are two different tools, so it is normal that they produce different output. You can filter the output of the genld command by using the -u flag to display processes that have old versions of loaded objects. An object is considered as an old object if the object image is different from the image that is currently installed on the file system. The -u flag is used after applying an update to list the processes that require a restart operation to use the new binaries and libraries.
Here the key point is the definition of "old versions of loaded objects". In your testing steps you replaced the /usr/ccs/lib/libc.a library by copying over the same library. This "replaces" the libc.a with the same library, but the inode changes which will make AIX think the library is updated and is a candidate for LLU. This is true for llvupdate, but it is NOT true for genld. In this situation ("replaces" the libc.a with the same library), genld will find the inode changes, then it will compare the content of those two inodes, and finally it will find they are same. So genld will think the library is NOT updated (and this is correct since the file content is not changed although the inode is changed). Now, having said that, yes, in some situation genld would behave like llvupdate. For example, if you replace the executable file by copying over the same file, genld -u will output:
# cp /usr/bin/topasrec /tmp/topasrec
# cp -p -f /tmp/topasrec /usr/bin/topasrec
# genld -u
Proc_pid: 5046728 Proc_name: topasrec
---
AIX IPSEC is now compatible with AIX LKU
I investigated this feature, but documentation on this was not (yet) very clear.
So tried some tests but so far it’s seems that this only applies to IPsec tunnels and NOT to ip-filtering.
Why would It be nice to have this feature also available for ip-filtering?
Remember the four stages again, we now have to stop ip-filtering in the prep stage.
It would be nice that this could be done automatically by LKU.
Current steps before LKU can be run:
-1 in the prep phase we now first unload the ip filters.
-2 then we remove the ipsec_v4 and ipsec_v6 devices.
Then run the LKU on the LPAR
-3 in the post phase we first enable the ipsec_v4 and ipsec_v6 devices
-4 we load the ip filters again.
What I tested for ip-filtering was the following:
Into the lvudate.data config file I put an extra line ipsec_auto_migrate = yes see below:
general:
kext_check = yes
ipsec_auto_migrate = yes
Then I did NOT unloaded the ip-filter rules and the did not removed the ipsec_v4 and ipsec_v6 devices.
Tried to run a lku, but this crashed, so therefore it seems that this feature only works for ipsec tunnels.
Of course I logged a case for this and I promise to come back on this feature in one of my following blogs.
Thank word(s) to Chris and Nayden:
As you probably noticed, Chris and I are have the same interest in those new lku subjects.
When I first started with LU, the first thing that I did was reading Chris his blogs.
Chris helped me many times by reviewing my blogs, including this one. He also reviewed several times my speakers presentation for the TechXchange 2024. Thanks Chris for all the work, and all your excellent blogs you created for us, we as community very much appreciate this!
@Nayden Stoyanov(Ned): Thank you again for investigation my findings, you answered and solved many difficult cases for us, it is a real pleasure to work with you!
Endnote:
As you may noticed, I like discussions to approve the security for AIX Patch management AIX Live Update and other AIX security subjects.
Any feedback or comments on this blog and other blogs is much appreciated.