Starting with IBM® AIX® 7.3 Technology Level 3, the Tech Preview of Live Library Update (LLU) is introduced. AIX LLU enhancement allows applications to start using the latest libraries, without having to stop and restart them. AIX 7.3 TL3 SP1 is showcasing this feature as a Tech Preview. The initial release supports LLU on most widely used two AIX libraries: “libc” and “libpthreads”.

Currently LLU is not enabled by default. User can override this setting for the processes by using LDR_CNTRL environment variable or enable it system wide using “raso” command.

To query the current LLU mode
raso -o llu_mode
To enable LLU
raso -o llu_mode=1
To disable LLU
raso -o llu_mode=0
To enable LLU conditionally (Default value in AIX 7.3 TL3 SP1)
raso -o llu_mode=2

More details on the raso command can be found in AIX documentation page.

After installing the AIX filesets or efixes, user can update the existing applications to use the new libraries (libc or libpthreads) by running “llvupdate” command on the processes.

Here is the basic flow diagram that explains LLU operation.

The shared object of a LLU capable library(ex. shr.o) is split into two objects - VAPI (shr.o) and Implementation(_shr.o):

• VAPI(Virtual API) object - Contains exported symbols/data and unexported global data that should be preserved across LLU operation.
• Implementation object - Contains the library implementation code and unexported library data which can be modified across LLU operation.

User applications need not be aware of split library as this is internally taken care by the loader based on LLU mode system setting.

During the LLU operation, pseudo (LLU) signals are used to interrupt process execution, all the threads of the process are frozen (after they are out of Implementation) and the Implementation object is replaced while retaining the VAPI object of the library.

Additional VAPI object symbols (if any) are added through delta VAPI object. Here is diagrammatic representation of LLU operation.

Example usage of “llvupdate” command:

1. View current process address space before LLU
   
   

   # procmap 9830866
   9830866 : sshd: root@pts/0
   10000000 1787K read/exec sshd
   20000168 784K read/write sshd
   ...
   ...
   d019c980 4580K read/exec /usr/lib/libc.a[_shr.o] -> Implementation's text segment
   f04c39d0 480K read/write /usr/lib/libc.a[_shr.o] -> Implementations's data segment
   ...
   ...
   d0100000 587K read/exec /usr/lib/libc.a[shr.o] -> VAPI's text segment
   f0440250 521K read/write /usr/lib/libc.a[shr.o] -> VAPI'S data segment
       Total 16226K

2. Run LLU preview and identify processes that needs LLU operation
   
   

   # llvupdate -P
   llvupdate preview
   An LLU-capable library is newer for process sshd pid 9830866. <<-- LLU target process
   Library needs to be updated /usr/lib/libc.a(_shr.o)
   Validating new module /usr/lib/libc.a(_shr.o)
   An LLU-capable library is newer for process ksh pid 11272566.
   Library needs to be updated /usr/lib/libc.a(_shr.o)

3. Trigger LLU on a given process with a single retry count
   
   

   # llvupdate -p 9830866 -n 1
   Non-interruptable live library update operation begins in
   10 seconds.
   llvupdate preview
   Validating new module /usr/lib/libc.a(_shr.o)
   Adding the process sshd pid 9830866 to the list
   ------------------------------------------------------------
   ************************** TRY 1
   **************************
   ======== NOTIFY PHASE BEGIN ======== | try 1 | tonotify:1
   ======== NOTIFY PHASE ENDED ======== | try 1 | suspended:1
   ..........................................................................................
   ======== QUERY PHASE BEGIN ======== | try 1 | suspended:1
   ======== QUERY PHASE ENDED ======== | try 1 | updated:1
   End of try 1/1, number of processes to be retried 0
   ====================
   ==== LLU Report ====
   9830866 sshd SUCCESS
   ==== LLU Report End ====
   SUCCESS: 1     NOT UPDATED: 0     FAILURE: 0
   ===============================================================
   LLU operation is completed. <<<--- LLU is complete

4. Trigger LLU on all LLU capable processes with single retry count
   
   

   # llvupdate -a -n 1

5. View process address space after LLU
   
   

   # procmap 9830866
   9830866 : sshd: root@pts/0
   10000000 1787K read/exec sshd
   20000168 784K read/write sshd
   ...
   ...
   d4744980 4580K read/exec /usr/lib/libc.a[_shr.o] -> Implementation's text segment (note the segment address change after LLU - first column)
   f19759d0 480K read/write /usr/lib/libc.a[_shr.o] -> Implementations's data segment (note the segment address changes after LLU - first column)
   ...
   ...
   d0100000 587K read/exec /usr/lib/libc.a[shr.o] -> VAPI's text segment (no change after LLU)
   f0440250 521K read/write /usr/lib/libc.a[shr.o] -> VAPI'S data segment (no changes after LLU)
       Total 16226K

More detailed information will be collected by LLU automatically:

1. The /var/adm/ras/liveupdate/logs/llvupdlog log file contains details about LLU operation. LLU command automatically takes backup of this file before attempting a new LLU.
2. The “llvupdate” command automatically enables system tracing before doing LLU and disables after LLU. These system traces are dumped
   into /var/adm/ras/liveupdate/trace_llu/trcfileLLU file.

LLU might fail in following scenarios:

1. Process is busy in system call or in kernel at the time of LLU
2. Process is continuously executing in implementation layer of library
   In the above failure scenarios LLU can be retried using “-n” option as shown below
   
   

   To set the number of retries.
   The default is 3 and max number can be 10
   llvupdate -p <pid> -n <retries>

For detailed usage scenarios, refer to llvupdate command documentation.

In certain rare scenarios of LLU failure, where application processes are left in uncleaned state, the cllvupdate command can be used to recover and clean the process state before attempting further LLUs. Here is the usage of cllvupdate command:

# cllvupdate -u

Upon any LLU failures, IBM AIX support can be contacted by collecting LLU logs using “snap -ac” command.

AIX also provides an option to trigger LKU and LLU together. This requires user to modify the /var/adm/ras/liveupdate/lvupdate.data file in the following way in “llvupdate” section.

llvupdate:
    llu = yes
    retries = 1
    timeout = 10

AIX also provides an option where ISVs can make third party libraries LLU capable. Please contact IBM AIX support team for further assistance.

Authors
Abhishek Paliwal (AIX Live Update Development) EMail : abhishek.paliwal1@ibm.com
Kavana Bhat (AIX VMM and Loaders/Linkers Architect) EMail : kavana.bhat@in.ibm.com
Sowmya G (AIX Linker Loader Development) EMail : sowmya.tg@in.ibm.com
Suresh Kumar A (Software Developer (AIX-BOS)) EMail : suresana@in.ibm.com
Vinod Kumar Boddukuri (AIX Live Update Architect) EMail : vinod.boddukuri@in.ibm.com