AIX

Overview

In this blog, we are going to explain recent enhancements made to server-side caching (aka Power Flash Cache) feature of AIX in  AIX 7.3 TL3; Server-side caching allows an AIX LPAR to use SSDs or Flash storage or NVMe drives or vPmem devices as read-only cache to improve read performance for spinning target disks. The cache can be significantly smaller than the data it is caching (i.e. the target devices). Users have to create a cache pool and cache partition on the attached cache devices using the cache_mgt commands. The cache partition is used to cache the target devices. One can start the cache for the target devices using the cache_mgt command.

When caching is enabled for target devices, additional memory is required on the LPAR since the caching software maintains the metadata associated with caching. The allocation for this metadata happens during the the start of the cache. The size of this metadata allocation is based upon the underlying cache partition size. In storage key environment (when storage protected keys are enabled), these metadata allocations will fail for certain cache sizes as per the current implementation of memory allocations.

One needs to reduce the number of storage protected keys on the system to start caching for the desired cache partition size.  Starting with AIX 7.3 TL3 one need not reduce the number of storage protected keys on the system (changing number of keys requires system restart) to start caching for the desired cache partition size.

skeyctl command on the LPAR can show the number of storage protection keys enabled on the system. Additional information about this command can be found here: IBM AIX: How to check skeyctl setting

The table below summarises the current limitations for cache partition size we have in storage keys environment and the same thing is also documented here.

It is evident from the above table that, we can go up to only 20TB cache partition size with 31 storage protected keys are enabled. To go beyond 20TB of cache partition size, one needs to reduce the storage protect keys enabled on the system as per the above table.

Demonstration:

Prior to this enhancement, (i.e.) on AIX levels below AIX 7.3 TL3, starting cache for the desired size will result in the following error:

1. Check the storage key status on the LPAR: 

     # skeyctl

     Storage Key attributes for current boot session:

     Max. number of hardware keys        = 31

     Number of hardware keys enabled     = 31

     Number of user keys                 = 7

     Kernel keys state                   = enabled

     Exclusive kernel key value          = disabled

2. Creating a cache pool

     # cache_mgt pool create -d hdisk1, hdisk2 -f

     Pool cmpool0 created with devices hdisk1, hdisk2.

3. Creating a cache partition of size 25TB

     # cache_mgt partition create -s 25600G

     Partition cmpart0 created in pool cmpool0.

4. Assign a target disk 

     # cache_mgt partition assign -t hdisk3

     Partition cmpart0 assigned to target hdisk3.

5. Start the cache 

     # cache_mgt cache start -t all

     Failed to start all the caches:
     Failed to start the cache for target hdisk2:
     Cache Engine library returned the following error
   

     Failed to create cache device cmpart0 due to lack of memory

AIX 7.3 TL3 onwards, cache start no longer fails with above error message

1. Check the storage key status on the LPAR: 

     # skeyctl

     Storage Key attributes for current boot session:

     Max. number of hardware keys        = 31

     Number of hardware keys enabled     = 31

     Number of user keys                 = 7

     Kernel keys state                   = enabled

     Exclusive kernel key value          = disabled

2. Creating a cache pool

     # cache_mgt pool create -d hdisk1, hdisk2 -f

     Pool cmpool0 created with devices hdisk1, hdisk2.

3. Creating a cache partition of size 25TB

     # cache_mgt partition create -s 25600G

     Partition cmpart0 created in pool cmpool0.

4. Assign a target disk 

     # cache_mgt partition assign -t hdisk3

     Partition cmpart0 assigned to target hdisk3.

5. Start the cache

     # cache_mgt cache start -t all

     All caches have been started.

6. Check cache status

     # cache_mgt cache list

     hdisk3, cmpart0, active

The above demonstration shows that even with 31 storage keys enabled on the LPAR, we successfully started a cache of 25TB. Previously, this configuration was limited to a maximum of 20TB. With this enhancement on AIX 7.3 TL3, restrictions associated with storage keys and cache size have been removed.

Reference:

1. More information regarding the earlier limitations can be found here.
2. Refer this to get the overview of AIX server-based caching.
3. skeyctl command related details is documented here.

About the Authors:

1. Vamshi Krishna Thatikonda (vamshikrishna@in.ibm.com)
2. Narayan Sethi (Narayan.Sethi@ibm.com)
3. Rahul Singh (rahsin11@in.ibm.com)