Large page optimization is a feature of IBM® AIX® Dynamic System Optimizer (ASO) that automatically promotes the page size to a larger 16 MB page size when it predicts improved performance. Large page optimization transparently upgrades 4 KB or 64 KB page sizes to a 16 MB page size. This larger page size can benefit workloads that use large chunks of data because it reduces the translation lookaside buffer (TLB) misses. When this optimization is activated, ASO determines if there is any benefit from optimization and if so, it automatically promotes heavily used regions of memory to 16 MB pages. This optimization potentially improves the performance of workloads that use those regions because it reduces the number of TLB misses.
Prior to AIX 7.3 Technology level 3, System V shared memory was only eligible for large page optimization. In AIX 7.3 Technology level 3, AIX has been enhanced to have this feature applied to text pages too. The rest of the document describes this feature, its configuration and benefits.
By default, this functionality is enabled in the ASO. ASO considers a process text region for 16MB page promotion under the following conditions:
· The size of process text must be greater than 16 MB.
· The base page size of the process text must be 64 KB.
· The process must be running in 64-bit environment
· The process must have a runtime exceeding 10 minutes.
· The cumulative CPU utilization of all processes sharing the same text must be high.
For example, lets create a sample application that has been created with 64K text pages and text size 372 MB. ‘svmon’ command can be used to view the address space of this application as shown below. Text pages are 64K in size.
(0) root @ bosden1-lp8: /kaushal
# LDR_CNTRL=TEXTPSIZE=64K ./LgBin &
[1] 2360298
(0) root @ bosden1-lp8: /kaushal
# svmon -P 2360298
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
2360298 LgBin 20886 14613 0 20864 Y N N
PageSize Inuse Pin Pgsp Virtual
s 4 KB 16198 14565 0 16176
m 64 KB 293 3 0 293
L 16 MB 0 0 0 0
S 16 GB 0 0 0 0
Vsid Esid Type Description PSize Inuse Pin Pgsp Virtual
20002 0 work kernel segment s 15051 14529 0 15051
a50025 90000000 work shared library text m 285 0 0 285
a70027 90020014 work shared library s 666 0 0 666
a000a 9ffffffd work shared library sm 410 36 0 410
984d18 f00000002 work process private m 5 3 0 5
db235b 10 work text data BSS heap m 2 0 0 2
f40374 9fffffff clnt USLA text,/dev/hd2:492 s 22 0 - -
d000d 9ffffffe work shared library sm 19 0 0 19
ac25ac 9001000a work shared library data sm 19 0 0 19
a74027 11 work text data BSS heap m 1 0 0 1
a32223 80020014 work USLA heap sm 5 0 0 5
e620e6 8fffffff work private load data s 4 0 0 4
982098 12 work text data BSS heap sm 1 0 0 1
b14231 ffffffff work application stack sm 1 0 0 1
ASO monitors the above application and after 10 min, promotes its text pages to 16MB and the same can be viewed with svmon as shown below:
# svmon -P 2360298
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
2360298 LgBin 111670 14613 0 111648 Y N N
PageSize Inuse Pin Pgsp Virtual
s 4 KB 16198 14565 0 16176
m 64 KB 5967 3 0 5967
L 16 MB 0 0 0 0
S 16 GB 0 0 0 0
Vsid Esid Type Description PSize Inuse Pin Pgsp Virtual
db235b 10 work text data BSS heap mL 4096 0 0 4096
a74027 11 work text data BSS heap mL 1581 0 0 1581
20002 0 work kernel segment s 15051 14529 0 15051
a50025 90000000 work shared library text m 285 0 0 285
a70027 90020014 work shared library s 666 0 0 666
a000a 9ffffffd work shared library sm 410 36 0 410
984d18 f00000002 work process private m 5 3 0 5
f40374 9fffffff clnt USLA text,/dev/hd2:492 s 22 0 - -
d000d 9ffffffe work shared library sm 19 0 0 19
ac25ac 9001000a work shared library data sm 19 0 0 19
a32223 80020014 work USLA heap sm 5 0 0 5
e620e6 8fffffff work private load data s 4 0 0 4
982098 12 work text data BSS heap sm 1 0 0 1
b14231 ffffffff work application stack sm 1 0 0 1
Users can explicitly request 16MB text promotion of their 64-bit applications using vm_pattr() system calls. The promotion will occur for 16MB ranges which are heavily used and are in memory. A sample code to achieve this can be found below. More on this can be found in AIX documentation - https://www.ibm.com/docs/kk/aix/7.3?topic=v-vm-pattr-system-call-kvm-pattr-kernel-service
#include <sys/vmpattr.h>
…
void main(int args, char *argv[])
{
struct vm_pa_psize_extended pa = {0};
...
int rc;
pa.pa_range.rng_start = 0x10; //Effective address of process text
pa.pa_range.rng_size = 256*1024*1024; //Size_of_process_text(Max 256 MB for a segment)
pa.pa_psize = -1;
pa.pa_info = NULL;
pa.pa_info_size = 0;
rc = vm_pattr(VM_PA_SET_PSIZE_EXTENDED, pid, &pa, sizeof(pa));
printf("Outcome of vm_pattr = %d\n", rc);
...
}
16MB page promotion of text was tried with TPC-E workload and around 2% TPS improvement was observed.
To conclude, we have observed substantial benefits with application text page promotion to 16MB. So, would encourage users to experiment and reap benefits of this feature.
About the authors
Kavana N Bhat (E-Mail: kavana.bhat@in.ibm.com) - AIX Kernel Team
Kaushal Kumar (E-Mail: kaushal.kumar2@ibm.com) - AIX Kernel Team
