Informix

Hi Øyvind,

the large pages in most Informix deployments is like icing on the cake, you have to get other areas tuned, after that you'll see most likely 10% or so improvement from using "large" (on Linux "huge") pages (and maybe face couple of interesting issues - if you also bind Informix to specific node(s) for the best affinity).

As other folks already mentioned, the savings mostly come from better translation of the (operating system's) virtual memory address to physical, which uses table (called TLB or similar) of limited size (note that size is hardware-specific).
When application (Informix) is using default operating system pages, usually 4k on Linux (should not be confused with Informix configurable page size) - the translation table is likely to overflow  around / after few GB of RAM, so having 400GB on the system (and using fair portion of it for Informix) has a good chance of seeing benefit from using large pages.

The Informix for Linux has IFX_LARGE_PAGES enabled by defaults, as long as you configured enough of them (on Linux the size of a huge page is usually 2GB), and, I think another requirement is for Informix to have RESIDENT flag set - they will be used. 

Here is a simple example using system with 128GB of RAM and Informix total memory footprint not exceeding 100GB:

-- with Informix stopped --

# cat /proc/meminfo

MemTotal: 131764620 kB
MemFree: 127360576 kB

# sysctl -w vm.nr_hugepages=50000

# cat /proc/meminfo

MemTotal: 131764620 kB
MemFree: 28652856 kB
....
HugePages_Total: 50000
HugePages_Free: 50000
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB

Note that 100GB are now not available for general use (MemFree) - so if other running apps needed (now or in future) more than  (MemTotal - allocated.LP.size) - it'll cause performance issues.

The remaining question - is it enough for Informix to see performance benefit from using large pages? Unfortunately the answer is no. One of the additional requirements is otherwise tuned and CPU bound workload. Another is workload type and the size of the "working set". While Informix may allocate/reserve tens or hundreds GBs of RAM - it's possible that only small portion of it is ACTIVELY used. If the "working set" size is small enough to not cause TLB misses - no effect will be seen from using large pages.

The "best" workload for seeing effect of the large pages is the TPC-C benchmark, since it accesses data in UNIFORM manner.
However even there there will be dependency from the workload pattern. As an example here is the data from several sets of (CPU-bound) OLTP benchmarks (TPC-C like) where each set uses different number of active client sessions, from 1 to 1200, number of sessions  is the first column:

              default          large
  CL          4k-page          2G-page      diff%
-------------------------------------------------
   1         56991.38         60144.81       5.5%
   2        109120.38        114728.69       5.1%
   3        156527.69        165304.43       5.6%
   4        203524.18        216079.75       6.2%
   5        249224.44        266030.82       6.7%
   6        290745.63        311187.94       7.0%
   8        364152.18        393351.18       8.0%
  10        403662.70        441657.76       9.4%
  12        438587.99        479961.70       9.4%
  14        468131.94        520049.46      11.1%
  16        501562.53        560274.93      11.7%
  20        502320.43        561788.31      11.8%
  24        503480.26        565137.04      12.2%
  28        503165.67        568734.12      13.0%
  32        501205.86        568941.13      13.5%
  40        499063.01        567008.11      13.6%
  48        496270.85        564567.48      13.8%
  56        492001.89        561357.70      14.1%
  64        487212.29        557250.38      14.4%
  80        476395.25        547373.80      14.9%
 100        464257.84        535203.78      15.3%
 200        440156.05        508607.34      15.6%
 400        433493.65        499677.93      15.3%
 800        420687.47        485471.21      15.4%
1000        407470.80        476033.15      16.8%
1200        397504.27        466980.94      17.5%
-------------------------------------------------

Second column is the throughput (transactions per minute) with default (4k) pages, next column is the same but with large (2GB) pages, and last column is the difference in %.

Working set size was about 32GB worth of pages (Informix data + index) and everything was fully cached in the bufferpool.

One benefit that can be derived from this information (I hope) is the conclusion that if there is no difference with/without large pages - the configuration can be further tuned for performance...

:-)

Hope it helps a bit,

--Vladimir

I guess IBM has it's own notion on how to format "code segments" making them totally useless...

Here is the table which will hopefully make more sense:

default large
CL 4k-page 2G-page diff%
-------------------------------------------------
1 56991.38 60144.81 5.5%
2 109120.38 114728.69 5.1%
3 156527.69 165304.43 5.6%
4 203524.18 216079.75 6.2%
5 249224.44 266030.82 6.7%
6 290745.63 311187.94 7.0%
8 364152.18 393351.18 8.0%
10 403662.70 441657.76 9.4%
12 438587.99 479961.70 9.4%
14 468131.94 520049.46 11.1%
16 501562.53 560274.93 11.7%
20 502320.43 561788.31 11.8%
24 503480.26 565137.04 12.2%
28 503165.67 568734.12 13.0%
32 501205.86 568941.13 13.5%
40 499063.01 567008.11 13.6%
48 496270.85 564567.48 13.8%
56 492001.89 561357.70 14.1%
64 487212.29 557250.38 14.4%
80 476395.25 547373.80 14.9%
100 464257.84 535203.78 15.3%
200 440156.05 508607.34 15.6%
400 433493.65 499677.93 15.3%
800 420687.47 485471.21 15.4%
1000 407470.80 476033.15 16.8%
1200 397504.27 466980.94 17.5%
-------------------------------------------------

....

              default          large  CL          4k-page          2G-page      diff%-------------------------------------------------   1         56991.38         60144.81       5.5%   2        109120.38        114728.69       5.1%   3        156527.69        165304.43       5.6%   4        203524.18        216079.75       6.2%   5        249224.44        266030.82       6.7%   6        290745.63        311187.94       7.0%   8        364152.18        393351.18       8.0%  10        403662.70        441657.76       9.4%  12        438587.99        479961.70       9.4%  14        468131.94        520049.46      11.1%  16        501562.53        560274.93      11.7%  20        502320.43        561788.31      11.8%  24        503480.26        565137.04      12.2%  28        503165.67        568734.12      13.0%  32        501205.86        568941.13      13.5%  40        499063.01        567008.11      13.6%  48        496270.85        564567.48      13.8%  56        492001.89        561357.70      14.1%  64        487212.29        557250.38      14.4%  80        476395.25        547373.80      14.9% 100        464257.84        535203.78      15.3% 200        440156.05        508607.34      15.6% 400        433493.65        499677.93      15.3% 800        420687.47        485471.21      15.4%1000        407470.80        476033.15      16.8%1200        397504.27        466980.94      17.5%-------------------------------------------------

We are on Linux (was RHEL6 during testing, is RHEL7 now). Hugepages was enabled at OS level, yes. And it was logged during startup as well.

We had 12.10 at the time, are on 14.10 now, but I guess that shouldn't make any difference. 

What difference do you notice? Less system time used (less overhead in kernel), or less user time by CP VPs?