Introduction:

VPMEM (Virtual Persistent MEMory) is an enhancement of IBM’s advanced virtualization platform (PowerVM) and introduces the ability to configure persistent volumes using the conventional DRAM memory modules available in every IBM POWER9 system onwards. Since vPMEM is built on DRAM technology, it has the same performance characteristics as DRAM, which enables IBM POWER users to speed up their applications. Virtual Persistent Memory volumes are persistent across LPAR reboots but the contents are lost across CEC (Physical Server) reboots.

The I/O model in vPMEM driver is currently synchronous i.e the I/O is issued and completed in the same context. Typical I/O completion in vPMEM involves just doing a memcpy operation to the appropriate disk (memory region) and calling iodone. I/O issue or completion does not involve DMA unlike in other drivers. I/O is started and completed on same CPU before the next I/O is started. This model is very well suited for synchronous I/O but asynchronous I/O will hit a performance hit.

Many applications, such as databases and file servers, uses asynchronous I/O to take advantage of the ability to overlap processing and I/O and can initiate a list of asynchronous I/O requests with a single function callusing  lio_listio or lio_listio64 subroutine. The mode parameter determines whether the subroutine returns when the I/O operations have been completed, or as soon as the operations have been queued. If the mode parameter is set to LIO_WAIT, the subroutine waits until all I/O is complete. If the mode parameter is set to LIO_NOWAIT or LIO_NOWAIT_AIOWAIT, the subroutine returns immediately.

List of asynchronous I/Os submitted through lio_listio can be initiated one after other without waiting for completion of any I/Os from list. However, with current design of vPMEM driver, list of asynchronous I/Os submitted through lio_listio behaves in synchronous manner i.e. a subsequent I/O from the list can be initiated only after completion of previous I/O from the list. Because of this synchronous behaviour, I/O latency gets increased on vPMEM device.  

Solution:

To make asynchronous I/O on vPMEM device work in true asynchronous fashion, a new kernel thread model is introduced.  With this model, I/O is delegated to new kernel thread and control returns back immediately. This decreases the delay in I/O initiation, which decreases the overall I/O latency.

A new flag is passed to vPMEM driver to delegate the I/O to new kernel thread.

Whether to pass new flag to vPMEM driver or not is controlled by the new ioo tunable aio_delegateiotodriverkth (for legacy AIO) or posix_aio_delegateiotodriverkth (for posix AIO).

More details about the tunables can we found at https://www.ibm.com/docs/en/aix/7.3?topic=i-ioo-command

Performance Test Results:

In our in-house performance testing, we have observed around ~44% improvement for 256 I/Os of size 32KBeach with lio_listio on vPMEM device.

Sample results:

Pre-requisites

This feature is supported from AIX 7.3 TL3 release and above.

References:

vPMEM Device Support in AIX 7.3
vPMEM Driver in AIX

Asynchronous I/O in AIX

About the authors

Arun Nagaraj (arnagraj@in.ibm.com) - AIX Filesystem and LVM Development

Prateek Kanswa (Prateek.Kanswa@ibm.com) AIX Filesystem Development

Tejas Prajapati (tejas.prajapati@ibm.com) – AIX Diagnostics

Phani Kumar Ayyagari (phanikumar@in.ibm.com) – AIX Diagnostics