Prerequisite
OS version: AIX 7.3 TL4 or above
Introduction
Prior to AIX 7.3 TL 4, mostly SCSI disks were used as the cluster repository. NVMe disks were not supported by CAA as the repository disks.
But there has been a growing customer demand to use NVMe disks as the cluster repository. This is because NVMe is designed for high-performance flash-based storage, offering significantly faster speeds and lower latency than the older SCSI. So, with AIX 7.3 TL4 onwards, CAA will be supporting NVMe disks as the cluster repository.
Creating a CAA cluster capable of supporting NVMe disks as the cluster repository
To create a cluster capable of supporting NVMe disks, “ –c nvme” option can be specified explicitly with the “mkcluster” command, as below:
mkcluster –m <nodename> -r <NVMe repository disk name> -c nvme –n <cluster name>.
If the “ –c nvme” option is not specified, it will follow the auto commit behavior. In other words, if an NVMe repository disk is specified, without specifying the “ –c nvme” option explicitly, CAA will still create a cluster capable of supporting NVMe disks.
CAA cluster configurations using NVMe disks as the cluster repository
(I) Creation of a 2-node CAA cluster with FC-NVMe disk as the cluster repository disk
|
Objective
|
Create a CAA Cluster using "-c nvme option", add a node using chcluster.
|
|
Setup
|
1. Assign 1 NVME Shared Disks of
size=5GB between the 2 nodes.
|
|
Procedure
|
Cluster Configuration:
Create a 2 node CAA cluster having NVMe disk as repo disk as below:
date;mkcluster -c nvme -r hdisk4 -m p10denb-lp2 -n TEST_CAA
In the above hdisk4 is NVMe disk.
Add a remote node as below:
date;chcluster -m +p10denb-lp3
|
|
Expected Results
|
List the lscluster -m where all nodes are listed
|
(II) Creation of a 2-SITE linked cluster, with each site having one node and a PCIe NVMe disk configured as the cluster repository disk within that site
|
Objective
|
Create a Linked CAA Cluster using "-c nvme" option, add a remote SITE using chcluster.
|
|
Setup
|
1.SITE B (REMOTE SITE): 1 node. Repository Disk: hdisk2 – PCIe NVMe disk
2.SITE A (LOCAL SITE): 1 node. Repository Disk: hdisk1 – PCIe NVMe disk
|
|
Procedure
|
Cluster Configuration:
Create a 2 SITE linked cluster having NVMe disk as repo disk as below:
date;mkcluster -c nvme -r hdisk1 -S LC_LOCALSITE -m p10denb-lp1 -n LC
In the above hdisk1 is PCIe NVMe disk.
Add a remote site as below:
date;chcluster -r hdisk2 -S +LC_REMOTESITE -m +p10denb-lp2 -n LC
Where hdisk2 is a PCIe NVMe disk
|
|
Expected Results
|
List the lscluster -m where all sites and the nodes in each site are listed
|
(III). Creation of a 2-SITE linked cluster, with each site having two nodes and a shared NVMe disk between the nodes.
|
Objective
|
Create a 2-node Linked CAA Cluster using "-c nvme" option, add a remote site having 2 nodes using chcluster.
|
|
Setup
|
1.SITE B (REMOTE SITE): 2 nodes
2.SITE A (LOCAL SITE): 2 nodes
3. Assign 1 NVMe Shared Disks of size=5GB between 2 nodes in each of the sites, SITE A & SITE B.
|
|
Procedure
|
Cluster Configuration:
Create a 2 SITE linked cluster having NVMe disk as repo disk as below:
date;mkcluster -c nvme -r hdisk4 -S LC_LOCALSITE -m p10denb-lp,p10denb-lp2 -n LC
In the above hdisk4 is NVMe disk.
Add a remote site as below:
date;chcluster -r hdisk3 -S +LC_REMOTESITE -m +p10denb-lp3,+p10denb-lp8 -n LC
Where hdisk3 is a NVMe disk
|
|
Expected Results
|
List the lscluster -m where all sites and the nodes in each site are listed
|
(IV) Creation of a 2-SITE linked cluster, where one site has NVMe disk as the cluster repository and the remote site has SCSI disk as the cluster repository.
|
Objective
|
Create a CAA Linked Cluster using "-c nvme option", with one SITE having NVMe & another site having SCSI DISK as repo
|
|
Setup
|
1. SITE B (REMOTE SITE): 2 nodes
2. SITE A (LOCAL SITE): 2 nodes
3. Assign 1 NVMe Shared Disks of size=5GB between 2 nodes in SITE A.
|
|
Procedure
|
Cluster Configuration:
Create a 2 SITE linked cluster having NVMe disk as repo disk as below:
(0) root @ p10denb-lp1: /
# date;mkcluster -c nvme -r hdisk4 -S LC_LOCALSITE -m p10denb-lp1,p10denb-lp2
In the above hdisk4 is NVMe disk.
Add a remote site with SCSI repo disk as below:
date;chcluster -r hdisk3 -S +LC_REMOTESITE -m +p10denb-lp3,+p10denb-lp8 -n LC
Where hdisk3 is SCSI disk
|
|
Expected Results
|
Linked cluster should be UP & running for Mixed disk.
|
(V) In mixed-level clusters, where nodes with AIX level less than AIX 7.3 TL4 are present, NVMe disks will not be supported
|
Objective
|
Create a CAA Cluster using -c nvme option, with one Node of 73H, add node of 73F
|
|
Setup
|
CAA cluster with one node 73H & other node 73F
|
|
Procedure
|
Cluster Configuration:
Create CAA cluster having NVMe disk as repo disk as below:
· mkcluster -r hdisk4 -m p10denb-lp1 -n CAA -c nvme
· Add a node using the below command:
date;chcluster -n LC -m p10denb-lp2
|
|
Expected Results
|
Add node will fail with below error:
# date;chcluster -n LC -m p10denb-lp2
Wed Jul 16 22:37:45 CDT 2025
ERROR: p10denb-lp2 is down-level node. It doesn't support NVME repo disk.
ERROR: ADD_NODE failed for p10denb-lp2
1035-306 chcluster: Could not modify cluster.
|
Conclusion
With CAA now supporting NVME disks as the cluster repository, customers can now leverage NVME’s faster speeds and lower latency for faster cluster operations.