Changing tiers for GRS volumes in the PowerVS
Introduction to GRS and flexible IOPS in PowerVS
The Global Replication Service (GRS) in IBM® Power® Virtual Server facilitates disaster recovery (DR) by allowing users to efficiently manage and replicate resources. GRS offers support through APIs, CLI, and is available with limited functionality in the UI.
More details about GRS can be found in the following link:
https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-getting-started-GRS
IBM Power Virtual Server provides the flexibility to choose IOPS-based storage tailored to your needs. The Flexible IOPS option eliminates the concept of specific disk types, replacing it with a storage pool system. These storage pools support multiple tiers, each offering different IOPS levels to match varying performance requirements. If a customer has deployed their VMs for a workload in a high IOPS environment but no longer requires that level of performance, they can switch the workload to a lower IOPS setting. This live switch helps optimize performance while reducing costs, ensuring that customers are billed appropriately for workloads that require lower IOPS. More details about FIOPS can be found in the following link:
https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-on-cloud-architecture#storage-tiers
Use Case Demo
Flexible IOPS tier change feature is not available for replication enabled volumes yet, this blog speaks about the way the tier for replication enabled volumes can be changed.
Consider a VM with volumes in tier 0, offering 250 IOPS per 10 GB, which comes at a higher cost. If the customer determines that such high IOPS performance is unnecessary and a brief data sync delay is acceptable, they can follow these steps to adjust IOPS accordingly.
Following are the details of a VM, which has three replication-enabled volumes in tier 0. The volumes at site-1 are also referred to as the primary volumes.
All the 3 volumes belong to a VG named “volume_group_site1”
The volumes have been onboarded to the secondary site as below:
PS C:\Users> ibmcloud pi vol on cr --source-crn dal10_crn --auxiliary-volumes aux_e-replication_data_volume-68716-2-b21bb8ed-80c245831259 --auxiliary-volumes aux_e-replication_data_volume-68716-1-ca0a66d1-8f3945831259 --auxiliary-volumes aux_e-GRS_VM-b6bda072-000097fa-boot-0-d63fa513-258545831259
Creating volume onboarding under account IBM cloud Account as user user@IBM...
ID 6b1c2218-0568-446d-9295-5df90f1a0100
Description -
Onboarded status and volume details:
PS C:\Users> ibmcloud pi vol on get 6b1c2218-0568-446d-9295-5df90f1a0100
Getting volume onboarding 6b1c2218-0568-446d-9295-5df90f1a0100 under account IBM cloud Account as user user@IBM...
ID 6b1c2218-0568-446d-9295-5df90f1a0100
Status SUCCESS
Description onboard_aux_vols_LEMCFX26
Volumes: aux_e-replication_data_volume-68716-2-b21bb8ed-80c245831259, aux_e-replication_data_volume-68716-1-ca0a66d1-8f3945831259, aux_e-GRS_VM-b6bda072-000097fa-boot-0-d63fa513-258545831259
Creation Time 2024-09-26T10:33:08.000Z
Progress 100
Onboarded Volumes aux_e-replication_data_volume-68716-1-ca0a66d1-8f3945831259,aux_e-replication_data_volume-68716-2-b21bb8ed-80c245831259,aux_e-GRS_VM-b6bda072-000097fa-boot-0-d63fa513-258545831259
Failures -
Volumes details are attached to a VM in the secondary site.
PS C:\Users> ibmcloud pi ins vol ls VSR_TC_1
Listing volumes attached to instance VSR_TC_1 under account IBM cloud Account as user user@IBM...
ID Name Address
09c6867e-c223-4fd1-af22-2c6e57931006 aux_e-replication_data_volume-68716-1-ca0a66d1-8f3945831259 /pcloud/v1/cloud-instances/a669e080-236d-40a9-ae78-fb8b8856aa45/volumes/09c6867e-c223-4fd1-af22-2c6e57931006
bdbfc269-b311-4058-9eff-f788896d5beb VSR_TC_1-e442fb11-00008210-boot-0 /pcloud/v1/cloud-instances/a669e080-236d-40a9-ae78-fb8b8856aa45/volumes/bdbfc269-b311-4058-9eff-f788896d5beb
e498a1c6-6eaa-4fa0-9555-e112a8a25d3e aux_e-GRS_VM-b6bda072-000097fa-boot-0-d63fa513-258545831259 /pcloud/v1/cloud-instances/a669e080-236d-40a9-ae78-fb8b8856aa45/volumes/e498a1c6-6eaa-4fa0-9555-e112a8a25d3e
fa8a22a6-fde8-4138-a276-980593f12142 aux_e-replication_data_volume-68716-2-b21bb8ed-80c245831259 /pcloud/v1/cloud-instances/a669e080-236d-40a9-ae78-fb8b8856aa45/volumes/fa8a22a6-fde8-4138-a276-980593f12142
The primary volume needs to be moved to tier3 which comes at 30 IOPS per 10 GB and the cost is much lesser than that of tier0.
First, the volume needs to be removed from volume group (VG) using the UUID of the volume and check the volume details to see if the volume is removed from VG.
PS C:\Users> ibmcloud pi vg update volume_group_site1 --remove-member-volume-ids ca0a66d1-8f39-4f18-b278-588e53587305
Updating volume group volume_group_site1 under account IBM cloud Account as user user@IBM...
OK
Volume Group ID volume_group_site1 update request was accepted.
PS C:\Users> ibmcloud pi volume get replication_data_volume-68716-1 --json
{
"auxVolumeName": "aux_e-replication_data_volume-68716-1-ca0a66d1-8f3945831259",
"auxiliary": false,
"bootable": false,
"creationDate": "2024-09-26T08:32:50.000Z",
"diskType": "tier0",
"freezeTime": "2024-09-26T11:12:30.000Z",
"ioThrottleRate": "250 iops",
"lastUpdateDate": "2024-09-26T11:19:45.000Z",
"masterVolumeName": "volume-replication_data_volume-68716-1-ca0a66d1-8f39",
"mirroringState": "consistent_copying",
"name": "replication_data_volume-68716-1",
"primaryRole": "master",
"pvmInstanceIDs": [
"b6bda072-b769-4d2a-a275-a5291eae474d"
],
"replicationEnabled": true,
"replicationSites": [
"us-east"
],
"replicationStatus": "enabled",
"replicationType": "global",
"shareable": false,
"size": 10,
"state": "in-use",
"volumeID": "ca0a66d1-8f39-4f18-b278-588e53587305",
"volumePool": "General-Flash-86",
"volumeType": "Tier0-General-Flash-86-GRS",
"wwn": "60050768108081F7D00000000009B741"
}
Volume doesn’t show the group details which was present earlier, hence volume is detached from the volume group.
Alternatively, this can be checked using the volume group details:
Now retype the volume to replication enabled as false:
This will disable the replication. Volume detail is as below:
PS C:\Users > ibmcloud pi volume get replication_data_volume-68716-1 --json
{
"bootable": false,
"creationDate": "2024-09-26T08:32:50.000Z",
"diskType": "tier0",
"ioThrottleRate": "250 iops",
"lastUpdateDate": "2024-09-26T11:26:26.000Z",
"name": "replication_data_volume-68716-1",
"pvmInstanceIDs": [
"b6bda072-b769-4d2a-a275-a5291eae474d"
],
"shareable": false,
"size": 10,
"state": "in-use",
"volumeID": "ca0a66d1-8f39-4f18-b278-588e53587305",
"volumePool": "General-Flash-86",
"volumeType": "Tier0-General-Flash-86",
"wwn": "60050768108081F7D00000000009B741"
}
As part of GRS best practices, go to the secondary site and delete the auxiliary volume to ensure proper data synchronization once replication is re-enabled.
Before deleting the auxiliary volume, it's important to first detach it from the volume group, as it takes 24 hours for the system to sync the details of the retyped primary volume.
PS C:\Users > ibmcloud pi vg update rccg-2e72-ba90a --remove-member-volume-ids 09c6867e-c223-4fd1-af22-2c6e57931006
Updating volume group rccg-2e72-ba90a under account IBM cloud Account as user user@IBM...
OK
Volume Group ID rccg-2e72-ba90a update request was accepted.
PS C:\Users > ibmcloud pi vg get a2df3927-e78e-42c9-8f60-4aed56ffd4c6 --long
Getting Volume Group a2df3927-e78e-42c9-8f60-4aed56ffd4c6 under account IBM cloud Account as user user@IBM...
ID a2df3927-e78e-42c9-8f60-4aed56ffd4c6
Name rccg-2e72-ba90a
Auxiliary true
Consistency Group Name rccg-2e72-ba90a
Replication Sites dal10
Replication Status enabled
Status available
Storage Pool General-Flash-23
Status Description -
Volume Member IDs e498a1c6-6eaa-4fa0-9555-e112a8a25d3e,fa8a22a6-fde8-4138-a276-980593f12142
Detach the auxiliary volume from VM and delete it:
PS C:\Users > ibmcloud pi ins vol det e442fb11-b295-4c4c-872f-a5746c527143 -v aux_e-replication_data_volume-68716-1-ca0a66d1-8f3945831259
Detaching volume aux_e-replication_data_volume-68716-1-ca0a66d1-8f3945831259 from instance e442fb11-b295-4c4c-872f-a5746c527143 under account IBM cloud Account as user user@IBM...
OK
Detach volume aux_e-replication_data_volume-68716-1-ca0a66d1-8f3945831259 from instance e442fb11-b295-4c4c-872f-a5746c527143 request was accepted.
PS C:\Users > ibmcloud pi volume delete aux_e-replication_data_volume-68716-1-ca0a66d1-8f3945831259
Deleting Volume aux_e-replication_data_volume-68716-1-ca0a66d1-8f3945831259 under account IBM cloud Account as user user@IBM...
OK
Volume ID aux_e-replication_data_volume-68716-1-ca0a66d1-8f3945831259 was deleted.
Come back to the primary volume, change the tier using CLI/API/UI as it is a normal volume now.
It got changed from tier0 to tier3 and IOPS rate is also changed from 250 IOPS per 10GB to 30 IOPS per 10GB.
PS C:\Users> ibmcloud pi vol get replication_data_volume-68716-1
Getting Volume replication_data_volume-68716-1 under account IBM cloud Account as user user@IBM...
ID ca0a66d1-8f39-4f18-b278-588e53587305
Name replication_data_volume-68716-1
Profile tier3
Status in-use
Size 10
Created 2024-09-26T08:32:50.000Z
Updated 2024-09-26T11:46:15.000Z
Shareable false
Bootable false
Storage Pool General-Flash-86
IO Throttle Rate 30 iops
Replication Enabled -
PVMInstanceIDs b6bda072-b769-4d2a-a275-a5291eae474d
User Tags -
WWN 60050768108081F7D00000000009B741
Now Replication can be enabled back.
The volume is now replication-enabled, but its mirroring state is currently 'inconsistent_copying' as data synchronization is in progress. Once the volume reaches the 'consistent_copy' state, it can be added back to the volume group.
PS C:\Users > ibmcloud pi vol get replication_data_volume-68716-1 --json
{
"auxVolumeName": "aux_e-replication_data_volume-68716-1-ca0a66d1-8f3945831259",
"auxiliary": false,
"bootable": false,
"creationDate": "2024-09-26T08:32:50.000Z",
"diskType": "tier3",
"freezeTime": "2024-09-26T12:03:00.000Z",
"ioThrottleRate": "30 iops",
"lastUpdateDate": "2024-09-26T11:49:34.000Z",
"masterVolumeName": "volume-replication_data_volume-68716-1-ca0a66d1-8f39",
"mirroringState": "consistent_copying",
"name": "replication_data_volume-68716-1",
"primaryRole": "master",
"pvmInstanceIDs": [
"b6bda072-b769-4d2a-a275-a5291eae474d"
],
"replicationEnabled": true,
"replicationSites": [
"us-east"
],
"replicationStatus": "enabled",
"replicationType": "global",
"shareable": false,
"size": 10,
"state": "in-use",
"volumeID": "ca0a66d1-8f39-4f18-b278-588e53587305",
"volumePool": "General-Flash-86",
"volumeType": "Tier3-General-Flash-86-GRS",
"wwn": "60050768108081F7D00000000009B741"
}
Adding it to the volume group.
PS C:\Users > ibmcloud pi vg update volume_group_site1 --add-member-volume-ids ca0a66d1-8f39-4f18-b278-588e53587305
Updating volume group volume_group_site1 under account IBM cloud Account as user user@IBM...
OK
Volume Group ID volume_group_site1 update request was accepted.
PS C:\Users > ibmcloud pi vol get replication_data_volume-68716-1 --json
{
"auxVolumeName": "aux_e-replication_data_volume-68716-1-ca0a66d1-8f3945831259",
"auxiliary": false,
"bootable": false,
"consistencyGroupName": "rccg-2e72-ba90a",
"creationDate": "2024-09-26T08:32:50.000Z",
"diskType": "tier3",
"freezeTime": "2024-09-26T12:12:00.000Z",
"groupID": "2e7249a1-900f-43c0-913d-b16e86eba90a",
"ioThrottleRate": "30 iops",
"lastUpdateDate": "2024-09-26T12:11:00.000Z",
"masterVolumeName": "volume-replication_data_volume-68716-1-ca0a66d1-8f39",
"mirroringState": "consistent_copying",
"name": "replication_data_volume-68716-1",
"primaryRole": "master",
"pvmInstanceIDs": [
"b6bda072-b769-4d2a-a275-a5291eae474d"
],
"replicationEnabled": true,
"replicationSites": [
"us-east"
],
"replicationStatus": "enabled",
"replicationType": "global",
"shareable": false,
"size": 10,
"state": "in-use",
"volumeID": "ca0a66d1-8f39-4f18-b278-588e53587305",
"volumePool": "General-Flash-86",
"volumeType": "Tier3-General-Flash-86-GRS",
"wwn": "60050768108081F7D00000000009B741"
}
Importantly the volume has now been changed to tier 3, but the auxiliary volume is no longer available at the secondary site. Therefore, onboard the volume in secondary site, it should be recreated with the same name but a different ID.
PS C:\Users > ibmcloud pi ws tg us-east_crn
Targeting service us-east_crn ...
PS C:\Users > ibmcloud pi vol on cr --source-crn dal10_crn --auxiliary-volumes aux_e-replication_data_volume-68716-1-ca0a66d1-8f3945831259
Creating volume onboarding under account IBM cloud Account as user user@IBM...
ID 2fbb9d45-bf32-42f3-bb24-1abf0fb220b5
Description -
PS C:\Users > ibmcloud pi vol on get 2fbb9d45-bf32-42f3-bb24-1abf0fb220b5
Getting volume onboarding 2fbb9d45-bf32-42f3-bb24-1abf0fb220b5 under account IBM cloud Account as user user@IBM...
ID 2fbb9d45-bf32-42f3-bb24-1abf0fb220b5
Status SUCCESS
Description onboard_aux_vols_9EGC9B2E
Volumes: aux_e-replication_data_volume-68716-1-ca0a66d1-8f3945831259
Creation Time 2024-09-26T12:18:28.000Z
Progress 100
Onboarded Volumes aux_e-replication_data_volume-68716-1-ca0a66d1-8f3945831259
Failures -
PS C:\Users > ibmcloud pi vol get f42e64fb-99c7-4399-be72-6329f4514b42 --json
{
"auxVolumeName": "aux_e-replication_data_volume-68716-1-ca0a66d1-8f3945831259",
"auxiliary": true,
"bootable": false,
"consistencyGroupName": "rccg-2e72-ba90a",
"creationDate": "2024-09-26T12:18:42.000Z",
"diskType": "tier3",
"freezeTime": "2024-09-26T08:12:00.000Z",
"groupID": "a2df3927-e78e-42c9-8f60-4aed56ffd4c6",
"ioThrottleRate": "30 iops",
"lastUpdateDate": "2024-09-26T12:18:43.000Z",
"masterVolumeName": "volume-replication_data_volume-68716-1-ca0a66d1-8f39",
"mirroringState": "consistent_copying",
"name": "aux_e-replication_data_volume-68716-1-ca0a66d1-8f3945831259",
"primaryRole": "master",
"pvmInstanceIDs": [],
"replicationEnabled": true,
"replicationSites": [
"dal10"
],
"replicationStatus": "enabled",
"replicationType": "global",
"shareable": false,
"size": 10,
"state": "available",
"volumeID": "f42e64fb-99c7-4399-be72-6329f4514b42",
"volumePool": "General-Flash-23",
"volumeType": "Tier3-General-Flash-23-GRS",
"wwn": "60050764008102897000000000037A9B"
}
This completes the process of tier change.
Summary:
- Create a replication-enabled volume, attach it to a VM and onboard it.
- Retype the volume back to non-replication.
- Delete or clean up the auxiliary volume.
- Change the tier of the volume.
- Retype the volume back to replication.
- Onboard the auxiliary volume.
With this method, changing the tier of any or all volumes in the VM from lower to higher IOPS or vice-versa can be achieved on the need basis.
References:
https://cloud.ibm.com/media/docs/downloads/power-iaas/Global_Replication_Services_Solution_using_IBM_Power_Virtual_Server.pdf