Introduction to Consistency Group Mirroring

Consistency group mirroring in Ceph allows for replicating a group of Ceph Block Device (RBD) images as a consistent set, ensuring data uniformity and consistency of images across different Ceph clusters in a crash-consistent manner. The primary goal is to ensure that all images within the consistency group are updated together to the secondary site, preserving dependencies between the data stored on them.

For example, Imagine a database application with multiple tables spread across several RBD images. To ensure data consistency, you can group these images into a consistency group. When mirroring is enabled, the system will replicate all the images in the consistency group to a remote site, guaranteeing that the database tables are consistent at the target site for recovery purposes.

Consistency Group Mirroring Type

Ceph uses snapshot mirroring for consistency groups, meaning it leverages RBD image snapshots to replicate the data.

Why You Need Consistency Group Mirroring

• Data Consistency: When mirroring individual RBD images, there's a risk of inconsistencies between different images if they're not synchronized at the same time. Consistency groups ensure that all images within the group are replicated together, maintaining a consistent state across the primary and secondary sites.
  

• Disaster Recovery: In a disaster recovery scenario, you might need to failover to a secondary site. Consistency group mirroring ensures that all the necessary data for an application is available in a consistent state at the secondary site, allowing for a smooth and reliable failover.
  

• Simplified Management: Managing mirroring at the consistency group level simplifies operations compared to managing individual images. You can perform operations like snapshots and failover on the entire group, rather than on individual images.
  

• Crash Consistency: Consistency groups ensure that all images within the group are in a crash-consistent state. This means that if a failure occurs, the replicated volumes will be in a state that can be recovered without data loss or corruption.

How it Works: Under the Hood

⭐Key Components 

1. Consistency Group (CG):

• A logical collection of RBD images

• Created and managed using rbd CLI

2. Snapshot-based Mirroring

• Uses Manual group mirror snapshot or scheduled Group mirror snapshot 

3. RBD Mirror Daemon

• Extended to support Consistency Group Mirroring (CGM)

4. Snapshot Scheduling 

• Policy that defines the frequency of group mirror snapshot creation and replication

🧠Workflow 

1. Enable Image mode mirroring on the pool for both clusters

2. Bootstrapping (exchange token) to peer both the clusters

3. Create a Consistency Group

4. Add multiple related RBD images to the group

5. Enable Group Mirroring

6. Define snapshot scheduling interval on the group

7. Rbd-mirror detects group mirror snapshots

8. Snapshots are mirrored to the remote cluster atomically

9. In Disaster Recovery event, remote images are promoted together as a single unit in a group
   
   

Key Features and Operations

🔑Enabling Mirroring: You can enable mirroring on a consistency group using the command-line interface (CLI)

Enabling Group Mirroring CLI: rbd mirror group enable <pool-name>/<group-name>

This command internally takes a mirror group snapshot. Whenever a mirror group snapshot is created either internally by system (for example: using above enable command) or externally by the user, the rbd Mirror daemon on the remote site pulls the snapshots. Dynamic addition or removal of images in the group is not allowed after enabling mirroring on the group. If you want to add new images or remove existing images, disable mirroring first, followed by adding or removing images from the group and re-enabling the mirroring on the group again. Only after enabling mirroring on a group, you can perform other operations like promote/demote, resync, setting group mirror snapshot schedule etc.

❌Disabling Mirroring: You can disable mirroring on a consistency group using the command-line interface (CLI)

Disabling Group Mirroring CLI: rbd mirror group disable <pool-name>/<group-name>

After disabling mirroring on the group, the group will stop participating in mirroring. You’wll also be able to delete the image or remove the image from the group. Once you disable mirroring on a group, the group and images belonging to the group will no longer be seen on the remote site. After disabling mirroring on a group you will not be able to perform any group related operations like promote/demote, resync, setting group mirror snapshot schedule etc.

One can use - -force in the above CLI to disable even if not primary.

🔍Status Monitoring: The CLI also allows you to check the mirroring status of a group.

Group Mirroring Monitoring CLI: rbd mirror group status <pool-name>/<group-name>

This command helps to inspect the health of mirroring, detect sync delays or failures, the replication progress in percentage for individual images in the group, replay state of the images, promotion and demotion states (Primary and secondary roles of the cluster) and a lot more.

Sample output:

⚠️ Common States in Group Monitoring

➕Promote/Demote: These operations switch the role of a cluster (e.g., from primary to secondary (Demote) or, from secondary to primary (Promote) ).

The Promote operation is performed on the secondary (target) consistency group to elevate it to the primary role, enabling it to take over as the active group.

Group Promote CLI: rbd mirror group promote <pool-name>/<group-name>

Use Case of Group Promote:

• Performed during a planned switchover or unplanned failover.

• Common in disaster recovery (DR) when the original primary site becomes unavailable.

• The promoted site becomes active and can start serving read/write operations.

• Makes the secondary storage site writable.

The Demote operation transitions a consistency group from a primary (source) role to a secondary (target) role, typically following a failover event or for maintenance activities. As a result, the group becomes read-only and is no longer writable.

Group Demote CLI: rbd mirror group demote <pool-name>/<group-name>

Use Case of Group Demote:

• Used when the original primary site has been recovered, and the DR site is currently active.

• Often part of a failback process to restore the original primary site's role.

• Makes the former primary read-only or replication target again.

• Requires synchronization to ensure the original source is updated with all changes.

• Typically precedes a re-establishment of mirroring direction.
  
  
  

🪡Resynchronization: This operation allows you to resynchronize the mirrored images if they get out of sync. After a failover, split-brain, or outage, a resync operation is often necessary to restore consistency between the primary and secondary clusters. A resync operation ensures that the mirrored consistency group images on the secondary site are in sync with the primary site.

Group Resync CLI: rbd mirror group resync <pool-name>/<group-name>

This command when run on a secondary cluster manually triggers a full resync from the primary to the secondary for that consistency group. You can monitor the progress of resync operation using CLI below.

Monitoring CLI: rbd mirror group status <pool-name>/<group-name>

🪞Snapshotting: Creating snapshots of consistency groups for mirroring is crucial for snapshot-based mirroring.

In addition to scheduled group mirroring snapshots, you have the flexibility to manually create snapshots for data replication using the CLI commands below.

Group Mirror Snapshot: rbd mirror group snapshot <pool-name>/<group-name>

Snapshot names are automatically generated. Snapshot ID is returned back as an output to above CLI. You can check created mirror group snapshot using below:

Group Snapshot List: rbd group snap list <pool-name>/<group-name>

⏱️Scheduling: You can schedule group mirror snapshot creation for consistency groups to automate the replication process in the frequency of a specified interval. This ensures regular backup or synchronization of the data on the images of the group. One can also remove, list and check the status of the mirror group snapshot schedule.

Add mirror group snapshot schedule using below CLI:

Create Group Mirror Snapshot Schedule: rbd mirror group snapshot schedule add --pool <pool-name> --group <group-name> <interval> [<start-time>]

Remove mirror group snapshot schedule using below CLI:

Remove Group Mirror Snapshot Schedule: rbd mirror group snapshot schedule remove --pool <pool-name> --group <group-name> <interval> [<start-time>]

List mirror group snapshot schedule using below CLI:

List Group Mirror Snapshot Schedule: rbd mirror group snapshot schedule list --pool <pool-name> --group <group-name>

Check mirror group snapshot schedule status using below CLI. This is to View the status for the next snapshot to be created:

Check Group Mirror Snapshot Schedule Status: rbd mirror group snapshot schedule status --pool <pool-name> --group <group-name>

♾️Scale Limits

The Consistency Group Mirroring feature is designed with scalability in mind, allowing flexible grouping and mirroring of images based on operational needs.

• A single consistency group can contain a maximum of 50 images.

• The overall limit for mirrored images within a cluster is 100 images, which includes both standalone mirrored images and those that are part of a consistency group.

Depending on the mirroring strategy, the number of groups supported will vary:

• In scenarios where each group contains only 1 image, up to 100 groups can be created.

• Conversely, to accommodate the maximum number of 100 mirrored images, the minimum number of groups required would be:
  100 images ÷ 50 images/group = 2 groups

It’s important to note that standalone mirrored images also consume from the same 100-image cluster-wide limit. As such, the number of images available for use in consistency groups will be reduced accordingly. For example, if 30 images are already mirrored as standalone images, only 70 images remain available for consistency group mirroring.

This shared quota ensures that administrators can mix and match standalone and grouped image mirroring flexibly, within the total capacity constraints of the system.

🪜Step-By-Step Guide

🎖️Pre-requisites

1. A minimum of two running IBM Storage Ceph clusters

2. Ceph Version 8.1 or later. Ensure both clusters (primary and secondary) run compatible versions

3. Running rbd-mirror daemon on secondary for one-way mirroring and on both (primary and secondary) for two-way mirroring.

4. Root level access to nodes

5. Group Mirroring works with Image mode only so make sure pool mirroring is enabled in the image mode. Refer Step#2
   
   

⚒️Configure mirroring

Step 1: Create rbd pool on both the clusters

Primary cluster:

Secondary cluster:

Step 2: Enable Image mode mirroring on pool on both clusters

Primary cluster:

Secondary cluster:

Step 3: Establish Peer connection between the clusters

On client node of primary cluster, generate the token:

Copy the token present in file “/root/bootstrap_token_site-a”.

On client node of secondary cluster:

Create file “/root/bootstrap_token_site-a” on the client node of the secondary cluster. Open the file and paste the copied token. Save the file.

Import the token using below CLI:

Step 4: Created Two Images in the pool on primary site.

⚓Exercise Group Mirroring

Step 1: Create Group on primary cluster

Verify group is created and mirroring is disabled by default.

Step 2: Add both created images to the group on primary site

Verify Images are added to the group.

Individual mirroring on the image can not be enabled once the images are part of the group.

Step 3: Enable Mirroring on the group on primary site

Verify mirroring is enabled on the group.

Enabling mirroring will create internally a mirror group snapshot.

Step 4: Monitor Mirroring status

Group state ‘up+stopped’ shows its a primary site and replay_state: ‘idle’ in ‘images’ section shows that both images have been replicated to the secondary site. If the replay_state is ‘syncing’, that means images are still under transit. Wait till the state shows ‘idle’.

Step 5: Verify Groups and Images on secondary cluster

Also, verify mirroring status on secondary site:

Group state as ‘up+replaying’ shows its a secondary site.

Step 6: Create manual mirror group snapshot

You can also verify the created snapshot.

The first snapshot in the list was created during enablement of mirroring on the group. Refer Step#3
The second snapshot listed is created manually above. Whenever a mirror group snapshot is created either manually or internally by system, images in the group will get replicated to the target site. The snapshot state in the list on the secondary site will show whether the replication of the snapshot is completed or not. If not, the state will be shown as “in-complete”. Wait till state changes to “complete”.

Step 7: Add & Monitor group mirroring Schedule

We can see a group mirror snapshot schedule of every 5 minutes is added at 8:55 UTC and the status shows the next snapshot time as 9:00 UTC, i.e. 5 minutes ahead of 8:55 UTC.

You can also choose to remove this schedule using below CLI:

Step 8: Demote the primary site & Promote secondary site

In a planned failover, you can demote the primary site to change its role to that of secondary and promote secondary site to change its role to primary (Read/Write). In an unplanned failover scenario, one can directly promote the secondary site with a –force flag.

Mirroring Primary field as false shows it is no more primary after performing a demote operation on the group.

Mirroring Primary field as true shows the site is primary after promoting operation on the group. You can also verify the same by checking group state in “rbd mirror group status <pool-name>/<group-name>”.

Step 9: Resynchronization

In an unplanned failover, once the former primary site is back. You can demote the former primary site followed by the resync operation on the former primary site which will pull the latest data from the current primary site. Further you can demote the current primary site and promote the one which is back from disaster. 

When the former primary is up, perform demote on former primary.

Resync on former primary site to pull latest data from the other cluster:

Wait for resync to complete,

Demote current primary,

Promote the site which is back from disaster

Step 10: Disable Group mirroring

Once disabled, you will not be able to see group and its images on the secondary site.

Limitations

• Images from different pool can not be added to the consistency group

• Clone images can not be the part of consistency group

• Dynamic addition and removal of images to the consistency group is not supported when mirroring on the group is in enabled state
  
  

Conclusion

Consistency Group Mirroring in Ceph RBD provides a robust, enterprise-grade solution for replicating multiple block images together. This capability is essential for protecting complex applications—like databases or virtual machines—that span multiple volumes and require crash-consistent snapshots during replication.

By grouping RBD images and leveraging snapshot-based mirroring, Ceph ensures that all related data is replicated atomically and consistently across clusters, enabling seamless disaster recovery, failover, and failback operations. The ability to schedule automated group snapshots and perform manual syncs or promotions gives administrators fine-grained control and operational flexibility.

As organizations increasingly rely on scalable, software-defined infrastructure, Ceph’s CG Mirroring offers a powerful mechanism to enhance data consistency and application resilience, all within an open-source, distributed storage platform. 

For more details please refer to the IBM Storage Ceph Documentation.