IBM Fusion Data Foundation stretch cluster on IBM Z: Pacemaker-managed arbiter across two availability zones
Platform: Red Hat® Enterprise Linux® (RHEL) 9.6 (IBM Z® / s390x)
Red Hat OpenShift Container Platform (RHOCP) version: 4.21
IBM Fusion® version: 2.13
IBM Fusion Data Foundation® (FDF) version: 2.21
Before you begin: Performance figures reflect a specific IBM Z reference environment, so treat them as indicative, not as product guarantees. This solution requires a single Fibre Channel Protocol (FCP) logical unit number (LUN) zoned to both Kernel-based Virtual Machine (KVM) hosts on the shared storage area network (SAN) fabric. It serves both the SBD fencing device and the VM's block disk. Without it, automatic failover will not function. Confirm that the LUN is accessible from both hosts before starting any software steps.
Note: This guide describes a reference implementation validated in a specific IBM Z environment. Verify support status with IBM and Red Hat before deploying in production.
Summary
IBM Fusion Data Foundation® (FDF) stretch cluster mode requires a dedicated third availability zone to host the Ceph® arbiter monitor. On IBM Z®, provisioning a dedicated zone for a single monitor process is often impractical. This guide shows how to run the arbiter control-plane node as a Pacemaker-managed KVM virtual machine (VM) on the two KVM hosts that already serve as your data zones. When the host running the arbiter VM fails, Pacemaker automatically fences the failed host and restarts the VM on the surviving host — no dedicated hardware required.
The guide covers the complete implementation in six phases: Pacemaker cluster setup, SBD storage-based fencing, VM disk migration to shared Logical Volume Manager (LVM) storage, libvirt live migration, automatic failover configuration, and IBM Fusion Data Foundation stretch cluster deployment. A reference environment on IBM Z completed automatic failover in approximately 23 seconds, with CephRBD write I/O resuming within 40 to 90 seconds. Manual recovery procedures are included for the scenario where automatic fencing cannot complete due to network isolation.
Requirements at a glance: Two IBM Z KVM hosts running Red Hat® Enterprise Linux® (RHEL) 9.6, one shared FCP LUN zoned to both hosts, Red Hat OpenShift Container Platform (RHOCP) 4.21, and IBM Fusion Data Foundation 2.21.
Introduction
IBM Fusion Data Foundation® (FDF) stretch cluster mode distributes storage across two availability zones within a single data centre to provide zone-level fault isolation. It requires a third arbiter zone, which acts as a tiebreaker that maintains Ceph® quorum when one availability zone fails. The solution is designed for deployments where the round-trip latency between data zones does not exceed 10 ms round-trip time (RTT). The arbiter zone follows the more relaxed etcd latency bounds. All three zones must have network connectivity to each other.
The challenge is that provisioning a dedicated third zone purely for an arbiter monitor is often impractical or expensive, especially on IBM Z®.
This guide shows a practical alternative using the KVM hosts already present in each availability zone: run the RHOCP arbiter control-plane node as a Pacemaker-managed KVM VM. When the KVM host running the arbiter fails, Pacemaker automatically restarts it on the surviving host. The arbiter zone effectively floats between the two KVM hosts, with no dedicated infrastructure required.
The result is a complete FDF stretch cluster on two KVM hosts, with automatic arbiter failover, quantified storage I/O impact, and a manual recovery path for the scenario where automatic fencing cannot complete.
Table of contents
- Architecture overview
- Prerequisites
- Phase 0: Red Hat subscription and repository setup
- Phase 1: Pacemaker cluster setup
- Phase 2: Storage and fencing configuration
- Phase 3: VM disk migration and CPU configuration
- Phase 4: Live migration configuration
- Phase 5: Automatic failover configuration
- FDF stretch cluster deployment
- Validation and testing
- Operational procedures
- Troubleshooting
- Appendix
Architecture overview
The problem this solves
FDF stretch cluster requires three topology zones: two data zones and one arbiter zone. In this design, the two KVM hosts are placed in the two data zones, while the arbiter control-plane VM keeps the Kubernetes® topology label arbiter regardless of which host currently runs it. Running it as a Pacemaker-managed KVM VM makes the arbiter zone a logical construct that floats between the two KVM hosts, always available on whichever host is running.
Topology
Node zone distribution
| Node |
Zone |
Role |
FDF Role |
| master-0 |
zone1 |
control-plane |
Ceph mon (zone1) |
| master-1 |
zone2 |
control-plane |
Ceph mon (zone2) |
| master-2 |
arbiter |
control-plane |
Ceph arbiter mon |
| worker-0, worker-2, worker-4 |
zone1 |
worker |
OSDs in zone1 |
| worker-1, worker-3 |
zone2 |
worker |
OSDs in zone2 |
Technology stack
| Component |
Technology |
Version |
| Cluster Manager |
Pacemaker |
2.1.9 |
| Cluster Communication |
Corosync |
3.1.9 |
| Fencing |
SBD (Storage-Based Death) |
1.5.2 |
| Watchdog |
softdog |
kernel |
| Storage |
LVM on FCP multipath |
— |
| Virtualization |
KVM/QEMU via libvirt |
10.10.0 |
| Live Migration |
Libvirt TCP port 16509 |
— |
| Operating System |
RHEL 9.6 (s390x) |
9.6 |
Implementation summary
| Phase |
Description |
Estimated Duration |
VM Downtime |
| Phase 0 |
Subscription and Repository Setup |
15 min |
None |
| Phase 1 |
Pacemaker Cluster Setup |
1 hour |
None |
| Phase 2 |
Storage and Fencing Configuration |
1 hour |
None |
| Phase 3 |
VM Disk Migration and CPU Configuration |
2 hours |
~30 min |
| Phase 4 |
Live Migration Configuration |
30 min |
None |
| Phase 5 |
Automatic Failover Configuration |
30 min |
None |
Total: around 5 to 6 hours, with one planned downtime window of approximately 30 minutes during Phase 3 (disk conversion). All other phases run without disrupting the VM.
Prerequisites
Hardware
- Two IBM Z® KVM hosts (s390x), minimum 16 vCPUs / 64 GB RAM each
- One shared FCP LUN (≥ 300 GB), presented to both KVM hosts via multipath
- RHOCP worker nodes with an additional raw block disk on each node for FDF OSDs
Software
- RHEL 9.6 (s390x) with active Red Hat® subscription on both KVM hosts
- An existing RHOCP cluster with:
- Three control-plane nodes (one per intended zone)
- At least four worker nodes (at least two per data zone)
master-2 currently running as a KVM VM on one of the two hosts
Network ports required
| Port |
Protocol |
Purpose |
| 2224 |
tcp |
pcsd (Pacemaker daemon) |
| 3121 |
tcp |
Pacemaker inter-node |
| 5403 |
udp |
Corosync |
| 16509 |
tcp |
Libvirt TCP live migration |
| 49152–49215 |
tcp |
Migration data transfer |
Naming conventions used in this guide
Throughout this guide, the two KVM hosts are referred to as kvm-host-1 and kvm-host-2. Replace these with your actual hostnames. Commands that run on one host only are labelled explicitly. Commands with no host label run on either host.
Phase 0: Red Hat subscription and repository setup
Step 0.1: Register Both Hosts
On both hosts:
subscription-manager register --username <username> --password <password>
Step 0.2: Pin Release Version and Enable Repositories
On both hosts:
echo "9.6" > /etc/dnf/vars/releasever
subscription-manager repos \
--enable=rhel-9-for-s390x-baseos-eus-rpms \
--enable=rhel-9-for-s390x-appstream-eus-rpms \
--enable=rhel-9-for-s390x-highavailability-eus-rpms
dnf clean all && dnf makecache
Validation
cat /etc/dnf/vars/releasever # Expected: 9.6
subscription-manager repos --list-enabled | grep "Repo ID"
dnf list available pcs pacemaker corosync sbd # No 404 errors
Phase 1: Pacemaker cluster setup
Step 1.1: Install Cluster Packages
On both hosts:
dnf install -y pcs pacemaker corosync fence-agents-all sbd
Step 1.2: Configure Firewall
On both hosts:
firewall-cmd --permanent --add-service=high-availability
firewall-cmd --reload
firewall-cmd --list-services | grep high-availability
Step 1.3: Set hacluster Password
On both hosts (use the same password on both):
passwd hacluster
Step 1.4: Start pcsd Service
On both hosts:
systemctl enable --now pcsd
systemctl status pcsd # Expected: active (running)
Step 1.5: Load softdog Watchdog
On both hosts:
modprobe softdog
echo "softdog" > /etc/modules-load.d/softdog.conf
lsmod | grep softdog
ls -l /dev/watchdog*
# Expected: /dev/watchdog and /dev/watchdog0 present
Step 1.6: Authenticate Cluster Nodes
On kvm-host-1 only:
pcs host auth kvm-host-1 kvm-host-2 -u hacluster -p 'YourPassword'
# Expected: kvm-host-1: Authorized / kvm-host-2: Authorized
Step 1.7: Create and Start the Cluster
On kvm-host-1 only:
pcs cluster setup ocp-arbiter-ha-cluster kvm-host-1 kvm-host-2
pcs cluster start --all
pcs cluster enable --all
sleep 10
pcs cluster status
Expected output:
* Stack: corosync (Pacemaker is running)
* Online: [ kvm-host-1 kvm-host-2 ]
Step 1.8: Configure Cluster Properties
On kvm-host-1 only:
# Temporarily disable STONITH until the SBD fencing device is configured
pcs property set stonith-enabled=false
# Required for 2-node clusters
pcs property set no-quorum-policy=ignore
Validation
pcs status
# Expected: both nodes Online, 0 resources, all daemons active/enabled
Phase 2: Storage and fencing configuration
⚠️ RHEL 9 LVM Prerequisites: RHEL 9 uses a new device file (/etc/lvm/devices/system.devices) that restricts LVM scanning to specific paths. On shared cluster storage, this prevents the second host from discovering the VG created on the first host. Disable it on both hosts before proceeding:
cat >> /etc/lvm/lvm.conf << 'EOF'
devices {
use_devicesfile = 0
}
EOF
This restores the traditional LVM discovery method (scan all block devices), essential for shared storage.
Step 2.1: Identify the Shared FCP LUN
On both hosts:
multipath -ll
Identify the multipath device that corresponds to the shared LUN. In the examples below, this device is referred to as <YOUR-SHARED-DISK-WWID>. Substitute your actual WWID. Verify: ls -l /dev/disk/by-id/ | grep scsi
The shared disk will be partitioned as follows:
/dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part1 — 10 MB → SBD fencing device
/dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part2 — remaining space → LVM volume group
Step 2.2: Partition the Shared Disk
On kvm-host-1 only:
Pre-check: Verify the disk is blank (or wipe it if reused from a previous setup):
SHARED_DISK="/dev/disk/by-id/<YOUR-SHARED-DISK-WWID>"
# Check for existing partition table or filesystem signatures
wipefs -n $SHARED_DISK # Show what would be wiped (non-destructive)
# If non-empty output appears above, wipe it (safe — only removes signatures):
wipefs -a $SHARED_DISK
# Verify disk is now blank
parted $SHARED_DISK print # Should show "Unrecognised disk label"
Create the partition table:
SHARED_DISK="/dev/disk/by-id/<YOUR-SHARED-DISK-WWID>"
parted -s $SHARED_DISK mklabel gpt
parted -s $SHARED_DISK mkpart primary 1MiB 11MiB # SBD partition
parted -s $SHARED_DISK mkpart primary 11MiB 100% # LVM partition
parted $SHARED_DISK print
lsblk $SHARED_DISK
# Expected: two partitions — 10 MB and remaining space
Step 2.3: Initialize SBD Device
On kvm-host-1 only:
⚠️ Use the partition, never the raw disk. If you accidentally run sbd … create on the raw disk (/dev/disk/by-id/<WWID> without -part1), it overwrites the partition table starting at byte 0, corrupting both partitions. Always target -part1 explicitly.
SBD_PART="/dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part1"
sbd -d $SBD_PART create
sbd -d $SBD_PART dump
# Expected: Header version 2.1, Number of slots: 255
Step 2.4: Configure SBD on Both Hosts
On both hosts:
cat > /etc/sysconfig/sbd << EOF
SBD_DEVICE="/dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part1"
SBD_WATCHDOG_DEV="/dev/watchdog"
SBD_WATCHDOG_TIMEOUT="5"
SBD_STARTMODE="always"
EOF
systemctl enable sbd # Do NOT start manually — Pacemaker starts SBD
systemctl is-enabled sbd # Expected: enabled
Note: systemctl start sbd returns "Operation refused, unit may be requested by dependency only". This is expected — SBD is started by Pacemaker's dependency chain.
Step 2.5: Enable STONITH and Activate SBD
On kvm-host-1 only:
pcs property set stonith-enabled=true
pcs cluster stop --all
pcs cluster start --all
sleep 15
systemctl status sbd
ssh kvm-host-2 "systemctl status sbd"
# Expected: active (running) on both hosts
Step 2.6: Create SBD STONITH Resource
On kvm-host-1 only:
pcs stonith create sbd_fencing fence_sbd \
devices=/dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part1 \
pcmk_delay_base=30s \
op monitor interval=300s timeout=30s
pcs stonith status
# Expected: sbd_fencing (stonith:fence_sbd): Started <node>
sbd -d /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part1 list
# Expected:
# 0 kvm-host-2.example.com clear
# 1 kvm-host-1.example.com clear
Step 2.7: Create LVM Volume Group and Logical Volume
On kvm-host-1 only:
⚠️ Reusing this partition? If this disk was previously used (lab reset, troubleshooting loop), old LVM metadata may exist. If pvcreate fails with "without -ff" or vgcreate fails with "already exists in filesystem", see Troubleshooting → "pvcreate fails: 'without -ff' on reused disk" and "vgcreate fails: 'already exists in filesystem'" for cleanup commands.
pvcreate /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part2
vgcreate vg_shared /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part2
lvcreate -L 130G -n lv_master2 vg_shared
lvs vg_shared
# Expected: lv_master2 vg_shared -wi-a----- 130.00g
Adjust the logical volume size to match the disk image of your master-2 VM.
Step 2.8: Activate LVM on the Second Host
On kvm-host-2:
pvscan --cache
vgscan --cache
vgchange -ay vg_shared
lvs
# Expected: lv_master2 visible and active
If the LV is not visible:
pvscan --cache /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part2
vgchange -ay vg_shared
Phase 2 Validation
systemctl is-active sbd # active — verify on both hosts
pcs stonith status # sbd_fencing Started
pcs property show stonith-enabled # true
lvs vg_shared # lv_master2 visible on both hosts
Phase 3: VM disk migration and CPU configuration
⚠️ Planned downtime required. This phase shuts down the master-2 VM to convert its disk from a local qcow2 image to a raw block device on shared LVM. The conversion itself takes approximately 15–30 minutes depending on disk size and storage performance.
Step 3.1: Verify RHOCP health before starting
oc get nodes # All nodes must be Ready before proceeding
Step 3.2: Shut Down the VM
On kvm-host-1:
virsh shutdown master-2
watch -n 10 'virsh list --all | grep master-2'
# Wait until state shows: shut off
Step 3.3: Back Up VM Configuration
virsh dumpxml master-2 > /root/master-2-backup-$(date +%Y%m%d-%H%M%S).xml
Step 3.4: Convert Disk to Shared Block Storage
virsh domblklist master-2 # Confirm the current disk path
SOURCE_DISK="/var/lib/libvirt/images/<your-cluster>/master-2.qcow2" # Path to master-2 source disk from previous command.
DEST_DISK="/dev/vg_shared/lv_master2"
qemu-img convert -p -O raw $SOURCE_DISK $DEST_DISK
qemu-img info $DEST_DISK
# Expected: format: raw, virtual size matching your source image
Step 3.5: Update VM Configuration
virsh dumpxml master-2 > /tmp/master-2-new.xml
# Update disk driver to raw
sed -i "s|<driver name='qemu' type='qcow2'.*/>|<driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>|" /tmp/master-2-new.xml
# Change disk type to block device
sed -i "s|<disk type='file'|<disk type='block'|" /tmp/master-2-new.xml
# Update disk source path
sed -i "s|<source file='.*'/>|<source dev='/dev/vg_shared/lv_master2'/>|" /tmp/master-2-new.xml
# Set CPU mode expected to work for live migration in this reference environment.
# This assumes both IBM Z KVM hosts expose compatible CPU features.
sed -i 's|<cpu mode=.host-model. check=.partial.*/>|<cpu mode="host-passthrough" check="none"/>|' /tmp/master-2-new.xml
# Verify changes before applying
grep -A3 "<disk type='block'" /tmp/master-2-new.xml
grep '<cpu' /tmp/master-2-new.xml
# Expected:
# <disk type='block' device='disk'>
# <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
# <source dev='/dev/vg_shared/lv_master2'/>
# <cpu mode='host-passthrough' check='none'/>
virsh define /tmp/master-2-new.xml
Step 3.6: Copy Configuration to Second Host
scp /tmp/master-2-new.xml kvm-host-2:/tmp/
ssh kvm-host-2 "virsh define /tmp/master-2-new.xml"
Step 3.7: Disable Autostart on Both Hosts
⚠️ Critical: Pacemaker owns the VM lifecycle. If autostart is enabled, the VM may start outside of Pacemaker's control on host boot, which bypasses split-brain protection.
On both hosts:
virsh autostart --disable master-2
virsh dominfo master-2 | grep Autostart
# Expected: Autostart: disable
Step 3.8: Start VM and verify RHOCP recovery
virsh start master-2
sleep 90
virsh list | grep master-2
oc get nodes # Wait 2–3 minutes — all RHOCP nodes should return to Ready
Phase 3 Validation
virsh domblklist master-2
# Expected: vda /dev/vg_shared/lv_master2
virsh dumpxml master-2 | grep '<cpu'
# Expected: <cpu mode='host-passthrough' check='none'/>
Phase 4: Live migration configuration
Step 4.1: Configure Libvirtd for TCP
On both hosts:
grep -q "^listen_tcp" /etc/libvirt/libvirtd.conf \
&& sed -i 's/^listen_tcp.*/listen_tcp = 1/' /etc/libvirt/libvirtd.conf \
|| echo 'listen_tcp = 1' >> /etc/libvirt/libvirtd.conf
# Warning: this enables unauthenticated, unencrypted libvirt TCP migration.
# Use it only on a trusted private network that is isolated from untrusted clients.
grep -q "^auth_tcp" /etc/libvirt/libvirtd.conf \
&& sed -i 's/^auth_tcp.*/auth_tcp = "none"/' /etc/libvirt/libvirtd.conf \
|| echo 'auth_tcp = "none"' >> /etc/libvirt/libvirtd.conf
grep -q "^tcp_port" /etc/libvirt/libvirtd.conf \
&& sed -i 's/^tcp_port.*/tcp_port = "16509"/' /etc/libvirt/libvirtd.conf \
|| echo 'tcp_port = "16509"' >> /etc/libvirt/libvirtd.conf
grep -q "^listen_tls" /etc/libvirt/libvirtd.conf \
&& sed -i 's/^listen_tls.*/listen_tls = 0/' /etc/libvirt/libvirtd.conf \
|| echo 'listen_tls = 0' >> /etc/libvirt/libvirtd.conf
Step 4.2: Enable Libvirtd TCP Socket
On both hosts:
systemctl stop libvirtd
systemctl enable libvirtd-tcp.socket
systemctl start libvirtd-tcp.socket
systemctl status libvirtd-tcp.socket
ss -tlnp | grep 16509 # Expected: port 16509 listening
Step 4.3: Open Firewall Ports
On both hosts:
firewall-cmd --permanent --add-port=16509/tcp
firewall-cmd --permanent --add-port=49152-49215/tcp
firewall-cmd --reload
firewall-cmd --list-ports | grep -E "16509|49152"
Step 4.4: Test TCP Connectivity
virsh -c qemu+tcp://kvm-host-2/system list
ssh kvm-host-2 "virsh -c qemu+tcp://kvm-host-1/system list"
# Expected: VM list returned from each remote host with no errors
Step 4.5: Test Live Migration
With master-2 running on kvm-host-1:
# Migrate to kvm-host-2
virsh migrate --live --verbose master-2 qemu+tcp://kvm-host-2/system
sleep 10
ssh kvm-host-2 "virsh list | grep master-2" # Expected: running on kvm-host-2
# Migrate back
ssh kvm-host-2 "virsh migrate --live --verbose master-2 qemu+tcp://kvm-host-1/system"
virsh list | grep master-2 # Expected: running on kvm-host-1
Success criteria: Port 16509 is listening on both hosts and bidirectional live migration completes without errors.
Phase 5: Automatic failover configuration
Step 5.1: Hand the VM to Pacemaker
virsh shutdown master-2
watch -n 5 'virsh list --all | grep master-2' # Wait: shut off
Step 5.2: Create the VirtualDomain Pacemaker Resource
On kvm-host-1 only:
pcs resource create master-2-vm VirtualDomain \
hypervisor='qemu:///system' \
config='/etc/libvirt/qemu/master-2.xml' \
migration_transport='tcp' \
meta allow-migrate=true \
op start timeout=120s \
op stop timeout=120s \
op monitor interval=30s timeout=30s \
op migrate_to timeout=120s \
op migrate_from timeout=120s
Step 5.3: Configure Resource Policies
pcs resource meta master-2-vm \
resource-stickiness=100 \
migration-threshold=3 \
failure-timeout=300s
| Attribute |
Value |
Effect |
resource-stickiness |
100 |
Prefers to remain on its current host |
migration-threshold |
3 |
Relocates after 3 consecutive failures |
failure-timeout |
300s |
Resets failure count after 5 minutes |
Step 5.4: Verify Resource Started
pcs status
Expected:
Full List of Resources:
* sbd_fencing (stonith:fence_sbd): Started kvm-host-1
* master-2-vm (ocf:heartbeat:VirtualDomain): Started kvm-host-1
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
sbd: active/enabled
Phase 5 Validation
virsh list | grep master-2 # Expected: running
pcs resource status master-2-vm # Expected: Started
oc get nodes | grep master-2 # Expected: Ready
FDF stretch cluster deployment
With the Pacemaker high availability (HA) layer in place and master-2 running under Pacemaker control, deploy IBM Fusion Data Foundation® in stretch cluster (arbiter) mode.
Step 6.1: Label nodes with topology zones
oc label node master-0.<cluster-domain> topology.kubernetes.io/zone=zone1
oc label node worker-0.<cluster-domain> topology.kubernetes.io/zone=zone1
oc label node worker-2.<cluster-domain> topology.kubernetes.io/zone=zone1
oc label node worker-4.<cluster-domain> topology.kubernetes.io/zone=zone1
oc label node master-1.<cluster-domain> topology.kubernetes.io/zone=zone2
oc label node worker-1.<cluster-domain> topology.kubernetes.io/zone=zone2
oc label node worker-3.<cluster-domain> topology.kubernetes.io/zone=zone2
oc label node master-2.<cluster-domain> topology.kubernetes.io/zone=arbiter
Verify:
oc get nodes -L topology.kubernetes.io/zone
Step 6.2: Deploy the IBM Fusion® operator
6.2.1: Obtain an Entitlement Key
An IBM entitlement key is required to pull IBM Fusion® images from the IBM Container Registry.
- Log in to https://myibm.ibm.com/products-services/containerlibrary
- Generate or copy your entitlement key.
6.2.2: Update the RHOCP global pull secret
The RHOCP global pull secret must be updated to include credentials for cp.icr.io.
1. Create a base64-encoded credential string:
echo -n "cp:<ENTITLEMENT_KEY>" | base64 -w0
2. Create an authority.json file with the encoded credentials:
{
"auth": "<BASE64_ENCODED_ENTITLEMENT_KEY>"
}
3. Merge the new credential into the existing pull secret:
Note: Install jq if not already present: yum install jq
oc get secret/pull-secret -n openshift-config -ojson \
| jq -r '.data[".dockerconfigjson"]' \
| base64 -d \
| jq '.[]."cp.icr.io" += input' - authority.json \
> temp_config.json
4. Verify temp_config.json contains a cp.icr.io entry, then apply:
oc set data secret/pull-secret -n openshift-config \
--from-file=.dockerconfigjson=temp_config.json
6.2.3: Create the IBM Operator CatalogSource
oc apply -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: ibm-operator-catalog
namespace: openshift-marketplace
spec:
displayName: IBM Operator Catalog
publisher: IBM
sourceType: grpc
image: icr.io/cpopen/ibm-operator-catalog:latest
updateStrategy:
registryPoll:
interval: 45m
EOF
6.2.4: Install the IBM Fusion® operator from OperatorHub
Navigate to Operators > OperatorHub in the RHOCP web console, search for IBM Fusion, and install it with the following settings:
- Update approval: Manual (IBM Fusion does not support automatic updates)
- All other settings: leave as default
Step 6.3: Deploy Fusion Data Foundation in local mode
Access the IBM Fusion web console: from the Applications menu in the RHOCP title bar, click IBM Fusion. Then navigate to:
Services > Data Foundation > Install > Install Service: Local > Update approval: Off > Install
This will deploy both the Fusion Data Foundation operator and the Local Storage operator.
Step 6.4: Create the StorageCluster
On the Local Storage page, click Get Started, then follow the wizard:
- Backing storage: Select Create a new StorageClass using the local storage devices
- Create local volume set: Name the LocalVolumeSet and StorageClass. Verify that four disks are detected across the worker nodes.
- Capacity and nodes page:
- Select Enable arbiter
- Select
arbiter as the arbiter zone
- Select all worker nodes as storage nodes
Note: Remaining configuration values can be kept at their defaults or adjusted as required. Refer to the official IBM documentation for advanced configuration options:
Configuring Fusion Data Foundation in local mode
The resulting StorageCluster CR will include:
spec:
arbiter:
enable: true
nodeTopologies:
arbiterLocation: arbiter
storageDeviceSets:
- count: 1
replica: 4 # 2 OSDs in zone1, 2 OSDs in zone2
...
Step 6.5: Verify FDF deployment
oc -n openshift-storage get storagecluster
# Expected: PHASE = Ready
oc patch storagecluster ocs-storagecluster -n openshift-storage --type json --patch '[{ "op": "replace", "path": "/spec/enableCephTools", "value": true }]'
oc -n openshift-storage rsh $(oc get pods -n openshift-storage -l app=rook-ceph-tools -o name) ceph -s
# Expected:
# stretch_mode: ENABLED
# health: HEALTH_OK
# mon: 5 daemons, quorum a,b,c,d,e
# osd: 4 osds: 4 up, 4 in
Validation and testing
Test 1: Cluster state
pcs status # Clean — no Failed Resource Actions
pcs constraint config --full # No unexpected constraints
pcs property show # stonith-enabled=true, no-quorum-policy=ignore
Test 2: Graceful standby — other VMs must not be affected
This test confirms that putting a node in standby migrates master-2-vm but leaves all other VMs on that host running normally.
# Confirm current location of master-2-vm
pcs status | grep master-2-vm
# Put kvm-host-2 in standby
pcs node standby kvm-host-2
sleep 30
# Verify master-2-vm has migrated to kvm-host-1
pcs status | grep master-2-vm
# Expected: Started kvm-host-1
# Verify all other VMs on kvm-host-2 are still running
ssh kvm-host-2 "virsh list --all"
# Expected: bastion, master-0, and all zone1 workers still running
# Restore kvm-host-2
pcs node unstandby kvm-host-2
sleep 15
pcs status
Test 3: Live migration via Pacemaker
pcs resource move master-2-vm kvm-host-2 --wait=120
pcs status | grep master-2-vm # Expected: Started kvm-host-2
pcs resource clear master-2-vm # Remove migration constraint
Reference environment observation: Live migration completed in approximately 25 seconds with no application downtime on the migrated VM. Results vary by hardware and storage throughput.
Test 4: Automatic failover
Warning: This test causes master-2 to be temporarily unavailable. Pacemaker will detect the cluster stack failure on the active host and restart the VM on the surviving host. kvm-host-2 will go offline and must be recovered afterward.
# Ensure master-2-vm is on kvm-host-2
pcs resource move master-2-vm kvm-host-2 --wait=120
pcs resource clear master-2-vm
# Simulate a cluster stack failure on the active host
ssh kvm-host-2 "systemctl stop pacemaker"
# Monitor recovery
watch -n 2 'pcs status'
# master-2-vm will appear on kvm-host-1 once fencing completes
# kvm-host-2 will show as OFFLINE
# Cleanup fencing and resources
pcs stonith history cleanup
pcs resource cleanup
pcs status # Expected: master-2-vm on kvm-host-1, kvm-host-2 offline
Restore kvm-host-2 to the cluster:
# On kvm-host-2, restart Pacemaker to rejoin the cluster
ssh kvm-host-2 "systemctl start pacemaker"
sleep 15
# Verify both nodes are online and healthy
pcs status # Expected: both nodes Online, resources clean
Reference environment observation: Automatic failover completed in approximately 23 seconds from host failure detection to master-2-vm running on the surviving host. RHOCP master-2 did not enter a NotReady state during this period. The only disruption was CephRBD write I/O, which resumed within 40 to 90 seconds. CephFS reads were unaffected throughout. Results vary by hardware, storage, and network.
Test 5: RHOCP cluster health
oc get nodes
# Expected: all nodes Ready
oc get co
# Expected: all operators Available=True, Progressing=False, Degraded=False
Operational procedures
Daily health check
pcs status # Verify: both nodes Online, all resources Started, no failures
oc get nodes # Verify: all nodes Ready
oc get co # Verify: all operators Available
Planned maintenance: KVM host standby
Use this procedure when patching or rebooting a KVM host to avoid unplanned failover.
# Step 1 — Move master-2-vm to the host that will remain active
pcs resource move master-2-vm kvm-host-2 --wait=120
pcs status | grep master-2-vm # Verify: Started kvm-host-2
# Step 2 — Put the maintenance host in standby
pcs node standby kvm-host-1
# Step 3 — Perform maintenance on kvm-host-1 ...
# Step 4 — Bring the host back
pcs node unstandby kvm-host-1
sleep 15
# Step 5 — Remove migration constraint
pcs resource clear master-2-vm
pcs status
After any unplanned host reboot
Pacemaker automatically restarts all resources when a node rejoins. Verify:
pcs status
# Expected: master-2-vm Started, sbd_fencing Started, no failures
If master-2-vm remains Stopped:
pcs resource cleanup master-2-vm
VM management rules
Always manage master-2 through Pacemaker. Direct virsh commands bypass split-brain protection and must not be used on a Pacemaker-managed VM.
| Action |
Correct Command |
Never Use |
| Start master-2 |
pcs resource enable master-2-vm |
virsh start master-2 |
| Stop master-2 |
pcs resource disable master-2-vm |
virsh shutdown master-2 |
| Migrate master-2 |
pcs resource move master-2-vm <node> |
virsh migrate ... |
| Check status |
pcs status |
— |
| Clear failures |
pcs resource cleanup master-2-vm |
— |
| Remove move constraint |
pcs resource clear master-2-vm |
— |
Troubleshooting
pcs cluster start --all returns HTTP error 400
softdog is not loaded on the second host:
ssh kvm-host-2 "modprobe softdog && systemctl restart pcsd"
pcs cluster start --all
master-2-vm fails to start
pcs resource failcount show master-2-vm
journalctl -u pacemaker -n 50 --no-pager | grep -i "master-2\|error"
ls -l /dev/vg_shared/lv_master2 # Confirm the LV is accessible
pcs resource cleanup master-2-vm
master-2-vm stuck in Stopping
pcs resource cleanup master-2-vm
SBD fencing not working
systemctl status sbd
sbd -d /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part1 dump
ls -l /dev/watchdog*
cat /etc/sysconfig/sbd
Hundreds of failed fencing actions in pcs status
Normal after a crash test or an unplanned node reboot. Clear with:
pcs stonith history cleanup
pcs resource cleanup
pvcreate fails: "without -ff" on reused disk
LVM metadata survives partial disk wipes. Clean it:
PART2="/dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part2"
vgremove -ff vg_shared 2>/dev/null
pvremove -ff $PART2
wipefs -a $PART2
pvcreate $PART2
vgcreate fails: "already exists in filesystem"
vgremove removed metadata but left device-mapper nodes. Remove them:
dmsetup ls | grep vg_shared
dmsetup remove vg_shared-lv_master2 # remove each LV node
dmsetup remove vg_shared 2>/dev/null # may already be gone
vgcreate vg_shared /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part2
Resource fails after node rejoins cluster
pcs resource cleanup
pcs status
# If master-2-vm is still Stopped:
systemctl restart libvirtd # On the node where it should start
pcs resource cleanup master-2-vm
KVM host network isolation: SBD fencing fails, master-2-vm stuck UNCLEAN
Symptom
One KVM host loses all network connectivity (for example, due to a datacenter network incident). The surviving host shows:
pcs status
Node kvm-host-2: UNCLEAN (offline)
master-2-vm (ocf:heartbeat:VirtualDomain): Started kvm-host-2 (UNCLEAN)
Failed Fencing Actions:
* reboot of kvm-host-2 ... last failed at ... ← repeating every ~15 min
Why automatic SBD fencing fails in this scenario
SBD fences a node by writing a poison pill to the shared disk. The target node's softdog watchdog monitors that device and triggers a reset when the fencing path completes successfully. However, if the isolated host still has storage connectivity (which is common in a pure network isolation scenario), the fence may not complete even though the host is unreachable over the network.
This is by design: SBD is a storage-based fencing mechanism, not a network-based one. Network isolation alone is not sufficient to trigger it when the host retains storage access. Pacemaker will not relocate master-2-vm until fencing is confirmed. This is intentional split-brain prevention behaviour.
Impact while unresolved
| Layer |
State |
| Pacemaker |
master-2-vm stuck Started kvm-host-2 (UNCLEAN) — will not relocate |
| RHOCP |
master-2 unreachable — arbiter zone effectively down |
| FDF |
HEALTH_WARN — Ceph loses arbiter mon quorum |
| Storage reads |
✅ Unaffected — CephFS continues from surviving zone |
| Storage writes |
❌ Blocked — CephRBD writes require arbiter quorum |
Resolution: Manual Fencing Procedure
⚠️ Critical: Step 2 is mandatory before Step 3. Running stonith_admin --confirm without physically stopping the host risks two instances of master-2 writing to the same etcd and RBD disk simultaneously, causing immediate data corruption.
Step 1: Verify the host is truly isolated
Confirm the host is unreachable via all paths: ping, SSH, and console.
ping kvm-host-2 # Expected: no response
ssh kvm-host-2 hostname # Expected: connection refused / timeout
Do not proceed if there is any possibility the host is still active.
Step 2: Manually stop the failing/isolated host
On IBM Z: deactivate the logical partition (LPAR) from the Hardware Management Console (HMC). On other platforms: power off the host via the management interface (Intelligent Platform Management Interface (IPMI), iLO, iDRAC, etc.). This guarantees the host cannot perform any further I/O on the shared storage.
Step 3: Confirm fence manually
stonith_admin --confirm kvm-host-2
This tells Pacemaker the node is confirmed dead. Pacemaker immediately starts master-2-vm on the surviving host.
Step 4: Verify recovery
pcs status
# Expected:
# master-2-vm Started kvm-host-1
virsh list --all
# Expected: master-2 running on kvm-host-1
oc get nodes
# Expected: master-2 Ready
# any OpenShift nodes hosted on kvm-host-2 remain NotReady
# — expected while kvm-host-2 remains down
Step 5: Clean up fencing history
pcs stonith history cleanup
pcs resource cleanup master-2-vm
pcs status # Expected: clean — no Failed Fencing Actions
Post-recovery state
|
State |
| Storage writes |
✅ Restored — master-2 (arbiter mon) is running on kvm-host-1 |
| Storage reads |
✅ Unaffected throughout |
| FDF health |
⚠️ HEALTH_WARN until kvm-host-2 rejoins and its affected OSDs/mons recover |
| FDF full recovery |
✅ HEALTH_OK after kvm-host-2 is brought back |
Bringing the isolated host back once the network issue is resolved
pcs cluster start kvm-host-2
sleep 15
pcs status # Expected: both nodes Online, all resources Started, no failures
oc get nodes # Expected: all nodes Ready
Appendix
Key reference values
| Item |
Example Value |
Your Value |
| Cluster name |
ocp-arbiter-ha-cluster |
|
| KVM Host 1 |
kvm-host-1.example.com |
|
| KVM Host 2 |
kvm-host-2.example.com |
|
| Shared disk WWID |
/dev/disk/by-id/<YOUR-SHARED-DISK-WWID> |
|
| SBD partition |
<WWID>-part1 |
|
| LVM partition |
<WWID>-part2 |
|
| LVM volume group |
vg_shared |
|
| VM disk logical volume |
lv_master2 |
|
| VM name |
master-2 |
|
| Pacemaker resource |
master-2-vm |
|
| STONITH resource |
sbd_fencing |
|
| Migration port |
16509/tcp |
|
Key commands reference
# Cluster status
pcs status
pcs cluster status
pcs constraint config --full
# Resource management
pcs resource enable master-2-vm
pcs resource disable master-2-vm
pcs resource move master-2-vm <node> --wait=120
pcs resource clear master-2-vm
pcs resource cleanup master-2-vm
# Node management
pcs node standby <node>
pcs node unstandby <node>
# Fencing
pcs stonith status
pcs stonith history cleanup
stonith_admin --confirm <node> # Manual fence confirmation only — see Troubleshooting
# SBD inspection
sbd -d /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part1 list
sbd -d /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part1 dump
# LVM
lvs vg_shared
vgchange -ay vg_shared
# Live migration (manual — use only outside of Pacemaker management)
virsh migrate --live --verbose master-2 qemu+tcp://kvm-host-2/system
Relevant log files
| Log |
Purpose |
journalctl -u pacemaker |
Pacemaker resource decisions and failover events |
journalctl -u corosync |
Cluster communication and quorum events |
journalctl -u libvirtd |
VM start, stop, and migration operations |
journalctl -u sbd |
SBD fencing events and watchdog activity |
Reference documentation