IBM Fusion

IBM Fusion

Ask questions, exchange ideas, and learn about IBM Fusion

 View Only

IBM Fusion Data Foundation stretch cluster on IBM Z: Pacemaker-managed arbiter across two availability zones

By Abdul Rauf Kuyyil Kandathil posted 09/22/26 07:35 AM

  

IBM Fusion Data Foundation stretch cluster on IBM Z: Pacemaker-managed arbiter across two availability zones

Platform: Red Hat® Enterprise Linux® (RHEL) 9.6 (IBM Z® / s390x)
Red Hat OpenShift Container Platform (RHOCP) version: 4.21
IBM Fusion® version: 2.13
IBM Fusion Data Foundation® (FDF) version: 2.21


Before you begin: Performance figures reflect a specific IBM Z reference environment, so treat them as indicative, not as product guarantees. This solution requires a single Fibre Channel Protocol (FCP) logical unit number (LUN) zoned to both Kernel-based Virtual Machine (KVM) hosts on the shared storage area network (SAN) fabric. It serves both the SBD fencing device and the VM's block disk. Without it, automatic failover will not function. Confirm that the LUN is accessible from both hosts before starting any software steps.

Note: This guide describes a reference implementation validated in a specific IBM Z environment. Verify support status with IBM and Red Hat before deploying in production.


Summary

IBM Fusion Data Foundation® (FDF) stretch cluster mode requires a dedicated third availability zone to host the Ceph® arbiter monitor. On IBM Z®, provisioning a dedicated zone for a single monitor process is often impractical. This guide shows how to run the arbiter control-plane node as a Pacemaker-managed KVM virtual machine (VM) on the two KVM hosts that already serve as your data zones. When the host running the arbiter VM fails, Pacemaker automatically fences the failed host and restarts the VM on the surviving host — no dedicated hardware required.

The guide covers the complete implementation in six phases: Pacemaker cluster setup, SBD storage-based fencing, VM disk migration to shared Logical Volume Manager (LVM) storage, libvirt live migration, automatic failover configuration, and IBM Fusion Data Foundation stretch cluster deployment. A reference environment on IBM Z completed automatic failover in approximately 23 seconds, with CephRBD write I/O resuming within 40 to 90 seconds. Manual recovery procedures are included for the scenario where automatic fencing cannot complete due to network isolation.

Requirements at a glance: Two IBM Z KVM hosts running Red Hat® Enterprise Linux® (RHEL) 9.6, one shared FCP LUN zoned to both hosts, Red Hat OpenShift Container Platform (RHOCP) 4.21, and IBM Fusion Data Foundation 2.21.


Introduction

IBM Fusion Data Foundation® (FDF) stretch cluster mode distributes storage across two availability zones within a single data centre to provide zone-level fault isolation. It requires a third arbiter zone, which acts as a tiebreaker that maintains Ceph® quorum when one availability zone fails. The solution is designed for deployments where the round-trip latency between data zones does not exceed 10 ms round-trip time (RTT). The arbiter zone follows the more relaxed etcd latency bounds. All three zones must have network connectivity to each other.

The challenge is that provisioning a dedicated third zone purely for an arbiter monitor is often impractical or expensive, especially on IBM Z®.

This guide shows a practical alternative using the KVM hosts already present in each availability zone: run the RHOCP arbiter control-plane node as a Pacemaker-managed KVM VM. When the KVM host running the arbiter fails, Pacemaker automatically restarts it on the surviving host. The arbiter zone effectively floats between the two KVM hosts, with no dedicated infrastructure required.

The result is a complete FDF stretch cluster on two KVM hosts, with automatic arbiter failover, quantified storage I/O impact, and a manual recovery path for the scenario where automatic fencing cannot complete.


Table of contents

  1. Architecture overview
  2. Prerequisites
  3. Phase 0: Red Hat subscription and repository setup
  4. Phase 1: Pacemaker cluster setup
  5. Phase 2: Storage and fencing configuration
  6. Phase 3: VM disk migration and CPU configuration
  7. Phase 4: Live migration configuration
  8. Phase 5: Automatic failover configuration
  9. FDF stretch cluster deployment
  10. Validation and testing
  11. Operational procedures
  12. Troubleshooting
  13. Appendix

Architecture overview

The problem this solves

FDF stretch cluster requires three topology zones: two data zones and one arbiter zone. In this design, the two KVM hosts are placed in the two data zones, while the arbiter control-plane VM keeps the Kubernetes® topology label arbiter regardless of which host currently runs it. Running it as a Pacemaker-managed KVM VM makes the arbiter zone a logical construct that floats between the two KVM hosts, always available on whichever host is running.

Topology

image

Node zone distribution

Node Zone Role FDF Role
master-0 zone1 control-plane Ceph mon (zone1)
master-1 zone2 control-plane Ceph mon (zone2)
master-2 arbiter control-plane Ceph arbiter mon
worker-0, worker-2, worker-4 zone1 worker OSDs in zone1
worker-1, worker-3 zone2 worker OSDs in zone2

Technology stack

Component Technology Version
Cluster Manager Pacemaker 2.1.9
Cluster Communication Corosync 3.1.9
Fencing SBD (Storage-Based Death) 1.5.2
Watchdog softdog kernel
Storage LVM on FCP multipath —
Virtualization KVM/QEMU via libvirt 10.10.0
Live Migration Libvirt TCP port 16509 —
Operating System RHEL 9.6 (s390x) 9.6

Implementation summary

Phase Description Estimated Duration VM Downtime
Phase 0 Subscription and Repository Setup 15 min None
Phase 1 Pacemaker Cluster Setup 1 hour None
Phase 2 Storage and Fencing Configuration 1 hour None
Phase 3 VM Disk Migration and CPU Configuration 2 hours ~30 min
Phase 4 Live Migration Configuration 30 min None
Phase 5 Automatic Failover Configuration 30 min None

Total: around 5 to 6 hours, with one planned downtime window of approximately 30 minutes during Phase 3 (disk conversion). All other phases run without disrupting the VM.


Prerequisites

Hardware

  • Two IBM Z® KVM hosts (s390x), minimum 16 vCPUs / 64 GB RAM each
  • One shared FCP LUN (≥ 300 GB), presented to both KVM hosts via multipath
  • RHOCP worker nodes with an additional raw block disk on each node for FDF OSDs

Software

  • RHEL 9.6 (s390x) with active Red Hat® subscription on both KVM hosts
  • An existing RHOCP cluster with:
    • Three control-plane nodes (one per intended zone)
    • At least four worker nodes (at least two per data zone)
    • master-2 currently running as a KVM VM on one of the two hosts

Network ports required

Port Protocol Purpose
2224 tcp pcsd (Pacemaker daemon)
3121 tcp Pacemaker inter-node
5403 udp Corosync
16509 tcp Libvirt TCP live migration
49152–49215 tcp Migration data transfer

Naming conventions used in this guide

Throughout this guide, the two KVM hosts are referred to as kvm-host-1 and kvm-host-2. Replace these with your actual hostnames. Commands that run on one host only are labelled explicitly. Commands with no host label run on either host.


Phase 0: Red Hat subscription and repository setup

Step 0.1: Register Both Hosts

On both hosts:

subscription-manager register --username <username> --password <password>

Step 0.2: Pin Release Version and Enable Repositories

On both hosts:

echo "9.6" > /etc/dnf/vars/releasever

subscription-manager repos \
  --enable=rhel-9-for-s390x-baseos-eus-rpms \
  --enable=rhel-9-for-s390x-appstream-eus-rpms \
  --enable=rhel-9-for-s390x-highavailability-eus-rpms

dnf clean all && dnf makecache

Validation

cat /etc/dnf/vars/releasever              # Expected: 9.6
subscription-manager repos --list-enabled | grep "Repo ID"
dnf list available pcs pacemaker corosync sbd   # No 404 errors

Phase 1: Pacemaker cluster setup

Step 1.1: Install Cluster Packages

On both hosts:

dnf install -y pcs pacemaker corosync fence-agents-all sbd

Step 1.2: Configure Firewall

On both hosts:

firewall-cmd --permanent --add-service=high-availability
firewall-cmd --reload
firewall-cmd --list-services | grep high-availability

Step 1.3: Set hacluster Password

On both hosts (use the same password on both):

passwd hacluster

Step 1.4: Start pcsd Service

On both hosts:

systemctl enable --now pcsd
systemctl status pcsd    # Expected: active (running)

Step 1.5: Load softdog Watchdog

On both hosts:

modprobe softdog
echo "softdog" > /etc/modules-load.d/softdog.conf

lsmod | grep softdog
ls -l /dev/watchdog*
# Expected: /dev/watchdog and /dev/watchdog0 present

Step 1.6: Authenticate Cluster Nodes

On kvm-host-1 only:

pcs host auth kvm-host-1 kvm-host-2 -u hacluster -p 'YourPassword'
# Expected: kvm-host-1: Authorized  /  kvm-host-2: Authorized

Step 1.7: Create and Start the Cluster

On kvm-host-1 only:

pcs cluster setup ocp-arbiter-ha-cluster kvm-host-1 kvm-host-2
pcs cluster start --all
pcs cluster enable --all
sleep 10
pcs cluster status

Expected output:

* Stack: corosync (Pacemaker is running)
* Online: [ kvm-host-1 kvm-host-2 ]

Step 1.8: Configure Cluster Properties

On kvm-host-1 only:

# Temporarily disable STONITH until the SBD fencing device is configured
pcs property set stonith-enabled=false

# Required for 2-node clusters
pcs property set no-quorum-policy=ignore

Validation

pcs status
# Expected: both nodes Online, 0 resources, all daemons active/enabled

Phase 2: Storage and fencing configuration

⚠️ RHEL 9 LVM Prerequisites: RHEL 9 uses a new device file (/etc/lvm/devices/system.devices) that restricts LVM scanning to specific paths. On shared cluster storage, this prevents the second host from discovering the VG created on the first host. Disable it on both hosts before proceeding:

cat >> /etc/lvm/lvm.conf << 'EOF'

devices {
  use_devicesfile = 0
}
EOF

This restores the traditional LVM discovery method (scan all block devices), essential for shared storage.

Step 2.1: Identify the Shared FCP LUN

On both hosts:

multipath -ll

Identify the multipath device that corresponds to the shared LUN. In the examples below, this device is referred to as <YOUR-SHARED-DISK-WWID>. Substitute your actual WWID. Verify: ls -l /dev/disk/by-id/ | grep scsi

The shared disk will be partitioned as follows:

  • /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part1 — 10 MB → SBD fencing device
  • /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part2 — remaining space → LVM volume group

Step 2.2: Partition the Shared Disk

On kvm-host-1 only:

Pre-check: Verify the disk is blank (or wipe it if reused from a previous setup):

SHARED_DISK="/dev/disk/by-id/<YOUR-SHARED-DISK-WWID>"

# Check for existing partition table or filesystem signatures
wipefs -n $SHARED_DISK    # Show what would be wiped (non-destructive)

# If non-empty output appears above, wipe it (safe — only removes signatures):
wipefs -a $SHARED_DISK

# Verify disk is now blank
parted $SHARED_DISK print    # Should show "Unrecognised disk label"

Create the partition table:

SHARED_DISK="/dev/disk/by-id/<YOUR-SHARED-DISK-WWID>"

parted -s $SHARED_DISK mklabel gpt
parted -s $SHARED_DISK mkpart primary 1MiB 11MiB      # SBD partition
parted -s $SHARED_DISK mkpart primary 11MiB 100%      # LVM partition

parted $SHARED_DISK print
lsblk $SHARED_DISK
# Expected: two partitions — 10 MB and remaining space

Step 2.3: Initialize SBD Device

On kvm-host-1 only:

⚠️ Use the partition, never the raw disk. If you accidentally run sbd … create on the raw disk (/dev/disk/by-id/<WWID> without -part1), it overwrites the partition table starting at byte 0, corrupting both partitions. Always target -part1 explicitly.

SBD_PART="/dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part1"

sbd -d $SBD_PART create
sbd -d $SBD_PART dump
# Expected: Header version 2.1, Number of slots: 255

Step 2.4: Configure SBD on Both Hosts

On both hosts:

cat > /etc/sysconfig/sbd << EOF
SBD_DEVICE="/dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part1"
SBD_WATCHDOG_DEV="/dev/watchdog"
SBD_WATCHDOG_TIMEOUT="5"
SBD_STARTMODE="always"
EOF

systemctl enable sbd    # Do NOT start manually — Pacemaker starts SBD

systemctl is-enabled sbd    # Expected: enabled

Note: systemctl start sbd returns "Operation refused, unit may be requested by dependency only". This is expected — SBD is started by Pacemaker's dependency chain.

Step 2.5: Enable STONITH and Activate SBD

On kvm-host-1 only:

pcs property set stonith-enabled=true

pcs cluster stop --all
pcs cluster start --all
sleep 15

systemctl status sbd
ssh kvm-host-2 "systemctl status sbd"
# Expected: active (running) on both hosts

Step 2.6: Create SBD STONITH Resource

On kvm-host-1 only:

pcs stonith create sbd_fencing fence_sbd \
  devices=/dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part1 \
  pcmk_delay_base=30s \
  op monitor interval=300s timeout=30s

pcs stonith status
# Expected: sbd_fencing (stonith:fence_sbd): Started <node>

sbd -d /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part1 list
# Expected:
# 0    kvm-host-2.example.com    clear
# 1    kvm-host-1.example.com    clear

Step 2.7: Create LVM Volume Group and Logical Volume

On kvm-host-1 only:

⚠️ Reusing this partition? If this disk was previously used (lab reset, troubleshooting loop), old LVM metadata may exist. If pvcreate fails with "without -ff" or vgcreate fails with "already exists in filesystem", see Troubleshooting → "pvcreate fails: 'without -ff' on reused disk" and "vgcreate fails: 'already exists in filesystem'" for cleanup commands.

pvcreate /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part2
vgcreate vg_shared /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part2

lvcreate -L 130G -n lv_master2 vg_shared

lvs vg_shared
# Expected: lv_master2 vg_shared -wi-a----- 130.00g

Adjust the logical volume size to match the disk image of your master-2 VM.

Step 2.8: Activate LVM on the Second Host

On kvm-host-2:

pvscan --cache
vgscan --cache
vgchange -ay vg_shared

lvs
# Expected: lv_master2 visible and active

If the LV is not visible:

pvscan --cache /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part2
vgchange -ay vg_shared

Phase 2 Validation

systemctl is-active sbd          # active — verify on both hosts
pcs stonith status               # sbd_fencing Started
pcs property show stonith-enabled  # true
lvs vg_shared                    # lv_master2 visible on both hosts

Phase 3: VM disk migration and CPU configuration

⚠️ Planned downtime required. This phase shuts down the master-2 VM to convert its disk from a local qcow2 image to a raw block device on shared LVM. The conversion itself takes approximately 15–30 minutes depending on disk size and storage performance.

Step 3.1: Verify RHOCP health before starting

oc get nodes    # All nodes must be Ready before proceeding

Step 3.2: Shut Down the VM

On kvm-host-1:

virsh shutdown master-2
watch -n 10 'virsh list --all | grep master-2'
# Wait until state shows: shut off

Step 3.3: Back Up VM Configuration

virsh dumpxml master-2 > /root/master-2-backup-$(date +%Y%m%d-%H%M%S).xml

Step 3.4: Convert Disk to Shared Block Storage

virsh domblklist master-2    # Confirm the current disk path

SOURCE_DISK="/var/lib/libvirt/images/<your-cluster>/master-2.qcow2" # Path to master-2 source disk from previous command.
DEST_DISK="/dev/vg_shared/lv_master2"

qemu-img convert -p -O raw $SOURCE_DISK $DEST_DISK

qemu-img info $DEST_DISK
# Expected: format: raw, virtual size matching your source image

Step 3.5: Update VM Configuration

virsh dumpxml master-2 > /tmp/master-2-new.xml

# Update disk driver to raw
sed -i "s|<driver name='qemu' type='qcow2'.*/>|<driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>|" /tmp/master-2-new.xml

# Change disk type to block device
sed -i "s|<disk type='file'|<disk type='block'|" /tmp/master-2-new.xml

# Update disk source path
sed -i "s|<source file='.*'/>|<source dev='/dev/vg_shared/lv_master2'/>|" /tmp/master-2-new.xml

# Set CPU mode expected to work for live migration in this reference environment.
# This assumes both IBM Z KVM hosts expose compatible CPU features.
sed -i 's|<cpu mode=.host-model. check=.partial.*/>|<cpu mode="host-passthrough" check="none"/>|' /tmp/master-2-new.xml

# Verify changes before applying
grep -A3 "<disk type='block'" /tmp/master-2-new.xml
grep '<cpu' /tmp/master-2-new.xml
# Expected:
# <disk type='block' device='disk'>
#   <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
#   <source dev='/dev/vg_shared/lv_master2'/>
# <cpu mode='host-passthrough' check='none'/>

virsh define /tmp/master-2-new.xml

Step 3.6: Copy Configuration to Second Host

scp /tmp/master-2-new.xml kvm-host-2:/tmp/
ssh kvm-host-2 "virsh define /tmp/master-2-new.xml"

Step 3.7: Disable Autostart on Both Hosts

⚠️ Critical: Pacemaker owns the VM lifecycle. If autostart is enabled, the VM may start outside of Pacemaker's control on host boot, which bypasses split-brain protection.

On both hosts:

virsh autostart --disable master-2
virsh dominfo master-2 | grep Autostart
# Expected: Autostart: disable

Step 3.8: Start VM and verify RHOCP recovery

virsh start master-2
sleep 90
virsh list | grep master-2

oc get nodes    # Wait 2–3 minutes — all RHOCP nodes should return to Ready

Phase 3 Validation

virsh domblklist master-2
# Expected: vda  /dev/vg_shared/lv_master2

virsh dumpxml master-2 | grep '<cpu'
# Expected: <cpu mode='host-passthrough' check='none'/>

Phase 4: Live migration configuration

Step 4.1: Configure Libvirtd for TCP

On both hosts:

grep -q "^listen_tcp" /etc/libvirt/libvirtd.conf \
  && sed -i 's/^listen_tcp.*/listen_tcp = 1/' /etc/libvirt/libvirtd.conf \
  || echo 'listen_tcp = 1' >> /etc/libvirt/libvirtd.conf

# Warning: this enables unauthenticated, unencrypted libvirt TCP migration.
# Use it only on a trusted private network that is isolated from untrusted clients.
grep -q "^auth_tcp" /etc/libvirt/libvirtd.conf \
  && sed -i 's/^auth_tcp.*/auth_tcp = "none"/' /etc/libvirt/libvirtd.conf \
  || echo 'auth_tcp = "none"' >> /etc/libvirt/libvirtd.conf

grep -q "^tcp_port" /etc/libvirt/libvirtd.conf \
  && sed -i 's/^tcp_port.*/tcp_port = "16509"/' /etc/libvirt/libvirtd.conf \
  || echo 'tcp_port = "16509"' >> /etc/libvirt/libvirtd.conf

grep -q "^listen_tls" /etc/libvirt/libvirtd.conf \
  && sed -i 's/^listen_tls.*/listen_tls = 0/' /etc/libvirt/libvirtd.conf \
  || echo 'listen_tls = 0' >> /etc/libvirt/libvirtd.conf

Step 4.2: Enable Libvirtd TCP Socket

On both hosts:

systemctl stop libvirtd
systemctl enable libvirtd-tcp.socket
systemctl start libvirtd-tcp.socket

systemctl status libvirtd-tcp.socket
ss -tlnp | grep 16509    # Expected: port 16509 listening

Step 4.3: Open Firewall Ports

On both hosts:

firewall-cmd --permanent --add-port=16509/tcp
firewall-cmd --permanent --add-port=49152-49215/tcp
firewall-cmd --reload
firewall-cmd --list-ports | grep -E "16509|49152"

Step 4.4: Test TCP Connectivity

virsh -c qemu+tcp://kvm-host-2/system list
ssh kvm-host-2 "virsh -c qemu+tcp://kvm-host-1/system list"
# Expected: VM list returned from each remote host with no errors

Step 4.5: Test Live Migration

With master-2 running on kvm-host-1:

# Migrate to kvm-host-2
virsh migrate --live --verbose master-2 qemu+tcp://kvm-host-2/system
sleep 10
ssh kvm-host-2 "virsh list | grep master-2"    # Expected: running on kvm-host-2

# Migrate back
ssh kvm-host-2 "virsh migrate --live --verbose master-2 qemu+tcp://kvm-host-1/system"
virsh list | grep master-2    # Expected: running on kvm-host-1

Success criteria: Port 16509 is listening on both hosts and bidirectional live migration completes without errors.


Phase 5: Automatic failover configuration

Step 5.1: Hand the VM to Pacemaker

virsh shutdown master-2
watch -n 5 'virsh list --all | grep master-2'    # Wait: shut off

Step 5.2: Create the VirtualDomain Pacemaker Resource

On kvm-host-1 only:

pcs resource create master-2-vm VirtualDomain \
  hypervisor='qemu:///system' \
  config='/etc/libvirt/qemu/master-2.xml' \
  migration_transport='tcp' \
  meta allow-migrate=true \
  op start timeout=120s \
  op stop timeout=120s \
  op monitor interval=30s timeout=30s \
  op migrate_to timeout=120s \
  op migrate_from timeout=120s

Step 5.3: Configure Resource Policies

pcs resource meta master-2-vm \
  resource-stickiness=100 \
  migration-threshold=3 \
  failure-timeout=300s
Attribute Value Effect
resource-stickiness 100 Prefers to remain on its current host
migration-threshold 3 Relocates after 3 consecutive failures
failure-timeout 300s Resets failure count after 5 minutes

Step 5.4: Verify Resource Started

pcs status

Expected:

Full List of Resources:
  * sbd_fencing   (stonith:fence_sbd):             Started kvm-host-1
  * master-2-vm   (ocf:heartbeat:VirtualDomain):   Started kvm-host-1

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
  sbd: active/enabled

Phase 5 Validation

virsh list | grep master-2          # Expected: running
pcs resource status master-2-vm    # Expected: Started
oc get nodes | grep master-2        # Expected: Ready

FDF stretch cluster deployment

With the Pacemaker high availability (HA) layer in place and master-2 running under Pacemaker control, deploy IBM Fusion Data Foundation® in stretch cluster (arbiter) mode.

Step 6.1: Label nodes with topology zones

oc label node master-0.<cluster-domain> topology.kubernetes.io/zone=zone1
oc label node worker-0.<cluster-domain>  topology.kubernetes.io/zone=zone1
oc label node worker-2.<cluster-domain>  topology.kubernetes.io/zone=zone1
oc label node worker-4.<cluster-domain>  topology.kubernetes.io/zone=zone1

oc label node master-1.<cluster-domain> topology.kubernetes.io/zone=zone2
oc label node worker-1.<cluster-domain>  topology.kubernetes.io/zone=zone2
oc label node worker-3.<cluster-domain>  topology.kubernetes.io/zone=zone2

oc label node master-2.<cluster-domain> topology.kubernetes.io/zone=arbiter

Verify:

oc get nodes -L topology.kubernetes.io/zone

Step 6.2: Deploy the IBM Fusion® operator

6.2.1: Obtain an Entitlement Key

An IBM entitlement key is required to pull IBM Fusion® images from the IBM Container Registry.

  1. Log in to https://myibm.ibm.com/products-services/containerlibrary
  2. Generate or copy your entitlement key.

6.2.2: Update the RHOCP global pull secret

The RHOCP global pull secret must be updated to include credentials for cp.icr.io.

1. Create a base64-encoded credential string:

echo -n "cp:<ENTITLEMENT_KEY>" | base64 -w0

2. Create an authority.json file with the encoded credentials:

{
    "auth": "<BASE64_ENCODED_ENTITLEMENT_KEY>"
}

3. Merge the new credential into the existing pull secret:

Note: Install jq if not already present: yum install jq

oc get secret/pull-secret -n openshift-config -ojson \
  | jq -r '.data[".dockerconfigjson"]' \
  | base64 -d \
  | jq '.[]."cp.icr.io" += input' - authority.json \
  > temp_config.json

4. Verify temp_config.json contains a cp.icr.io entry, then apply:

oc set data secret/pull-secret -n openshift-config \
  --from-file=.dockerconfigjson=temp_config.json

6.2.3: Create the IBM Operator CatalogSource

oc apply -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ibm-operator-catalog
  namespace: openshift-marketplace
spec:
  displayName: IBM Operator Catalog
  publisher: IBM
  sourceType: grpc
  image: icr.io/cpopen/ibm-operator-catalog:latest
  updateStrategy:
    registryPoll:
      interval: 45m
EOF

6.2.4: Install the IBM Fusion® operator from OperatorHub

Navigate to Operators > OperatorHub in the RHOCP web console, search for IBM Fusion, and install it with the following settings:

  • Update approval: Manual (IBM Fusion does not support automatic updates)
  • All other settings: leave as default

Step 6.3: Deploy Fusion Data Foundation in local mode

Access the IBM Fusion web console: from the Applications menu in the RHOCP title bar, click IBM Fusion. Then navigate to:

Services > Data Foundation > Install > Install Service: Local > Update approval: Off > Install

This will deploy both the Fusion Data Foundation operator and the Local Storage operator.

Step 6.4: Create the StorageCluster

On the Local Storage page, click Get Started, then follow the wizard:

  1. Backing storage: Select Create a new StorageClass using the local storage devices
  2. Create local volume set: Name the LocalVolumeSet and StorageClass. Verify that four disks are detected across the worker nodes.
  3. Capacity and nodes page:
    • Select Enable arbiter
    • Select arbiter as the arbiter zone
    • Select all worker nodes as storage nodes

Note: Remaining configuration values can be kept at their defaults or adjusted as required. Refer to the official IBM documentation for advanced configuration options:
Configuring Fusion Data Foundation in local mode

The resulting StorageCluster CR will include:

spec:
  arbiter:
    enable: true
  nodeTopologies:
    arbiterLocation: arbiter
  storageDeviceSets:
  - count: 1
    replica: 4    # 2 OSDs in zone1, 2 OSDs in zone2
    ...

Step 6.5: Verify FDF deployment

oc -n openshift-storage get storagecluster
# Expected: PHASE = Ready

oc patch storagecluster ocs-storagecluster -n openshift-storage --type json --patch '[{ "op": "replace", "path": "/spec/enableCephTools", "value": true }]'

oc -n openshift-storage rsh $(oc get pods -n openshift-storage -l app=rook-ceph-tools -o name) ceph -s
# Expected:
#   stretch_mode: ENABLED
#   health: HEALTH_OK
#   mon: 5 daemons, quorum a,b,c,d,e
#   osd: 4 osds: 4 up, 4 in

Validation and testing

Test 1: Cluster state

pcs status                     # Clean — no Failed Resource Actions
pcs constraint config --full   # No unexpected constraints
pcs property show              # stonith-enabled=true, no-quorum-policy=ignore

Test 2: Graceful standby — other VMs must not be affected

This test confirms that putting a node in standby migrates master-2-vm but leaves all other VMs on that host running normally.

# Confirm current location of master-2-vm
pcs status | grep master-2-vm

# Put kvm-host-2 in standby
pcs node standby kvm-host-2
sleep 30

# Verify master-2-vm has migrated to kvm-host-1
pcs status | grep master-2-vm
# Expected: Started kvm-host-1

# Verify all other VMs on kvm-host-2 are still running
ssh kvm-host-2 "virsh list --all"
# Expected: bastion, master-0, and all zone1 workers still running

# Restore kvm-host-2
pcs node unstandby kvm-host-2
sleep 15
pcs status

Test 3: Live migration via Pacemaker

pcs resource move master-2-vm kvm-host-2 --wait=120
pcs status | grep master-2-vm    # Expected: Started kvm-host-2

pcs resource clear master-2-vm    # Remove migration constraint

Reference environment observation: Live migration completed in approximately 25 seconds with no application downtime on the migrated VM. Results vary by hardware and storage throughput.

Test 4: Automatic failover

Warning: This test causes master-2 to be temporarily unavailable. Pacemaker will detect the cluster stack failure on the active host and restart the VM on the surviving host. kvm-host-2 will go offline and must be recovered afterward.

# Ensure master-2-vm is on kvm-host-2
pcs resource move master-2-vm kvm-host-2 --wait=120
pcs resource clear master-2-vm

# Simulate a cluster stack failure on the active host
ssh kvm-host-2 "systemctl stop pacemaker"

# Monitor recovery
watch -n 2 'pcs status'
# master-2-vm will appear on kvm-host-1 once fencing completes
# kvm-host-2 will show as OFFLINE

# Cleanup fencing and resources
pcs stonith history cleanup
pcs resource cleanup
pcs status    # Expected: master-2-vm on kvm-host-1, kvm-host-2 offline

Restore kvm-host-2 to the cluster:

# On kvm-host-2, restart Pacemaker to rejoin the cluster
ssh kvm-host-2 "systemctl start pacemaker"
sleep 15

# Verify both nodes are online and healthy
pcs status    # Expected: both nodes Online, resources clean

Reference environment observation: Automatic failover completed in approximately 23 seconds from host failure detection to master-2-vm running on the surviving host. RHOCP master-2 did not enter a NotReady state during this period. The only disruption was CephRBD write I/O, which resumed within 40 to 90 seconds. CephFS reads were unaffected throughout. Results vary by hardware, storage, and network.

Test 5: RHOCP cluster health

oc get nodes
# Expected: all nodes Ready

oc get co
# Expected: all operators Available=True, Progressing=False, Degraded=False

Operational procedures

Daily health check

pcs status       # Verify: both nodes Online, all resources Started, no failures
oc get nodes     # Verify: all nodes Ready
oc get co        # Verify: all operators Available

Planned maintenance: KVM host standby

Use this procedure when patching or rebooting a KVM host to avoid unplanned failover.

# Step 1 — Move master-2-vm to the host that will remain active
pcs resource move master-2-vm kvm-host-2 --wait=120
pcs status | grep master-2-vm    # Verify: Started kvm-host-2

# Step 2 — Put the maintenance host in standby
pcs node standby kvm-host-1

# Step 3 — Perform maintenance on kvm-host-1 ...

# Step 4 — Bring the host back
pcs node unstandby kvm-host-1
sleep 15

# Step 5 — Remove migration constraint
pcs resource clear master-2-vm
pcs status

After any unplanned host reboot

Pacemaker automatically restarts all resources when a node rejoins. Verify:

pcs status
# Expected: master-2-vm Started, sbd_fencing Started, no failures

If master-2-vm remains Stopped:

pcs resource cleanup master-2-vm

VM management rules

Always manage master-2 through Pacemaker. Direct virsh commands bypass split-brain protection and must not be used on a Pacemaker-managed VM.

Action Correct Command Never Use
Start master-2 pcs resource enable master-2-vm virsh start master-2
Stop master-2 pcs resource disable master-2-vm virsh shutdown master-2
Migrate master-2 pcs resource move master-2-vm <node> virsh migrate ...
Check status pcs status —
Clear failures pcs resource cleanup master-2-vm —
Remove move constraint pcs resource clear master-2-vm —

Troubleshooting

pcs cluster start --all returns HTTP error 400

softdog is not loaded on the second host:

ssh kvm-host-2 "modprobe softdog && systemctl restart pcsd"
pcs cluster start --all

master-2-vm fails to start

pcs resource failcount show master-2-vm
journalctl -u pacemaker -n 50 --no-pager | grep -i "master-2\|error"
ls -l /dev/vg_shared/lv_master2    # Confirm the LV is accessible

pcs resource cleanup master-2-vm

master-2-vm stuck in Stopping

pcs resource cleanup master-2-vm

SBD fencing not working

systemctl status sbd
sbd -d /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part1 dump
ls -l /dev/watchdog*
cat /etc/sysconfig/sbd

Hundreds of failed fencing actions in pcs status

Normal after a crash test or an unplanned node reboot. Clear with:

pcs stonith history cleanup
pcs resource cleanup

pvcreate fails: "without -ff" on reused disk

LVM metadata survives partial disk wipes. Clean it:

PART2="/dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part2"
vgremove -ff vg_shared 2>/dev/null
pvremove -ff $PART2
wipefs -a $PART2
pvcreate $PART2

vgcreate fails: "already exists in filesystem"

vgremove removed metadata but left device-mapper nodes. Remove them:

dmsetup ls | grep vg_shared
dmsetup remove vg_shared-lv_master2   # remove each LV node
dmsetup remove vg_shared 2>/dev/null  # may already be gone
vgcreate vg_shared /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part2

Resource fails after node rejoins cluster

pcs resource cleanup
pcs status
# If master-2-vm is still Stopped:
systemctl restart libvirtd    # On the node where it should start
pcs resource cleanup master-2-vm

KVM host network isolation: SBD fencing fails, master-2-vm stuck UNCLEAN

Symptom

One KVM host loses all network connectivity (for example, due to a datacenter network incident). The surviving host shows:

pcs status
  Node kvm-host-2: UNCLEAN (offline)
  master-2-vm (ocf:heartbeat:VirtualDomain): Started kvm-host-2 (UNCLEAN)

Failed Fencing Actions:
  * reboot of kvm-host-2 ... last failed at ...   ← repeating every ~15 min

Why automatic SBD fencing fails in this scenario

SBD fences a node by writing a poison pill to the shared disk. The target node's softdog watchdog monitors that device and triggers a reset when the fencing path completes successfully. However, if the isolated host still has storage connectivity (which is common in a pure network isolation scenario), the fence may not complete even though the host is unreachable over the network.

This is by design: SBD is a storage-based fencing mechanism, not a network-based one. Network isolation alone is not sufficient to trigger it when the host retains storage access. Pacemaker will not relocate master-2-vm until fencing is confirmed. This is intentional split-brain prevention behaviour.

Impact while unresolved

Layer State
Pacemaker master-2-vm stuck Started kvm-host-2 (UNCLEAN) — will not relocate
RHOCP master-2 unreachable — arbiter zone effectively down
FDF HEALTH_WARN — Ceph loses arbiter mon quorum
Storage reads ✅ Unaffected — CephFS continues from surviving zone
Storage writes ❌ Blocked — CephRBD writes require arbiter quorum

Resolution: Manual Fencing Procedure

⚠️ Critical: Step 2 is mandatory before Step 3. Running stonith_admin --confirm without physically stopping the host risks two instances of master-2 writing to the same etcd and RBD disk simultaneously, causing immediate data corruption.

Step 1: Verify the host is truly isolated

Confirm the host is unreachable via all paths: ping, SSH, and console.

ping kvm-host-2              # Expected: no response
ssh kvm-host-2 hostname      # Expected: connection refused / timeout

Do not proceed if there is any possibility the host is still active.

Step 2: Manually stop the failing/isolated host

On IBM Z: deactivate the logical partition (LPAR) from the Hardware Management Console (HMC). On other platforms: power off the host via the management interface (Intelligent Platform Management Interface (IPMI), iLO, iDRAC, etc.). This guarantees the host cannot perform any further I/O on the shared storage.

Step 3: Confirm fence manually

stonith_admin --confirm kvm-host-2

This tells Pacemaker the node is confirmed dead. Pacemaker immediately starts master-2-vm on the surviving host.

Step 4: Verify recovery

pcs status
# Expected:
#   master-2-vm  Started kvm-host-1

virsh list --all
# Expected: master-2 running on kvm-host-1

oc get nodes
# Expected: master-2 Ready
#           any OpenShift nodes hosted on kvm-host-2 remain NotReady
#           — expected while kvm-host-2 remains down

Step 5: Clean up fencing history

pcs stonith history cleanup
pcs resource cleanup master-2-vm
pcs status    # Expected: clean — no Failed Fencing Actions

Post-recovery state

State
Storage writes ✅ Restored — master-2 (arbiter mon) is running on kvm-host-1
Storage reads ✅ Unaffected throughout
FDF health ⚠️ HEALTH_WARN until kvm-host-2 rejoins and its affected OSDs/mons recover
FDF full recovery ✅ HEALTH_OK after kvm-host-2 is brought back

Bringing the isolated host back once the network issue is resolved

pcs cluster start kvm-host-2
sleep 15
pcs status      # Expected: both nodes Online, all resources Started, no failures
oc get nodes    # Expected: all nodes Ready

Appendix

Key reference values

Item Example Value Your Value
Cluster name ocp-arbiter-ha-cluster
KVM Host 1 kvm-host-1.example.com
KVM Host 2 kvm-host-2.example.com
Shared disk WWID /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>
SBD partition <WWID>-part1
LVM partition <WWID>-part2
LVM volume group vg_shared
VM disk logical volume lv_master2
VM name master-2
Pacemaker resource master-2-vm
STONITH resource sbd_fencing
Migration port 16509/tcp

Key commands reference

# Cluster status
pcs status
pcs cluster status
pcs constraint config --full

# Resource management
pcs resource enable master-2-vm
pcs resource disable master-2-vm
pcs resource move master-2-vm <node> --wait=120
pcs resource clear master-2-vm
pcs resource cleanup master-2-vm

# Node management
pcs node standby <node>
pcs node unstandby <node>

# Fencing
pcs stonith status
pcs stonith history cleanup
stonith_admin --confirm <node>    # Manual fence confirmation only — see Troubleshooting

# SBD inspection
sbd -d /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part1 list
sbd -d /dev/disk/by-id/<YOUR-SHARED-DISK-WWID>-part1 dump

# LVM
lvs vg_shared
vgchange -ay vg_shared

# Live migration (manual — use only outside of Pacemaker management)
virsh migrate --live --verbose master-2 qemu+tcp://kvm-host-2/system

Relevant log files

Log Purpose
journalctl -u pacemaker Pacemaker resource decisions and failover events
journalctl -u corosync Cluster communication and quorum events
journalctl -u libvirtd VM start, stop, and migration operations
journalctl -u sbd SBD fencing events and watchdog activity

Reference documentation

0 comments
10 views

Permalink