IBM QRadar is a leading SIEM tool that leverages intelligent security analytics to gain actionable insight into most critical threats. QRadar makes it easy to remediate threats faster. QRadar can be deployed in public cloud or on premises.
With continuous advancement, sophistication and persistence of cybersecurity threats, there is an inevitable demand to keep QRadar working without downtime. To cater to this demand, QRadar comes equipped with high availability feature where QRadar would continue to collect, store and process data even when hardware fails and/or network failure.
For High Availability (HA) in QRadar, the primary host connects with secondary host to form a HA cluster. In this cluster, secondary takes the workload of primary host when primary host fails. When there is a transfer of workload from primary to secondary then secondary host is said to have become active. Secondary HA host maintains access to same data as that of primary HA host, to cater to the situation where primary HA host fails.
HA can be used on QRadar that is deployed on hardware or virtual appliances whether it is appliance or software installation. High Availability of QRadar is not supported in cloud environments. As High Availability feature is introduced in QRadar to reduce downtime, here in this blog we would discuss on steps for hardware migration of HA cluster that would have minimum downtime and would not require to break HA to perform hardware migration on each hosts separately.
Before we start with Hardware migration, we need to ensure on the following checks:
> Secondary host is in standby state and primary host is active state.
This can be checked from UI in system and license management. It should show output as shown below:
From backend, this can be checked using following command:
/opt/qradar/ha/bin/ha cstate
Output of above command should be as follows:
[root@vm1xxx7-primary ~]# /opt/qradar/ha/bin/ha cstate
Local: R:PRIMARY S:ACTIVE/ONLINE CS:NONE P:1.0 HBC:UP RTT:1 I:0 SI:27299008
Remote: R:SECONDARY S:STANDBY/ONLINE CS:NONE P:1.0 HBC:UP RTT:1 I:9773 SI:27337561
HBC: ALIVE/0
LSN: drbd_status => 1.0 I:0
LSN: ha_services => 1.0 I:0
LSN: drbd_sync => 1.0 I:0
LSN: mount_status => 1.0 I:0
LSN: app_services => 1.0 I:0
LSN: drbd_io_perf => 1.0 I:0
LSN: link_status => 1.0 I:0
LSN: cluster_ip => 1.0 I:0
RSN: drbd_status => 1.0 I:0
RSN: drbd_sync => 1.0 I:0
RSN: mount_status => 1.0 I:0
RSN: drbd_io_perf => 1.0 I:0
RSN: link_status => 1.0 I:0
> The IP address remains same in old appliance and new appliance of both (primary and secondary) hosts.
> The new appliance should have same or higher hardware specifications to that of the old appliance.
Hardware migration of secondary host
> Create new 500 appliance for secondary host.
- Insert the bootable USB flash drive into the USB port of your appliance.
- Restart the appliance to boot up the new hardware/ VM appliance from a USB flash drive.
- After the appliance starts, the USB flash drive prepares the appliance for installation. This process can take up to an hour to complete.
- When the Red Hat Enterprise Linux menu is displayed, select one of the following options:
· If you connected a keyboard and monitor, select Install Red Hat Enterprise Linux 7.7.
· If you connected a notebook with a serial connection, select Install Red Hat Enterprise Linux 7.7 using Serial console without format prompt or Install Red Hat Enterprise Linux 7.7 using Serial console with format prompt.
- Type SETUP to begin the installation.
- Type root at the login prompt to launch the installation wizard.
- Accept the End User License Agreement.
- Select the appliance type: High Availability Appliance.
- Follow the instructions in the wizard.
- Configure the QRadar root password.
- Review your software version. If your primary HA host patch version is newer than the software on this appliance, download and install the SFS (software fix/patch) from Fix Central (www.ibm.com/support/fixcentral/) to upgrade this appliance to match the software version.
- Log in to the QRadar user interface.
> Log in to the QRadar user interface.
> Select Main menu > Admin > System and License Management > Systems.
> Highlight the secondary HA host that you are going to migrate and select High Availability > Set Offline.
> Now, shutdown old secondary host when configuring the new secondary host.
> Bring secondary host in the network and power it on.
> Get the new secondary host in place of old secondary host.
- Log in to the QRadar user interface.
- Select Main menu > Admin > System and License Management > Systems.
- Highlight the secondary HA host that you are restoring and select High Availability > Restore System.
> Restore system would start HA configuration restore process to build HA configuration on the new appliance and begin data synchronisation between the hosts where primary host would be sending the data to secondary host. Once synchronisation gets completed, secondary host comes back in standby state.
Based on the amount of data and the crossover connectivity, the HA restore process would take variable amount of time. Once synchronisation completes, secondary host would come back in standby state.
Hardware migration of primary host
> Failover to secondary host to get primary host in standby state.
- Ensure the following before you conduct a manual failover:
The primary and secondary HA hosts are synchronised.
The secondary HA host has a status of standby.
- Select Main menu > Admin > System and License Management > Systems.
- Highlight the primary HA host which is planned for hardware migration activity and select High Availability > Restore System.
After the secondary host becomes active, you can perform maintenance on the primary host.
> Installing new primary host with same IP address as that of old primary host
> Shutdown old primary host when configuring the new primary host.
> Bring primary host in the network and power it on.
- Insert the bootable USB flash drive into the USB port of your appliance.
- Restart the appliance to boot up the new hardware/ VM appliance from a USB flash drive.
- After the appliance starts, the USB flash drive prepares the appliance for installation. This process can take up to an hour to complete.
- When the Red Hat Enterprise Linux menu is displayed, select one of the following options:
· If you connected a keyboard and monitor, select Install Red Hat Enterprise Linux 7.7.
· If you connected a notebook with a serial connection, select Install Red Hat Enterprise Linux 7.7 using Serial console without format prompt or Install Red Hat Enterprise Linux 7.7 using Serial console with format prompt.
- Type SETUP to begin the installation.
- Type root at the login prompt to start the installation wizard.
- Accept the End User License Agreement.
- Select the appliance type:
- Appliance Install
- Software Install
Important: You must choose the same appliance type as the failed primary. Do not chose an HA standby appliance.
- In the Type of Setup window, select HA Recovery Setup.
- Follow the instructions in the wizard.
11. Configure the QRadar network settings.
a. In the Cluster Virtual IP Address Setup window, enter the cluster virtual IP address.
b. In the Network Information Setup window, enter the original hostname and the IP address of the primary HA host.
Note: When an HA cluster is created, “-primary” is appended to the original hostname of the primary HA host. Do not include “-primary” when you enter the original hostname in the Network Information Setup window.
- Configure the QRadar root password.
- Review your software version. If your secondary HA host patch version is newer than the software on this appliance, download and install the SFS (software fix) from Fix Central (www.ibm.com/support/fixcentral/) to upgrade this appliance to match the software version.
> Shutdown old primary host when configuring the new primary host.
> Bring primary host in the network and power it on.
> Get the new primary host in place of old primary host.
1. Log in to the QRadar user interface.
2. Select Main menu > Admin > System and License Management > Systems.
3. Highlight the primary HA host that you are restoring and select High Availability > Restore System.
> Restore system would start HA configuration restore process to build HA configuration on the new appliance and begin data synchronisation between the hosts, secondary host would be sending the data that came in after it took the active role to primary host. Once synchronisation gets completed, primary host comes back in standby state.
Based on the amount of data and the crossover connectivity, the HA restore process would take variable amount of time. Once synchronisation completes, primary host would come back in standby state.
> Failover back to primary host.
1. Ensure the following before you conduct a manual failover:
a. The primary and secondary HA hosts are synchronized.
b. The primary HA host has a status of standby.
2. Select Main menu > Admin > System and License Management > Systems.
3. Highlight the secondary HA host of the HA cluster and select High Availability > Set Offline.
4. Once, primary host is active, highlight the secondary HA host again and select High Availability > Set Online.
This is to make the secondary host come up as Standby host.
With this you would be able to successfully perform hardware migration on QRadar HA cluster with minimum downtime.
Through this blog we showed you how you can perform Hardware Migration in QRadar HA cluster.
If you have any questions regarding any of the points mentioned above or want to discuss this further, feel free to get in touch with me.
SHEONA SINHA: sheona.sinha@ibm.com
Special thanks to BOUDHAYAN CHAKRABARTY (bochakra@in.ibm.com) for reviewing and approving this article.