Troubleshooting QRadar Firmware Upgrades
IBM QRadar is sold as an appliance using Lenovo and Dell boxes. From Lenovo, IBM uses M3, M4, M5 and M6 appliances on which QRadar is deployed.
Together with regular QRadar version upgrades, these QRadar appliances need hardware firmware upgrades as well for optimum functioning. These firmware upgrades are releases as separate packages (depending on the type of the appliance) after testing them. Since these firmware versions are first released by the respective OEMs (Lenovo, Dell) and then they are tested by IBM QRadar team, there might be scenarios where the tested version of the firmware is a bit older compared to the latest version of the firmware from the OEM. This is perfectly normal.
For Lenovo appliances, the tested and supported firmware for QRadar appliances are released in the below link:
https://www.ibm.com/support/pages/qradar-firmware-list-xseries-appliances-updated
Firmware versions are released as ISO files/upgrade files in the above link.
Before discussing the different scenarios for a successful firmware upgrade, lets first understand the two commonly used access methods for Lenovo Appliances:
XClarity Controller:
For M6 appliances XCC (XClarity Controller) is used. M6 appliances have BMC port like the other appliances have IMM port for connection. XCC is the web UI, using which M6 appliances are accessed remotely.
Integrated Management Module (IMM):
On the back panel of each appliance type, are serial connector and Ethernet connectors. To manage these connectors and other system-management functions, a management module is required. For this purpose of management on M3, M4 and M5 appliances IMM 2 (Integrated Management Module 2) is used. The IMM or IMM2 can be configured to share an Ethernet port with IBM QRadar management interface or it can also be configured in dedicated mode to reduce the risk of losing connection.
Based on the type of Appliance in question, you would need access to XCC or IMM for any Firmware upgrades. You can refer to the below links for setting up IMM or XCC:
https://www.ibm.com/support/pages/qradar-managing-qradar-appliances-imm
https://www.ibm.com/support/pages/qradar-changing-imm-networking-configuration
https://www.ibm.com/support/pages/qradar-troubleshooting-connectivity-imm-or-xcc-qradar-appliances
Firmware upgrade issues and their solutions:
- Firmware upgrade fails with error message – Machine type is different.
This error message shows that the upgrade failed because the machine type in the upgrade image (used for the upgrade) and the machine type of the host machine are different.
To confirm if this is the case, run the following commands on the ssh session to the appliance:
- /opt/lenevo/toolcenter/asu/asu64 show SYSTEM_PROD_DATA - - kcs
- dmidecode -t1
In the above image the Machine types shown are different. Follow the below steps to resolve this:
- Ensure that IMM firmware is already upgraded.
- Mount the iso file which was downloaded for machine type 8871.
- Once mounted, extract the below file from the ISO folder (version number might be different in the file name) - oem_fw_uefi_tceg48a-3.30_anyos_32-64.uxz
- Now upgrade the UEFI firmware using the above file
- UEFI firmware upgrade will require a reboot of the OS of the host. Reboot the box
- After reboot is completed, upgrade the rest of the firmware using the steps as mentioned in https://www.ibm.com/support/pages/qradar-firmware-list-xseries-appliances-updated
The change that we did above was the UEFI component was upgraded separately before upgrading the rest of firmware components using ISO.
- IMM is upgraded but fails to update the IMM version on IMM UI.
You might get an error message like the one present below even after restarting the IMM:
“The firmware builds below require restart actions. Before the actions are taken, the pending firmware will keep the previous build and the table below still show the previous build name. The primary IMM build has been updated to xxxxxx, and it must restart IMM to take effect. It’s also recommended to clear the browser cache before you access the updated web console.”
After the IMM upgrade if the above message is seen, follow the below steps:
Ensure that the browser cache is cleared, and browser is restarted.
Then
A. Try to restart the IMM by pressing the button “Restart IMM” on UI.
B. If IMM restart does not help, follow the steps:
- Shutdown the appliance using power off from IMM Web UI.
- Remove both the power cables from the socket of the appliance. For this we need to access the appliance physically.
- Wait for 5 mins.
- Attach the power cables back and then boot up the box.
IMM version should now be seen as updated in the IMM Web UI. This is called Cold Reboot of the appliance.
- IMM upgrade file fails to verify.
IMM upgrade may fail even if you choose the correct version of upgrade file. You will see an error message like the one below:
- Using Force Option while upgrading firmware.
IBM QRadar’s supported Firmware versions might be a bit old compared to the latest ones released by Lenovo. This is normal since IBM QRadar team tests out the firmware first before releasing it. Only the firmware versions which have been officially tested and released by IBM QRadar team should be used for IBM QRadar appliances.
In some cases, you need to select the check box “Enable updating to back-level firmware” in the Update Setting page and proceed with the Firmware upgrade.
This allows firmware to be upgraded to the latest version of the Supported firmware version which might not be the latest as per Lenovo.
After you click on Next, the below options are displayed. Select the options as given in the below screenshot and proceed with the Firmware upgrade:
- Upgrade failed for several firmware components
Sometimes, after a Firmware upgrade, you might get an error like the one below which shows that multiple components upgrade has failed:
The error remark shown is “Connection failed” which means multiple components upgrade has failed.
In such scenario, you should verify 'LAN over USB' is enabled or not. The detailed steps are mentioned in the technote - https://www.ibm.com/support/pages/node/278761
You may also try to cold reboot the box using the following steps:
- Shutdown the appliance using power off from IMM Web UI.
- Remove both the power cables from the socket of the appliance. For this we need to access the appliance physically.
- Wait for 5 mins.
- Attach the power cables back and then boot up the box.
Once done, you can proceed with upgrading the Firmware again using the Firmware ISO.
During the Firmware upgrade, there might be other issues that you face which are not covered in this Blog Post. In such cases, we would recommend you engage IBM Support (https://www.ibm.com/support/pages/qradar-how-open-and-manage-cases) to fix the issue.
For any questions or comments regarding any of the points mentioned above or if you want to discuss this further, feel free to get in touch with us and we would be more than happy to answer any of your queries:
Ashish Kothekar (Ashish) – ashish.kothekar@in.ibm.com
Boudhayan Chakrabarty (Bob) – bochakra@in.ibm.com
Sheona Sinha (Sheona) – sheona.sinha@ibm.com