IBM QRadar

Hey all, as mentioned in a Reddit thread there is an issue a few users have reported where the upgrade to 7.5.0 UP9 takes longer than expected to complete.

A flash notice was issued to all users for awareness to the problem that includes a method to check the number of files that need to be converted from UID 99 (nobody) to UID 65534 (new UID in RHEL8 for nobody). This UID change is why some users are experiencing extended upgrade times as the system has to read potentially millions of files and update their user/groups to the new value. Depending on the number of files and the ages of the appliance the upgrade can take a long time (1hr per 1 million file updates) to complete post-upgrade conversions.

Users who plan to upgrade to 7.5.0 UP9 can take one of the following steps:

1. Run the all_servers.sh command in the flash notice. This will output the number of files owned by nobody that the UP9 upgrade needs to convert. Depending on the counts returned, you can either proceed with the UP9 upgrade or wait for the UP9 re-released SFS file.
2. Wait for a re-release of QRadar 7.5.0 UP9. Dev has recompiled the UP9 release and is testing it with a minor change to improve the performance of the conversion of the UIDs in the post patch portion of the upgrade. This will significantly reduce or mostly eliminate the issue for users who have millions of records/files owned by nobody that need to be converted to UID 65534.

In summary if you have a planned upgrade to 7.5.0 Update Package 9 you want to check your appliances to determine the number of files that need conversion to the new UID and potentially delay your upgrade. Development plans to replace the UP9 release on IBM Fix Central to correct this issue.

Links

• Flash notice: https://www.ibm.com/support/pages/node/7160458
• Known issue: https://www.ibm.com/mysupport/s/defect/aCIKe00000002hb/dt392091
• Other thread where this issue is discussed: https://www.reddit.com/r/QRadar/comments/1e0smg0/news_qradar_750_update_package_9_is_released/

FAQ for this issue

Q. I've already started by upgrade, what can I do?
Nothing, just let the upgrade complete without interruption. The upgrade will likely seem to be stuck on a post-patch process 35/36. For example,

[INFO](patchmode) Running postpatch scripts
Applying postpatch script: (35/36)

It is critical that you let the upgrade complete. If you attempt to interrupt an upgrade in process, this often causes rebuilds of the appliance that take much longer than letting the patch complete as it can introduce data integrity issues. Let the upgrade run if in progress and it will eventually complete.

Q. Is QRadar on Cloud affected?
Yes, but the DevOps teams are converting these UIDs to prevent the issue for QRadar on Cloud users. So, there is no work required by QRadar on Cloud administrators.

Q. Can I change the ownership of these files myself to reduce the upgrade time frame?
No, this is definitely not recommended. It is better to wait if you have appliances that report a larger number of files owned by nobody. As services and file replication can be impacted, which is why the SFS installer completes these actions while services are stopped. Trying to fix this yourself is not recommended as a new version of UP9 will be made available to prevent this issue. It is best to wait and ensure that you are not replicating files incorrectly or having DRBD block replicate files with the wrong UID that might require extra clean up.

Q. How can I find out when the new UP9 is posted?
I'll post an update here in this thread, the flash notice will be updated, or you can subscribe to the known issue (DT392091).

Q. What if I've already upgraded to the existing UP9 and did not experience this issue, is there more software for me to install?
No, the re-release of UP9 is going to have the same build number with the exception of the timestamp at the end of the file is going to display 20240716xxxx, instead of 20240703xxxx. These are essentially the same builds with a few updated lines of code to improve the performance of the UID conversion. There is no need to update or reinstall for any reason if you already applied the original UP9 release after the re-release version is posted to IBM Fix Central.

If there are follow-up questions, let me know.

Hey all, as mentioned in a Reddit thread there is an issue a few users have reported where the upgrade to 7.5.0 UP9 takes longer than expected to complete.

A flash notice was issued to all users for awareness to the problem that includes a method to check the number of files that need to be converted from UID 99 (nobody) to UID 65534 (new UID in RHEL8 for nobody). This UID change is why some users are experiencing extended upgrade times as the system has to read potentially millions of files and update their user/groups to the new value. Depending on the number of files and the ages of the appliance the upgrade can take a long time (1hr per 1 million file updates) to complete post-upgrade conversions.

Users who plan to upgrade to 7.5.0 UP9 can take one of the following steps:

1. Run the all_servers.sh command in the flash notice. This will output the number of files owned by nobody that the UP9 upgrade needs to convert. Depending on the counts returned, you can either proceed with the UP9 upgrade or wait for the UP9 re-released SFS file.
2. Wait for a re-release of QRadar 7.5.0 UP9. Dev has recompiled the UP9 release and is testing it with a minor change to improve the performance of the conversion of the UIDs in the post patch portion of the upgrade. This will significantly reduce or mostly eliminate the issue for users who have millions of records/files owned by nobody that need to be converted to UID 65534.

In summary if you have a planned upgrade to 7.5.0 Update Package 9 you want to check your appliances to determine the number of files that need conversion to the new UID and potentially delay your upgrade. Development plans to replace the UP9 release on IBM Fix Central to correct this issue.

Links

• Flash notice: https://www.ibm.com/support/pages/node/7160458
• Known issue: https://www.ibm.com/mysupport/s/defect/aCIKe00000002hb/dt392091
• Other thread where this issue is discussed: https://www.reddit.com/r/QRadar/comments/1e0smg0/news_qradar_750_update_package_9_is_released/

FAQ for this issue

Q. I've already started by upgrade, what can I do?
Nothing, just let the upgrade complete without interruption. The upgrade will likely seem to be stuck on a post-patch process 35/36. For example,

[INFO](patchmode) Running postpatch scripts
Applying postpatch script: (35/36)

It is critical that you let the upgrade complete. If you attempt to interrupt an upgrade in process, this often causes rebuilds of the appliance that take much longer than letting the patch complete as it can introduce data integrity issues. Let the upgrade run if in progress and it will eventually complete.

Q. Is QRadar on Cloud affected?
Yes, but the DevOps teams are converting these UIDs to prevent the issue for QRadar on Cloud users. So, there is no work required by QRadar on Cloud administrators.

Q. Can I change the ownership of these files myself to reduce the upgrade time frame?
No, this is definitely not recommended. It is better to wait if you have appliances that report a larger number of files owned by nobody. As services and file replication can be impacted, which is why the SFS installer completes these actions while services are stopped. Trying to fix this yourself is not recommended as a new version of UP9 will be made available to prevent this issue. It is best to wait and ensure that you are not replicating files incorrectly or having DRBD block replicate files with the wrong UID that might require extra clean up.

Q. How can I find out when the new UP9 is posted?
I'll post an update here in this thread, the flash notice will be updated, or you can subscribe to the known issue (DT392091).

Q. What if I've already upgraded to the existing UP9 and did not experience this issue, is there more software for me to install?
No, the re-release of UP9 is going to have the same build number with the exception of the timestamp at the end of the file is going to display 20240716xxxx, instead of 20240703xxxx. These are essentially the same builds with a few updated lines of code to improve the performance of the UID conversion. There is no need to update or reinstall for any reason if you already applied the original UP9 release after the re-release version is posted to IBM Fix Central.

If there are follow-up questions, let me know.