IBM Security QRadar SOAR

 View Only
Expand all | Collapse all

Support for RHEL 8 Update

  • 1.  Support for RHEL 8 Update

    Posted Mon July 24, 2023 08:40 AM

    In our previous post, we explained how SOAR will support RHEL8. In this post we will reveal more information about the recommended approach and considerations for migrating your SOAR system to RHEL8.

    For any questions on the below, please get in touch via your IBM contacts, or reply to this post, or email me direct on martin.feeney@ie.ibm.com.

    For Virtual Appliance (OVA) deployment 

    Migrate your SOAR data and configuration to a new provisioned RHEL8 instance using backup and restore tools

    • In v50, the SOAR platform supports both RHEL7 and RHEL8. You can upgrade your current RHEL7 based SOAR system to v50, then backup it's data and configuration via the soarSystemBackup / soarSystemRestore tool to perform the migration. This is our recommended approach.

    • Please note you can only perform the migration on v50 because v51 won't be able to run on RHEL7.

    • Please also note we're recommending the new soarSystem rather than resSystem backup and restore utility. This is to reduce downtime. If you have not yet adopted the new backup and restore utilities introduced in V46, then either familiarise yourself with these, or alternatively you can use the old resSystem utilities but the downtime will be longer.

    • With this approach, the new system could have a different IP address/FQDN than the existing one. Please review the following areas and determine what configuration changes are required after migration to restore connection with these systems (any dependency on ip address, FQDN, or certificates). If the new system has new IP/FQDN you will need to regenerate certificates for many of these. Alternatively if you re-use the same domain name (FQDN) for new SOAR system you will avoid this effort.

      • System firewall settings and proxy configuration

      • The SOAR web interface

      • AppHost pairing or Integration server settings

      • SAML authentication

      • Inbound Email connection

      • DR settings

      • Any customer specific 3rd party integration

    The following are the general steps to migrate existing RHEL7 to RHEL8 using soar backup and restore tools. 

    • Step 1 in the diagram is not necessarily part of the migration downtime. You can upgrade your existing instance to v50 to enjoy the new features and then do the migration at another time.

    • You can rehearse the migration and estimate the downtime using soarSystemBackup tool's online backup capability ("-O, --online" option).

    • If anything goes wrong during the migration, you can simply change back to use the existing instance.

    IBM Security Expert Labs

    If the upgrade process described above (or below) presents challenges in terms of the required expertise to implement, consider contacting IBM Security Expert Labs who are familiar with the process and could provide this service for you. Reach out to your IBM contacts or sel@us.ibm.com if you would like to discuss further.

    For Software (Bring-Your-Own-RHEL) deployment

    1. Migrate your SOAR data and configuration to a new provisioned RHEL8 instance using backup and restore tools

    The process described above would still be the recommended approach for BYORHEL. Upgrade your current RHEL7 instance to v50, and migrate the data to a different RHEL8 instance with v50 software installed.

    2. Perform in-place upgrade if you are familiar with RHEL in-place upgrade process using LEAPP tool (Your own RH subscription or RHEL ISO image is required)

    Please note our validation of the process outlined here was performed using vanilla RHEL7 OS instances with no extra HW or SW configurations. Your own environment may well have configurations we've not encountered and the responsibility for upgrading to RHEL8 and validating its success is between you and Redhat support.

    The SOAR v50 installation run file can serve the following purposes. 

      1. Update v49 SOAR to v50 on RHEL7

      2. Install new v50 on RHEL7.

      3. Install new v50 on RHEL8.

     

    The RHEL7 to RHEL8 in-place upgrade has to be done via Redhat OS update tool, LEAPP.

    If you are using your own RHEL VM (not the OVA) and familiar with the LEAPP tool, and prefer in-place upgrade your system, you will need Redhat's subscription or RHEL8 ISO, and strong Linux/RHEL knowledge to perform the upgrade. The following Redhat's documentation would provide the instruction to you to perform the OS upgrade. As mentioned previously, support for this process comes from Redhat.

    The overall sequence to in-place upgrade a v49 system on RHEL7 to v50 system on REHL8 would follow the steps below.

    • Same as the migration approach, you still can upgrade SOAR to v50 (step 1) first then perform OS upgrade at another time.

    • It's recommended to take a VM snapshot before conducting in-place upgrade because there is no easy way to rollback to RHEL7 if anything goes wrong during that process.

    • After step 4 OS upgrade, the SOAR rpms depending on RHEL7 will be removed by LEAPP, while the SOAR data, will persist in the system. The rpm removal will also remove a number SOAR software configurations that will need to be replaced later

    • After the OS upgrade completed, re-insall SOAR v50 software on RHEL8 systems, and all the data will be picked up.

    Finally, RHEL OS upgrade is not a simple process that can vary on different infra environments, we suggest you rehearse the upgrade process on test environments to discover potential obstacles.



    ------------------------------
    Martin Feeney
    Product Manager, IBM QRadar Security SOAR
    martin.feeney@ie.ibm.com
    ------------------------------


  • 2.  RE: Support for RHEL 8 Update

    Posted Wed July 26, 2023 05:31 AM

    You should also note that if you are running old python 2 based integrations on your existing appliance, these are not supported on RHEL8 so plans should be drawn up to upgrade them to python 3.

    Furthermore it has always been best practice to run integrations on a dedicated system, so as part of this migration please consider this, and perhaps consider the container based App Host which breaks the dependency between python runtime and the OS, making it easier to manage and upgrade. See the Site Planner doc, App Host Deployment Guide and Integration Server Guide for more details.



    ------------------------------
    Martin Feeney
    Product Manager, IBM Security SOAR
    martin.feeney@ie.ibm.com
    Ireland
    ------------------------------

    Original Message


  • 3.  RE: Support for RHEL 8 Update

    Posted Wed October 25, 2023 10:27 AM

    Hello Martin

    Thanks you for the guidance about SOAR-OS upgrade from RHEL7 to RHEL8.

    I would like to confirm more explicitly what conditions determine which method to choose.

    For Software (Bring-Your-Own-RHEL) deployment, two methods: "Backup and restore tools" and "In-place upgrade" are described.

    My current understandings with some experiments are:

      a) LVM configuration is used for soar programs
         (some free spaces in Volume Group to accomodate logical volumne for /crypt/attachments and logical volumes for snapshots)
      b) organization data created under en_US.UTF-8

    1) "Backup and restore tools"

      conditions: both (a) and (b) are met

    2) "In-place upgrade"

      conditions: the other cases of (1)

    Is this correct?



    ------------------------------
    Yohji Amano
    ------------------------------

    Original Message


  • 4.  RE: Support for RHEL 8 Update

    Posted Wed October 25, 2023 11:07 AM

    Hi Yohij, you are correct.

    You can use 1) soarSystemBackup and soarSystemRestore tools to migrate your SOAR from RHEL7 to a new RHEL8 instance and this is our recommended approach.

    Instead of 1), you can also choose 2) to in-place upgrade your OS and then reinstall SOAR, but please noted the OS upgrade process is not supported by IBM SOAR team. 

    More details for migrating a v50 BYOR SOAR to RHEL8 can be found here:

    https://www.ibm.com/docs/en/sqsp/50?topic=8-migrating-standalone-software-installation



    ------------------------------
    Gilbert Liao
    ------------------------------

    Original Message


  • 5.  RE: Support for RHEL 8 Update

    Posted Wed October 25, 2023 08:30 PM

    Hi Gilbert

    Thank you for your quick answers. 

    I'll take your affirmation would include the conditions listed in my previous post.



    ------------------------------
    Yohji Amano
    ------------------------------

    Original Message


  • 6.  RE: Support for RHEL 8 Update

    Posted Thu October 26, 2023 11:32 AM

    Hi Yohji,

    I'd like to clarify that b) organization data created under en_US.UTF-8 in your previous comment is not a mandatory requirement for RHEL8 migration. 

    Thanks.



    ------------------------------
    Gilbert Liao
    ------------------------------

    Original Message


  • 7.  RE: Support for RHEL 8 Update

    Posted Fri October 27, 2023 04:33 PM

    Hi Gilbert, thank you for clarifying the conditions.

    The reason why I posted this article was I actually failed to restore the organization data created under ja_JP.UTF-8 locale while I successfully restored the data created under en_US.UTF-8. I have tried two VMs with ja_JP.UTF-8. (Backup was OK. But Restore fail) and both cases were NG. (Error seemed to be observed with postgres connection.)

    So I wonder whether or not there may be some restrictions regarding locale settings.



    ------------------------------
    Yohji Amano
    ------------------------------

    Original Message


  • 8.  RE: Support for RHEL 8 Update

    Posted Sun October 29, 2023 08:35 AM
    Edited by Gilbert Liao Sun October 29, 2023 12:19 PM

    Hi Yohji,

    Can I clarify with you what  organization data created under ja_JP.UTF-8 locale mean? Does that mean

    a) the org is created using resuitl with ja locale. e.g.,  resutil neworg -name "テレビ" -orglocale "ja"

    or

    b) the backup file is generated using soarSystemBackup when VM's locale is set to ja_JP.UTF-8?

    Thank you.



    ------------------------------
    Gilbert Liao
    ------------------------------

    Original Message


  • 9.  RE: Support for RHEL 8 Update

    Posted Mon October 30, 2023 01:45 AM

    Hi Gilbert

    Thanks for your follow up. 

    Please disregard my previous messages since they may include my misunderstandings.

    Through further investigations, I've got why I failed to restore:

    The problem was that postgres log (/var/lib/pgsql/12/data/log/postgresql-<weekday 1st 3 letters>.log) was written in Japanese.

    11228 2023-10-28 05:14:22.929 JST > LOG:  リカバリ処理は復元ポイント"postgresql-backup-2023-10-28-050839"、時刻2023-10-28 05:09:15.134505+09 に停止します

    This was because I installed SOAR at ja_JP.UTF-8 locale.

    Then postgres configuration file (/var/lib/pgsql/12/data/postgresql.conf) was configured like lc_messages = 'ja_JP.UTF-8', lc_monetary = 'ja_JP.UTF-8' and so on. Due to these settings the log messages were logged in Japanese.  (They may suppress the locale settings in the scripts.)

    On the other hand resRestoreDatabaseToRestorePoint was invoked during soarSystemRestore command execution. Then I noticed that resRestoreDatabaseToRestorePoint suppose that the log message was ouput in English since grep is used like "recovery stopping at restore point".

    Due to mismatch,  resRestoreDatabaseToRestorePoint determined that something wrong happened during execution and stopped furthers.

    Based on the above results and with the workaround which may not be proper ways, I could complete soarSystemRestore command with the same backup file.



    ------------------------------
    Yohji Amano
    ------------------------------

    Original Message


  • 10.  RE: Support for RHEL 8 Update

    Posted Tue October 31, 2023 11:57 AM

    Hi Yohji,

    Thanks for your details investigation.

    We don't expect postgresql to be configured in non-english locale so we will enhance the installer to make sure that.

    Another follow-up question. Is the postgresql in your source system (the one soarSystemBackup running on) also configured in 'ja_JP.UTF-8'?



    ------------------------------
    Gilbert Liao
    ------------------------------

    Original Message


  • 11.  RE: Support for RHEL 8 Update

    Posted Wed November 01, 2023 06:37 AM

    1. To your another follow-up question.  -> yes.

    Thorough investigations, my observations are as follows:

    On source system

         install soar with ja_JP.UTF-8 locale.

                 This implicitly configures postgresql with ja_JP.UTF-8 locale.

         During soarSystemBackup    

                 postgresql.conf is  counted as one of migration resources. It's included in backup tar file.

    On target system

         During soarSystemRestore

                  - The configuration from source system was deployed to target system, which implicitly means

                     postgresql.conf was retrieved in ja_JP.UTF-8 locale on target system.

                  - At startup after retstoring the backup,  postgresql-<weekday 1st 3 letters>.log is written in the ja_JP.UTF-8, which come from

                     the source system. 

                  - restore tool expects English message in the log but Japanese message with same context was logged.

    2. so we will enhance the installer to make sure that.

       Be warned that en_US.UTF-8 locale is not necessarily available on RHEL8. User might need to install glibc-langpack-en additionally.

       In that case, system will not work thought simple configuration change to en_US.UTF-8 locale.



    ------------------------------
    Yohji Amano
    ------------------------------

    Original Message