Hi dear colleagues!
For long years our company has been implementing and provide SOC in Ukraine. We chose IBM Qradar as the SIEM in our SOC. During the implementation process we also create such document like Disaster Recovery Plan (DRP) for the SIEM system. In DRP we describe different scenarios of restore processes after disaster in customer infrastructure. All works great while IBM Qradar App host exist :) After we started to use IBM Qradar App host we lose opportunity to play the most critical scenario of restore process - "Full restore of all Qradar components".

Now I will describe the problem that we get during the test restore process:

1 step - Installing of the new environment with the same network parameters in isolated network. Result - all completed without any errors.

2 step - Restore of the configurations and data. Result - Success with ERROR "Host tokens did not match". We understood why this error exist but make decision to open the ticket to IBM Support and work together with this problem.

Work with IBM Support Engineer - mister X :)

3 step - Updating of hosts tokens. Result - all completed with error "App host state N/A"

4 step - IBM Engineer recommended to re-add App host. Result - Fail with ERROR "The system fill that all apps are running on N/A App Host".

5 step - Transferring of all apps from App Host to Qradar Console by unknown script. Result - Could not perform while ERROR exist "IBM Qradar App Host state - N/A"

6 step - Transferring by unknown IBM script. Result - Fail with ERROR "Leak of RAM on IBM Qradar Console". It is not a secret for everyone that we buy App host install license that's only 10 % of IBM Qradar Console are used for running Apps :). No matter. We have good relations with our system administrator and he increased RAM up to 96 GB.

7 step - Transferring by unknown IBM script. Result - Success with ERROR "app framework on the console was not healthy".

8 step - Fix the problem with app framework. This process include many undocumented changes in PostgreSQL, running unknown IBM scripts, restarting of all services. Result - Success

All this process take 14 hours for 1st client (6 hosts in deployment) and 5 hours for 2-nd client (2 hosts in deployment).

And now I want to ask You such questions:

Is it normal restore process for enterprise product?

Did you test this scenario in your environment? What was the result?

Should we ask IBM to make detailed manual of restore process for distributed Qradar deployment?

Have You any recommendations?

Thank You.

The best answer for IBM Support that I ever get 🤯

"This is caused by a known defect related to a critical file not being restored correctly. It is not user error and we are aware of the issue however there currently is no fix."

This is backup of enterprise product 🥺!

------------------------------
Vadim Novikov
SOC Engineer
IT Specialist
Kiev
+380972970792

Original Message:
Sent: Fri December 18, 2020 11:05 PM
From: Vadim Novikov
Subject: Nonworking restoring proceess for distributed IBM Qradar Deployment

Hi dear colleagues!
For long years our company has been implementing and provide SOC in Ukraine. We chose IBM Qradar as the SIEM in our SOC. During the implementation process we also create such document like Disaster Recovery Plan (DRP) for the SIEM system. In DRP we describe different scenarios of restore processes after disaster in customer infrastructure. All works great while IBM Qradar App host exist :) After we started to use IBM Qradar App host we lose opportunity to play the most critical scenario of restore process - "Full restore of all Qradar components".

Now I will describe the problem that we get during the test restore process:

1 step - Installing of the new environment with the same network parameters in isolated network. Result - all completed without any errors.

2 step - Restore of the configurations and data. Result - Success with ERROR "Host tokens did not match". We understood why this error exist but make decision to open the ticket to IBM Support and work together with this problem.

Work with IBM Support Engineer - mister X :)

3 step - Updating of hosts tokens. Result - all completed with error "App host state N/A"

4 step - IBM Engineer recommended to re-add App host. Result - Fail with ERROR "The system fill that all apps are running on N/A App Host".

5 step - Transferring of all apps from App Host to Qradar Console by unknown script. Result - Could not perform while ERROR exist "IBM Qradar App Host state - N/A"

6 step - Transferring by unknown IBM script. Result - Fail with ERROR "Leak of RAM on IBM Qradar Console". It is not a secret for everyone that we buy App host install license that's only 10 % of IBM Qradar Console are used for running Apps :). No matter. We have good relations with our system administrator and he increased RAM up to 96 GB.

7 step - Transferring by unknown IBM script. Result - Success with ERROR "app framework on the console was not healthy".

8 step - Fix the problem with app framework. This process include many undocumented changes in PostgreSQL, running unknown IBM scripts, restarting of all services. Result - Success

All this process take 14 hours for 1st client (6 hosts in deployment) and 5 hours for 2-nd client (2 hosts in deployment).

And now I want to ask You such questions:

Is it normal restore process for enterprise product?

Did you test this scenario in your environment? What was the result?

Should we ask IBM to make detailed manual of restore process for distributed Qradar deployment?

Have You any recommendations?

Thank You.

'had my part of headaches during last few months trying several times to migrate the console from an old to the new appliance (several managed hosts, though no App Host present). I believe one crucial problem was that sometimes during the restore the managed hosts are not recognized as being there on the network and afterwards a number of issues can occur.... this is something I believe to be present for quite a while (and still is), but could be fixed in the next minor release. Anyway, you can try using /opt/qradar/support/defect-inspector to check through the qradar.error log if something obvious was noticed, but also maybe it could be possible option to submit a severity 4 case to ask for a defect inspection by a support engineer (so some fixes for know stuff would be implemented in advance).

The best answer for IBM Support that I ever get 🤯

"This is caused by a known defect related to a critical file not being restored correctly. It is not user error and we are aware of the issue however there currently is no fix."

This is backup of enterprise product 🥺!

------------------------------
Vadim Novikov
SOC Engineer
IT Specialist
Kiev
+380972970792

Original Message:
Sent: Fri December 18, 2020 11:05 PM
From: Vadim Novikov
Subject: Nonworking restoring proceess for distributed IBM Qradar Deployment

Hi dear colleagues!
For long years our company has been implementing and provide SOC in Ukraine. We chose IBM Qradar as the SIEM in our SOC. During the implementation process we also create such document like Disaster Recovery Plan (DRP) for the SIEM system. In DRP we describe different scenarios of restore processes after disaster in customer infrastructure. All works great while IBM Qradar App host exist :) After we started to use IBM Qradar App host we lose opportunity to play the most critical scenario of restore process - "Full restore of all Qradar components".

Now I will describe the problem that we get during the test restore process:

1 step - Installing of the new environment with the same network parameters in isolated network. Result - all completed without any errors.

2 step - Restore of the configurations and data. Result - Success with ERROR "Host tokens did not match". We understood why this error exist but make decision to open the ticket to IBM Support and work together with this problem.

Work with IBM Support Engineer - mister X :)

3 step - Updating of hosts tokens. Result - all completed with error "App host state N/A"

4 step - IBM Engineer recommended to re-add App host. Result - Fail with ERROR "The system fill that all apps are running on N/A App Host".

5 step - Transferring of all apps from App Host to Qradar Console by unknown script. Result - Could not perform while ERROR exist "IBM Qradar App Host state - N/A"

6 step - Transferring by unknown IBM script. Result - Fail with ERROR "Leak of RAM on IBM Qradar Console". It is not a secret for everyone that we buy App host install license that's only 10 % of IBM Qradar Console are used for running Apps :). No matter. We have good relations with our system administrator and he increased RAM up to 96 GB.

7 step - Transferring by unknown IBM script. Result - Success with ERROR "app framework on the console was not healthy".

8 step - Fix the problem with app framework. This process include many undocumented changes in PostgreSQL, running unknown IBM scripts, restarting of all services. Result - Success

All this process take 14 hours for 1st client (6 hosts in deployment) and 5 hours for 2-nd client (2 hosts in deployment).

And now I want to ask You such questions:

Is it normal restore process for enterprise product?

Did you test this scenario in your environment? What was the result?

Should we ask IBM to make detailed manual of restore process for distributed Qradar deployment?

Have You any recommendations?

Thank You.

The best answer for IBM Support that I ever get 🤯

"This is caused by a known defect related to a critical file not being restored correctly. It is not user error and we are aware of the issue however there currently is no fix."

This is backup of enterprise product 🥺!

------------------------------
Vadim Novikov
SOC Engineer
IT Specialist
Kiev
+380972970792

Original Message:
Sent: Fri December 18, 2020 11:05 PM
From: Vadim Novikov
Subject: Nonworking restoring proceess for distributed IBM Qradar Deployment

Hi dear colleagues!
For long years our company has been implementing and provide SOC in Ukraine. We chose IBM Qradar as the SIEM in our SOC. During the implementation process we also create such document like Disaster Recovery Plan (DRP) for the SIEM system. In DRP we describe different scenarios of restore processes after disaster in customer infrastructure. All works great while IBM Qradar App host exist :) After we started to use IBM Qradar App host we lose opportunity to play the most critical scenario of restore process - "Full restore of all Qradar components".

Now I will describe the problem that we get during the test restore process:

1 step - Installing of the new environment with the same network parameters in isolated network. Result - all completed without any errors.

2 step - Restore of the configurations and data. Result - Success with ERROR "Host tokens did not match". We understood why this error exist but make decision to open the ticket to IBM Support and work together with this problem.

Work with IBM Support Engineer - mister X :)

3 step - Updating of hosts tokens. Result - all completed with error "App host state N/A"

4 step - IBM Engineer recommended to re-add App host. Result - Fail with ERROR "The system fill that all apps are running on N/A App Host".

5 step - Transferring of all apps from App Host to Qradar Console by unknown script. Result - Could not perform while ERROR exist "IBM Qradar App Host state - N/A"

6 step - Transferring by unknown IBM script. Result - Fail with ERROR "Leak of RAM on IBM Qradar Console". It is not a secret for everyone that we buy App host install license that's only 10 % of IBM Qradar Console are used for running Apps :). No matter. We have good relations with our system administrator and he increased RAM up to 96 GB.

7 step - Transferring by unknown IBM script. Result - Success with ERROR "app framework on the console was not healthy".

8 step - Fix the problem with app framework. This process include many undocumented changes in PostgreSQL, running unknown IBM scripts, restarting of all services. Result - Success

All this process take 14 hours for 1st client (6 hosts in deployment) and 5 hours for 2-nd client (2 hosts in deployment).

And now I want to ask You such questions:

Is it normal restore process for enterprise product?

Did you test this scenario in your environment? What was the result?

Should we ask IBM to make detailed manual of restore process for distributed Qradar deployment?

Have You any recommendations?

Thank You.

Hi Vadim,

I'm the one that wrote the quote you're referencing, so I'd like to clarify something. In our terminology, a "fix" means that Development has identified the source of the issue and has updated the code to permanently resolve the issue. A "workaround" is a temporary procedure that Support is able to perform in order to alleviate the issue, but it may or may not be a permanent resolution.

As I explained to your colleague who opened the case that you've pulled this quote from, this is in reference to a known issue where we do not have a "fix", but we do have a "workaround". If you continue to experience this issue please do not hesitate to open a case, and we'll be happy perform the workaround for you, as well as update the internal defect to make sure that Development is aware of the impact.

The reason there isn't an APAR for this issue is because it hasn't been requested. Not all known defects get published publicly. I've requested one for you and will provide an update.

That being said, your original post here includes a scenario which uses a config restore, but that does not mean that all config restores or disaster recovery scenarios are affected. I see that you have opened a new case requesting some information regarding an officially documented procedure. Depending on your version of QRadar, here are the links that we currently have:

https://www.ibm.com/support/knowledgecenter/SS42VS_7.4/com.ibm.qradar.doc/c_qradar_ha_data_redundancy_overview.html

https://www.ibm.com/support/knowledgecenter/SS42VS_7.3.2/com.ibm.qradar.doc/c_qradar_ha_data_redundancy_overview.html

https://www.ibm.com/support/pages/qradar-siem-hardware-migration-scenarios

If these are not helpful for your scenario we'll be more than happy to continue the discussion in your case. 

Thanks,
David Spell
L2 QRadar Product Support

The best answer for IBM Support that I ever get 🤯

"This is caused by a known defect related to a critical file not being restored correctly. It is not user error and we are aware of the issue however there currently is no fix."

This is backup of enterprise product 🥺!

------------------------------
Vadim Novikov
SOC Engineer
IT Specialist
Kiev
+380972970792

Original Message:
Sent: Fri December 18, 2020 11:05 PM
From: Vadim Novikov
Subject: Nonworking restoring proceess for distributed IBM Qradar Deployment

Hi dear colleagues!
For long years our company has been implementing and provide SOC in Ukraine. We chose IBM Qradar as the SIEM in our SOC. During the implementation process we also create such document like Disaster Recovery Plan (DRP) for the SIEM system. In DRP we describe different scenarios of restore processes after disaster in customer infrastructure. All works great while IBM Qradar App host exist :) After we started to use IBM Qradar App host we lose opportunity to play the most critical scenario of restore process - "Full restore of all Qradar components".

Now I will describe the problem that we get during the test restore process:

1 step - Installing of the new environment with the same network parameters in isolated network. Result - all completed without any errors.

2 step - Restore of the configurations and data. Result - Success with ERROR "Host tokens did not match". We understood why this error exist but make decision to open the ticket to IBM Support and work together with this problem.

Work with IBM Support Engineer - mister X :)

3 step - Updating of hosts tokens. Result - all completed with error "App host state N/A"

4 step - IBM Engineer recommended to re-add App host. Result - Fail with ERROR "The system fill that all apps are running on N/A App Host".

5 step - Transferring of all apps from App Host to Qradar Console by unknown script. Result - Could not perform while ERROR exist "IBM Qradar App Host state - N/A"

6 step - Transferring by unknown IBM script. Result - Fail with ERROR "Leak of RAM on IBM Qradar Console". It is not a secret for everyone that we buy App host install license that's only 10 % of IBM Qradar Console are used for running Apps :). No matter. We have good relations with our system administrator and he increased RAM up to 96 GB.

7 step - Transferring by unknown IBM script. Result - Success with ERROR "app framework on the console was not healthy".

8 step - Fix the problem with app framework. This process include many undocumented changes in PostgreSQL, running unknown IBM scripts, restarting of all services. Result - Success

All this process take 14 hours for 1st client (6 hosts in deployment) and 5 hours for 2-nd client (2 hosts in deployment).

And now I want to ask You such questions:

Is it normal restore process for enterprise product?

Did you test this scenario in your environment? What was the result?

Should we ask IBM to make detailed manual of restore process for distributed Qradar deployment?

Have You any recommendations?

Thank You.