IBM QRadar

 View Only

Introducing the IBM QRadar Data Synchronization App

By Joel Violette posted Tue September 08, 2020 03:00 PM

  

Today marks the release of the IBM QRadar Data Synchronization App. This post will give a brief overview of its key capabilities and the problems it solves. There have been many questions concerning its functionality and use case coverage; many of these common questions are answered at the end.

Disaster Recovery (DR) is a key aspect to the resiliency of a QRadar deployment. There is a wide variety of solutions currently deployed in the field for DR, including: redundant console only configurations, event and flow forwarding based solutions, and even full event distribution to two deployments (often termed "dual home"). These solutions vary greatly in terms of complexity, cost, and effectiveness. However, for the most part, customers rely on significant customization, usually offered by IBM professional services, in the setup and configuration on their DR solution.


Our previous blog post touched on the technical details of the upcoming IBM QRadar Data Synchronization App. This post will describe the scope of v1 of the app, how the app fits with the initiatives of our Disaster Recovery solution, and plans for the app in v2 and beyond.

Important: The IBM QRadar Data Synchronization solution is a paid-for service. To be within IBM license compliance and for the IBM QRadar Data Synchronization application to be effective you must purchase a QRadar Data Sync License(s). Contact your sales representative for more information.

Note: The IBM QRadar Data Synchronization app is currently incompatible with QRadar Network Insights appliances. Enhancements to support QRadar Network Insights appliances are currently planned.

What is the IBM QRadar Data Synchronization App?


The IBM QRadar Data Synchronization App provides a resilience solution for QRadar deployments in disaster events, ensuring operations can continue to function as normally as possible in such scenarios.

The core of a QRadar deployment is its store of historical data to support searching and correlation, as well as its real-time processing of incoming events and flows against that data. Therefore the two core sets of functionality that must be available for base SIEM operations are the event/flow data store and its processing services.


The IBM QRadar Data Synchronization App covers the two main components of resilience as follows:

1. Data store resilience

When the app is set up, it begins to replicate Ariel Data (Event and Flow Data) from the Main Site to the Destination Site, beginning at a customizable point in the past. When the initial synchronization period is complete, the Destination Site will be prepared with the Ariel Data (Event and Flow Data) from the Main Site. After this stage is complete, new incoming event and flow data to the Main Site will be copied across to the Destination Site at a definable interval.

2. Processing services resilience

This is the role the Destination Site fills. Upon setup of the IBM QRadar Data Synchronization App, the Destination Site's processing services are suppressed, leaving the Destination Site as a redundant set of systems, dormant, ready to take on the processing load of the Main Site in a disaster event. In a case where the Main Site goes down, the user simply activates the Destination Site, which initiates all of its collection and processing services. Since the Ariel copy functionality will have already populated this system with Ariel Data (Event and Flow Data), the Destination Site becomes a fully functioning QRadar SIEM.

What other features are included with the IBM QRadar Data Synchronization App?

Easy UI set up wizard

Inside the app we have an easy-to-follow setup wizard which walks the user through setup and configuration on both Main and Destination sites.


Host Mapping UI


The IBM QRadar Data Synchronization app has a Host Mapping UI which shows all mappings of Main Site hosts to Destination Site hosts. This interface includes Automatic Host Discovery, which is a feature that ends the need to manually import a map of the deployment. It does this by automatically populating the interface with all hosts in the deployment. Also included is the Host Pairing UI, where the user can easily browse, filter, pair, unpair, and view metrics of all hosts in the deployment.


Event and flow data synchronization


As mentioned above, for event and flow data synchronization, we have the automatic synchronization of Ariel Data (Event and Flow Data) from each host on the Main Site to its mapped host on the Destination Site. The app supports custom synchronization frequency, retroactive data sync, and leveraging retention buckets for data exclusion.

Destination Site Activation


The model here is for Destination Site Activation to require human initiation. The user clicks a single button on the Destination Site app, and activation sets in motion. The suppressed services on the Destination Site are automatically started, and after an initiation period of 10-15 minutes, event collecting, processing, and rule activation will begin on the newly activated Destination Site.


Ariel Copy Failback


At this point, when the Destination Site is active, the user can initiate reverse Ariel synchronization. This feature allows the Destination Site hosts to synchronize back to the Main Site any incoming event/flow data that has been collected on the Destination Site after Destination Site Activation.

What are the limitations of v1?

Since the IBM QRadar Data Synchronization App development team is rolling out features incrementally, not all eventual functionality is included in v1 of the app. Features and functionality not supported in v1 include:

  • No support for redundant Console-Only DR. This is a scenario wherein the user has a full deployment on the main site and only a console on the DR site, and upon DR activation all the hosts redirect to the DR console.
  • No support for dual-home DR Solutions. Due to service suppression on one site the DR app is not suitable for this type of DR solution, any events sent to the Destination Site would be dropped until the Destination Site is activated.
  • No automatic migration of users. Currently the user must manually configure the users on the DR Site.
  • No support for automatic rehoming of offsite EC's. An alternative solution is to use Disconnected Log Collector (DLC) which will support automatic rehoming in the future.
  • No migration of App Configuration and Data.
  • No support for Domains and Multitenancy. This requires significantly advanced remapping and additional logic for Ariel synchronization

    What is coming up in future versions?

    • Domains & Multitenancy
    • App Configuration and Data
    • Custom Configuration Data (Log Source Extensions, etc.)
    • DLC automatic rehoming
    • Wincollect automatic rehoming
    • Redundant Console

    Common Questions Around the IBM QRadar Data Synchronization App

    What requirements must my environment meet?

    1. Main Site and Destination Site deployments must be at QRadar version 7.4.0 FixPack 3.
    2. Destination Site deployment must be a fully duplicated deployment (1:1 host ratio) for hosts that contain or collect Ariel (event and flow) data. 
      1. Requiring 1:1 mapping: EPs, FPs, Combo (EP/EP), ECs, FCs, Consoles, and Data Nodes
      2. Not requiring 1:1 mapping: QRM, QVM, QIF, and App Host.
      3. A HA Cluster is considered one host. A HA cluster can be paired with a non-HA host.
    3. Destination Site hosts require equal or greater storage to their paired Main Site hosts.
    4. Deployment can not have domains configured (not supported for v1).

    When the Main Site goes down, how is event / flow traffic forwarded to the Destination site ECs / FCs?

    You can either repoint each data source one by one, or if you've configured your event sources by DNS name instead of IP address, you would update the DNS record to the new IP.

    Since the IBM QRadar Data Synchronization App only supports a 1:1 mapping of hosts, can you pair a VM host with a hardware host?

    There is no hardware limitation on mappings - you can pair hardware to VM, but would want to ensure the VM has specs to handle the load once the Destination Site is activated.

    Does the Destination Site get activated automatically if the Main Site goes down?

    No. The Destination Site is always activated by the user in the Destination Site app UI.

    How do I know if everything is working as it should?

    The Host Summary view in the Host Mapping UI includes the current connectivity status between each host and its pair, and the Ariel copy history information, a list of all recently completed Ariel Data (Event and Flow Data) synchronization actions.

    When will the IBM QRadar Data Synchronization App be available?

    It is out now on the IBM Security App Exchange. Download it here.

    Learn more:
    - IBM QRadar Data Sync App Official Documentation
    - 25 Common Questions about the IBM QRadar Data Synchronization App


    #Featured-area-2
    #Highlights
    #Highlights-home
    #QRadar
    5 comments
    1362 views

    Permalink

    Comments

    Joel Violette
    Tue July 20, 2021 03:23 PM

    Hello! Thanks for your questions. I've moved to a new project, and will pass them along to Eric Larsen who took over my role.

    I've asked the team and new blog will be released with version 3.0.0, and I will link it here to update when it is released.

    M R
    Sun July 04, 2021 12:07 PM

    Hello Joel,

    How we can know that Data synch in running and when i will finished

    Darren H.
    Fri February 05, 2021 04:43 AM

    Hello Joel,

    Can you [or Shane Lundy] update this post as it is a few months since the launch andV 2.0.0 is out there with newer features.

    In addition, can you also state the event loss and timings for the product? There is no mention of any measurable elements or metrics on fail over or recovery.

    Many thanks!

    Joel Violette
    Wed October 14, 2020 09:28 AM

    Hi Ryan, the failover site will be licensed with Data Sync licenses; licensing for the failover (destination) site is on a flat per-node basis, not based on EPS.

    Ryan Hitch
    Wed September 30, 2020 11:33 AM

    What licensing is required at the failover site? For example, if my active site has an all in one console do I need to buy a full all in one console license for the passive site or is there a cheaper DR site option since this console will be sitting there idle 99% of the time.