Chronology of a search in IBM QRadar
In a distributed deployment of IBM QRadar event collection is done by Event Collectors while Flows are collected either by a Flow Collector or a QRadar Network Insight (QNI) appliance.
If there are no such Collectors present in the environment, Event Processors would be responsible for collecting events from different log sources and Flow Processors would be responsible for collecting Flow Records from different third party Flow Sources. In case of an All-in-One (AIO) deployment, the above event collection and Third party Flow collection would be done by this AIO itself.
The events or Flows collected are then stored on the Event Processor or the Flow Processor respectively. These Event or Flow processors have the Ariel Database on which the events or Flows that have been collected are stored. For searching purpose, these event or flow processors of a service named as the Ariel Query Server (AQS) while the Console from where you would initiate the search have the Ariel Proxy Server (APS) running on them. In case of an AIO, the AQS and APS both run the on the same box.
Data Nodes are special appliances which add storage space for Event Processors. Together with that they also speed up searches since the queries are sent directly to Data Nodes from the Console. Due to the Data Node responding back parallelly together with the Event Processor to which it has been attached to, the search results come in faster. Once you connect a Data Node to an Event Processor a process known as Data Rebalancing starts which equally distributes the existing data residing on the Event Processor (to which this Data Node is connected to) between the Event Processor and the Data Node. After the initial Data rebalancing is completed, further rebalancing of data continues happening between the Event Processor and the Data Node to make sure that data is distributed equally between them. You can refer to the below link to know more about IBM QRadar Data Nodes:
https://www.ibm.com/support/pages/what-qradar-data-node-appliance
Below diagram represents the flow of data (events/flows) in a deployment of IBM QRadar which includes Data Nodes as well:
- The arrows shown in blue show the flow of search requests and their results.
- The arrows in the orange show data rebalancing between Data Nodes and Event Processors attached to them.
- The arrows in the green show data collection when Event Collectors and Flow Collectors/QNI are involved.
Irrespective of the type of deployment of IBM QRadar you have, any search that you run (whether Event Search or Flow Search), they follow the below chronology before the search results are displayed on the Console User Interface:
- An analyst runs a QRadar Search on the QRadar console from either the Log Activity Tab or the Network Activity Tab (based on whether he wants to search for events or flows).
- IBM QRadar Console’s tomcat service (responsible for the QRadar’s User Interface) sends the search request to the Ariel Proxy Server running on the Console.
- Ariel Proxy Server has the list of QRadar Managed Hosts where data is stored i.e. the Event or Flow Processors as well as Data Nodes which have the Ariel Query Server running.
- Ariel Proxy Server sends the search request to the different Ariel Query Servers running on the Processors and Data Nodes.
- Ariel Query Server running on these Processors and Data Nodes then queries the data from the Ariel Database as per the filters that are present in the search query
- Each Ariel Query Server sends the data back to Ariel Proxy Server running on the Console
- Ariel Proxy Server on the Console consolidates the data that it received from the different Ariel Proxy Servers running on the Processors and Data Nodes and then sends it to the tomcat service for the results to be displayed on the QRadar Console User Interface.
Of course there are other sub-processes as well which help in querying, filtering, retrieving, and displaying the data. However, the above gives a birds eye view of how data in searched for and retrieved in IBM QRadar.
For any questions or comments regarding any of the points mentioned above or if you want to discuss this further, feel free to get in touch with us and we would be more than happy to answer any of your queries:
Authored by :
Ashish Kothekar (Ashish) – ashish.kothekar@in.ibm.com
Boudhayan Chakrabarty (Bob) – bochakra@in.ibm.com