IBM QRadar is a Security Information and Event Management (SIEM) solution from IBM that provides security awareness and compliance monitoring.
With IBM QRadar, you can get the present Security posture of you environment using a combination of Event, Flow, Vulnerability, Threat Indicators etc processing capabilities.
Summary:
Consider a SOC environment, where you have hundreds of thousand log sources integrated in your SOC environment, which could belong to a single customer, such as a Bank, or, where SOC team is an MSSP and providing SIEM solutions to large number of clients.
In such situations, to identify logs sources that were sending events and suddenly stopped sending events becomes difficult to track and then resolve to get the log sources up and running again.
SOC Analysts often need to detect log sources which stop reporting events to IBM QRadar in order to fix and resume event reporting. While they may have various monitoring products to detect this, it is sometimes required that this detection happens on QRadar itself.
Such requirement is often met by creating DSSE (Device Stopped Sending Events) rules.
This can be done by using a Tracker rule and a watcher rule.
Basically, the tracker rule functions on reference maps, keeps track of active log sources and loads into reference set ActiveSystemSending. When elements from reference set gets expired it triggers watcher rule to generate offense.
The purpose of this article is to provide descriptive details on creating rules for device stopped sending events.
Rule #1 - Tracker Rule
A tracker rule records the log sources and events as they come through the system. This rule uses reference map as a record keeper where all log source Names and their ID’s. If a log source ID exists in the SystemsToWatch reference map, then the value gets placed in the ActiveSystemsSending reference set.
Step 1: Create Reference Map:
1. Export CSV file with log source name and log source ID.
select logsourceid, LOGSOURCENAME(logsourceid) from events group by logsourceid last 8 Hours
Once search is complete, Go to Actions > Export to CSV > Visible Columns
Open this file in Excel and add Key1 column name for log source ID and Data column name for log source name and save the file.
2. Create and provision the reference map via the CLI (can also be done using the REST API)
/opt/qradar/bin/ReferenceDataUtil.sh create SystemsToWatch MAP ALN
Arg: create
Arg: SystemsToWatch
Arg: MAP
Arg: ALN
Successfully created Reference Data Collection. ReferenceDataCacheMap{id=172, namespace=Shared,
name=SystemsToWatch, collectionType=MAP, elementType=ALN, createdTime=2019-01-09 16:57:29}
3. Transfer CSV file from local server into qradar's /tmp
4. Check the list of all log sources by cat /tmp/log-source-export.csv
key1,data
312,FirewallCluster
313,FirewallCluster
5. /opt/qradar/bin/ReferenceDataUtil.sh load SystemsToWatch /tmp/log-source-export.csv
Arg: load
Arg: SystemsToWatch
Arg: /tmp/log-source-export.csv
Processed 2 records from /tmp/log-source-export.csv
6. /opt/qradar/bin/ReferenceDataUtil.sh list SystemsToWatch displayContents
Arg: list
Arg: SystemsToWatch
Arg: displayContents
ReferenceDataCacheMap{id=172, namespace=Shared, name=SystemsToWatch, collectionType=MAP,
elementType=ALN, createdTime=2019-01-09 16:57:29}
Key1=312 Data=FirewallCluster
Key1=313 Data=FirewallCluster
Alternatively, when the number of similar log sources that you want to monitor count is very high, you can use the Reference Data Management App to import CSV file.
=============================================================================
Step 2: Create Reference Set “ActiveSystemSending”
1. Create Reference set name ActiveSystemSending, select type Alphanumeric Ignore Case. Give Time to live element since last seen, select When elements expire select Log each element in a separate log entry and select shared tenant.
=============================================================================
Step 3: Create CEP “SystemName”
1. Create AQL based CEP.
REFERENCEMAP('SystemsToWatch',logsourceid)
=============================================================================
Step 4: Verify the reference map showing the results on UI, the same we are going to be used in the rule.
select logsourceid, LOGSOURCENAME(logsourceid) from events where REFERENCEMAP('SystemsToWatch', logsourceid) IS NOT NULL last 2 hours
=============================================================================
Step 5: Create the Tracker rule.
1. Rule Condition:
2. Rule Response:
=============================================================================
Once you do this active log sources will start to populate into reference set ActiveSystemSending, it will start “Time To Live” on that log source. TTL will complete the given time after that log source element will get expired and it will create a seperate log entry like below:
[pool-7-thread-1] com.q1labs.core.shared.referencedata.ReferenceDataExpiryThread: [INFO] [NOT:0070006100][xx.xx.xx.xx/- -] [-/- -]Expired ReferenceData element [{LinuxServer @ redhat1}; from collection: ActiveSystemsSending
[{LinuxServer @ redhat1} -> This is log source name and collection: ActiveSystemsSending -> Reference set Name.
Once watcher rule find such a log entry it will generate offense.
=============================================================================
Step 6: Required CEPs for watcher rule.
Before creating watcher rule, we need to create 2 Extraction based CEP's
1. This one will confirm expired element entry is from correct reference set/collection.
CEP for ActiveSystemsending = collection:\s(\w+)\]
In case if you have given any other name, please create custom CEP which will check and verify correct collection name.
2. For indexing purpose.
Rule Response selection, For indexing purpose.
The Log Source Name that is mentioned in this procedure is ExpiredReferenceElement and it is configured as follows.
=============================================================================
Step 7: Create Watcher rule as follows.
=============================================================================
Optional:
AND when the event matches DSSE_Device_Name (custom) is not any of [N/A or n/a or null or NULL]
AND when any of DSSE_Device_Name (custom) are contained in any of My_Test_Device_List
AND NOT when any of DSSE_Device_Name (custom) are contained in any of ActiveSystemSending
In this example, the DSSE_Device_Name (custom) will extract the hostname/IP of the device that we monitor to check if the element expired is one of the monitored systems.
When you have a lot of reference sets that contains IP, URL, IPv6, hash expires because of time to live value set, then CEP might not match for such an element and might contain N/A, n/a, null, or NULL in the DSSE_Device_Name (custom) cep, which we do want to avoid.
At time, we observed that when system is heavily loaded, their might be a small time difference when the DSSE_Device_Name (custom) CEP is still present in the ActiveSystemSending reference set before it is removed as part of TTL value for the amount of time we monitor the device.
In such situation, we compare the presence of the monitored system in the DSSE_Device_Name (custom) cep with a super-set of all the monitored devices of similar type which is populated in the My_Test_Device_List reference set. And, the device must have been removed from the ActiveSystemSending reference set as part of the TTL value being met.
Once it’s done Create a dummy log source to test the functionality of these rules or else monitor for some time.
If at any point in time, you have any questions, have any comments or want to discuss this further, feel free to get in touch with us and we would be more than happy to answer any of your queries:
Vaishnavi Mangalarap – Vaishnavi.mangalarap@ibm.com
Jenson John - jenson.john@in.ibm.com
Special thanks to Prateek Jain (prateek.j@in.ibm.com) for reviewing our Blog.