1 . Introduction
IBM QRadar is the leader in Gartner’s Magic Quadrant for Security Information and Event Management solutions. QRadar correlates data from disparate sources including raw packets of data from the wire and coverts these billions and trillions of events and raw packets of data into a few actionable items. Out-of-the-box QRadar supports hundreds of security solutions, varied applications installed on-prem as well as on cloud, system and audit logging from different platforms. Those platforms which are not supported out-of-the-box can still be easily integrated with QRadar using tools like DSM Editor.
When the data from such disparate log sources is sent to QRadar, the data itself is in different formats. To normalized the data of different formats, QRadar has Device Support Modules also known as DSMs. These DSMs are used to parse the data and convert it into QRadar's consumable format. These DSMs are software modules which are updated regularly by IBM.
From a payload, DSM extracts information required by QRadar. This information is used to run correlation rules in QRadar. For example: From a certain payload, we can extract the username, time of the event, and access details of the user. If the user tried to login into a server 50 times within 5 minutes and failed because of invalid credentials, then all the information like time, event count, username and action performed are captured by QRadar. This information can be used to create rules for invalid login usage as well as can be used to create reports etc.
Sometimes, the payload may contain more information than what QRadar extracts by default. In such scenarios, you can use the Custom Event Property (CEP) feature of QRadar. For example, if a certain payload contains hostnames in FQDN format and the default DSM does not parse this information, then CEPs can be created by users to extract the required information from the payload.
2. Where can we use CEP in QRadar?
In QRadar, we use custom event properties when we want to extract the non-normalized fields from event payloads.
Some log sources send unique information that is important for the administrators and want this data to appear in the user interface while running searches, reports, etc. This is where CEPs come in allowing the admin to use regex to extract the important field from the event payload and display it in the user interface. Once the CEP's are captured, they are all labelled with the term (custom) and can be observed in the ‘Log Activity Tab’.
You can create the Custom Event property by two ways.
1) Click on Admin --> Under "Data Sources" click on "Custom Event Properties"
2) Click on Log Activity --> Click on Search (New search/Edit Search) then click on "Manage Custom Properties".
3. Impact of non-optimized CEP (example, performance degradation):
While creating CEP's, you need to make sure that those CEPs are optimized CEPs. Slow-processing CEPs degrade pipeline performance and can send events directly to storage. Due to unoptimized CEPs, rules and searches may not produce the desired results.
Example:
If there is a performance issue, you will get notifications on your QRadar dashboard saying "Performance degradation has been detected in the event pipeline. Events were routed directly to storage.”:
Nov xx 11:34:20 ::ffff:x.x.x.x [ecs-ec.ecs-ec] [[type=com.eventgnosis.system.ThreadedEventProcessor][parent=SIEMECDC-primary.xxxxxx.net:ecs-ec/EC/Parsing/DSM_Normalize]] com.q1labs.semsources.filters.normalize.DSMFilter: [WARN] [NOT:0080004101][x.x.x.x/- -] [-/- -]Device Parsing has sent a total of 262488 event(s) directly to storage. 123632 event(s) have been sent in the last 60 seconds. Queue is at 100 percent capacity.
Oct xx 11:16:05 ::ffff:x.x.x.x [ecs-ep.ecs-ep] [[type=com.eventgnosis.system.ThreadedEventProcessor][parent=siem.xxx.xx:ecs-ep/EP/Processor2]] com.q1labs.semsources.cre.CRE: [WARN] [NOT:0080004101][x.x.x.x/- -] [-/- -]Custom Rule Engine has sent a total of 3306785 event(s) directly to storage. 240852 event(s) were sent in the last 60 seconds. Queue is at 100 percent capacity.
Nov xx 12:05:05 ::ffff:x.x.x.x [ecs-ec.ecs-ec] [Timer-15] com.q1labs.semsources.filters.normalize.DSMFilter: [WARN] [NOT:0080014100][x.x.x.x/- -] [-/- -]Expensive Log Source or Log Source Extensions Based On Average Throughput in the last 60 seconds (most to least expensive) - ArubaClearPass=413.0eps, H3CComware=562.0eps, GenericDSM=571.0eps, CustomFIIntegraCustom_ext=785.0eps, VmWare=1283.0eps, LinuxServer=1504.0eps, CiscoCallManager=1657.0eps, F5APM=2366.0eps
4. RegEx and Capture Group
4.1. Introduction
Before you understand how to optimize the CEP, there are a couple of terms you should be aware of:
- RegEx
- Capture Group
What is RegEx?
- Regular Expression (RegEx) is a string of text that allows you to create patterns that help match, locate and manage text.
- In QRadar, we use regex to extract the piece of data from the payload.
What is Capture Group?
- A part of a pattern that can be enclosed in parentheses “()” is called a capture group
- There can be one or more capture groups
4.2. RegEx Patterns
Here are some RegEx patterns which will come in handy while creating Regular Expressions:
|
Pattern
|
Description
|
|
\d
|
One digit from 0 to 9
|
|
\D
|
One character which is not digit
|
|
\w
|
One word character. (ASCII Letter, digit, underscore)
|
|
\W
|
One character which is not word
|
|
\s
|
Whitespace character. (Spaces, Tabs, New line)
|
|
\S
|
One character which is not whitespace.
|
|
\t
|
Tab
|
|
+
|
One or more characters. Example: \w+ or \d+
|
|
*
|
Zero or more characters. Example: \w* or \d*
|
|
?
|
Once or None
|
|
.
|
Any character except line break. Example: .*?
|
|
\
|
Escape a special character
|
|
|
|
OR Operand
|
|
[…]
|
One or More character in bracket
|
|
[\d\w]
|
One character that is digit or word
|
|
^
|
Start or string. (But when [^ inside bracket], it means not)
|
|
[^x]
|
One character that is not x
|
|
{n}
|
Exactly “n” number of times. Example: \d{3}
|
|
[a-z] or [A-Z] or [0-9]
|
One character within defined range
|
4.3. Writing RegEx
Based on the RegEx patterns which has been shown in section 4.2, let us extract the below highlighted information from the sample payload:
DeviceType=Estreamer DeviceAddress=x.x.222.111 CurrentTime=1648543496131 recordType=IPS_IMPACT_ALERT recordLength=335 timestamp=29 Mar 2022 01:44:54 netmapDomainRef=0 impactAlertData.eventId=44869 impactAlertData.detectionEngineId=2 impactAlertData.eventSecond=1648543493 impactAlertData.impact=7 impactAlertData.sourceAddress=x.x.111.22 impactAlertData.destinationAddress=x.x.222.55 impactAlertData.description=[1:58562:1] "SERVER-WEBAPP Oracle WebLogic Server remote code execution attempt" [Impact: Potentially Vulnerable] From "CAPTURE-DATA.111." at Tue Mar 29 08:44:53 2022 UTC [Classification: Web Application Attack] [Priority: 1] {tcp} x.x.111.44:47536 (united states)->x.x.222.55:80 (unknown)
| DeviceAddress=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) |
| Capture Group: 1 |
| recordType=(\w+) |
| Capture Group: 1 |
| recordLength=(\d+) |
| Capture Group: 1 |
| timestamp=(.*?)\snetmapDomainRef |
| Capture Group: 1 |
| impactAlertData\.description=\[([\d\:]+)\] |
| Capture Group: 1 |
4.4. Optimized Vs Unoptimized regex
Even if there are multiple ways to extract the same information from a payload, it is important to make sure that the RegEx that you are writing is optimized. Lets capture the information which is highlighted below in green from the same payload in different ways to find out which would be Optimized and which one would not be:
4.4.1. Example 1: Let us capture CAPTURE-DATA.111. from the payload
RegEx 1:
| From\s\"([\w\.\-]+)\" |
| Capture steps: 1 |
RegEx 2:
| From\s\"(.*?)\" |
| Capture steps: 1 |
If you observe RegEx 1 and RegEx 2, you will notice the number of steps needed to capture the required information via RegEx 1 (11 steps) is less compared to RegEx 2 (50 steps). Hence, RegEx 1 is an optimised Regular Expression.
4.4.2. Example 2: Capturing tcp from the payload
RegEx 1:
| \s\{(\w+)\} |
| Capture steps: 1 |
RegEx 2:
| \]\s\{(\w+)\} |
| Capture steps: 1 |
There are 117 steps required to capture the required information in RegEx 1, but only 17 steps with RegEx 2.
To prevent any QRadar Event Pipeline performance issues, you need to use Optimized RegExes while creating CEPs in QRadar. We talked about how to create Optimized Regular Expressions above.
If you have any questions regarding any of the points mentioned above or want to discuss this further, feel free to get in touch with us.