Special Thanks to @Noaa Kless for all the hard work she put into the project. You'll find some of her comments throughout this blog

Hi guys

If you clicked on this link, you probably don't need to be convinced on the value of SIGMA rules. Everybody talks about it, a lot of rules repositories exist (the main one can be found here)... There is no doubt they are a good tool. 

But if you are not sure what they are, here is my take on it...

SIGMA rules are detection rules, built in an agnostic format, that helps researchers and admins understand what they are looking for, no matter which tool they are working on. It could very well be written in English, but instead it uses a coding format and goes straight to the point, can be written in minutes, and most important, it can be parsed to work in any environment.

SIGMA rules are community driven, so you get the wisdom and rapidity of the crowd working with you. 
You probably have noticed it, every time a new significant threat or vulnerability gets announced, a few hours later we have responses coming from the community, and they are often the same: a Snort signature, a YARA rule... And a SIGMA rule ! 

While I said that SIGMA rules are agnostic, the title of this blog post mentions "content optimized for QRadar", and that's probably what brought you here, so let's talk about it ! 

Welcome to the new pySigma script !

A few tools exist today to convert SIGMA rules to AQL, and they do serve that purpose, but the queries generated are not always adapted to run correctly on QRadar, I will explain that in a minute...

In case you haven't gotten the news, the original sigmac script that initially been written as part of the SIGMA project is going end of life at the end of 2023, it is being replaced by pySigma which works on a new architecture (documentation can be found here). 

The pySigma script usage is straight forward, you feed it with a YAML file (or directory of YAML files) and it outputs the corresponding AQL, which allows to create detection rules or searches for threat hunting.
Two main problems can be highlighted in the original behaviour: A lot of the time the payload is used to query the data, which is not great for performances, and in some cases the properties used are not the right ones, which can result in false-negatives. 

So we've created a new script ! It can be found here on Github.

What does the script do differently ?

A bunch of things that can make this blog hard to consume... So here are a few quick links that will help you navigate ;)

Mapping of the Custom Properties
Mapping of the Log Sources
Use of LIKE instead of ILIKE
Subnet search
Increase the number of true positives

Mapping of the Custom Properties

Over the years we have released many properties on the App Exchange, we've made sure that all the properties were normalized as much as possible across all the devices, just like you always find the same properties in the DSMs. We even released the IBM QRadar Custom Properties Dictionary, so anybody could build their own property expressions and tie them to the normalized property definitions.

This normalization work allowed us to map the properties used in the SIGMA rules to the properties used in QRadar, this is what it looks like:

Mapping of the Log Sources

We didn't stop at the CEPs, to make the searches even faster, we've mapped the products name and services to the QRadar Log Source types.
Here are a few examples:

This mapping allows to filter on a specific Log Source Type, even if no property of the rule is known and the payload is used, which will decrease drastically the processing time.

Explanation from Noaa: This mapping helps us to get even better performance, because it can map the rule’s log source type name to the matching QRadar log source type id, so the filtering can be done on the ‘devicetype’ field Instead of the ‘LOGSOURCETYPENAME()’ function.

Use of LIKE instead of ILIKE

This is a simple change, but it makes a fairly big difference.

When you are using ILIKE, you have to test for each characters in upper and lower case. 
By using the function LOWER(), you can use the function LIKE and test the chain of characters in lower case only, and get the same results.

Subnet search

Another problem we have noticed is that some SIGMA rules contain IP addresses with wildcards, which means that a text search has to be performed using the "startswith" operator. 
To improve the performances we changed these values in CIDR notation allowing to use the IP address field which is optimized.

Increase the number of true positives

Unstructured fields used to be treated as equalities, while they are rarely going to be an exact match, despite being better for performances, it might be hiding true positives. 
To remediate this problem, we changed the test for it to be a "contains" (like %%) as shown in the following example:
Input

title:OracleWebLogicExploit 

[...]

logsource: 

category:webserver 

detection: 

selection: 

cs-uri-query:'*/config/keystore/*.js*' 

condition:selection 

  

Output

SELECT * FROM events WHERE LOWER(URL) LIKE '%/config/keystore/%.js%' 

If the tested field is the payload and the value is a number, the name field will be added to the tested value in the search to avoid false positives (i.e: number in date, timestamp, etc) as in the example below:
Input

title:ExchangeExploitationCVE-2021-28480 

[...]
logsource: 

category:webserver 

detection: 

selection: 

cs-uri-query|contains:'/owa/calendar/a' 

cs-method:'POST' 

filter: 

sc-status:503 

condition:selectionandnotfilter

  

Output

SELECT * FROM events WHERE LOWER(URL) LIKE '%/owa/calendar/a%' AND Method='POST' AND NOT (LOWER(UTF8(payload)) LIKE '%sc-status%503%') 

Note: This conversion will output a valid AQL query but will display a warning to highlight that the parameters have been changed.


If the field is a subpart of a linux command (i.e: a0, a1, etc), the name of the filed and value are part of the payload search as in the following example:

Input

title:RemoveImmutableFileAttribute-Auditd 

[...]
logsource: 

product:linux 

service:auditd 

detection: 

selection: 

type:'EXECVE' 

a0|contains:'chattr' 

a1|contains:'-i' 

condition:selection

Output 

SELECT * FROM events WHERE devicetype=11 AND LOWER(UTF8(payload)) LIKE '%execve%' AND LOWER(Command) LIKE '%a0%chattr%' AND LOWER(Command) LIKE '%a1%-i%' 

Note: This conversion will output a valid AQL query but will display a warning to highlight that the parameters have been changed


We came across a few cases where mapping was not possible and a payload query is the only way to convert the SIGMA rule.
For those cases, the -p argument has been added to the script execution:

• If you choose -p qradar-aql-fields, no payload search will be included, the rules will not be converted and an SigmaTransformationError exception will be displayed.
• If you choose -p qradar-aql-payload, you will get the AQL using a payload search. In case we find an impossible operation for a payload test such as a boolean, a CIDR, Null test or regex, a SigmaTransformationError exception is generated and the AQL is not generated.

The ‘pySigma-backend-QRadar-AQL’ is integrated with sigma-cli, it needs to be installed prior to the command execution (refer to the Getting Started section of the page).
The command line works as below:
sigma convert -t ibm-qradar-aql -p <qradar-aql-fields | qradar-aql-payload> <rule path> -o <output file name>

Conclusion

For full explanation, and for mapping details, all the information is in the readme file. 
If you notice that we are missing a case or if you have any suggestion, feel free to create a pull request as described in the readme file under Mapping Contribution section.

The script will continue to evolve and the new updates made to pySigma will help achieve better and more optimized mappings.
If you are curious about the updates made to the Sigma project (new rule types, sets, etc), I invite you to follow Florian Roth quarterly updates on Medium. 

If you are interested in reading more about QRadar Security Content, you can find the complete list of blog entries here.