IBM Security QRadar

Hi Community,

in Qradar there is the concept of stateful and stateless tests.
Stateful tests look for a count of events over a period of time.

A) "over a period of time" refers to the time when Qradar receives the event, right? Is that a way to set this test to watch log source time? This is related to the fact that a rule is triggered when, for example, there is no connection between QRadar and Wincollect and Events are buffered. After communication has been established, all buffered Events are sent to Qradar and those rules get triggered because Qradar observes many events at the same time.

B) Is there a limit for including rules in other rules (not talking about Building blocks)? For example,
Rule A gets triggered.
Rule B gets triggered if Rule was triggered in the last x minutes + another condition ( it works)
Rule C should be triggered if Rule B was triggered in the last x minutes + another condition ( it doesn't work)

Thank you

Regards,
Bruno

------------------------------
Bruno Oliveira
------------------------------

've found this

https://www.ibm.com/mysupport/s/question/0D50z00006PEFGTCA5/can-i-use-log-source-time-instead-of-start-time-in-rules?language=de

------------------------------
Bruno Oliveira
------------------------------

Original Message:
Sent: Tue June 09, 2020 04:14 AM
From: Bruno Oliveira
Subject: Stateful Tests

Hi Community,

in Qradar there is the concept of stateful and stateless tests.
Stateful tests look for a count of events over a period of time.

A) "over a period of time" refers to the time when Qradar receives the event, right? Is that a way to set this test to watch log source time? This is related to the fact that a rule is triggered when, for example, there is no connection between QRadar and Wincollect and Events are buffered. After communication has been established, all buffered Events are sent to Qradar and those rules get triggered because Qradar observes many events at the same time.

B) Is there a limit for including rules in other rules (not talking about Building blocks)? For example,
Rule A gets triggered.
Rule B gets triggered if Rule was triggered in the last x minutes + another condition ( it works)
Rule C should be triggered if Rule B was triggered in the last x minutes + another condition ( it doesn't work)

Thank you

Regards,
Bruno

------------------------------
Bruno Oliveira
------------------------------

Hi Bruno,

For A), yes, the real-time correlation engine acts on events as they are received. We can't use log source time for this because there's no guarantee Log Source Time will be accurate for all events. If system's clocks are off or if they reside in different time zones and their timestamp doesn't specify a timezone, then the Log Source Time can effectively jump forward or backward in time. Therefore for every event we received with a log source time that didn't match the current actual time, we'd need to issue searches back in time to find all events that occurred around the same time (according to log source time). If we received an event whose log source time was in the future, we'd need to hold it in memory until "real time" caught up, or somehow set a flag to tell the correlation engine to search back in time when we catch up. This sort of stuff coudl severely impact the perfromance fo the real-time pipeline, which is why we don't do it. But this is why we have historical correlation. I'm going to copy/paste in my answer from a previous similar question here in the community:

"If a real-time event stream is not an option, the solution to address use cases like this where you are using stateful tests (thing X happens after thing Y in Z amount of time) is to use our Historical Correlation feature. This is available in the Actions menu of Log Activity and Network Activity for use in Events or Flows. Essentially you define a saved search which targets the events of interest which you are receiving in batch (so your Sophos Central events in this case) and link it to a set of rules (or all rules if you wish, but in this case you probably want to pick just the relevant stateful ones that aren't functioning properly) and then using your historical correlation profile you can replay those events through the select rules as if they were received in realtime. If you select "Device Time" in the "Correlate Events By" dropdown, this means the historical correlation processor will examine the Log Source Time property of all events returned by the saved search and pass the events through the rules in the order they actually occurred, with appropriate delays to simulate the time at which they actually happened, relative to one another. Note that rules used in historical correlation will only generate offenses, they can't take other actions/responses.

You can choose to schedule your profile to run regularly so you don't have to kick it off manually periodically.
You may also want to create a Routing Rule with action "Bypass Correlation" that matches your Sophos Central events such that you don't get false positives rule hits on those events from the realtime correlation engine - this ensures that they only get processed by the historical correlation processor."

For B), as far as I know there is no limit for nesting rules within each other.

Cheers
Colin


------------------------------
COLIN HAY
IBM Security
------------------------------

Original Message:
Sent: Tue June 09, 2020 04:14 AM
From: Bruno Oliveira
Subject: Stateful Tests

Hi Community,

in Qradar there is the concept of stateful and stateless tests.
Stateful tests look for a count of events over a period of time.

A) "over a period of time" refers to the time when Qradar receives the event, right? Is that a way to set this test to watch log source time? This is related to the fact that a rule is triggered when, for example, there is no connection between QRadar and Wincollect and Events are buffered. After communication has been established, all buffered Events are sent to Qradar and those rules get triggered because Qradar observes many events at the same time.

B) Is there a limit for including rules in other rules (not talking about Building blocks)? For example,
Rule A gets triggered.
Rule B gets triggered if Rule was triggered in the last x minutes + another condition ( it works)
Rule C should be triggered if Rule B was triggered in the last x minutes + another condition ( it doesn't work)

Thank you

Regards,
Bruno

------------------------------
Bruno Oliveira
------------------------------

Thank you Colin!

It helped a lot.

Regards,
Bruno

------------------------------
Bruno Oliveira
------------------------------

Original Message:
Sent: Wed June 10, 2020 03:20 PM
From: COLIN HAY
Subject: Stateful Tests

Hi Bruno,

For A), yes, the real-time correlation engine acts on events as they are received. We can't use log source time for this because there's no guarantee Log Source Time will be accurate for all events. If system's clocks are off or if they reside in different time zones and their timestamp doesn't specify a timezone, then the Log Source Time can effectively jump forward or backward in time. Therefore for every event we received with a log source time that didn't match the current actual time, we'd need to issue searches back in time to find all events that occurred around the same time (according to log source time). If we received an event whose log source time was in the future, we'd need to hold it in memory until "real time" caught up, or somehow set a flag to tell the correlation engine to search back in time when we catch up. This sort of stuff coudl severely impact the perfromance fo the real-time pipeline, which is why we don't do it. But this is why we have historical correlation. I'm going to copy/paste in my answer from a previous similar question here in the community:

"If a real-time event stream is not an option, the solution to address use cases like this where you are using stateful tests (thing X happens after thing Y in Z amount of time) is to use our Historical Correlation feature. This is available in the Actions menu of Log Activity and Network Activity for use in Events or Flows. Essentially you define a saved search which targets the events of interest which you are receiving in batch (so your Sophos Central events in this case) and link it to a set of rules (or all rules if you wish, but in this case you probably want to pick just the relevant stateful ones that aren't functioning properly) and then using your historical correlation profile you can replay those events through the select rules as if they were received in realtime. If you select "Device Time" in the "Correlate Events By" dropdown, this means the historical correlation processor will examine the Log Source Time property of all events returned by the saved search and pass the events through the rules in the order they actually occurred, with appropriate delays to simulate the time at which they actually happened, relative to one another. Note that rules used in historical correlation will only generate offenses, they can't take other actions/responses.

You can choose to schedule your profile to run regularly so you don't have to kick it off manually periodically.
You may also want to create a Routing Rule with action "Bypass Correlation" that matches your Sophos Central events such that you don't get false positives rule hits on those events from the realtime correlation engine - this ensures that they only get processed by the historical correlation processor."

For B), as far as I know there is no limit for nesting rules within each other.

Cheers
Colin


------------------------------
COLIN HAY
IBM Security

Original Message:
Sent: Tue June 09, 2020 04:14 AM
From: Bruno Oliveira
Subject: Stateful Tests

Hi Community,

in Qradar there is the concept of stateful and stateless tests.
Stateful tests look for a count of events over a period of time.

A) "over a period of time" refers to the time when Qradar receives the event, right? Is that a way to set this test to watch log source time? This is related to the fact that a rule is triggered when, for example, there is no connection between QRadar and Wincollect and Events are buffered. After communication has been established, all buffered Events are sent to Qradar and those rules get triggered because Qradar observes many events at the same time.

B) Is there a limit for including rules in other rules (not talking about Building blocks)? For example,
Rule A gets triggered.
Rule B gets triggered if Rule was triggered in the last x minutes + another condition ( it works)
Rule C should be triggered if Rule B was triggered in the last x minutes + another condition ( it doesn't work)

Thank you

Regards,
Bruno

------------------------------
Bruno Oliveira
------------------------------

Hi Colin,

just to make it clear. 

CRE doesnt user any of the QRadar times such as StartTime, StorageTime and LogSourceTime for stateful tests. But the time in which the CRE sees the events. For Example.

We have an AIO connected to one EC. 

Events from Log Source x come every second (1 EPS) to QRadar through the EC.

There is a rule like : If Events from Log Source X are seen from 4 pm to 4:10 pm, an offense is generated.

Suppose that at 3:59pm there is a network problem between the AiO and the EC. Events are buffered on the EC. At 4:11 pm the network problem is solved and events are sent to the AiO. Start and LogSource times are like 3:59 to 4:11, but storage time for all those events are like after 4:11 pm, because they are store on the AiO.

There is no offense generated because the CRE uses its time and didn't see any events between 4:00 pm and 4:10, even though start times are between 4:00 and 4:10pm

Is that right?

------------------------------
Bruno Oliveira
------------------------------

Original Message:
Sent: Wed June 10, 2020 03:20 PM
From: COLIN HAY
Subject: Stateful Tests

Hi Bruno,

For A), yes, the real-time correlation engine acts on events as they are received. We can't use log source time for this because there's no guarantee Log Source Time will be accurate for all events. If system's clocks are off or if they reside in different time zones and their timestamp doesn't specify a timezone, then the Log Source Time can effectively jump forward or backward in time. Therefore for every event we received with a log source time that didn't match the current actual time, we'd need to issue searches back in time to find all events that occurred around the same time (according to log source time). If we received an event whose log source time was in the future, we'd need to hold it in memory until "real time" caught up, or somehow set a flag to tell the correlation engine to search back in time when we catch up. This sort of stuff coudl severely impact the perfromance fo the real-time pipeline, which is why we don't do it. But this is why we have historical correlation. I'm going to copy/paste in my answer from a previous similar question here in the community:

"If a real-time event stream is not an option, the solution to address use cases like this where you are using stateful tests (thing X happens after thing Y in Z amount of time) is to use our Historical Correlation feature. This is available in the Actions menu of Log Activity and Network Activity for use in Events or Flows. Essentially you define a saved search which targets the events of interest which you are receiving in batch (so your Sophos Central events in this case) and link it to a set of rules (or all rules if you wish, but in this case you probably want to pick just the relevant stateful ones that aren't functioning properly) and then using your historical correlation profile you can replay those events through the select rules as if they were received in realtime. If you select "Device Time" in the "Correlate Events By" dropdown, this means the historical correlation processor will examine the Log Source Time property of all events returned by the saved search and pass the events through the rules in the order they actually occurred, with appropriate delays to simulate the time at which they actually happened, relative to one another. Note that rules used in historical correlation will only generate offenses, they can't take other actions/responses.

You can choose to schedule your profile to run regularly so you don't have to kick it off manually periodically.
You may also want to create a Routing Rule with action "Bypass Correlation" that matches your Sophos Central events such that you don't get false positives rule hits on those events from the realtime correlation engine - this ensures that they only get processed by the historical correlation processor."

For B), as far as I know there is no limit for nesting rules within each other.

Cheers
Colin


------------------------------
COLIN HAY
IBM Security

Original Message:
Sent: Tue June 09, 2020 04:14 AM
From: Bruno Oliveira
Subject: Stateful Tests

Hi Community,

in Qradar there is the concept of stateful and stateless tests.
Stateful tests look for a count of events over a period of time.

A) "over a period of time" refers to the time when Qradar receives the event, right? Is that a way to set this test to watch log source time? This is related to the fact that a rule is triggered when, for example, there is no connection between QRadar and Wincollect and Events are buffered. After communication has been established, all buffered Events are sent to Qradar and those rules get triggered because Qradar observes many events at the same time.

B) Is there a limit for including rules in other rules (not talking about Building blocks)? For example,
Rule A gets triggered.
Rule B gets triggered if Rule was triggered in the last x minutes + another condition ( it works)
Rule C should be triggered if Rule B was triggered in the last x minutes + another condition ( it doesn't work)

Thank you

Regards,
Bruno

------------------------------
Bruno Oliveira
------------------------------

That's correct. The real-time CRE works in real time, i.e. when it is given the event. So if it gets an event at 4:11, it doesn't matter when it actually occurred (*unless* you're doing a test specifically involving the Log Source Time property) - it will consider the event as having occurred at 4:11, and thus will not match a test which checks if it happened between 4:00 and 4:10. But the same rule used in historical correlation should trigger

------------------------------
COLIN HAY
IBM Security
------------------------------

Original Message:
Sent: Thu June 18, 2020 03:29 AM
From: Bruno Oliveira
Subject: Stateful Tests

Hi Colin,

just to make it clear. 

CRE doesnt user any of the QRadar times such as StartTime, StorageTime and LogSourceTime for stateful tests. But the time in which the CRE sees the events. For Example.

We have an AIO connected to one EC. 

Events from Log Source x come every second (1 EPS) to QRadar through the EC.

There is a rule like : If Events from Log Source X are seen from 4 pm to 4:10 pm, an offense is generated.

Suppose that at 3:59pm there is a network problem between the AiO and the EC. Events are buffered on the EC. At 4:11 pm the network problem is solved and events are sent to the AiO. Start and LogSource times are like 3:59 to 4:11, but storage time for all those events are like after 4:11 pm, because they are store on the AiO.

There is no offense generated because the CRE uses its time and didn't see any events between 4:00 pm and 4:10, even though start times are between 4:00 and 4:10pm

Is that right?

------------------------------
Bruno Oliveira

Original Message:
Sent: Wed June 10, 2020 03:20 PM
From: COLIN HAY
Subject: Stateful Tests

Hi Bruno,

For A), yes, the real-time correlation engine acts on events as they are received. We can't use log source time for this because there's no guarantee Log Source Time will be accurate for all events. If system's clocks are off or if they reside in different time zones and their timestamp doesn't specify a timezone, then the Log Source Time can effectively jump forward or backward in time. Therefore for every event we received with a log source time that didn't match the current actual time, we'd need to issue searches back in time to find all events that occurred around the same time (according to log source time). If we received an event whose log source time was in the future, we'd need to hold it in memory until "real time" caught up, or somehow set a flag to tell the correlation engine to search back in time when we catch up. This sort of stuff coudl severely impact the perfromance fo the real-time pipeline, which is why we don't do it. But this is why we have historical correlation. I'm going to copy/paste in my answer from a previous similar question here in the community:

"If a real-time event stream is not an option, the solution to address use cases like this where you are using stateful tests (thing X happens after thing Y in Z amount of time) is to use our Historical Correlation feature. This is available in the Actions menu of Log Activity and Network Activity for use in Events or Flows. Essentially you define a saved search which targets the events of interest which you are receiving in batch (so your Sophos Central events in this case) and link it to a set of rules (or all rules if you wish, but in this case you probably want to pick just the relevant stateful ones that aren't functioning properly) and then using your historical correlation profile you can replay those events through the select rules as if they were received in realtime. If you select "Device Time" in the "Correlate Events By" dropdown, this means the historical correlation processor will examine the Log Source Time property of all events returned by the saved search and pass the events through the rules in the order they actually occurred, with appropriate delays to simulate the time at which they actually happened, relative to one another. Note that rules used in historical correlation will only generate offenses, they can't take other actions/responses.

You can choose to schedule your profile to run regularly so you don't have to kick it off manually periodically.
You may also want to create a Routing Rule with action "Bypass Correlation" that matches your Sophos Central events such that you don't get false positives rule hits on those events from the realtime correlation engine - this ensures that they only get processed by the historical correlation processor."

For B), as far as I know there is no limit for nesting rules within each other.

Cheers
Colin


------------------------------
COLIN HAY
IBM Security

Original Message:
Sent: Tue June 09, 2020 04:14 AM
From: Bruno Oliveira
Subject: Stateful Tests

Hi Community,

in Qradar there is the concept of stateful and stateless tests.
Stateful tests look for a count of events over a period of time.

A) "over a period of time" refers to the time when Qradar receives the event, right? Is that a way to set this test to watch log source time? This is related to the fact that a rule is triggered when, for example, there is no connection between QRadar and Wincollect and Events are buffered. After communication has been established, all buffered Events are sent to Qradar and those rules get triggered because Qradar observes many events at the same time.

B) Is there a limit for including rules in other rules (not talking about Building blocks)? For example,
Rule A gets triggered.
Rule B gets triggered if Rule was triggered in the last x minutes + another condition ( it works)
Rule C should be triggered if Rule B was triggered in the last x minutes + another condition ( it doesn't work)

Thank you

Regards,
Bruno

------------------------------
Bruno Oliveira
------------------------------