IBM Security QRadar

Hello, we need to understand how QRadar's engines work to fulfill our customers' demands.

Why does Console apply Overflow filter (license limit) after the Processor in distributed architecture? (please check overall.png)

As we know, QRadar applies first throttle license restriction at the Collector (where it receives the events, 1st time).
Then, the processor checks the license again (2nd time). Then console checks again according to overall.png (3rd time).
However, as our license allocation is different on Processor and Console, console shouldn't check again for the processor's license limit. Yet, the IBM training shows and explains these differently (please check the attached overall.png, we had asked the same more explicitly in it.)

Related IBM Training which overall.png has been taken from: https://learn.ibm.com/mod/video/view.php?id=270766 (Dissecting a captured event course)

1-) Can you explain how license restriction works in a distributed environment?
2-) What is the difference between console EPS limit checking and Processor EPS limit checking?

Thank you,

Best regards,

All appliances that are capable of receiving events Console (All-in-One or distributed), Event Processor, Event Collector each have a copy of ecs-ec-ingress, which is the service responsible for taking events off of the wire and adding them to the event queue to be parsed. As each appliance has an allocation from the overall license, this governs the rate at which each appliance can process events and any overages go to the 5GB spillover queue. 

The license is checked at each appliance that receives events. (Note: Event Collectors do not get allocated licenses, but they inherit the license from the EP they are connected to).

For example, I have an overall license for 10,000 EPS. This 10,000 EPS is allocated across the Console and Event Processors in the deployment:

• Console 1,000 EPS allocated
• Event Processor 1 (4,500 EPS allocated)
  • Event Collect 1 (license inherited from Event Processor 1)
• Event Processor 2 (4,500 EPS allocated)
  • Event Collect 1 (license inherited from Event Processor 2)

----------------------- total license = 10,000 EPS ----------------------------

1-) Can you explain how license restriction works in a distributed environment?
In a distributed environment, any appliance with ecs-ec-ingress can listen or retrieve events. As this process lives on the Console, EC, and EP, any of these appliances can receive events, so if the license is exceeded, such as going over 4,500 EPS on an EP or 1,000 EPS on the Console, the data goes in to the spillover queue. This also allows users to potentially load balance Syslog events when all appliances can listen. The deployment receives an overall license. This license is then allocated across appliances as required and can be changed from System and License Management UI on the Admin tab. 

2-) What is the difference between console EPS limit checking and Processor EPS limit checking?
None, the Console can receive, parse, and store events just like an Event Processor. Some users with small deployments might only have an Event Collector and a Console where larger deployments might have dozens of EPs receiving, parsing, and storing events, but the Console should still get a a minimum EPS allocation to prevent things hitting the ingress spillover queue. The Console is like an EP, but runs the UI, apps, and includes the magistrate to run the offense model, and keeps the core configuration. There is no reason you cannot send events to the Console, but in a distributed environment, most admins have the EPs doing the heavy lifting so that the Console has more resources for the jobs only it can complete (UI, offenses, apps, configuration/deployment tasks). 

We've got a tech note in support about services that run on each appliance type. Anything with ecs-ec-ingress receives events and enforces the incoming event rate per your license you allocated to your EP (remember, the EC inherits its license from the EP it is connected to). The ingress service runs on multiple appliance types, including the Console even when you have a distributed deployment. 

If you have follow-up questions let me know.

Hello, we need to understand how QRadar's engines work to fulfill our customers' demands.

Why does Console apply Overflow filter (license limit) after the Processor in distributed architecture? (please check overall.png)

As we know, QRadar applies first throttle license restriction at the Collector (where it receives the events, 1st time).
Then, the processor checks the license again (2nd time). Then console checks again according to overall.png (3rd time).
However, as our license allocation is different on Processor and Console, console shouldn't check again for the processor's license limit. Yet, the IBM training shows and explains these differently (please check the attached overall.png, we had asked the same more explicitly in it.)

Related IBM Training which overall.png has been taken from: https://learn.ibm.com/mod/video/view.php?id=270766 (Dissecting a captured event course)

1-) Can you explain how license restriction works in a distributed environment?
2-) What is the difference between console EPS limit checking and Processor EPS limit checking?

Thank you,

Best regards,

All appliances that are capable of receiving events Console (All-in-One or distributed), Event Processor, Event Collector each have a copy of ecs-ec-ingress, which is the service responsible for taking events off of the wire and adding them to the event queue to be parsed. As each appliance has an allocation from the overall license, this governs the rate at which each appliance can process events and any overages go to the 5GB spillover queue. 

The license is checked at each appliance that receives events. (Note: Event Collectors do not get allocated licenses, but they inherit the license from the EP they are connected to).

For example, I have an overall license for 10,000 EPS. This 10,000 EPS is allocated across the Console and Event Processors in the deployment:

• Console 1,000 EPS allocated
• Event Processor 1 (4,500 EPS allocated)
  • Event Collect 1 (license inherited from Event Processor 1)
• Event Processor 2 (4,500 EPS allocated)
  • Event Collect 1 (license inherited from Event Processor 2)

----------------------- total license = 10,000 EPS ----------------------------

1-) Can you explain how license restriction works in a distributed environment?
In a distributed environment, any appliance with ecs-ec-ingress can listen or retrieve events. As this process lives on the Console, EC, and EP, any of these appliances can receive events, so if the license is exceeded, such as going over 4,500 EPS on an EP or 1,000 EPS on the Console, the data goes in to the spillover queue. This also allows users to potentially load balance Syslog events when all appliances can listen. The deployment receives an overall license. This license is then allocated across appliances as required and can be changed from System and License Management UI on the Admin tab. 

2-) What is the difference between console EPS limit checking and Processor EPS limit checking?
None, the Console can receive, parse, and store events just like an Event Processor. Some users with small deployments might only have an Event Collector and a Console where larger deployments might have dozens of EPs receiving, parsing, and storing events, but the Console should still get a a minimum EPS allocation to prevent things hitting the ingress spillover queue. The Console is like an EP, but runs the UI, apps, and includes the magistrate to run the offense model, and keeps the core configuration. There is no reason you cannot send events to the Console, but in a distributed environment, most admins have the EPs doing the heavy lifting so that the Console has more resources for the jobs only it can complete (UI, offenses, apps, configuration/deployment tasks). 

We've got a tech note in support about services that run on each appliance type. Anything with ecs-ec-ingress receives events and enforces the incoming event rate per your license you allocated to your EP (remember, the EC inherits its license from the EP it is connected to). The ingress service runs on multiple appliance types, including the Console even when you have a distributed deployment. 

If you have follow-up questions let me know.

Hello, we need to understand how QRadar's engines work to fulfill our customers' demands.

Why does Console apply Overflow filter (license limit) after the Processor in distributed architecture? (please check overall.png)

As we know, QRadar applies first throttle license restriction at the Collector (where it receives the events, 1st time).
Then, the processor checks the license again (2nd time). Then console checks again according to overall.png (3rd time).
However, as our license allocation is different on Processor and Console, console shouldn't check again for the processor's license limit. Yet, the IBM training shows and explains these differently (please check the attached overall.png, we had asked the same more explicitly in it.)

Related IBM Training which overall.png has been taken from: https://learn.ibm.com/mod/video/view.php?id=270766 (Dissecting a captured event course)

1-) Can you explain how license restriction works in a distributed environment?
2-) What is the difference between console EPS limit checking and Processor EPS limit checking?

Thank you,

Best regards,

Hello Jonathan,

Have you had a chance to check our below questions?

Thanks,

Best regards,

Onur Tufan

All appliances that are capable of receiving events Console (All-in-One or distributed), Event Processor, Event Collector each have a copy of ecs-ec-ingress, which is the service responsible for taking events off of the wire and adding them to the event queue to be parsed. As each appliance has an allocation from the overall license, this governs the rate at which each appliance can process events and any overages go to the 5GB spillover queue. 

The license is checked at each appliance that receives events. (Note: Event Collectors do not get allocated licenses, but they inherit the license from the EP they are connected to).

For example, I have an overall license for 10,000 EPS. This 10,000 EPS is allocated across the Console and Event Processors in the deployment:

• Console 1,000 EPS allocated
• Event Processor 1 (4,500 EPS allocated)
  • Event Collect 1 (license inherited from Event Processor 1)
• Event Processor 2 (4,500 EPS allocated)
  • Event Collect 1 (license inherited from Event Processor 2)

----------------------- total license = 10,000 EPS ----------------------------

1-) Can you explain how license restriction works in a distributed environment?
In a distributed environment, any appliance with ecs-ec-ingress can listen or retrieve events. As this process lives on the Console, EC, and EP, any of these appliances can receive events, so if the license is exceeded, such as going over 4,500 EPS on an EP or 1,000 EPS on the Console, the data goes in to the spillover queue. This also allows users to potentially load balance Syslog events when all appliances can listen. The deployment receives an overall license. This license is then allocated across appliances as required and can be changed from System and License Management UI on the Admin tab. 

2-) What is the difference between console EPS limit checking and Processor EPS limit checking?
None, the Console can receive, parse, and store events just like an Event Processor. Some users with small deployments might only have an Event Collector and a Console where larger deployments might have dozens of EPs receiving, parsing, and storing events, but the Console should still get a a minimum EPS allocation to prevent things hitting the ingress spillover queue. The Console is like an EP, but runs the UI, apps, and includes the magistrate to run the offense model, and keeps the core configuration. There is no reason you cannot send events to the Console, but in a distributed environment, most admins have the EPs doing the heavy lifting so that the Console has more resources for the jobs only it can complete (UI, offenses, apps, configuration/deployment tasks). 

We've got a tech note in support about services that run on each appliance type. Anything with ecs-ec-ingress receives events and enforces the incoming event rate per your license you allocated to your EP (remember, the EC inherits its license from the EP it is connected to). The ingress service runs on multiple appliance types, including the Console even when you have a distributed deployment. 

If you have follow-up questions let me know.

Hello, we need to understand how QRadar's engines work to fulfill our customers' demands.

Why does Console apply Overflow filter (license limit) after the Processor in distributed architecture? (please check overall.png)

As we know, QRadar applies first throttle license restriction at the Collector (where it receives the events, 1st time).
Then, the processor checks the license again (2nd time). Then console checks again according to overall.png (3rd time).
However, as our license allocation is different on Processor and Console, console shouldn't check again for the processor's license limit. Yet, the IBM training shows and explains these differently (please check the attached overall.png, we had asked the same more explicitly in it.)

Related IBM Training which overall.png has been taken from: https://learn.ibm.com/mod/video/view.php?id=270766 (Dissecting a captured event course)

1-) Can you explain how license restriction works in a distributed environment?
2-) What is the difference between console EPS limit checking and Processor EPS limit checking?

Thank you,

Best regards,

Hey Onur,

Sorry for the delay, I was heads down on a bunch of work. The license check for events occurs where they arrive based on the license allocation assigned to that host. Ecs-ec-ingress is the primary gatekeeper as to how many events the appliance can take in, this has two checks:

1. Incoming EPS to determine how fast events are coming in against the license as anything over needs to go to the spillover queue (bursting events)
2. A monitoring check to see if you are under license is so we know how to empty the spillover queue (events that need to be processed) and how many.

As each appliance can receive events, for example, you might send Windows events to your Event Collector, then send Firewall events directly to your Event Processor, and send custom events from an App to your Console. Each of these appliances have an ecs-ec-ingress service, so you could send events to each appliance, but they only apply this 2-part license check on events they directly receive. If an Event Collector receives Windows events, then hands them off to the ecs-ec service to be parsed, we don't check the license again as the events are already in the pipeline. The image you showed is at best confusing or misleading. As of the 3 arrows you marked in your image, if the Windows events are received by the Event Collector, that is the appliance where ecs-ec-ingress is acting as the gatekeeping to do the license checks. This might help explain things: 


After the events are in the pipeline, there are buffers that handle moving events to other services between appliances, these are buffers than license checks. For example if an ecs-ec service is down or an ecs-ep service is busy or full, the events need somewhere to go and the what is represented as a "Overflow filter (enforced license limit)" should just be thought of as a buffer between services and not an X EPS check. For example, if someone restarts a service, those events are already past the license check, but we need a buffer between services so that data is not lost, if someone were to restart ecs-ec or when services get backed up when the system is really busy. The diagram is not accurate in my opinion as far as it is more confusing. If an event enters the pipeline at the top of the stack (the Event Collector) it is handed off between each service, but we aren't checking against the license again on the Event Processor as we want to empty that queue as fast as possible to get the event collected, parsed, go through the rules engine (CRE), and stored to disk. 

Hope this helps. If you have follow-up questions, let me know. 

Hello Jonathan,

Have you had a chance to check our below questions?

Thanks,

Best regards,

Onur Tufan

All appliances that are capable of receiving events Console (All-in-One or distributed), Event Processor, Event Collector each have a copy of ecs-ec-ingress, which is the service responsible for taking events off of the wire and adding them to the event queue to be parsed. As each appliance has an allocation from the overall license, this governs the rate at which each appliance can process events and any overages go to the 5GB spillover queue. 

The license is checked at each appliance that receives events. (Note: Event Collectors do not get allocated licenses, but they inherit the license from the EP they are connected to).

For example, I have an overall license for 10,000 EPS. This 10,000 EPS is allocated across the Console and Event Processors in the deployment:

• Console 1,000 EPS allocated
• Event Processor 1 (4,500 EPS allocated)
  • Event Collect 1 (license inherited from Event Processor 1)
• Event Processor 2 (4,500 EPS allocated)
  • Event Collect 1 (license inherited from Event Processor 2)

----------------------- total license = 10,000 EPS ----------------------------

1-) Can you explain how license restriction works in a distributed environment?
In a distributed environment, any appliance with ecs-ec-ingress can listen or retrieve events. As this process lives on the Console, EC, and EP, any of these appliances can receive events, so if the license is exceeded, such as going over 4,500 EPS on an EP or 1,000 EPS on the Console, the data goes in to the spillover queue. This also allows users to potentially load balance Syslog events when all appliances can listen. The deployment receives an overall license. This license is then allocated across appliances as required and can be changed from System and License Management UI on the Admin tab. 

2-) What is the difference between console EPS limit checking and Processor EPS limit checking?
None, the Console can receive, parse, and store events just like an Event Processor. Some users with small deployments might only have an Event Collector and a Console where larger deployments might have dozens of EPs receiving, parsing, and storing events, but the Console should still get a a minimum EPS allocation to prevent things hitting the ingress spillover queue. The Console is like an EP, but runs the UI, apps, and includes the magistrate to run the offense model, and keeps the core configuration. There is no reason you cannot send events to the Console, but in a distributed environment, most admins have the EPs doing the heavy lifting so that the Console has more resources for the jobs only it can complete (UI, offenses, apps, configuration/deployment tasks). 

We've got a tech note in support about services that run on each appliance type. Anything with ecs-ec-ingress receives events and enforces the incoming event rate per your license you allocated to your EP (remember, the EC inherits its license from the EP it is connected to). The ingress service runs on multiple appliance types, including the Console even when you have a distributed deployment. 

If you have follow-up questions let me know.

Hello, we need to understand how QRadar's engines work to fulfill our customers' demands.

Why does Console apply Overflow filter (license limit) after the Processor in distributed architecture? (please check overall.png)

As we know, QRadar applies first throttle license restriction at the Collector (where it receives the events, 1st time).
Then, the processor checks the license again (2nd time). Then console checks again according to overall.png (3rd time).
However, as our license allocation is different on Processor and Console, console shouldn't check again for the processor's license limit. Yet, the IBM training shows and explains these differently (please check the attached overall.png, we had asked the same more explicitly in it.)

Related IBM Training which overall.png has been taken from: https://learn.ibm.com/mod/video/view.php?id=270766 (Dissecting a captured event course)

1-) Can you explain how license restriction works in a distributed environment?
2-) What is the difference between console EPS limit checking and Processor EPS limit checking?

Thank you,

Best regards,