Hello Experts,

I am facing a problem measuring the EPS, I am receiving normal number of events that don't exceed the license as per the dashboard and AQL query"select logsourcename(logsourceid) as LogSource, SUM(eventcount)/300 as EPS from events group by logsourceid order by EPS desc last 5 MINUTES" but still having notifications of dropping events, note that I am not receiving notifications for expensive rules or custom properties which means that processing runs smoothly, kindly advice regarding how this could happen and if possible to measure received events in the event collection(eco-ec-ingress) service.

Regards,
Ahmed El Sayed

------------------------------
Ahmed Elsayed
------------------------------

Ahmed,
your query does not result in exact EPS values but in a rough mean value over 5 min. Pls change it to something like "select logsourcename(logsourceid) as LogSource, SUM(eventcount) as EPC from events group by logsourceid order by EPC desc last 15 MINUTES" and divide the overall event count by 900 manually and compare it to your EPS license. Spillover effects and license giveback is calculated on a per minute basis and may effect the results of your query negatively depending if you already eat up your license by 90% or just 50%. This may make a big difference! If you see events dropped messages your system is probably right. Use standard system monitoring query and adjust time value to 15min instead of 6 hours default. Root cause maybe a license issue but there maybe other side effects as those you described already, e.g. too many rule engine events.
Regards
Karl

------------------------------
[Karl] [Jaeger] [Business Partner]
[QRadar Specialist]
[pro4bizz]
[Karlsruhe] [Germany]
[4972190981722]
------------------------------

Hello Karl,

Kindly note that I've ran the query for different time intervals sometimes for 24 hours and divided the result by the corresponding number of seconds of that interval, and noted that EPS is not exceeding the license but still having dropped events, also there is no notifications for either expensive rules or expensive properties, so I was thinking that the resources of the appliance is consumed so that causing the events to be dropped from the pipeline, is it possible to have such behavior.

Thanks,
Regards.


------------------------------
Ahmed Elsayed
------------------------------

Original Message:
Sent: Mon January 25, 2021 12:21 PM
From: Karl Jaeger
Subject: EPS

Ahmed,
your query does not result in exact EPS values but in a rough mean value over 5 min. Pls change it to something like "select logsourcename(logsourceid) as LogSource, SUM(eventcount) as EPC from events group by logsourceid order by EPC desc last 15 MINUTES" and divide the overall event count by 900 manually and compare it to your EPS license. Spillover effects and license giveback is calculated on a per minute basis and may effect the results of your query negatively depending if you already eat up your license by 90% or just 50%. This may make a big difference! If you see events dropped messages your system is probably right. Use standard system monitoring query and adjust time value to 15min instead of 6 hours default. Root cause maybe a license issue but there maybe other side effects as those you described already, e.g. too many rule engine events.
Regards
Karl

------------------------------
[Karl] [Jaeger] [Business Partner]
[QRadar Specialist]
[pro4bizz]
[Karlsruhe] [Germany]
[4972190981722]
------------------------------

Ahmed,
without supplying any more details this is just a guessing game. Pls reread my comments above. You are going circles. As outlined before QRadar does not calculate licenses on a 24h basis . EPS License used is calculated, based on EPS average per 1 minute interval. If you got peaks one minute you are using license giveback the next one to process those events stored in spillover queue. If events are still flowing 2nd minute QRadar is pushing back event processing to minute three if there is a drop in the event stream. If not events are still pushed to cache. Of course this is limited cause at some point events get dropped.

Are you seeing events dropped from the pipeline or license exceeded? There are 100 different error scenarios. Pls provide event counts for each minute and error message you see to enable community to help you. Use time line graph like the one above and walk to the peak using mouseover data.
Regards
Karl

------------------------------
[Karl] [Jaeger] [Business Partner]
[QRadar Specialist]
[pro4bizz]
[Karlsruhe] [Germany]
[4972190981722]
------------------------------

Original Message:
Sent: Tue January 26, 2021 03:26 AM
From: Ahmed Elsayed
Subject: EPS

Hello Karl,

Kindly note that I've ran the query for different time intervals sometimes for 24 hours and divided the result by the corresponding number of seconds of that interval, and noted that EPS is not exceeding the license but still having dropped events, also there is no notifications for either expensive rules or expensive properties, so I was thinking that the resources of the appliance is consumed so that causing the events to be dropped from the pipeline, is it possible to have such behavior.

Thanks,
Regards.


------------------------------
Ahmed Elsayed
------------------------------


Original Message:
Sent: Mon January 25, 2021 12:21 PM
From: Karl Jaeger
Subject: EPS

Ahmed,
your query does not result in exact EPS values but in a rough mean value over 5 min. Pls change it to something like "select logsourcename(logsourceid) as LogSource, SUM(eventcount) as EPC from events group by logsourceid order by EPC desc last 15 MINUTES" and divide the overall event count by 900 manually and compare it to your EPS license. Spillover effects and license giveback is calculated on a per minute basis and may effect the results of your query negatively depending if you already eat up your license by 90% or just 50%. This may make a big difference! If you see events dropped messages your system is probably right. Use standard system monitoring query and adjust time value to 15min instead of 6 hours default. Root cause maybe a license issue but there maybe other side effects as those you described already, e.g. too many rule engine events.
Regards
Karl

------------------------------
[Karl] [Jaeger] [Business Partner]
[QRadar Specialist]
[pro4bizz]
[Karlsruhe] [Germany]
[4972190981722]

Try to use Pulse App https://exchange.xforce.ibmcloud.com/hub/extension/9df9eb09dbbad7bd42d738cc9748b5db
One of the dashboards shows EPS rate in real-time
You need just add that dashboard in app settings.

------------------------------
Aigerim Kozybayeva
------------------------------

Original Message:
Sent: Tue January 26, 2021 03:26 AM
From: Ahmed Elsayed
Subject: EPS

Hello Karl,

Kindly note that I've ran the query for different time intervals sometimes for 24 hours and divided the result by the corresponding number of seconds of that interval, and noted that EPS is not exceeding the license but still having dropped events, also there is no notifications for either expensive rules or expensive properties, so I was thinking that the resources of the appliance is consumed so that causing the events to be dropped from the pipeline, is it possible to have such behavior.

Thanks,
Regards.


------------------------------
Ahmed Elsayed
------------------------------


Original Message:
Sent: Mon January 25, 2021 12:21 PM
From: Karl Jaeger
Subject: EPS

Ahmed,
your query does not result in exact EPS values but in a rough mean value over 5 min. Pls change it to something like "select logsourcename(logsourceid) as LogSource, SUM(eventcount) as EPC from events group by logsourceid order by EPC desc last 15 MINUTES" and divide the overall event count by 900 manually and compare it to your EPS license. Spillover effects and license giveback is calculated on a per minute basis and may effect the results of your query negatively depending if you already eat up your license by 90% or just 50%. This may make a big difference! If you see events dropped messages your system is probably right. Use standard system monitoring query and adjust time value to 15min instead of 6 hours default. Root cause maybe a license issue but there maybe other side effects as those you described already, e.g. too many rule engine events.
Regards
Karl

------------------------------
[Karl] [Jaeger] [Business Partner]
[QRadar Specialist]
[pro4bizz]
[Karlsruhe] [Germany]
[4972190981722]