Hi guys,

We want to drop some events at the EC level before coming to the EP. Is there a method for this, has anyone tried it before?

Thanks in advance

Hi Adem,

It is possible to drop events at the EC level (before the EP) using routing rules in "online" mode.

Although this satisfies your need to drop at the EC, I notice you specified "without routing rules" in your topic; I'm not sure why you want to avoid routing rules, but if for whatever reason you cannot use them, your only other option would be to prevent the events from being ingested at all. This can be handled in multiple ways, depending on the mechanism used to collect the data. Our WinCollect agent can have filters configured such that it only sends certain events. Other syslog sources may have similar filtering capabilities, it depends entirely on what is transmitting the syslog events. Many of our protocols that "pull" event data via various REST APIs, JDBC queries, and so may have filtering capabilities at the query level, but again it depends on the specific integration. Likewise the protocols that use a pub/sub paradigm may have filtering capabilities, or useful data vs unuseful data can be separated at the topic/subscription/eventhub/etc level such that only useful event data can be retrieved.

In short, online routing rules running in the EC are probably your best option, if you are willing to use them. If not, your goal may be doable, but it depends on the details.

Cheers

Colin

Hi guys,

We want to drop some events at the EC level before coming to the EP. Is there a method for this, has anyone tried it before?

Thanks in advance

Hi Colin,

Firstly, thank you for your feedback and detailed answer. The reason we want to avoid using routing roule here is that we think it will overload the system resources on the console. What is your opinion about this?

Hi Adem,

It is possible to drop events at the EC level (before the EP) using routing rules in "online" mode.

Although this satisfies your need to drop at the EC, I notice you specified "without routing rules" in your topic; I'm not sure why you want to avoid routing rules, but if for whatever reason you cannot use them, your only other option would be to prevent the events from being ingested at all. This can be handled in multiple ways, depending on the mechanism used to collect the data. Our WinCollect agent can have filters configured such that it only sends certain events. Other syslog sources may have similar filtering capabilities, it depends entirely on what is transmitting the syslog events. Many of our protocols that "pull" event data via various REST APIs, JDBC queries, and so may have filtering capabilities at the query level, but again it depends on the specific integration. Likewise the protocols that use a pub/sub paradigm may have filtering capabilities, or useful data vs unuseful data can be separated at the topic/subscription/eventhub/etc level such that only useful event data can be retrieved.

In short, online routing rules running in the EC are probably your best option, if you are willing to use them. If not, your goal may be doable, but it depends on the details.

Cheers

Colin

Hi guys,

We want to drop some events at the EC level before coming to the EP. Is there a method for this, has anyone tried it before?

Thanks in advance

Hi Adem,

Sorry for the delayed response. Generally speaking, routing rules are not considered to be very expensive computationally, especially for the Drop action (the Forward action is more expensive). But the relative cost of a routing rule is dependent on the complexity of the match criteria - if using inexpensive operations like checking the value of log source type, log source, QID, username, source IP, etc they are quite fast. If you would be doing a "Payload Contains" check or applying regex to either the full payload or to a property/field value, this will increase the computational cost. Can you provide any details as to the criteria for which you would be dropping events?

Cheers

Colin

Hi Colin,

Firstly, thank you for your feedback and detailed answer. The reason we want to avoid using routing roule here is that we think it will overload the system resources on the console. What is your opinion about this?

Hi Adem,

It is possible to drop events at the EC level (before the EP) using routing rules in "online" mode.

Although this satisfies your need to drop at the EC, I notice you specified "without routing rules" in your topic; I'm not sure why you want to avoid routing rules, but if for whatever reason you cannot use them, your only other option would be to prevent the events from being ingested at all. This can be handled in multiple ways, depending on the mechanism used to collect the data. Our WinCollect agent can have filters configured such that it only sends certain events. Other syslog sources may have similar filtering capabilities, it depends entirely on what is transmitting the syslog events. Many of our protocols that "pull" event data via various REST APIs, JDBC queries, and so may have filtering capabilities at the query level, but again it depends on the specific integration. Likewise the protocols that use a pub/sub paradigm may have filtering capabilities, or useful data vs unuseful data can be separated at the topic/subscription/eventhub/etc level such that only useful event data can be retrieved.

In short, online routing rules running in the EC are probably your best option, if you are willing to use them. If not, your goal may be doable, but it depends on the details.

Cheers

Colin

Hi guys,

We want to drop some events at the EC level before coming to the EP. Is there a method for this, has anyone tried it before?

Thanks in advance

Hi Adem,

Sorry for the delayed response. Generally speaking, routing rules are not considered to be very expensive computationally, especially for the Drop action (the Forward action is more expensive). But the relative cost of a routing rule is dependent on the complexity of the match criteria - if using inexpensive operations like checking the value of log source type, log source, QID, username, source IP, etc they are quite fast. If you would be doing a "Payload Contains" check or applying regex to either the full payload or to a property/field value, this will increase the computational cost. Can you provide any details as to the criteria for which you would be dropping events?

Cheers

Colin

Hi Colin,

Firstly, thank you for your feedback and detailed answer. The reason we want to avoid using routing roule here is that we think it will overload the system resources on the console. What is your opinion about this?

Hi Adem,

It is possible to drop events at the EC level (before the EP) using routing rules in "online" mode.

Although this satisfies your need to drop at the EC, I notice you specified "without routing rules" in your topic; I'm not sure why you want to avoid routing rules, but if for whatever reason you cannot use them, your only other option would be to prevent the events from being ingested at all. This can be handled in multiple ways, depending on the mechanism used to collect the data. Our WinCollect agent can have filters configured such that it only sends certain events. Other syslog sources may have similar filtering capabilities, it depends entirely on what is transmitting the syslog events. Many of our protocols that "pull" event data via various REST APIs, JDBC queries, and so may have filtering capabilities at the query level, but again it depends on the specific integration. Likewise the protocols that use a pub/sub paradigm may have filtering capabilities, or useful data vs unuseful data can be separated at the topic/subscription/eventhub/etc level such that only useful event data can be retrieved.

In short, online routing rules running in the EC are probably your best option, if you are willing to use them. If not, your goal may be doable, but it depends on the details.

Cheers

Colin

Hi guys,

We want to drop some events at the EC level before coming to the EP. Is there a method for this, has anyone tried it before?

Thanks in advance

Hi Adem,

Sorry for the delayed response. Generally speaking, routing rules are not considered to be very expensive computationally, especially for the Drop action (the Forward action is more expensive). But the relative cost of a routing rule is dependent on the complexity of the match criteria - if using inexpensive operations like checking the value of log source type, log source, QID, username, source IP, etc they are quite fast. If you would be doing a "Payload Contains" check or applying regex to either the full payload or to a property/field value, this will increase the computational cost. Can you provide any details as to the criteria for which you would be dropping events?

Cheers

Colin

Hi Colin,

Firstly, thank you for your feedback and detailed answer. The reason we want to avoid using routing roule here is that we think it will overload the system resources on the console. What is your opinion about this?

Hi Adem,

It is possible to drop events at the EC level (before the EP) using routing rules in "online" mode.

Although this satisfies your need to drop at the EC, I notice you specified "without routing rules" in your topic; I'm not sure why you want to avoid routing rules, but if for whatever reason you cannot use them, your only other option would be to prevent the events from being ingested at all. This can be handled in multiple ways, depending on the mechanism used to collect the data. Our WinCollect agent can have filters configured such that it only sends certain events. Other syslog sources may have similar filtering capabilities, it depends entirely on what is transmitting the syslog events. Many of our protocols that "pull" event data via various REST APIs, JDBC queries, and so may have filtering capabilities at the query level, but again it depends on the specific integration. Likewise the protocols that use a pub/sub paradigm may have filtering capabilities, or useful data vs unuseful data can be separated at the topic/subscription/eventhub/etc level such that only useful event data can be retrieved.

In short, online routing rules running in the EC are probably your best option, if you are willing to use them. If not, your goal may be doable, but it depends on the details.

Cheers

Colin

Hi guys,

We want to drop some events at the EC level before coming to the EP. Is there a method for this, has anyone tried it before?

Thanks in advance

Hi Heinrich,

I believe if you use Generic CSV as your Event Format option, the Amazon S3 protocol will automatically suppress the header lines, and no further action will be required. However this will reformat the events into either a name=value format or into JSON, depending on which option you choose.

If you want to preserve the original CSV format, keep with the LINEBYLINE format. In this case the header lines will be treated as events and you'll have to drop them later in the pipeline.

The method you mentioned will not work. The option which used to be labeled "Drop this event" has been renamed to "Bypass further rule correlation for this event" because that's what the action actually does - it flags the event to not be tested by any additional correlation rules.

Instead, you should use Routing Rules as discussed above. Go to the Admin tab and click the Routing Rules icon. Create a new routing rule, select "Online" as the mode and select the Event Collector host which is receiving your Amazon events. In the Event Filters section, you essentially define search criteria to match only the events you want to drop. A payload contains test is quite expensive, especially by itself, so you should add a filter for the log source type (or better yet, the specific log source) receiving the header line events, this ensures the rule will only be evaluated further ahainst events for that log source type or log source. Then, to ensure you're only dropping the "junk" events that you don't want, ideally you'd use a Low level Category or QID filter to match just the bad events (this assumes they are getting assigned a category like "Unknown" or a distinctive Event name that indicates they are unparsed.

Cheers

Colin

Hi Adem,

Sorry for the delayed response. Generally speaking, routing rules are not considered to be very expensive computationally, especially for the Drop action (the Forward action is more expensive). But the relative cost of a routing rule is dependent on the complexity of the match criteria - if using inexpensive operations like checking the value of log source type, log source, QID, username, source IP, etc they are quite fast. If you would be doing a "Payload Contains" check or applying regex to either the full payload or to a property/field value, this will increase the computational cost. Can you provide any details as to the criteria for which you would be dropping events?

Cheers

Colin

Hi Colin,

Firstly, thank you for your feedback and detailed answer. The reason we want to avoid using routing roule here is that we think it will overload the system resources on the console. What is your opinion about this?

Hi Adem,

It is possible to drop events at the EC level (before the EP) using routing rules in "online" mode.

Although this satisfies your need to drop at the EC, I notice you specified "without routing rules" in your topic; I'm not sure why you want to avoid routing rules, but if for whatever reason you cannot use them, your only other option would be to prevent the events from being ingested at all. This can be handled in multiple ways, depending on the mechanism used to collect the data. Our WinCollect agent can have filters configured such that it only sends certain events. Other syslog sources may have similar filtering capabilities, it depends entirely on what is transmitting the syslog events. Many of our protocols that "pull" event data via various REST APIs, JDBC queries, and so may have filtering capabilities at the query level, but again it depends on the specific integration. Likewise the protocols that use a pub/sub paradigm may have filtering capabilities, or useful data vs unuseful data can be separated at the topic/subscription/eventhub/etc level such that only useful event data can be retrieved.

In short, online routing rules running in the EC are probably your best option, if you are willing to use them. If not, your goal may be doable, but it depends on the details.

Cheers

Colin

Hi guys,

We want to drop some events at the EC level before coming to the EP. Is there a method for this, has anyone tried it before?

Thanks in advance