IBM QRadar

• are the logs arriving at the EP ? (tcpdump is a good tool to investigate it)
• are they parsed? (you should see events not parsed in the SIM Generic Log logsource related to your EP

• if the logsource is SYSLOG based, the payload MUST adhere to the RFC164 or RFC5424
• the logsource identifier is correct

Generally speaking, you shouldn't try to change or force QRadar configuration from command line as the first attempt to solve this kind of problems.

Hope this help.
Best regards,
Mario

• are the logs arriving at the EP ? (tcpdump is a good tool to investigate it)
• are they parsed? (you should see events not parsed in the SIM Generic Log logsource related to your EP

• if the logsource is SYSLOG based, the payload MUST adhere to the RFC164 or RFC5424
• the logsource identifier is correct

Generally speaking, you shouldn't try to change or force QRadar configuration from command line as the first attempt to solve this kind of problems.

Hope this help.
Best regards,
Mario

Hi Benjamin (and Mario and Karl),

There's a lot going on in this thread and some comments are completely accurate so I'm going to comment on several statements and then provide some additional advice.

Benjamin: "I checked the Console, using rpm -qa | grep -i forcepoint i saw the DSM-ForcepointVseries-********noarch. but i cant see this same dsm on my EP."

As Mario indicated, this is expected. DSM RPMs are only installed on the console and the artifacts they deliver are then deployed out to all managed hosts via deploy actions.

Mario: "if the logsource is SYSLOG based, the payload MUST adhere to the RFC164 or RFC5424"

This is not entirely correct. The syslog header on the events must conform to RFC 3164 or 5424 for QRadar to parse the syslog header and use the IP/hostname/identifier in it as the source of the event. This extracted value is used to "route" the event to a configured log source by matching it to a Log Source Identifier. If no matching log source exists, the event will appear in Log Activity associated with a SIM Generic Log log source, as it is a "catch all" bucket for any events which don't match the Log Source Identifier of any available log sources.

If a syslog event does not conform to the RFCs, QRadar will still accept it and attempt to parse it, but in this case the source of the event will default to the IP address from which it was sent.

From QRadar version 7.4.0 onward, there is a Log Source Identifier property/field in the Log Activity event details view, so for events being collected by a SIM Generic log source, you can see what the system considered the source to be - this tells you what Log Source Identifier value to use when creating a log source to handle the event.

Benjamin: "From tcpdump the logs are arriving at the EP, however looking at the log activity tab, i can see the log source with SIM-generic to the EP, not parsed nor mapped. In addition i manually created the same log source, and the logs are not parsed nor mapped despite using the forcepoint v-series DSM"

If you can see the events in tcpdump from the EP and can see the events in Log Activity (regardless of which log source they are associated to), this means the events are being successfully sent to QRadar and read off the network. The fact that the events are being routed to SIM Generic means there is no log source configured with the correct Log Source Identifier. If there was such a log source, the events would be linked to it, even if the associated DSM couldn't parse the events (they would appear with category = Stored).

Benjamin, after you manually created the log source, are the events being linked to it as Stored, or are they still going to SIM generic? If still SIM Generic, it means you need to fix your Log Source Identifier. If they are with the correct log source but are Stored (and not parsed correctly), then there is a parsing problem occurring.

Benjamin: "I am surprised why i am getting another log source auto detected after creating a log source manaully. I thought QRadar is not meant to discover another log source since i have created one already. "

A few questions here:
1. What log source type is the autodetected log source, and what type is the manually created log source (I assume it's Forcepoint V Series, but just want to be sure)

2. Does the autodetected log source have the same Log Source Identifier as the manually created log source?

QRadar will autodetect log source(s) on the same Log Source Identifier as a manually created log source if events are coming into the system which no log source can successfully parse. For example, say you manually create a log source with LSI (Log Source Identifier) 10.10.10.10. Any events originating from that IP will be routed to that log source for attempted parsing. If the log source can successfully parse the events, the events are not considered for autodetection, they simply stay with that log source. However, if they are not parseable by the existing log source, they will appear in Log Activity as Stored events (meaning they were just stored, not parsed) and will also be analyzed for the purpose of autodetection. If enough such events are received which can be parsed by another log source type (DSM), then QRadar will autodetect a log source of that type for 10.10.10.10, with a lower parsing order than the manually created log source (meaning the original log source has higher precedence/priority for any events coming from 10.10.10.10). We do this because it is possible to have multiple log sources for the same LSI - there could be a log source for the OS (Linux, Windows, etc) and then additional log sources for applications, databases, etc running on that OS.

Benjamin: "The IP address on both log sources is the same."

When you say "the IP address is the same", you mean that the LSI is the same? If so this answers one of my questions above. But this suggests that they are not the same log source type, as QRadar does not allow two log sources of the same type to coexist on the same LSI.

Benjamin: "For the manually created log source, the payload is in LEEF format, but the values are empty, like src= dst= usrname=, while the log source that was auto detected, the payload seems fine. i.e. it has values like src=x.x.x.x dst=x.x.x.x usrname=xxx."

Ok so you're saying that both your manually configured log source and the autodetected log source are actively receiving events, and they're both LEEF? Then I assume the LEEF headers must be different, and that one log source (presumably the autodetected one) is parsing events successfully?

If the LEEF attributes for the manually created log source are empty, this could explain why it isn't parsing. Is the LEEF header also empty? This sounds like a configuration problem on the sending side where it's just sending bad data for some events. It would be really helpful to see a payload sample from each log source.

Karl: "that would be true if you disabled autodetection or changed log source parsing order"

Parsing order doesn't actually affect autodetection in any way, it just defines the order in which a set of log sources which share a Log Source Identifier attempt to parse an event which matches their LSI. The first one to successfully parse the events "wins" and any log sources lower in parsing precedence don't get the chance to try. As long as one log source can accept the event, the autodetection engine will never see it, it's only if all log sources reject it that the autodetection engine gets it.

Karl: "does logdata come in with IP in log header? pls double check with tcpdump. If not you need to specify hostname as logsource identifier rather than IP address"

To be clear, you only need to use the hostname if it is present in an RFC3164 or RFC5424-compliant syslog header. If there is no header or if the header is non-compliant, you need to use the IP address for the LSI, and if there is a compliant header, you need to use whatever value is in the header (could be IP address, could be hostname, could be something else, depending on how the sending system is configured)



My advice:

First, answers to some of the questions above would help me provide a clear explanation.
Second, a general bit of wisdom for cases like this is to first determine if the events are even entering the system (that has been done - we know they are), and if so, launch the DSM Editor with the intended log source type and some sample events from Log Activity to check if the intended DSM can handle the event format. The DSM Editor allows you to pass raw events directly to a target DSM, without needing to worry about Log Source Identifier misconfiguration, so it eliminates a variable. If the DSM Editor's Log Activity Preview says that the event format is good, then the problem is almost certainly a routing problem, where the log source's Log Source Identifier is set incorrectly. If the DSM Editor's preview indicates that the DSM cannot handle the events, then this is the root problem that will cause autodetection to fail and manually-created log sources of that type to fail, and it means the 3rd party system is sending data in a format that the DSM doesn't understand, which typically can be solved with a config change on the sending side.

Cheers
Colin

Hi Benjamin (and Mario and Karl),

There's a lot going on in this thread and some comments are completely accurate so I'm going to comment on several statements and then provide some additional advice.

Benjamin: "I checked the Console, using rpm -qa | grep -i forcepoint i saw the DSM-ForcepointVseries-********noarch. but i cant see this same dsm on my EP."

As Mario indicated, this is expected. DSM RPMs are only installed on the console and the artifacts they deliver are then deployed out to all managed hosts via deploy actions.

Mario: "if the logsource is SYSLOG based, the payload MUST adhere to the RFC164 or RFC5424"

This is not entirely correct. The syslog header on the events must conform to RFC 3164 or 5424 for QRadar to parse the syslog header and use the IP/hostname/identifier in it as the source of the event. This extracted value is used to "route" the event to a configured log source by matching it to a Log Source Identifier. If no matching log source exists, the event will appear in Log Activity associated with a SIM Generic Log log source, as it is a "catch all" bucket for any events which don't match the Log Source Identifier of any available log sources.

If a syslog event does not conform to the RFCs, QRadar will still accept it and attempt to parse it, but in this case the source of the event will default to the IP address from which it was sent.

From QRadar version 7.4.0 onward, there is a Log Source Identifier property/field in the Log Activity event details view, so for events being collected by a SIM Generic log source, you can see what the system considered the source to be - this tells you what Log Source Identifier value to use when creating a log source to handle the event.

Benjamin: "From tcpdump the logs are arriving at the EP, however looking at the log activity tab, i can see the log source with SIM-generic to the EP, not parsed nor mapped. In addition i manually created the same log source, and the logs are not parsed nor mapped despite using the forcepoint v-series DSM"

If you can see the events in tcpdump from the EP and can see the events in Log Activity (regardless of which log source they are associated to), this means the events are being successfully sent to QRadar and read off the network. The fact that the events are being routed to SIM Generic means there is no log source configured with the correct Log Source Identifier. If there was such a log source, the events would be linked to it, even if the associated DSM couldn't parse the events (they would appear with category = Stored).

Benjamin, after you manually created the log source, are the events being linked to it as Stored, or are they still going to SIM generic? If still SIM Generic, it means you need to fix your Log Source Identifier. If they are with the correct log source but are Stored (and not parsed correctly), then there is a parsing problem occurring.

Benjamin: "I am surprised why i am getting another log source auto detected after creating a log source manaully. I thought QRadar is not meant to discover another log source since i have created one already. "

A few questions here:
1. What log source type is the autodetected log source, and what type is the manually created log source (I assume it's Forcepoint V Series, but just want to be sure)

2. Does the autodetected log source have the same Log Source Identifier as the manually created log source?

QRadar will autodetect log source(s) on the same Log Source Identifier as a manually created log source if events are coming into the system which no log source can successfully parse. For example, say you manually create a log source with LSI (Log Source Identifier) 10.10.10.10. Any events originating from that IP will be routed to that log source for attempted parsing. If the log source can successfully parse the events, the events are not considered for autodetection, they simply stay with that log source. However, if they are not parseable by the existing log source, they will appear in Log Activity as Stored events (meaning they were just stored, not parsed) and will also be analyzed for the purpose of autodetection. If enough such events are received which can be parsed by another log source type (DSM), then QRadar will autodetect a log source of that type for 10.10.10.10, with a lower parsing order than the manually created log source (meaning the original log source has higher precedence/priority for any events coming from 10.10.10.10). We do this because it is possible to have multiple log sources for the same LSI - there could be a log source for the OS (Linux, Windows, etc) and then additional log sources for applications, databases, etc running on that OS.

Benjamin: "The IP address on both log sources is the same."

When you say "the IP address is the same", you mean that the LSI is the same? If so this answers one of my questions above. But this suggests that they are not the same log source type, as QRadar does not allow two log sources of the same type to coexist on the same LSI.

Benjamin: "For the manually created log source, the payload is in LEEF format, but the values are empty, like src= dst= usrname=, while the log source that was auto detected, the payload seems fine. i.e. it has values like src=x.x.x.x dst=x.x.x.x usrname=xxx."

Ok so you're saying that both your manually configured log source and the autodetected log source are actively receiving events, and they're both LEEF? Then I assume the LEEF headers must be different, and that one log source (presumably the autodetected one) is parsing events successfully?

If the LEEF attributes for the manually created log source are empty, this could explain why it isn't parsing. Is the LEEF header also empty? This sounds like a configuration problem on the sending side where it's just sending bad data for some events. It would be really helpful to see a payload sample from each log source.

Karl: "that would be true if you disabled autodetection or changed log source parsing order"

Parsing order doesn't actually affect autodetection in any way, it just defines the order in which a set of log sources which share a Log Source Identifier attempt to parse an event which matches their LSI. The first one to successfully parse the events "wins" and any log sources lower in parsing precedence don't get the chance to try. As long as one log source can accept the event, the autodetection engine will never see it, it's only if all log sources reject it that the autodetection engine gets it.

Karl: "does logdata come in with IP in log header? pls double check with tcpdump. If not you need to specify hostname as logsource identifier rather than IP address"

To be clear, you only need to use the hostname if it is present in an RFC3164 or RFC5424-compliant syslog header. If there is no header or if the header is non-compliant, you need to use the IP address for the LSI, and if there is a compliant header, you need to use whatever value is in the header (could be IP address, could be hostname, could be something else, depending on how the sending system is configured)



My advice:

First, answers to some of the questions above would help me provide a clear explanation.
Second, a general bit of wisdom for cases like this is to first determine if the events are even entering the system (that has been done - we know they are), and if so, launch the DSM Editor with the intended log source type and some sample events from Log Activity to check if the intended DSM can handle the event format. The DSM Editor allows you to pass raw events directly to a target DSM, without needing to worry about Log Source Identifier misconfiguration, so it eliminates a variable. If the DSM Editor's Log Activity Preview says that the event format is good, then the problem is almost certainly a routing problem, where the log source's Log Source Identifier is set incorrectly. If the DSM Editor's preview indicates that the DSM cannot handle the events, then this is the root problem that will cause autodetection to fail and manually-created log sources of that type to fail, and it means the 3rd party system is sending data in a format that the DSM doesn't understand, which typically can be solved with a config change on the sending side.

Cheers
Colin

Hi Benjamin (and Mario and Karl),

There's a lot going on in this thread and some comments are completely accurate so I'm going to comment on several statements and then provide some additional advice.

Benjamin: "I checked the Console, using rpm -qa | grep -i forcepoint i saw the DSM-ForcepointVseries-********noarch. but i cant see this same dsm on my EP."

As Mario indicated, this is expected. DSM RPMs are only installed on the console and the artifacts they deliver are then deployed out to all managed hosts via deploy actions.

Mario: "if the logsource is SYSLOG based, the payload MUST adhere to the RFC164 or RFC5424"

This is not entirely correct. The syslog header on the events must conform to RFC 3164 or 5424 for QRadar to parse the syslog header and use the IP/hostname/identifier in it as the source of the event. This extracted value is used to "route" the event to a configured log source by matching it to a Log Source Identifier. If no matching log source exists, the event will appear in Log Activity associated with a SIM Generic Log log source, as it is a "catch all" bucket for any events which don't match the Log Source Identifier of any available log sources.

If a syslog event does not conform to the RFCs, QRadar will still accept it and attempt to parse it, but in this case the source of the event will default to the IP address from which it was sent.

From QRadar version 7.4.0 onward, there is a Log Source Identifier property/field in the Log Activity event details view, so for events being collected by a SIM Generic log source, you can see what the system considered the source to be - this tells you what Log Source Identifier value to use when creating a log source to handle the event.

Benjamin: "From tcpdump the logs are arriving at the EP, however looking at the log activity tab, i can see the log source with SIM-generic to the EP, not parsed nor mapped. In addition i manually created the same log source, and the logs are not parsed nor mapped despite using the forcepoint v-series DSM"

If you can see the events in tcpdump from the EP and can see the events in Log Activity (regardless of which log source they are associated to), this means the events are being successfully sent to QRadar and read off the network. The fact that the events are being routed to SIM Generic means there is no log source configured with the correct Log Source Identifier. If there was such a log source, the events would be linked to it, even if the associated DSM couldn't parse the events (they would appear with category = Stored).

Benjamin, after you manually created the log source, are the events being linked to it as Stored, or are they still going to SIM generic? If still SIM Generic, it means you need to fix your Log Source Identifier. If they are with the correct log source but are Stored (and not parsed correctly), then there is a parsing problem occurring.

Benjamin: "I am surprised why i am getting another log source auto detected after creating a log source manaully. I thought QRadar is not meant to discover another log source since i have created one already. "

A few questions here:
1. What log source type is the autodetected log source, and what type is the manually created log source (I assume it's Forcepoint V Series, but just want to be sure)

2. Does the autodetected log source have the same Log Source Identifier as the manually created log source?

QRadar will autodetect log source(s) on the same Log Source Identifier as a manually created log source if events are coming into the system which no log source can successfully parse. For example, say you manually create a log source with LSI (Log Source Identifier) 10.10.10.10. Any events originating from that IP will be routed to that log source for attempted parsing. If the log source can successfully parse the events, the events are not considered for autodetection, they simply stay with that log source. However, if they are not parseable by the existing log source, they will appear in Log Activity as Stored events (meaning they were just stored, not parsed) and will also be analyzed for the purpose of autodetection. If enough such events are received which can be parsed by another log source type (DSM), then QRadar will autodetect a log source of that type for 10.10.10.10, with a lower parsing order than the manually created log source (meaning the original log source has higher precedence/priority for any events coming from 10.10.10.10). We do this because it is possible to have multiple log sources for the same LSI - there could be a log source for the OS (Linux, Windows, etc) and then additional log sources for applications, databases, etc running on that OS.

Benjamin: "The IP address on both log sources is the same."

When you say "the IP address is the same", you mean that the LSI is the same? If so this answers one of my questions above. But this suggests that they are not the same log source type, as QRadar does not allow two log sources of the same type to coexist on the same LSI.

Benjamin: "For the manually created log source, the payload is in LEEF format, but the values are empty, like src= dst= usrname=, while the log source that was auto detected, the payload seems fine. i.e. it has values like src=x.x.x.x dst=x.x.x.x usrname=xxx."

Ok so you're saying that both your manually configured log source and the autodetected log source are actively receiving events, and they're both LEEF? Then I assume the LEEF headers must be different, and that one log source (presumably the autodetected one) is parsing events successfully?

If the LEEF attributes for the manually created log source are empty, this could explain why it isn't parsing. Is the LEEF header also empty? This sounds like a configuration problem on the sending side where it's just sending bad data for some events. It would be really helpful to see a payload sample from each log source.

Karl: "that would be true if you disabled autodetection or changed log source parsing order"

Parsing order doesn't actually affect autodetection in any way, it just defines the order in which a set of log sources which share a Log Source Identifier attempt to parse an event which matches their LSI. The first one to successfully parse the events "wins" and any log sources lower in parsing precedence don't get the chance to try. As long as one log source can accept the event, the autodetection engine will never see it, it's only if all log sources reject it that the autodetection engine gets it.

Karl: "does logdata come in with IP in log header? pls double check with tcpdump. If not you need to specify hostname as logsource identifier rather than IP address"

To be clear, you only need to use the hostname if it is present in an RFC3164 or RFC5424-compliant syslog header. If there is no header or if the header is non-compliant, you need to use the IP address for the LSI, and if there is a compliant header, you need to use whatever value is in the header (could be IP address, could be hostname, could be something else, depending on how the sending system is configured)



My advice:

First, answers to some of the questions above would help me provide a clear explanation.
Second, a general bit of wisdom for cases like this is to first determine if the events are even entering the system (that has been done - we know they are), and if so, launch the DSM Editor with the intended log source type and some sample events from Log Activity to check if the intended DSM can handle the event format. The DSM Editor allows you to pass raw events directly to a target DSM, without needing to worry about Log Source Identifier misconfiguration, so it eliminates a variable. If the DSM Editor's Log Activity Preview says that the event format is good, then the problem is almost certainly a routing problem, where the log source's Log Source Identifier is set incorrectly. If the DSM Editor's preview indicates that the DSM cannot handle the events, then this is the root problem that will cause autodetection to fail and manually-created log sources of that type to fail, and it means the 3rd party system is sending data in a format that the DSM doesn't understand, which typically can be solved with a config change on the sending side.

Cheers
Colin

Hi Benjamin (and Mario and Karl),

There's a lot going on in this thread and some comments are completely accurate so I'm going to comment on several statements and then provide some additional advice.

Benjamin: "I checked the Console, using rpm -qa | grep -i forcepoint i saw the DSM-ForcepointVseries-********noarch. but i cant see this same dsm on my EP."

As Mario indicated, this is expected. DSM RPMs are only installed on the console and the artifacts they deliver are then deployed out to all managed hosts via deploy actions.

Mario: "if the logsource is SYSLOG based, the payload MUST adhere to the RFC164 or RFC5424"

This is not entirely correct. The syslog header on the events must conform to RFC 3164 or 5424 for QRadar to parse the syslog header and use the IP/hostname/identifier in it as the source of the event. This extracted value is used to "route" the event to a configured log source by matching it to a Log Source Identifier. If no matching log source exists, the event will appear in Log Activity associated with a SIM Generic Log log source, as it is a "catch all" bucket for any events which don't match the Log Source Identifier of any available log sources.

If a syslog event does not conform to the RFCs, QRadar will still accept it and attempt to parse it, but in this case the source of the event will default to the IP address from which it was sent.

From QRadar version 7.4.0 onward, there is a Log Source Identifier property/field in the Log Activity event details view, so for events being collected by a SIM Generic log source, you can see what the system considered the source to be - this tells you what Log Source Identifier value to use when creating a log source to handle the event.

Benjamin: "From tcpdump the logs are arriving at the EP, however looking at the log activity tab, i can see the log source with SIM-generic to the EP, not parsed nor mapped. In addition i manually created the same log source, and the logs are not parsed nor mapped despite using the forcepoint v-series DSM"

If you can see the events in tcpdump from the EP and can see the events in Log Activity (regardless of which log source they are associated to), this means the events are being successfully sent to QRadar and read off the network. The fact that the events are being routed to SIM Generic means there is no log source configured with the correct Log Source Identifier. If there was such a log source, the events would be linked to it, even if the associated DSM couldn't parse the events (they would appear with category = Stored).

Benjamin, after you manually created the log source, are the events being linked to it as Stored, or are they still going to SIM generic? If still SIM Generic, it means you need to fix your Log Source Identifier. If they are with the correct log source but are Stored (and not parsed correctly), then there is a parsing problem occurring.

Benjamin: "I am surprised why i am getting another log source auto detected after creating a log source manaully. I thought QRadar is not meant to discover another log source since i have created one already. "

A few questions here:
1. What log source type is the autodetected log source, and what type is the manually created log source (I assume it's Forcepoint V Series, but just want to be sure)

2. Does the autodetected log source have the same Log Source Identifier as the manually created log source?

QRadar will autodetect log source(s) on the same Log Source Identifier as a manually created log source if events are coming into the system which no log source can successfully parse. For example, say you manually create a log source with LSI (Log Source Identifier) 10.10.10.10. Any events originating from that IP will be routed to that log source for attempted parsing. If the log source can successfully parse the events, the events are not considered for autodetection, they simply stay with that log source. However, if they are not parseable by the existing log source, they will appear in Log Activity as Stored events (meaning they were just stored, not parsed) and will also be analyzed for the purpose of autodetection. If enough such events are received which can be parsed by another log source type (DSM), then QRadar will autodetect a log source of that type for 10.10.10.10, with a lower parsing order than the manually created log source (meaning the original log source has higher precedence/priority for any events coming from 10.10.10.10). We do this because it is possible to have multiple log sources for the same LSI - there could be a log source for the OS (Linux, Windows, etc) and then additional log sources for applications, databases, etc running on that OS.

Benjamin: "The IP address on both log sources is the same."

When you say "the IP address is the same", you mean that the LSI is the same? If so this answers one of my questions above. But this suggests that they are not the same log source type, as QRadar does not allow two log sources of the same type to coexist on the same LSI.

Benjamin: "For the manually created log source, the payload is in LEEF format, but the values are empty, like src= dst= usrname=, while the log source that was auto detected, the payload seems fine. i.e. it has values like src=x.x.x.x dst=x.x.x.x usrname=xxx."

Ok so you're saying that both your manually configured log source and the autodetected log source are actively receiving events, and they're both LEEF? Then I assume the LEEF headers must be different, and that one log source (presumably the autodetected one) is parsing events successfully?

If the LEEF attributes for the manually created log source are empty, this could explain why it isn't parsing. Is the LEEF header also empty? This sounds like a configuration problem on the sending side where it's just sending bad data for some events. It would be really helpful to see a payload sample from each log source.

Karl: "that would be true if you disabled autodetection or changed log source parsing order"

Parsing order doesn't actually affect autodetection in any way, it just defines the order in which a set of log sources which share a Log Source Identifier attempt to parse an event which matches their LSI. The first one to successfully parse the events "wins" and any log sources lower in parsing precedence don't get the chance to try. As long as one log source can accept the event, the autodetection engine will never see it, it's only if all log sources reject it that the autodetection engine gets it.

Karl: "does logdata come in with IP in log header? pls double check with tcpdump. If not you need to specify hostname as logsource identifier rather than IP address"

To be clear, you only need to use the hostname if it is present in an RFC3164 or RFC5424-compliant syslog header. If there is no header or if the header is non-compliant, you need to use the IP address for the LSI, and if there is a compliant header, you need to use whatever value is in the header (could be IP address, could be hostname, could be something else, depending on how the sending system is configured)



My advice:

First, answers to some of the questions above would help me provide a clear explanation.
Second, a general bit of wisdom for cases like this is to first determine if the events are even entering the system (that has been done - we know they are), and if so, launch the DSM Editor with the intended log source type and some sample events from Log Activity to check if the intended DSM can handle the event format. The DSM Editor allows you to pass raw events directly to a target DSM, without needing to worry about Log Source Identifier misconfiguration, so it eliminates a variable. If the DSM Editor's Log Activity Preview says that the event format is good, then the problem is almost certainly a routing problem, where the log source's Log Source Identifier is set incorrectly. If the DSM Editor's preview indicates that the DSM cannot handle the events, then this is the root problem that will cause autodetection to fail and manually-created log sources of that type to fail, and it means the 3rd party system is sending data in a format that the DSM doesn't understand, which typically can be solved with a config change on the sending side.

Cheers
Colin

Hello Benjamin,

I understand your wish to know as much as possible about the QRadar.

Most of the topics are covered by our official documentation

https://www.ibm.com/support/knowledgecenter/en/SS42VS_7.4/com.ibm.qradar.doc/qradar_IC_welcome.html

integrated by technote

https://www.ibm.com/community/qradar/home/knowledge/

Some topics are related to how QRadar works internally and they aren't public. I also ignore the precise distribution mechanism, but I'm sure that it's the way QRadar works. For sure, the needed component are distributed automatically to the EP (and also to the EC if present). It doesn't matter if they are jar files, xml files, config files or whatever else. I don't need to interact with this internal component, it is like a black-box and I accept this.

I think (and hope) that you trust QRadar since it is one of the leading SIEM in the market. QRadar is a robust platform based on appliances, this particularity includes, among other things, that the manual operation on the environment are limited and strictly related to manage specific issues. If you were requested to install something manually (and it is a real rare case) you should follow the instructions provided and nothing else.

Hope this clarify my point of view.

Regards,
Mario

Hi Benjamin (and Mario and Karl),

There's a lot going on in this thread and some comments are completely accurate so I'm going to comment on several statements and then provide some additional advice.

Benjamin: "I checked the Console, using rpm -qa | grep -i forcepoint i saw the DSM-ForcepointVseries-********noarch. but i cant see this same dsm on my EP."

As Mario indicated, this is expected. DSM RPMs are only installed on the console and the artifacts they deliver are then deployed out to all managed hosts via deploy actions.

Mario: "if the logsource is SYSLOG based, the payload MUST adhere to the RFC164 or RFC5424"

This is not entirely correct. The syslog header on the events must conform to RFC 3164 or 5424 for QRadar to parse the syslog header and use the IP/hostname/identifier in it as the source of the event. This extracted value is used to "route" the event to a configured log source by matching it to a Log Source Identifier. If no matching log source exists, the event will appear in Log Activity associated with a SIM Generic Log log source, as it is a "catch all" bucket for any events which don't match the Log Source Identifier of any available log sources.

If a syslog event does not conform to the RFCs, QRadar will still accept it and attempt to parse it, but in this case the source of the event will default to the IP address from which it was sent.

From QRadar version 7.4.0 onward, there is a Log Source Identifier property/field in the Log Activity event details view, so for events being collected by a SIM Generic log source, you can see what the system considered the source to be - this tells you what Log Source Identifier value to use when creating a log source to handle the event.

Benjamin: "From tcpdump the logs are arriving at the EP, however looking at the log activity tab, i can see the log source with SIM-generic to the EP, not parsed nor mapped. In addition i manually created the same log source, and the logs are not parsed nor mapped despite using the forcepoint v-series DSM"

If you can see the events in tcpdump from the EP and can see the events in Log Activity (regardless of which log source they are associated to), this means the events are being successfully sent to QRadar and read off the network. The fact that the events are being routed to SIM Generic means there is no log source configured with the correct Log Source Identifier. If there was such a log source, the events would be linked to it, even if the associated DSM couldn't parse the events (they would appear with category = Stored).

Benjamin, after you manually created the log source, are the events being linked to it as Stored, or are they still going to SIM generic? If still SIM Generic, it means you need to fix your Log Source Identifier. If they are with the correct log source but are Stored (and not parsed correctly), then there is a parsing problem occurring.

Benjamin: "I am surprised why i am getting another log source auto detected after creating a log source manaully. I thought QRadar is not meant to discover another log source since i have created one already. "

A few questions here:
1. What log source type is the autodetected log source, and what type is the manually created log source (I assume it's Forcepoint V Series, but just want to be sure)

2. Does the autodetected log source have the same Log Source Identifier as the manually created log source?

QRadar will autodetect log source(s) on the same Log Source Identifier as a manually created log source if events are coming into the system which no log source can successfully parse. For example, say you manually create a log source with LSI (Log Source Identifier) 10.10.10.10. Any events originating from that IP will be routed to that log source for attempted parsing. If the log source can successfully parse the events, the events are not considered for autodetection, they simply stay with that log source. However, if they are not parseable by the existing log source, they will appear in Log Activity as Stored events (meaning they were just stored, not parsed) and will also be analyzed for the purpose of autodetection. If enough such events are received which can be parsed by another log source type (DSM), then QRadar will autodetect a log source of that type for 10.10.10.10, with a lower parsing order than the manually created log source (meaning the original log source has higher precedence/priority for any events coming from 10.10.10.10). We do this because it is possible to have multiple log sources for the same LSI - there could be a log source for the OS (Linux, Windows, etc) and then additional log sources for applications, databases, etc running on that OS.

Benjamin: "The IP address on both log sources is the same."

When you say "the IP address is the same", you mean that the LSI is the same? If so this answers one of my questions above. But this suggests that they are not the same log source type, as QRadar does not allow two log sources of the same type to coexist on the same LSI.

Benjamin: "For the manually created log source, the payload is in LEEF format, but the values are empty, like src= dst= usrname=, while the log source that was auto detected, the payload seems fine. i.e. it has values like src=x.x.x.x dst=x.x.x.x usrname=xxx."

Ok so you're saying that both your manually configured log source and the autodetected log source are actively receiving events, and they're both LEEF? Then I assume the LEEF headers must be different, and that one log source (presumably the autodetected one) is parsing events successfully?

If the LEEF attributes for the manually created log source are empty, this could explain why it isn't parsing. Is the LEEF header also empty? This sounds like a configuration problem on the sending side where it's just sending bad data for some events. It would be really helpful to see a payload sample from each log source.

Karl: "that would be true if you disabled autodetection or changed log source parsing order"

Parsing order doesn't actually affect autodetection in any way, it just defines the order in which a set of log sources which share a Log Source Identifier attempt to parse an event which matches their LSI. The first one to successfully parse the events "wins" and any log sources lower in parsing precedence don't get the chance to try. As long as one log source can accept the event, the autodetection engine will never see it, it's only if all log sources reject it that the autodetection engine gets it.

Karl: "does logdata come in with IP in log header? pls double check with tcpdump. If not you need to specify hostname as logsource identifier rather than IP address"

To be clear, you only need to use the hostname if it is present in an RFC3164 or RFC5424-compliant syslog header. If there is no header or if the header is non-compliant, you need to use the IP address for the LSI, and if there is a compliant header, you need to use whatever value is in the header (could be IP address, could be hostname, could be something else, depending on how the sending system is configured)



My advice:

First, answers to some of the questions above would help me provide a clear explanation.
Second, a general bit of wisdom for cases like this is to first determine if the events are even entering the system (that has been done - we know they are), and if so, launch the DSM Editor with the intended log source type and some sample events from Log Activity to check if the intended DSM can handle the event format. The DSM Editor allows you to pass raw events directly to a target DSM, without needing to worry about Log Source Identifier misconfiguration, so it eliminates a variable. If the DSM Editor's Log Activity Preview says that the event format is good, then the problem is almost certainly a routing problem, where the log source's Log Source Identifier is set incorrectly. If the DSM Editor's preview indicates that the DSM cannot handle the events, then this is the root problem that will cause autodetection to fail and manually-created log sources of that type to fail, and it means the 3rd party system is sending data in a format that the DSM doesn't understand, which typically can be solved with a config change on the sending side.

Cheers
Colin

Hi Benjamin (and Mario and Karl),

There's a lot going on in this thread and some comments are completely accurate so I'm going to comment on several statements and then provide some additional advice.

Benjamin: "I checked the Console, using rpm -qa | grep -i forcepoint i saw the DSM-ForcepointVseries-********noarch. but i cant see this same dsm on my EP."

As Mario indicated, this is expected. DSM RPMs are only installed on the console and the artifacts they deliver are then deployed out to all managed hosts via deploy actions.

Mario: "if the logsource is SYSLOG based, the payload MUST adhere to the RFC164 or RFC5424"

This is not entirely correct. The syslog header on the events must conform to RFC 3164 or 5424 for QRadar to parse the syslog header and use the IP/hostname/identifier in it as the source of the event. This extracted value is used to "route" the event to a configured log source by matching it to a Log Source Identifier. If no matching log source exists, the event will appear in Log Activity associated with a SIM Generic Log log source, as it is a "catch all" bucket for any events which don't match the Log Source Identifier of any available log sources.

If a syslog event does not conform to the RFCs, QRadar will still accept it and attempt to parse it, but in this case the source of the event will default to the IP address from which it was sent.

From QRadar version 7.4.0 onward, there is a Log Source Identifier property/field in the Log Activity event details view, so for events being collected by a SIM Generic log source, you can see what the system considered the source to be - this tells you what Log Source Identifier value to use when creating a log source to handle the event.

Benjamin: "From tcpdump the logs are arriving at the EP, however looking at the log activity tab, i can see the log source with SIM-generic to the EP, not parsed nor mapped. In addition i manually created the same log source, and the logs are not parsed nor mapped despite using the forcepoint v-series DSM"

If you can see the events in tcpdump from the EP and can see the events in Log Activity (regardless of which log source they are associated to), this means the events are being successfully sent to QRadar and read off the network. The fact that the events are being routed to SIM Generic means there is no log source configured with the correct Log Source Identifier. If there was such a log source, the events would be linked to it, even if the associated DSM couldn't parse the events (they would appear with category = Stored).

Benjamin, after you manually created the log source, are the events being linked to it as Stored, or are they still going to SIM generic? If still SIM Generic, it means you need to fix your Log Source Identifier. If they are with the correct log source but are Stored (and not parsed correctly), then there is a parsing problem occurring.

Benjamin: "I am surprised why i am getting another log source auto detected after creating a log source manaully. I thought QRadar is not meant to discover another log source since i have created one already. "

A few questions here:
1. What log source type is the autodetected log source, and what type is the manually created log source (I assume it's Forcepoint V Series, but just want to be sure)

2. Does the autodetected log source have the same Log Source Identifier as the manually created log source?

QRadar will autodetect log source(s) on the same Log Source Identifier as a manually created log source if events are coming into the system which no log source can successfully parse. For example, say you manually create a log source with LSI (Log Source Identifier) 10.10.10.10. Any events originating from that IP will be routed to that log source for attempted parsing. If the log source can successfully parse the events, the events are not considered for autodetection, they simply stay with that log source. However, if they are not parseable by the existing log source, they will appear in Log Activity as Stored events (meaning they were just stored, not parsed) and will also be analyzed for the purpose of autodetection. If enough such events are received which can be parsed by another log source type (DSM), then QRadar will autodetect a log source of that type for 10.10.10.10, with a lower parsing order than the manually created log source (meaning the original log source has higher precedence/priority for any events coming from 10.10.10.10). We do this because it is possible to have multiple log sources for the same LSI - there could be a log source for the OS (Linux, Windows, etc) and then additional log sources for applications, databases, etc running on that OS.

Benjamin: "The IP address on both log sources is the same."

When you say "the IP address is the same", you mean that the LSI is the same? If so this answers one of my questions above. But this suggests that they are not the same log source type, as QRadar does not allow two log sources of the same type to coexist on the same LSI.

Benjamin: "For the manually created log source, the payload is in LEEF format, but the values are empty, like src= dst= usrname=, while the log source that was auto detected, the payload seems fine. i.e. it has values like src=x.x.x.x dst=x.x.x.x usrname=xxx."

Ok so you're saying that both your manually configured log source and the autodetected log source are actively receiving events, and they're both LEEF? Then I assume the LEEF headers must be different, and that one log source (presumably the autodetected one) is parsing events successfully?

If the LEEF attributes for the manually created log source are empty, this could explain why it isn't parsing. Is the LEEF header also empty? This sounds like a configuration problem on the sending side where it's just sending bad data for some events. It would be really helpful to see a payload sample from each log source.

Karl: "that would be true if you disabled autodetection or changed log source parsing order"

Parsing order doesn't actually affect autodetection in any way, it just defines the order in which a set of log sources which share a Log Source Identifier attempt to parse an event which matches their LSI. The first one to successfully parse the events "wins" and any log sources lower in parsing precedence don't get the chance to try. As long as one log source can accept the event, the autodetection engine will never see it, it's only if all log sources reject it that the autodetection engine gets it.

Karl: "does logdata come in with IP in log header? pls double check with tcpdump. If not you need to specify hostname as logsource identifier rather than IP address"

To be clear, you only need to use the hostname if it is present in an RFC3164 or RFC5424-compliant syslog header. If there is no header or if the header is non-compliant, you need to use the IP address for the LSI, and if there is a compliant header, you need to use whatever value is in the header (could be IP address, could be hostname, could be something else, depending on how the sending system is configured)



My advice:

First, answers to some of the questions above would help me provide a clear explanation.
Second, a general bit of wisdom for cases like this is to first determine if the events are even entering the system (that has been done - we know they are), and if so, launch the DSM Editor with the intended log source type and some sample events from Log Activity to check if the intended DSM can handle the event format. The DSM Editor allows you to pass raw events directly to a target DSM, without needing to worry about Log Source Identifier misconfiguration, so it eliminates a variable. If the DSM Editor's Log Activity Preview says that the event format is good, then the problem is almost certainly a routing problem, where the log source's Log Source Identifier is set incorrectly. If the DSM Editor's preview indicates that the DSM cannot handle the events, then this is the root problem that will cause autodetection to fail and manually-created log sources of that type to fail, and it means the 3rd party system is sending data in a format that the DSM doesn't understand, which typically can be solved with a config change on the sending side.

Cheers
Colin

Hi Benjamin (and Mario and Karl),

There's a lot going on in this thread and some comments are completely accurate so I'm going to comment on several statements and then provide some additional advice.

Benjamin: "I checked the Console, using rpm -qa | grep -i forcepoint i saw the DSM-ForcepointVseries-********noarch. but i cant see this same dsm on my EP."

As Mario indicated, this is expected. DSM RPMs are only installed on the console and the artifacts they deliver are then deployed out to all managed hosts via deploy actions.

Mario: "if the logsource is SYSLOG based, the payload MUST adhere to the RFC164 or RFC5424"

This is not entirely correct. The syslog header on the events must conform to RFC 3164 or 5424 for QRadar to parse the syslog header and use the IP/hostname/identifier in it as the source of the event. This extracted value is used to "route" the event to a configured log source by matching it to a Log Source Identifier. If no matching log source exists, the event will appear in Log Activity associated with a SIM Generic Log log source, as it is a "catch all" bucket for any events which don't match the Log Source Identifier of any available log sources.

If a syslog event does not conform to the RFCs, QRadar will still accept it and attempt to parse it, but in this case the source of the event will default to the IP address from which it was sent.

From QRadar version 7.4.0 onward, there is a Log Source Identifier property/field in the Log Activity event details view, so for events being collected by a SIM Generic log source, you can see what the system considered the source to be - this tells you what Log Source Identifier value to use when creating a log source to handle the event.

Benjamin: "From tcpdump the logs are arriving at the EP, however looking at the log activity tab, i can see the log source with SIM-generic to the EP, not parsed nor mapped. In addition i manually created the same log source, and the logs are not parsed nor mapped despite using the forcepoint v-series DSM"

If you can see the events in tcpdump from the EP and can see the events in Log Activity (regardless of which log source they are associated to), this means the events are being successfully sent to QRadar and read off the network. The fact that the events are being routed to SIM Generic means there is no log source configured with the correct Log Source Identifier. If there was such a log source, the events would be linked to it, even if the associated DSM couldn't parse the events (they would appear with category = Stored).

Benjamin, after you manually created the log source, are the events being linked to it as Stored, or are they still going to SIM generic? If still SIM Generic, it means you need to fix your Log Source Identifier. If they are with the correct log source but are Stored (and not parsed correctly), then there is a parsing problem occurring.

Benjamin: "I am surprised why i am getting another log source auto detected after creating a log source manaully. I thought QRadar is not meant to discover another log source since i have created one already. "

A few questions here:
1. What log source type is the autodetected log source, and what type is the manually created log source (I assume it's Forcepoint V Series, but just want to be sure)

2. Does the autodetected log source have the same Log Source Identifier as the manually created log source?

QRadar will autodetect log source(s) on the same Log Source Identifier as a manually created log source if events are coming into the system which no log source can successfully parse. For example, say you manually create a log source with LSI (Log Source Identifier) 10.10.10.10. Any events originating from that IP will be routed to that log source for attempted parsing. If the log source can successfully parse the events, the events are not considered for autodetection, they simply stay with that log source. However, if they are not parseable by the existing log source, they will appear in Log Activity as Stored events (meaning they were just stored, not parsed) and will also be analyzed for the purpose of autodetection. If enough such events are received which can be parsed by another log source type (DSM), then QRadar will autodetect a log source of that type for 10.10.10.10, with a lower parsing order than the manually created log source (meaning the original log source has higher precedence/priority for any events coming from 10.10.10.10). We do this because it is possible to have multiple log sources for the same LSI - there could be a log source for the OS (Linux, Windows, etc) and then additional log sources for applications, databases, etc running on that OS.

Benjamin: "The IP address on both log sources is the same."

When you say "the IP address is the same", you mean that the LSI is the same? If so this answers one of my questions above. But this suggests that they are not the same log source type, as QRadar does not allow two log sources of the same type to coexist on the same LSI.

Benjamin: "For the manually created log source, the payload is in LEEF format, but the values are empty, like src= dst= usrname=, while the log source that was auto detected, the payload seems fine. i.e. it has values like src=x.x.x.x dst=x.x.x.x usrname=xxx."

Ok so you're saying that both your manually configured log source and the autodetected log source are actively receiving events, and they're both LEEF? Then I assume the LEEF headers must be different, and that one log source (presumably the autodetected one) is parsing events successfully?

If the LEEF attributes for the manually created log source are empty, this could explain why it isn't parsing. Is the LEEF header also empty? This sounds like a configuration problem on the sending side where it's just sending bad data for some events. It would be really helpful to see a payload sample from each log source.

Karl: "that would be true if you disabled autodetection or changed log source parsing order"

Parsing order doesn't actually affect autodetection in any way, it just defines the order in which a set of log sources which share a Log Source Identifier attempt to parse an event which matches their LSI. The first one to successfully parse the events "wins" and any log sources lower in parsing precedence don't get the chance to try. As long as one log source can accept the event, the autodetection engine will never see it, it's only if all log sources reject it that the autodetection engine gets it.

Karl: "does logdata come in with IP in log header? pls double check with tcpdump. If not you need to specify hostname as logsource identifier rather than IP address"

To be clear, you only need to use the hostname if it is present in an RFC3164 or RFC5424-compliant syslog header. If there is no header or if the header is non-compliant, you need to use the IP address for the LSI, and if there is a compliant header, you need to use whatever value is in the header (could be IP address, could be hostname, could be something else, depending on how the sending system is configured)



My advice:

First, answers to some of the questions above would help me provide a clear explanation.
Second, a general bit of wisdom for cases like this is to first determine if the events are even entering the system (that has been done - we know they are), and if so, launch the DSM Editor with the intended log source type and some sample events from Log Activity to check if the intended DSM can handle the event format. The DSM Editor allows you to pass raw events directly to a target DSM, without needing to worry about Log Source Identifier misconfiguration, so it eliminates a variable. If the DSM Editor's Log Activity Preview says that the event format is good, then the problem is almost certainly a routing problem, where the log source's Log Source Identifier is set incorrectly. If the DSM Editor's preview indicates that the DSM cannot handle the events, then this is the root problem that will cause autodetection to fail and manually-created log sources of that type to fail, and it means the 3rd party system is sending data in a format that the DSM doesn't understand, which typically can be solved with a config change on the sending side.

Cheers
Colin

Hi Benjamin (and Mario and Karl),

There's a lot going on in this thread and some comments are completely accurate so I'm going to comment on several statements and then provide some additional advice.

Benjamin: "I checked the Console, using rpm -qa | grep -i forcepoint i saw the DSM-ForcepointVseries-********noarch. but i cant see this same dsm on my EP."

As Mario indicated, this is expected. DSM RPMs are only installed on the console and the artifacts they deliver are then deployed out to all managed hosts via deploy actions.

Mario: "if the logsource is SYSLOG based, the payload MUST adhere to the RFC164 or RFC5424"

This is not entirely correct. The syslog header on the events must conform to RFC 3164 or 5424 for QRadar to parse the syslog header and use the IP/hostname/identifier in it as the source of the event. This extracted value is used to "route" the event to a configured log source by matching it to a Log Source Identifier. If no matching log source exists, the event will appear in Log Activity associated with a SIM Generic Log log source, as it is a "catch all" bucket for any events which don't match the Log Source Identifier of any available log sources.

If a syslog event does not conform to the RFCs, QRadar will still accept it and attempt to parse it, but in this case the source of the event will default to the IP address from which it was sent.

From QRadar version 7.4.0 onward, there is a Log Source Identifier property/field in the Log Activity event details view, so for events being collected by a SIM Generic log source, you can see what the system considered the source to be - this tells you what Log Source Identifier value to use when creating a log source to handle the event.

Benjamin: "From tcpdump the logs are arriving at the EP, however looking at the log activity tab, i can see the log source with SIM-generic to the EP, not parsed nor mapped. In addition i manually created the same log source, and the logs are not parsed nor mapped despite using the forcepoint v-series DSM"

If you can see the events in tcpdump from the EP and can see the events in Log Activity (regardless of which log source they are associated to), this means the events are being successfully sent to QRadar and read off the network. The fact that the events are being routed to SIM Generic means there is no log source configured with the correct Log Source Identifier. If there was such a log source, the events would be linked to it, even if the associated DSM couldn't parse the events (they would appear with category = Stored).

Benjamin, after you manually created the log source, are the events being linked to it as Stored, or are they still going to SIM generic? If still SIM Generic, it means you need to fix your Log Source Identifier. If they are with the correct log source but are Stored (and not parsed correctly), then there is a parsing problem occurring.

Benjamin: "I am surprised why i am getting another log source auto detected after creating a log source manaully. I thought QRadar is not meant to discover another log source since i have created one already. "

A few questions here:
1. What log source type is the autodetected log source, and what type is the manually created log source (I assume it's Forcepoint V Series, but just want to be sure)

2. Does the autodetected log source have the same Log Source Identifier as the manually created log source?

QRadar will autodetect log source(s) on the same Log Source Identifier as a manually created log source if events are coming into the system which no log source can successfully parse. For example, say you manually create a log source with LSI (Log Source Identifier) 10.10.10.10. Any events originating from that IP will be routed to that log source for attempted parsing. If the log source can successfully parse the events, the events are not considered for autodetection, they simply stay with that log source. However, if they are not parseable by the existing log source, they will appear in Log Activity as Stored events (meaning they were just stored, not parsed) and will also be analyzed for the purpose of autodetection. If enough such events are received which can be parsed by another log source type (DSM), then QRadar will autodetect a log source of that type for 10.10.10.10, with a lower parsing order than the manually created log source (meaning the original log source has higher precedence/priority for any events coming from 10.10.10.10). We do this because it is possible to have multiple log sources for the same LSI - there could be a log source for the OS (Linux, Windows, etc) and then additional log sources for applications, databases, etc running on that OS.

Benjamin: "The IP address on both log sources is the same."

When you say "the IP address is the same", you mean that the LSI is the same? If so this answers one of my questions above. But this suggests that they are not the same log source type, as QRadar does not allow two log sources of the same type to coexist on the same LSI.

Benjamin: "For the manually created log source, the payload is in LEEF format, but the values are empty, like src= dst= usrname=, while the log source that was auto detected, the payload seems fine. i.e. it has values like src=x.x.x.x dst=x.x.x.x usrname=xxx."

Ok so you're saying that both your manually configured log source and the autodetected log source are actively receiving events, and they're both LEEF? Then I assume the LEEF headers must be different, and that one log source (presumably the autodetected one) is parsing events successfully?

If the LEEF attributes for the manually created log source are empty, this could explain why it isn't parsing. Is the LEEF header also empty? This sounds like a configuration problem on the sending side where it's just sending bad data for some events. It would be really helpful to see a payload sample from each log source.

Karl: "that would be true if you disabled autodetection or changed log source parsing order"

Parsing order doesn't actually affect autodetection in any way, it just defines the order in which a set of log sources which share a Log Source Identifier attempt to parse an event which matches their LSI. The first one to successfully parse the events "wins" and any log sources lower in parsing precedence don't get the chance to try. As long as one log source can accept the event, the autodetection engine will never see it, it's only if all log sources reject it that the autodetection engine gets it.

Karl: "does logdata come in with IP in log header? pls double check with tcpdump. If not you need to specify hostname as logsource identifier rather than IP address"

To be clear, you only need to use the hostname if it is present in an RFC3164 or RFC5424-compliant syslog header. If there is no header or if the header is non-compliant, you need to use the IP address for the LSI, and if there is a compliant header, you need to use whatever value is in the header (could be IP address, could be hostname, could be something else, depending on how the sending system is configured)



My advice:

First, answers to some of the questions above would help me provide a clear explanation.
Second, a general bit of wisdom for cases like this is to first determine if the events are even entering the system (that has been done - we know they are), and if so, launch the DSM Editor with the intended log source type and some sample events from Log Activity to check if the intended DSM can handle the event format. The DSM Editor allows you to pass raw events directly to a target DSM, without needing to worry about Log Source Identifier misconfiguration, so it eliminates a variable. If the DSM Editor's Log Activity Preview says that the event format is good, then the problem is almost certainly a routing problem, where the log source's Log Source Identifier is set incorrectly. If the DSM Editor's preview indicates that the DSM cannot handle the events, then this is the root problem that will cause autodetection to fail and manually-created log sources of that type to fail, and it means the 3rd party system is sending data in a format that the DSM doesn't understand, which typically can be solved with a config change on the sending side.

Cheers
Colin

Hi Benjamin (and Mario and Karl),

There's a lot going on in this thread and some comments are completely accurate so I'm going to comment on several statements and then provide some additional advice.

Benjamin: "I checked the Console, using rpm -qa | grep -i forcepoint i saw the DSM-ForcepointVseries-********noarch. but i cant see this same dsm on my EP."

As Mario indicated, this is expected. DSM RPMs are only installed on the console and the artifacts they deliver are then deployed out to all managed hosts via deploy actions.

Mario: "if the logsource is SYSLOG based, the payload MUST adhere to the RFC164 or RFC5424"

This is not entirely correct. The syslog header on the events must conform to RFC 3164 or 5424 for QRadar to parse the syslog header and use the IP/hostname/identifier in it as the source of the event. This extracted value is used to "route" the event to a configured log source by matching it to a Log Source Identifier. If no matching log source exists, the event will appear in Log Activity associated with a SIM Generic Log log source, as it is a "catch all" bucket for any events which don't match the Log Source Identifier of any available log sources.

If a syslog event does not conform to the RFCs, QRadar will still accept it and attempt to parse it, but in this case the source of the event will default to the IP address from which it was sent.

From QRadar version 7.4.0 onward, there is a Log Source Identifier property/field in the Log Activity event details view, so for events being collected by a SIM Generic log source, you can see what the system considered the source to be - this tells you what Log Source Identifier value to use when creating a log source to handle the event.

Benjamin: "From tcpdump the logs are arriving at the EP, however looking at the log activity tab, i can see the log source with SIM-generic to the EP, not parsed nor mapped. In addition i manually created the same log source, and the logs are not parsed nor mapped despite using the forcepoint v-series DSM"

If you can see the events in tcpdump from the EP and can see the events in Log Activity (regardless of which log source they are associated to), this means the events are being successfully sent to QRadar and read off the network. The fact that the events are being routed to SIM Generic means there is no log source configured with the correct Log Source Identifier. If there was such a log source, the events would be linked to it, even if the associated DSM couldn't parse the events (they would appear with category = Stored).

Benjamin, after you manually created the log source, are the events being linked to it as Stored, or are they still going to SIM generic? If still SIM Generic, it means you need to fix your Log Source Identifier. If they are with the correct log source but are Stored (and not parsed correctly), then there is a parsing problem occurring.

Benjamin: "I am surprised why i am getting another log source auto detected after creating a log source manaully. I thought QRadar is not meant to discover another log source since i have created one already. "

A few questions here:
1. What log source type is the autodetected log source, and what type is the manually created log source (I assume it's Forcepoint V Series, but just want to be sure)

2. Does the autodetected log source have the same Log Source Identifier as the manually created log source?

QRadar will autodetect log source(s) on the same Log Source Identifier as a manually created log source if events are coming into the system which no log source can successfully parse. For example, say you manually create a log source with LSI (Log Source Identifier) 10.10.10.10. Any events originating from that IP will be routed to that log source for attempted parsing. If the log source can successfully parse the events, the events are not considered for autodetection, they simply stay with that log source. However, if they are not parseable by the existing log source, they will appear in Log Activity as Stored events (meaning they were just stored, not parsed) and will also be analyzed for the purpose of autodetection. If enough such events are received which can be parsed by another log source type (DSM), then QRadar will autodetect a log source of that type for 10.10.10.10, with a lower parsing order than the manually created log source (meaning the original log source has higher precedence/priority for any events coming from 10.10.10.10). We do this because it is possible to have multiple log sources for the same LSI - there could be a log source for the OS (Linux, Windows, etc) and then additional log sources for applications, databases, etc running on that OS.

Benjamin: "The IP address on both log sources is the same."

When you say "the IP address is the same", you mean that the LSI is the same? If so this answers one of my questions above. But this suggests that they are not the same log source type, as QRadar does not allow two log sources of the same type to coexist on the same LSI.

Benjamin: "For the manually created log source, the payload is in LEEF format, but the values are empty, like src= dst= usrname=, while the log source that was auto detected, the payload seems fine. i.e. it has values like src=x.x.x.x dst=x.x.x.x usrname=xxx."

Ok so you're saying that both your manually configured log source and the autodetected log source are actively receiving events, and they're both LEEF? Then I assume the LEEF headers must be different, and that one log source (presumably the autodetected one) is parsing events successfully?

If the LEEF attributes for the manually created log source are empty, this could explain why it isn't parsing. Is the LEEF header also empty? This sounds like a configuration problem on the sending side where it's just sending bad data for some events. It would be really helpful to see a payload sample from each log source.

Karl: "that would be true if you disabled autodetection or changed log source parsing order"

Parsing order doesn't actually affect autodetection in any way, it just defines the order in which a set of log sources which share a Log Source Identifier attempt to parse an event which matches their LSI. The first one to successfully parse the events "wins" and any log sources lower in parsing precedence don't get the chance to try. As long as one log source can accept the event, the autodetection engine will never see it, it's only if all log sources reject it that the autodetection engine gets it.

Karl: "does logdata come in with IP in log header? pls double check with tcpdump. If not you need to specify hostname as logsource identifier rather than IP address"

To be clear, you only need to use the hostname if it is present in an RFC3164 or RFC5424-compliant syslog header. If there is no header or if the header is non-compliant, you need to use the IP address for the LSI, and if there is a compliant header, you need to use whatever value is in the header (could be IP address, could be hostname, could be something else, depending on how the sending system is configured)



My advice:

First, answers to some of the questions above would help me provide a clear explanation.
Second, a general bit of wisdom for cases like this is to first determine if the events are even entering the system (that has been done - we know they are), and if so, launch the DSM Editor with the intended log source type and some sample events from Log Activity to check if the intended DSM can handle the event format. The DSM Editor allows you to pass raw events directly to a target DSM, without needing to worry about Log Source Identifier misconfiguration, so it eliminates a variable. If the DSM Editor's Log Activity Preview says that the event format is good, then the problem is almost certainly a routing problem, where the log source's Log Source Identifier is set incorrectly. If the DSM Editor's preview indicates that the DSM cannot handle the events, then this is the root problem that will cause autodetection to fail and manually-created log sources of that type to fail, and it means the 3rd party system is sending data in a format that the DSM doesn't understand, which typically can be solved with a config change on the sending side.

Cheers
Colin

Hi Benjamin (and Mario and Karl),

There's a lot going on in this thread and some comments are completely accurate so I'm going to comment on several statements and then provide some additional advice.

Benjamin: "I checked the Console, using rpm -qa | grep -i forcepoint i saw the DSM-ForcepointVseries-********noarch. but i cant see this same dsm on my EP."

As Mario indicated, this is expected. DSM RPMs are only installed on the console and the artifacts they deliver are then deployed out to all managed hosts via deploy actions.

Mario: "if the logsource is SYSLOG based, the payload MUST adhere to the RFC164 or RFC5424"

This is not entirely correct. The syslog header on the events must conform to RFC 3164 or 5424 for QRadar to parse the syslog header and use the IP/hostname/identifier in it as the source of the event. This extracted value is used to "route" the event to a configured log source by matching it to a Log Source Identifier. If no matching log source exists, the event will appear in Log Activity associated with a SIM Generic Log log source, as it is a "catch all" bucket for any events which don't match the Log Source Identifier of any available log sources.

If a syslog event does not conform to the RFCs, QRadar will still accept it and attempt to parse it, but in this case the source of the event will default to the IP address from which it was sent.

From QRadar version 7.4.0 onward, there is a Log Source Identifier property/field in the Log Activity event details view, so for events being collected by a SIM Generic log source, you can see what the system considered the source to be - this tells you what Log Source Identifier value to use when creating a log source to handle the event.

Benjamin: "From tcpdump the logs are arriving at the EP, however looking at the log activity tab, i can see the log source with SIM-generic to the EP, not parsed nor mapped. In addition i manually created the same log source, and the logs are not parsed nor mapped despite using the forcepoint v-series DSM"

If you can see the events in tcpdump from the EP and can see the events in Log Activity (regardless of which log source they are associated to), this means the events are being successfully sent to QRadar and read off the network. The fact that the events are being routed to SIM Generic means there is no log source configured with the correct Log Source Identifier. If there was such a log source, the events would be linked to it, even if the associated DSM couldn't parse the events (they would appear with category = Stored).

Benjamin, after you manually created the log source, are the events being linked to it as Stored, or are they still going to SIM generic? If still SIM Generic, it means you need to fix your Log Source Identifier. If they are with the correct log source but are Stored (and not parsed correctly), then there is a parsing problem occurring.

Benjamin: "I am surprised why i am getting another log source auto detected after creating a log source manaully. I thought QRadar is not meant to discover another log source since i have created one already. "

A few questions here:
1. What log source type is the autodetected log source, and what type is the manually created log source (I assume it's Forcepoint V Series, but just want to be sure)

2. Does the autodetected log source have the same Log Source Identifier as the manually created log source?

QRadar will autodetect log source(s) on the same Log Source Identifier as a manually created log source if events are coming into the system which no log source can successfully parse. For example, say you manually create a log source with LSI (Log Source Identifier) 10.10.10.10. Any events originating from that IP will be routed to that log source for attempted parsing. If the log source can successfully parse the events, the events are not considered for autodetection, they simply stay with that log source. However, if they are not parseable by the existing log source, they will appear in Log Activity as Stored events (meaning they were just stored, not parsed) and will also be analyzed for the purpose of autodetection. If enough such events are received which can be parsed by another log source type (DSM), then QRadar will autodetect a log source of that type for 10.10.10.10, with a lower parsing order than the manually created log source (meaning the original log source has higher precedence/priority for any events coming from 10.10.10.10). We do this because it is possible to have multiple log sources for the same LSI - there could be a log source for the OS (Linux, Windows, etc) and then additional log sources for applications, databases, etc running on that OS.

Benjamin: "The IP address on both log sources is the same."

When you say "the IP address is the same", you mean that the LSI is the same? If so this answers one of my questions above. But this suggests that they are not the same log source type, as QRadar does not allow two log sources of the same type to coexist on the same LSI.

Benjamin: "For the manually created log source, the payload is in LEEF format, but the values are empty, like src= dst= usrname=, while the log source that was auto detected, the payload seems fine. i.e. it has values like src=x.x.x.x dst=x.x.x.x usrname=xxx."

Ok so you're saying that both your manually configured log source and the autodetected log source are actively receiving events, and they're both LEEF? Then I assume the LEEF headers must be different, and that one log source (presumably the autodetected one) is parsing events successfully?

If the LEEF attributes for the manually created log source are empty, this could explain why it isn't parsing. Is the LEEF header also empty? This sounds like a configuration problem on the sending side where it's just sending bad data for some events. It would be really helpful to see a payload sample from each log source.

Karl: "that would be true if you disabled autodetection or changed log source parsing order"

Parsing order doesn't actually affect autodetection in any way, it just defines the order in which a set of log sources which share a Log Source Identifier attempt to parse an event which matches their LSI. The first one to successfully parse the events "wins" and any log sources lower in parsing precedence don't get the chance to try. As long as one log source can accept the event, the autodetection engine will never see it, it's only if all log sources reject it that the autodetection engine gets it.

Karl: "does logdata come in with IP in log header? pls double check with tcpdump. If not you need to specify hostname as logsource identifier rather than IP address"

To be clear, you only need to use the hostname if it is present in an RFC3164 or RFC5424-compliant syslog header. If there is no header or if the header is non-compliant, you need to use the IP address for the LSI, and if there is a compliant header, you need to use whatever value is in the header (could be IP address, could be hostname, could be something else, depending on how the sending system is configured)



My advice:

First, answers to some of the questions above would help me provide a clear explanation.
Second, a general bit of wisdom for cases like this is to first determine if the events are even entering the system (that has been done - we know they are), and if so, launch the DSM Editor with the intended log source type and some sample events from Log Activity to check if the intended DSM can handle the event format. The DSM Editor allows you to pass raw events directly to a target DSM, without needing to worry about Log Source Identifier misconfiguration, so it eliminates a variable. If the DSM Editor's Log Activity Preview says that the event format is good, then the problem is almost certainly a routing problem, where the log source's Log Source Identifier is set incorrectly. If the DSM Editor's preview indicates that the DSM cannot handle the events, then this is the root problem that will cause autodetection to fail and manually-created log sources of that type to fail, and it means the 3rd party system is sending data in a format that the DSM doesn't understand, which typically can be solved with a config change on the sending side.

Cheers
Colin

Hi Benjamin (and Mario and Karl),

There's a lot going on in this thread and some comments are completely accurate so I'm going to comment on several statements and then provide some additional advice.

Benjamin: "I checked the Console, using rpm -qa | grep -i forcepoint i saw the DSM-ForcepointVseries-********noarch. but i cant see this same dsm on my EP."

As Mario indicated, this is expected. DSM RPMs are only installed on the console and the artifacts they deliver are then deployed out to all managed hosts via deploy actions.

Mario: "if the logsource is SYSLOG based, the payload MUST adhere to the RFC164 or RFC5424"

This is not entirely correct. The syslog header on the events must conform to RFC 3164 or 5424 for QRadar to parse the syslog header and use the IP/hostname/identifier in it as the source of the event. This extracted value is used to "route" the event to a configured log source by matching it to a Log Source Identifier. If no matching log source exists, the event will appear in Log Activity associated with a SIM Generic Log log source, as it is a "catch all" bucket for any events which don't match the Log Source Identifier of any available log sources.

If a syslog event does not conform to the RFCs, QRadar will still accept it and attempt to parse it, but in this case the source of the event will default to the IP address from which it was sent.

From QRadar version 7.4.0 onward, there is a Log Source Identifier property/field in the Log Activity event details view, so for events being collected by a SIM Generic log source, you can see what the system considered the source to be - this tells you what Log Source Identifier value to use when creating a log source to handle the event.

Benjamin: "From tcpdump the logs are arriving at the EP, however looking at the log activity tab, i can see the log source with SIM-generic to the EP, not parsed nor mapped. In addition i manually created the same log source, and the logs are not parsed nor mapped despite using the forcepoint v-series DSM"

If you can see the events in tcpdump from the EP and can see the events in Log Activity (regardless of which log source they are associated to), this means the events are being successfully sent to QRadar and read off the network. The fact that the events are being routed to SIM Generic means there is no log source configured with the correct Log Source Identifier. If there was such a log source, the events would be linked to it, even if the associated DSM couldn't parse the events (they would appear with category = Stored).

Benjamin, after you manually created the log source, are the events being linked to it as Stored, or are they still going to SIM generic? If still SIM Generic, it means you need to fix your Log Source Identifier. If they are with the correct log source but are Stored (and not parsed correctly), then there is a parsing problem occurring.

Benjamin: "I am surprised why i am getting another log source auto detected after creating a log source manaully. I thought QRadar is not meant to discover another log source since i have created one already. "

A few questions here:
1. What log source type is the autodetected log source, and what type is the manually created log source (I assume it's Forcepoint V Series, but just want to be sure)

2. Does the autodetected log source have the same Log Source Identifier as the manually created log source?

QRadar will autodetect log source(s) on the same Log Source Identifier as a manually created log source if events are coming into the system which no log source can successfully parse. For example, say you manually create a log source with LSI (Log Source Identifier) 10.10.10.10. Any events originating from that IP will be routed to that log source for attempted parsing. If the log source can successfully parse the events, the events are not considered for autodetection, they simply stay with that log source. However, if they are not parseable by the existing log source, they will appear in Log Activity as Stored events (meaning they were just stored, not parsed) and will also be analyzed for the purpose of autodetection. If enough such events are received which can be parsed by another log source type (DSM), then QRadar will autodetect a log source of that type for 10.10.10.10, with a lower parsing order than the manually created log source (meaning the original log source has higher precedence/priority for any events coming from 10.10.10.10). We do this because it is possible to have multiple log sources for the same LSI - there could be a log source for the OS (Linux, Windows, etc) and then additional log sources for applications, databases, etc running on that OS.

Benjamin: "The IP address on both log sources is the same."

When you say "the IP address is the same", you mean that the LSI is the same? If so this answers one of my questions above. But this suggests that they are not the same log source type, as QRadar does not allow two log sources of the same type to coexist on the same LSI.

Benjamin: "For the manually created log source, the payload is in LEEF format, but the values are empty, like src= dst= usrname=, while the log source that was auto detected, the payload seems fine. i.e. it has values like src=x.x.x.x dst=x.x.x.x usrname=xxx."

Ok so you're saying that both your manually configured log source and the autodetected log source are actively receiving events, and they're both LEEF? Then I assume the LEEF headers must be different, and that one log source (presumably the autodetected one) is parsing events successfully?

If the LEEF attributes for the manually created log source are empty, this could explain why it isn't parsing. Is the LEEF header also empty? This sounds like a configuration problem on the sending side where it's just sending bad data for some events. It would be really helpful to see a payload sample from each log source.

Karl: "that would be true if you disabled autodetection or changed log source parsing order"

Parsing order doesn't actually affect autodetection in any way, it just defines the order in which a set of log sources which share a Log Source Identifier attempt to parse an event which matches their LSI. The first one to successfully parse the events "wins" and any log sources lower in parsing precedence don't get the chance to try. As long as one log source can accept the event, the autodetection engine will never see it, it's only if all log sources reject it that the autodetection engine gets it.

Karl: "does logdata come in with IP in log header? pls double check with tcpdump. If not you need to specify hostname as logsource identifier rather than IP address"

To be clear, you only need to use the hostname if it is present in an RFC3164 or RFC5424-compliant syslog header. If there is no header or if the header is non-compliant, you need to use the IP address for the LSI, and if there is a compliant header, you need to use whatever value is in the header (could be IP address, could be hostname, could be something else, depending on how the sending system is configured)



My advice:

First, answers to some of the questions above would help me provide a clear explanation.
Second, a general bit of wisdom for cases like this is to first determine if the events are even entering the system (that has been done - we know they are), and if so, launch the DSM Editor with the intended log source type and some sample events from Log Activity to check if the intended DSM can handle the event format. The DSM Editor allows you to pass raw events directly to a target DSM, without needing to worry about Log Source Identifier misconfiguration, so it eliminates a variable. If the DSM Editor's Log Activity Preview says that the event format is good, then the problem is almost certainly a routing problem, where the log source's Log Source Identifier is set incorrectly. If the DSM Editor's preview indicates that the DSM cannot handle the events, then this is the root problem that will cause autodetection to fail and manually-created log sources of that type to fail, and it means the 3rd party system is sending data in a format that the DSM doesn't understand, which typically can be solved with a config change on the sending side.

Cheers
Colin