IBM Security QRadar

Hi Everyone,

I have a question about Compare Field.

In the documents as we all know, It said:
A numeric value or time stamp field from the table or view that identifies
new events that are added to the table between queries. Enables the
protocol to identify events that were previously polled by the protocol to
ensure that duplicate events are not created.

However, is there any explanation about the situation that if two columns (ID and Time Stamp) exist, which one is best to write in Compare Field area?

Thanks in advance.

------------------------------
Halil BALIM
------------------------------

Hi Halil,

In my opinion, when there is a "time stamp", use it. For my experience, this works best.

Best regards,
Kiril

------------------------------
Kiril Bonev
System Specialist
CNsys PLC
Sofia
+359 2 958 36 00
------------------------------

Original Message:
Sent: Thu March 26, 2020 03:33 AM
From: Halil BALIM
Subject: Compare Field

Hi Everyone,

I have a question about Compare Field.

In the documents as we all know, It said:
A numeric value or time stamp field from the table or view that identifies
new events that are added to the table between queries. Enables the
protocol to identify events that were previously polled by the protocol to
ensure that duplicate events are not created.

However, is there any explanation about the situation that if two columns (ID and Time Stamp) exist, which one is best to write in Compare Field area?

Thanks in advance.

------------------------------
Halil BALIM
------------------------------

Hi Halil,

It depends a bit on the nature of each of the columns (ID and timestamp). If the timestamp is in a unit/precision of minutes, it's not so good to use, because if the JDBC query runs and a new row is then added for the same minute, the protocol will miss that event on the next poll. If the timestamp is accurate to a particular second, it's much less likely that you'll miss any data. But in this sense an ID column (providing that it is a numeric counter and not some kind of UUID or other non-sequential value) is the best option because it will *always* increment with each row added to the table, so you will never ever miss data.

So generally I'd recommend using ID. The only drawback of an ID is if its max size is small and there's a chance it will rollover back to 1 - then you'll stop getting new data entirely - this will obviously never happen with a timestamp. But if you don't expect the ID to ever roll over due to its max size and the rate at which stuff is added to the table, I still prefer ID because it's most accurate and will never be duplicated across multiple rows.

Cheers
Colin

------------------------------
COLIN HAY
------------------------------

Original Message:
Sent: Thu March 26, 2020 03:33 AM
From: Halil BALIM
Subject: Compare Field

Hi Everyone,

I have a question about Compare Field.

In the documents as we all know, It said:
A numeric value or time stamp field from the table or view that identifies
new events that are added to the table between queries. Enables the
protocol to identify events that were previously polled by the protocol to
ensure that duplicate events are not created.

However, is there any explanation about the situation that if two columns (ID and Time Stamp) exist, which one is best to write in Compare Field area?

Thanks in advance.

------------------------------
Halil BALIM
------------------------------

Hi Colin,

Thank you for your clear explanation.

Regards.

------------------------------
Halil BALIM
------------------------------

Original Message:
Sent: Fri March 27, 2020 11:45 AM
From: COLIN HAY
Subject: Compare Field

Hi Halil,

It depends a bit on the nature of each of the columns (ID and timestamp). If the timestamp is in a unit/precision of minutes, it's not so good to use, because if the JDBC query runs and a new row is then added for the same minute, the protocol will miss that event on the next poll. If the timestamp is accurate to a particular second, it's much less likely that you'll miss any data. But in this sense an ID column (providing that it is a numeric counter and not some kind of UUID or other non-sequential value) is the best option because it will *always* increment with each row added to the table, so you will never ever miss data.

So generally I'd recommend using ID. The only drawback of an ID is if its max size is small and there's a chance it will rollover back to 1 - then you'll stop getting new data entirely - this will obviously never happen with a timestamp. But if you don't expect the ID to ever roll over due to its max size and the rate at which stuff is added to the table, I still prefer ID because it's most accurate and will never be duplicated across multiple rows.

Cheers
Colin

------------------------------
COLIN HAY

Original Message:
Sent: Thu March 26, 2020 03:33 AM
From: Halil BALIM
Subject: Compare Field

Hi Everyone,

I have a question about Compare Field.

In the documents as we all know, It said:
A numeric value or time stamp field from the table or view that identifies
new events that are added to the table between queries. Enables the
protocol to identify events that were previously polled by the protocol to
ensure that duplicate events are not created.

However, is there any explanation about the situation that if two columns (ID and Time Stamp) exist, which one is best to write in Compare Field area?

Thanks in advance.

------------------------------
Halil BALIM
------------------------------

Hi Colin,

Is there a way we can completely ignore the compare field when query from Qradar. I have a table in MSSQL which refreshes every day. From Qradar I just want it to run query like "select * from table" without compare field during every fetch, is it possible?

------------------------------
Pravin Ayare
------------------------------

Original Message:
Sent: Fri March 27, 2020 11:45 AM
From: COLIN HAY
Subject: Compare Field

Hi Halil,

It depends a bit on the nature of each of the columns (ID and timestamp). If the timestamp is in a unit/precision of minutes, it's not so good to use, because if the JDBC query runs and a new row is then added for the same minute, the protocol will miss that event on the next poll. If the timestamp is accurate to a particular second, it's much less likely that you'll miss any data. But in this sense an ID column (providing that it is a numeric counter and not some kind of UUID or other non-sequential value) is the best option because it will *always* increment with each row added to the table, so you will never ever miss data.

So generally I'd recommend using ID. The only drawback of an ID is if its max size is small and there's a chance it will rollover back to 1 - then you'll stop getting new data entirely - this will obviously never happen with a timestamp. But if you don't expect the ID to ever roll over due to its max size and the rate at which stuff is added to the table, I still prefer ID because it's most accurate and will never be duplicated across multiple rows.

Cheers
Colin

------------------------------
COLIN HAY

Original Message:
Sent: Thu March 26, 2020 03:33 AM
From: Halil BALIM
Subject: Compare Field

Hi Everyone,

I have a question about Compare Field.

In the documents as we all know, It said:
A numeric value or time stamp field from the table or view that identifies
new events that are added to the table between queries. Enables the
protocol to identify events that were previously polled by the protocol to
ensure that duplicate events are not created.

However, is there any explanation about the situation that if two columns (ID and Time Stamp) exist, which one is best to write in Compare Field area?

Thanks in advance.

------------------------------
Halil BALIM
------------------------------

Hi Pravin,

Currently it is not possible to configure a log source with the JDBC protocol type and no "Compare Field" set. However, it would be possible to adjust the protocol to make that field optional to enable this type of use case. I'd suggest you log a Request For Enhancement (see https://www.ibm.com/support/pages/submit-request-enhancement-rfe) for this and comment on it to request that it be made public so other users can upvote the request.

Cheers
Colin

------------------------------
COLIN HAY
IBM Security
------------------------------

Original Message:
Sent: Wed July 15, 2020 07:58 AM
From: Pravin Ayare
Subject: Compare Field

Hi Colin,

Is there a way we can completely ignore the compare field when query from Qradar. I have a table in MSSQL which refreshes every day. From Qradar I just want it to run query like "select * from table" without compare field during every fetch, is it possible?

------------------------------
Pravin Ayare

Original Message:
Sent: Fri March 27, 2020 11:45 AM
From: COLIN HAY
Subject: Compare Field

Hi Halil,

It depends a bit on the nature of each of the columns (ID and timestamp). If the timestamp is in a unit/precision of minutes, it's not so good to use, because if the JDBC query runs and a new row is then added for the same minute, the protocol will miss that event on the next poll. If the timestamp is accurate to a particular second, it's much less likely that you'll miss any data. But in this sense an ID column (providing that it is a numeric counter and not some kind of UUID or other non-sequential value) is the best option because it will *always* increment with each row added to the table, so you will never ever miss data.

So generally I'd recommend using ID. The only drawback of an ID is if its max size is small and there's a chance it will rollover back to 1 - then you'll stop getting new data entirely - this will obviously never happen with a timestamp. But if you don't expect the ID to ever roll over due to its max size and the rate at which stuff is added to the table, I still prefer ID because it's most accurate and will never be duplicated across multiple rows.

Cheers
Colin

------------------------------
COLIN HAY

Original Message:
Sent: Thu March 26, 2020 03:33 AM
From: Halil BALIM
Subject: Compare Field

Hi Everyone,

I have a question about Compare Field.

In the documents as we all know, It said:
A numeric value or time stamp field from the table or view that identifies
new events that are added to the table between queries. Enables the
protocol to identify events that were previously polled by the protocol to
ensure that duplicate events are not created.

However, is there any explanation about the situation that if two columns (ID and Time Stamp) exist, which one is best to write in Compare Field area?

Thanks in advance.

------------------------------
Halil BALIM
------------------------------