IBM QRadar

Hello Everyone,

I am confused about Data retention on QRadar. When I configured EVent Retention with "When storage space is required" policy, I can't find the threshold option such as 80% or 90%. I don't know what the default storage to delete is. And if event is stored for 2 years and I configure retention is 6 months, What will happen ? It will delete the oldest six months or the most recent six months ?

Thanks a lot.

There is monitoring running in the back checking for disk space usage on multiple partitions. Collected Events and Flows are kept under /store/ariel/. Also, by default backups are placed under /store/backup. For these partitions you get a warning for over 90% usage. At 95% usage some services might shut down. Have a look at :
https://www.ibm.com/support/pages/qradar-troubleshooting-disk-space-usage-problems
https://www.ibm.com/community/101/qradar/diskspace/
https://www.ibm.com/docs/en/SS42VS_7.5/pdf/b_qradar_system_notifications.pdf
(Note: backups will stop at 90% space usage on the partition)

Hello Everyone,

I am confused about Data retention on QRadar. When I configured EVent Retention with "When storage space is required" policy, I can't find the threshold option such as 80% or 90%. I don't know what the default storage to delete is. And if event is stored for 2 years and I configure retention is 6 months, What will happen ? It will delete the oldest six months or the most recent six months ?

Thanks a lot.

Hi Dusan,

It means Qradar keeps online date (/store/ariel/) under 90% (default) if I configure Event Retention with "When storage space is required", is that right ? And if event is stored for 2 years and I configure retention is 6 months, What will happen ? It will delete the oldest six months or the most recent six months ? Please let me know.

Thanks a lot

There is monitoring running in the back checking for disk space usage on multiple partitions. Collected Events and Flows are kept under /store/ariel/. Also, by default backups are placed under /store/backup. For these partitions you get a warning for over 90% usage. At 95% usage some services might shut down. Have a look at :
https://www.ibm.com/support/pages/qradar-troubleshooting-disk-space-usage-problems
https://www.ibm.com/community/101/qradar/diskspace/
https://www.ibm.com/docs/en/SS42VS_7.5/pdf/b_qradar_system_notifications.pdf
(Note: backups will stop at 90% space usage on the partition)

Hello Everyone,

I am confused about Data retention on QRadar. When I configured EVent Retention with "When storage space is required" policy, I can't find the threshold option such as 80% or 90%. I don't know what the default storage to delete is. And if event is stored for 2 years and I configure retention is 6 months, What will happen ? It will delete the oldest six months or the most recent six months ?

Thanks a lot.

Hi Hung,

Select When storage space is required to keep data that matches the Keep data placed in this bucket for parameter in storage until the disk monitoring system detects that storage is required.

Deletions that are based on storage space begin when the free disk space drops to 15% or less, and the deletions continue until the free disk space is 18% or the policy time frame that is set in the Keep data placed in this bucket for field runs out. For example, if the used disk space reaches 85% for records, data is deleted until the used percentage drops to 82%. When storage is required, only data that matches the Keep data placed in this bucket for field is deleted.

Product Documentation: https://www.ibm.com/docs/en/qsip/7.5?topic=retention-configuring-buckets

Hope it helps.

Hi Dusan,

It means Qradar keeps online date (/store/ariel/) under 90% (default) if I configure Event Retention with "When storage space is required", is that right ? And if event is stored for 2 years and I configure retention is 6 months, What will happen ? It will delete the oldest six months or the most recent six months ? Please let me know.

Thanks a lot

There is monitoring running in the back checking for disk space usage on multiple partitions. Collected Events and Flows are kept under /store/ariel/. Also, by default backups are placed under /store/backup. For these partitions you get a warning for over 90% usage. At 95% usage some services might shut down. Have a look at :
https://www.ibm.com/support/pages/qradar-troubleshooting-disk-space-usage-problems
https://www.ibm.com/community/101/qradar/diskspace/
https://www.ibm.com/docs/en/SS42VS_7.5/pdf/b_qradar_system_notifications.pdf
(Note: backups will stop at 90% space usage on the partition)

Hello Everyone,

I am confused about Data retention on QRadar. When I configured EVent Retention with "When storage space is required" policy, I can't find the threshold option such as 80% or 90%. I don't know what the default storage to delete is. And if event is stored for 2 years and I configure retention is 6 months, What will happen ? It will delete the oldest six months or the most recent six months ?

Thanks a lot.

And just a note: changing a retention for a bucket (say from 12 to 6 months) will not be retroactively applied on the already ingested data, but on the data from that point when the change was made forward. 

Hi Hung,

Select When storage space is required to keep data that matches the Keep data placed in this bucket for parameter in storage until the disk monitoring system detects that storage is required.

Deletions that are based on storage space begin when the free disk space drops to 15% or less, and the deletions continue until the free disk space is 18% or the policy time frame that is set in the Keep data placed in this bucket for field runs out. For example, if the used disk space reaches 85% for records, data is deleted until the used percentage drops to 82%. When storage is required, only data that matches the Keep data placed in this bucket for field is deleted.

Product Documentation: https://www.ibm.com/docs/en/qsip/7.5?topic=retention-configuring-buckets

Hope it helps.

Hi Dusan,

It means Qradar keeps online date (/store/ariel/) under 90% (default) if I configure Event Retention with "When storage space is required", is that right ? And if event is stored for 2 years and I configure retention is 6 months, What will happen ? It will delete the oldest six months or the most recent six months ? Please let me know.

Thanks a lot

There is monitoring running in the back checking for disk space usage on multiple partitions. Collected Events and Flows are kept under /store/ariel/. Also, by default backups are placed under /store/backup. For these partitions you get a warning for over 90% usage. At 95% usage some services might shut down. Have a look at :
https://www.ibm.com/support/pages/qradar-troubleshooting-disk-space-usage-problems
https://www.ibm.com/community/101/qradar/diskspace/
https://www.ibm.com/docs/en/SS42VS_7.5/pdf/b_qradar_system_notifications.pdf
(Note: backups will stop at 90% space usage on the partition)

Hello Everyone,

I am confused about Data retention on QRadar. When I configured EVent Retention with "When storage space is required" policy, I can't find the threshold option such as 80% or 90%. I don't know what the default storage to delete is. And if event is stored for 2 years and I configure retention is 6 months, What will happen ? It will delete the oldest six months or the most recent six months ?

Thanks a lot.

Actually - that is not correct.

QRadar decides which bucket to place an event at ingestion time - that does not get applied retroactively.  (I.E. if you change a log source's bucket from bucket A to bucket B then the events already in bucket A will not be moved to bucket B)

But the retention period used per bucket is always the 'current' retention period for that bucket.  If you change the retention period of bucket A from 12 to 6 months then ALL of the data in bucket A will be eligible for deletion after 6 months.  (whether it actually gets deleted or not will depend on what the bucket settings are and how full the disk is).

QRadar always deletes the oldest data first.

Paul

And just a note: changing a retention for a bucket (say from 12 to 6 months) will not be retroactively applied on the already ingested data, but on the data from that point when the change was made forward. 

Hi Hung,

Select When storage space is required to keep data that matches the Keep data placed in this bucket for parameter in storage until the disk monitoring system detects that storage is required.

Deletions that are based on storage space begin when the free disk space drops to 15% or less, and the deletions continue until the free disk space is 18% or the policy time frame that is set in the Keep data placed in this bucket for field runs out. For example, if the used disk space reaches 85% for records, data is deleted until the used percentage drops to 82%. When storage is required, only data that matches the Keep data placed in this bucket for field is deleted.

Product Documentation: https://www.ibm.com/docs/en/qsip/7.5?topic=retention-configuring-buckets

Hope it helps.

Hi Dusan,

It means Qradar keeps online date (/store/ariel/) under 90% (default) if I configure Event Retention with "When storage space is required", is that right ? And if event is stored for 2 years and I configure retention is 6 months, What will happen ? It will delete the oldest six months or the most recent six months ? Please let me know.

Thanks a lot

There is monitoring running in the back checking for disk space usage on multiple partitions. Collected Events and Flows are kept under /store/ariel/. Also, by default backups are placed under /store/backup. For these partitions you get a warning for over 90% usage. At 95% usage some services might shut down. Have a look at :
https://www.ibm.com/support/pages/qradar-troubleshooting-disk-space-usage-problems
https://www.ibm.com/community/101/qradar/diskspace/
https://www.ibm.com/docs/en/SS42VS_7.5/pdf/b_qradar_system_notifications.pdf
(Note: backups will stop at 90% space usage on the partition)

Hello Everyone,

I am confused about Data retention on QRadar. When I configured EVent Retention with "When storage space is required" policy, I can't find the threshold option such as 80% or 90%. I don't know what the default storage to delete is. And if event is stored for 2 years and I configure retention is 6 months, What will happen ? It will delete the oldest six months or the most recent six months ?

Thanks a lot.

I have created a new bucket with lesser retention period specific for an event processor, will it be retroactively applied.

Actually - that is not correct.

QRadar decides which bucket to place an event at ingestion time - that does not get applied retroactively.  (I.E. if you change a log source's bucket from bucket A to bucket B then the events already in bucket A will not be moved to bucket B)

But the retention period used per bucket is always the 'current' retention period for that bucket.  If you change the retention period of bucket A from 12 to 6 months then ALL of the data in bucket A will be eligible for deletion after 6 months.  (whether it actually gets deleted or not will depend on what the bucket settings are and how full the disk is).

QRadar always deletes the oldest data first.

Paul

And just a note: changing a retention for a bucket (say from 12 to 6 months) will not be retroactively applied on the already ingested data, but on the data from that point when the change was made forward. 

Hi Hung,

Select When storage space is required to keep data that matches the Keep data placed in this bucket for parameter in storage until the disk monitoring system detects that storage is required.

Deletions that are based on storage space begin when the free disk space drops to 15% or less, and the deletions continue until the free disk space is 18% or the policy time frame that is set in the Keep data placed in this bucket for field runs out. For example, if the used disk space reaches 85% for records, data is deleted until the used percentage drops to 82%. When storage is required, only data that matches the Keep data placed in this bucket for field is deleted.

Product Documentation: https://www.ibm.com/docs/en/qsip/7.5?topic=retention-configuring-buckets

Hope it helps.

Hi Dusan,

It means Qradar keeps online date (/store/ariel/) under 90% (default) if I configure Event Retention with "When storage space is required", is that right ? And if event is stored for 2 years and I configure retention is 6 months, What will happen ? It will delete the oldest six months or the most recent six months ? Please let me know.

Thanks a lot

There is monitoring running in the back checking for disk space usage on multiple partitions. Collected Events and Flows are kept under /store/ariel/. Also, by default backups are placed under /store/backup. For these partitions you get a warning for over 90% usage. At 95% usage some services might shut down. Have a look at :
https://www.ibm.com/support/pages/qradar-troubleshooting-disk-space-usage-problems
https://www.ibm.com/community/101/qradar/diskspace/
https://www.ibm.com/docs/en/SS42VS_7.5/pdf/b_qradar_system_notifications.pdf
(Note: backups will stop at 90% space usage on the partition)

Hello Everyone,

I am confused about Data retention on QRadar. When I configured EVent Retention with "When storage space is required" policy, I can't find the threshold option such as 80% or 90%. I don't know what the default storage to delete is. And if event is stored for 2 years and I configure retention is 6 months, What will happen ? It will delete the oldest six months or the most recent six months ?

Thanks a lot.