IBM Security QRadar

 View Only
  • 1.  Put data log on different storage tiers based on retention

    Posted Wed April 10, 2024 11:06 AM

    Hello everyone,

    I'm new to QRadar and I need to check if there is a product feature or configuration that allow me do the following:

    I need to keep up data logs with one year of retention, but I want to save the last month on a storage tier based on flash disks, the other eleven months on a storage tier based on traditional, rotative disks.

    Do you know if it is something duable and, if affermative, can you give me any directions?

    Thanks a lot



    ------------------------------
    Fabio Guzzi
    ------------------------------



  • 2.  RE: Put data log on different storage tiers based on retention

    Posted Wed April 10, 2024 05:16 PM
    Edited by Jonathan Pechta Wed April 10, 2024 05:16 PM

    I'm not aware of an easy method to do this in QRadar. The events and flows are kept in /store/ariel/ paths and stored locally on the appliance that parses, indexes, and stores the events to be searched. The only option I've seen around this type of feature in our docs is for Fibre Channel multipath offboard storage: https://www.ibm.com/docs/en/qsip/7.5?topic=fcs-moving-store-file-system-multipath-fibre-channel-solution

    However, this option reduces storage redundancy and not what you are looking for as a method to reduce costs by moving /store to different storage performance pricing/tiers. I'm not aware of any other supported option other than the Fibre Channel solution that is documented.

    Typically, unless it is documented it is considered unsupported. 



    ------------------------------
    Jonathan Pechta
    IBM Security - Community of Practice Lead
    jonathan.pechta1@ibm.com
    ------------------------------



  • 3.  RE: Put data log on different storage tiers based on retention

    Posted Thu April 11, 2024 02:23 AM

    Thank you Jonathan!

    I was thinking to create two QR deployments (A and B) and configure a 30-days bucket on the first one and 365-bucket on the second one.

    QR deployment A would be provisioned with SDD storage while QR deployment B with traditional HDD 

    The same log source would forward data to two event collectors, the first one becoming to QR deployment A while the second one becoming to QR deployment B.

    I'm aware this way I would introduce some inefficiency (the first month would be on both deployments) but the advantages in terms of storage costs would be still relevant.

    I would use QR deployment A for short term query (80% in my case) and QR deployment B for long term query (less frequent and less urgent so I can accept less performance).

    Does it make sense?

    Thanks  



    ------------------------------
    Fabio Guzzi
    ------------------------------



  • 4.  RE: Put data log on different storage tiers based on retention

    Posted Thu April 11, 2024 03:15 AM

    Hi Fabio,

    What you describe in you're reply above with 2 deployments would achieve what you want.   However you will require 2 sets of QRadar licenses.

    Also if both systems are required to generate offenses and be used for monitoring then you would need to ensure that you have an effective Change Control process in place.

    If A is only going to be used for monitoring and B for searching then there would not be a need to keep the rules in sync, you would however have to keep any CEP's in sync.

    Also would there be any pull logsources in the deployment or would they all be push?

    Thanks



    ------------------------------
    John Dawson
    Qradar Support Architect
    IBM
    ------------------------------



  • 5.  RE: Put data log on different storage tiers based on retention

    Posted Thu April 11, 2024 03:40 AM
    Edited by Fabio Guzzi Thu April 11, 2024 03:41 AM

    Thanks for your useful feedback John.

    I will take into consideration what you mentioned about impacts on licenses and rules.

    In terms of logsources types, currently I have only push logsources but I cannot exclude future integrations with pull logsources.

    Can you see any issues in case of pull logsources?

    Thanks



    ------------------------------
    Fabio Guzzi
    ------------------------------



  • 6.  RE: Put data log on different storage tiers based on retention

    IBM Champion
    Posted Thu April 18, 2024 08:08 AM

    A solution that may be less expensive than two QR systems would be to use two data nodes or event processors for storage, set the retention policies different on each.  Problem is it would balance storage between the datanodes.  If you did multiple EP's that may do it. 



    ------------------------------
    Frank Eargle
    IBM QRadar Champion
    ------------------------------



  • 7.  RE: Put data log on different storage tiers based on retention

    Posted Thu April 18, 2024 12:17 PM

    Thank you Frank, I agree better with only one QR deployment!

    Not sure I got what you mean when you say "Problem is it would balance storage between the datanodes", could you please further explain?

    Another question is: is there a way to connect the same EC to multiple EPs and send them the same log record? That way I could act on collector side only, avoiding to change the log source configuration with multiple forward directives.

    Thanks

    Fabio



    ------------------------------
    Fabio Guzzi
    ------------------------------



  • 8.  RE: Put data log on different storage tiers based on retention

    Posted Thu April 18, 2024 12:28 PM

    Hi Fabio,

    Unfortunately this will not work.  Retention buckets/Policies only work on logsources or logsource groups.  They do not work on individual boxes. 

    You say that you need a months data on fast storage and the rest on cheaper storage.  Will it be the months worth of data that is being searched and the rest is for compliance reasons?  If so you could take data backups, store them off board and restore them if a certain period of data, outside of the month stored on fast storage, needs to be searched. 

    The Backup and Recovery documentation can be found here - https://www.ibm.com/docs/en/qsip/7.5?topic=administration-backup-recovery

    Thanks



    ------------------------------
    John Dawson
    Qradar Support Architect
    IBM
    ------------------------------



  • 9.  RE: Put data log on different storage tiers based on retention

    Posted Thu April 18, 2024 01:52 PM

    As John points out, that isn't possible as there is no 'per-host' retention policy mechanism.

    What you could do would be to have an EP with 'fast' storage and a Data Node attached to it with 'slow' storage.  Set the Data Node to 'Archive Mode' and manually move the data from the EP to the DN when it becomes a month old.

    It would be a custom solution, but it would achieve your requirements.

    Paul



    ------------------------------
    Paul Ford-Hutchinson
    ------------------------------