IBM Security QRadar

 View Only

  • 1.  Cleaning up historical correlation catalogs

    Posted Thu September 09, 2021 09:42 AM

    When using QRadar's historical correlation, it seems that catalogs are not cleaned up even when deleting the historical correlation profile.

    Specifically, if I do the following:

    1. Create a saved search and a historical correlation profile that is expected to return results

    2. Run the HC profile that was created in step 1

    3. Click View History on the HC profile that was created in step 1, and observe the catalog name for the Run that is shown

    4. Close the View History dialog

    5. Delete the HC profile that was created in step 1

    6. Go to Log Activity and run the query select * from HC_XXX, where HC_XXX is the catalog name that was observed in step 3

    7. Observe that results from the catalog are returned, even though the historical correlation has been deleted


    Given the above:

    1. Where are the HC catalogs actually stored? Do they take up space?

    2. What is the proper way of cleaning up a HC profile along with its associated catalogs?



    ------------------------------
    Yair Manor
    ------------------------------


  • 2.  RE: Cleaning up historical correlation catalogs

    Posted Wed November 03, 2021 11:19 AM

    A Historical Correlation profile is in essence a non-aggregated saved search, a list of enabled rules, scheduling, and metadata information. The profile is stored in the database and contains the parameters for the data stored within QRadar. HC uses the existing data within QRadar and uses the existing events run the data through the rules to generate historical correlation offenses.

    I would confirm if the profile is deleted from the API to confirm the profile is removed.

    GET /api/historical_correlation/profiles

    I would think that these would be cleaned up at the top of the hour, but this might also be referencing the saved search that is still available. You might confirm if you still have managed search results (Log Activity > New Search > Manage Search Results). The profile is stored in the database and if you've deleted your profile and the data still displays, it might need to age out or this might be a defect of some sort. You didn't mention your version, but if you are still seeing issues. I would start by confirming if the profile is available in the API after it is deleted from the user interface. If the API still lists the name of your HC profile after removal from the UI, then it is likely a defect that should be logged with support. You didn't mention your version, but if it continues, open a case.



    ------------------------------
    Jonathan Pechta
    QRadar Support Content Lead
    Support forums: ibm.biz/qradarforums
    jonathan.pechta1@ibm.com
    ------------------------------

    Original Message


  • 3.  RE: Cleaning up historical correlation catalogs

    Posted Wed November 03, 2021 11:36 AM
    Thank you @Jonathan Pechta - I have confirmed that the HC Profile is not available via API, the saved search is removed, and Managed Search Results don't show anything. I have waited for over 1 month (which is the retention period of searches) and the data still persists. I have a support case open which has recently been escalated to development as a defect.

    I'll keep this thread updated with the results.​

    ------------------------------
    Yair Manor
    ------------------------------

    Original Message


  • 4.  RE: Cleaning up historical correlation catalogs

    Posted Fri November 19, 2021 01:11 PM
    To close the loop following the resolution of the support case I opened with IBM:

    1. HC output files are stored in /store/hc and can take up a significant amount of space depending on the HC results.
    2. DB entries for HC runs are removed from the DB 15 days after their HC profile is deleted. However, the files in /store/hc are retained beyond that.
    3. QRadar runs a periodic cleanup (by default: once per day) which will clean up the /store partition once it crosses 85% disk usage. IBM claim that when that happens, the cleanup process will remove run results of HC runs that have been deleted and removed from the DB. It will not remove results of HC runs whose profile still exists or whose profile was deleted less than 15 days ago.
    4. In my own tests, when I manually filled up the partition, all HC results under /store/hc were removed, including for HC profiles which had not been deleted. I am not sure if this behavior is by design or a defect, although from my perspective it is acceptable.


    ------------------------------
    Yair Manor
    ------------------------------

    Original Message