IBM QRadar SOAR

IBM QRadar SOAR

Join this online user group to communicate across Security product users and IBM experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only

  • 1.  Bulk deleting incidents in Resilient

    Posted Fri October 29, 2021 05:23 AM

    Hello

    I need to be able to delete all incidents in Resilient.

    I tried using the following API endpoint, but if I provide more than a few 100 incident ids it takes forever. I tried providing around 7000 incident ids and Resilient stopped responding.


    Is there a way to clear the incident database quickly? We need to do this more than once, since we clone our production environment to a test environment from time to time, but are not allowed to have incident data there.

    Right now the deletion through API takes hours and we have to keep it running over night.



    ------------------------------
    Andreas Fiehn
    ------------------------------


  • 2.  RE: Bulk deleting incidents in Resilient

    Posted Mon November 01, 2021 07:49 AM
    I don't have a specific answer, but some thoughts:

    1) I think when this happens most of the time is taken doing Rules and Playbook Processing. You could try disabling all the Rules and Playbooks before incident deletion. 
    2) Keeping the batch size down will increase throughput.
    3) It sounds like you are trying to create the same environment in your lab system as production but without data. If so, what about just doing an Export of the configuration data and importing it to a new organization in the lab? 

    Ben

    ------------------------------
    Ben Lurie
    ------------------------------

    Original Message


  • 3.  RE: Bulk deleting incidents in Resilient

    Posted Wed November 03, 2021 06:24 AM

    Thank you Ben!

    It is exactly our what we are trying to do and we will take a look at the Export of configuration.Thank you


    Sometimes we need an exact copy of everything, so it would be nice with a fast way to make deletions. Do you know if there is a way to disable all the rules through API?



    ------------------------------
    Andreas Fiehn
    ------------------------------

    Original Message


  • 4.  RE: Bulk deleting incidents in Resilient

    Posted Mon November 01, 2021 08:26 AM
    Hi Andreas, I have the same problem here. SOAR is incredibly slow to create/delete incidents, I really don't know why.

    Using the REST API, from my experience, you don't add much throughput for thread pools > 5.

    One thing I'd try is to dump the database contents before removing one incident, then remove it, and then take another dump and compare both, so you could figure out yourself what tables are affected and how.

    Then you could just write your own set of SQL commands to run, after taking down all the resilient services first, of course. You can access the DB directly using a command such as sudo -u postgres -i psql co3

    Not an easy task, but without a really fast batch delete feature, this is the best idea that comes to my mind.

    Of course, this is not a procedure recommended by IBM, take it on your own risk.

    ------------------------------
    Leonardo Kenji Shikida
    ------------------------------

    Original Message


  • 5.  RE: Bulk deleting incidents in Resilient

    Posted Wed November 03, 2021 06:30 AM

    Hi Leonardo,
    Thanks for the reply! I guess it could work, although it will probably take some time to set up :)



    ------------------------------
    Andreas Fiehn
    ------------------------------

    Original Message


  • 6.  RE: Bulk deleting incidents in Resilient

    Posted Thu November 04, 2021 04:31 AM

    Hi,

    Like Ben, if you wish to get a copy of your prod, with no incident, I suggest you install a new SOAR, same version, and export all the configuration.

    Administrator > Organization > Export > select all

    and import the timestamped .res file in your new dev

    Second solution to is remove the incidents
    I will use a python code to RestAPI to this endpoint, loop 10 incident by 10 incident ID's 700 hundred times for 7000 incidents.
    "Should" be done by the night :) - never try it to this quatity not did a time measure to delete 10, and multiply by 700
    You could try also by 50 :)





    ------------------------------
    BENOIT ROSTAGNI
    ------------------------------

    Original Message


  • 7.  RE: Bulk deleting incidents in Resilient

    Posted Wed December 22, 2021 09:45 AM
    Hi Andreas,

    You can achieve that by looping single incident delete request with incident IDs. Get all incident IDs by exporting all incidents to excel. Then use your web browsers developer console and switch to network tab, create a single incident delete request from UI and capture the web request. Right click to that request and copy it as cURL request or anything that you are more familiar with. Iterate through this request with incident IDs an you will be able to delete all incidents on Resilient. It still takes time but you dont have to interact with process more than one time.

    I hope this suits what you need.

    ------------------------------
    Burak Karaca
    ------------------------------

    Original Message


  • 8.  RE: Bulk deleting incidents in Resilient

    Posted Wed December 22, 2021 11:49 AM
    Hi Andreas
     
    From my experience using resilient REST API, you don't have any more gain using more than 5 threads in a pool in order to parallelize  your requests. Much more than that, you'll flood resilient and it will stop responding until you restart all its services.
     
    Indeed, each authenticated REST request takes around 1 second to execute, so if you have 7000 incidents and 5 threads taking 1 second each, you'll end up taking 1400 x 1 second = 23 minutes to delete them all.
     
    If you need something faster than this, you can try deleting directly from the resilient DB, but in this case, I don't know exactly which tables should you change and how, because it's a completely undocumented procedure, and certainly not endorsed by IBM.
     
    To directly access the resilient postgresql DB, you can do something like
     
    sudo -u postgres -i psql -c "select * from monapp.configvar" co3
     
    If you take a snapshot of all tables before and after an incident removal, you'll probably be able to figure out which tables you need to modify and how. 
     
    It's a lot of work, but it may be the way for you, if you really need something fast for your development environment.
     
    TIA
    Leonardo Kenji Shikida
    Security Delivery Specialist
     
    Phone: 55 31 99994 0875





    Original Message


  • 9.  RE: Bulk deleting incidents in Resilient

    Posted Wed December 22, 2021 09:52 AM
    Edited by Burak Karaca Thu December 23, 2021 05:03 AM
    This was just a duplicated message, i editted it so.