IBM Security QRadar SOAR

All,

I have a support case open for playbooks that are taking a long time to complete. However, once they learned that I created the playbook they're quick to say custom work in unsupported. I've been trying to convince them that the bottleneck seems to be on the IBM SOAR side.

I have matched up app/function logs that show the functions ran and returned data and then the playbook takes ~5-10 minutes before the next step of the playbook is completed. Obviously the 'next steps' of each playbook are different, but I've found examples where the next steps are a condition (did the function fail or not based on the 'success' flag of the response) and then a script to post notes and potentially close a task. 

We have a situation where the same playbook is called three times on the same IBM SOAR case. Two of the instances took 12 seconds to complete. The third took 15 minutes. I verified that the same paths within the playbook were followed by all three instances.

Does anyone have ideas about what could be causing playbooks to be running slowly or any investigative steps to take to try and find out?

Any help would be greatly appreciated.

------------------------------
Liam Mahoney
------------------------------

Hi

How many instances of the playbook are being running simultaneously?

TIA

Leo

------------------------------
[]

Leonardo Kenji Shikida
------------------------------

Original Message:
Sent: Mon January 23, 2023 11:43 AM
From: Liam Mahoney
Subject: Slow Playbooks

All,

I have a support case open for playbooks that are taking a long time to complete. However, once they learned that I created the playbook they're quick to say custom work in unsupported. I've been trying to convince them that the bottleneck seems to be on the IBM SOAR side.

I have matched up app/function logs that show the functions ran and returned data and then the playbook takes ~5-10 minutes before the next step of the playbook is completed. Obviously the 'next steps' of each playbook are different, but I've found examples where the next steps are a condition (did the function fail or not based on the 'success' flag of the response) and then a script to post notes and potentially close a task. 

We have a situation where the same playbook is called three times on the same IBM SOAR case. Two of the instances took 12 seconds to complete. The third took 15 minutes. I verified that the same paths within the playbook were followed by all three instances.

Does anyone have ideas about what could be causing playbooks to be running slowly or any investigative steps to take to try and find out?

Any help would be greatly appreciated.

------------------------------
Liam Mahoney
------------------------------

Hi Leonardo,

There is no limit for playbook execution across incidents. However, only one specific playbook at be run per incident at a time. For example, playbook A and B can run simultaneously for one incident, but only one playbook A will run at a time for one incident.

Hope this helps,
Mark

------------------------------
Mark Scherfling
------------------------------

Original Message:
Sent: Tue January 24, 2023 11:35 AM
From: Leonardo Kenji Shikida
Subject: Slow Playbooks

Hi

How many instances of the playbook are being running simultaneously?

TIA

Leo

------------------------------
[]

Leonardo Kenji Shikida

Original Message:
Sent: Mon January 23, 2023 11:43 AM
From: Liam Mahoney
Subject: Slow Playbooks

All,

I have a support case open for playbooks that are taking a long time to complete. However, once they learned that I created the playbook they're quick to say custom work in unsupported. I've been trying to convince them that the bottleneck seems to be on the IBM SOAR side.

I have matched up app/function logs that show the functions ran and returned data and then the playbook takes ~5-10 minutes before the next step of the playbook is completed. Obviously the 'next steps' of each playbook are different, but I've found examples where the next steps are a condition (did the function fail or not based on the 'success' flag of the response) and then a script to post notes and potentially close a task. 

We have a situation where the same playbook is called three times on the same IBM SOAR case. Two of the instances took 12 seconds to complete. The third took 15 minutes. I verified that the same paths within the playbook were followed by all three instances.

Does anyone have ideas about what could be causing playbooks to be running slowly or any investigative steps to take to try and find out?

Any help would be greatly appreciated.

------------------------------
Liam Mahoney
------------------------------

Leonardo,

Hard to say for sure as this number fluctuates throughout the day. I would say it's pretty common for there to be multiple instances of the same playbook running simultaneously on the same incident. In this scenario these playbooks are configured to either run on task or artifact objects. So multiple tasks / artifacts are added to the incident, each one of them has a playbook that runs for it.

For example, one of the playbooks we've been experiencing this with is our artifact enrichment playbook. Each artifact that is created triggers an instance of one of the artifact enrichment playbooks. If we get a few incidents created within a span of a few seconds of each other and each incident has multiple artifacts we're looking at quite a few instances of that playbook running at the same time.

------------------------------
Liam Mahoney
------------------------------

Original Message:
Sent: Tue January 24, 2023 11:35 AM
From: Leonardo Kenji Shikida
Subject: Slow Playbooks

Hi

How many instances of the playbook are being running simultaneously?

TIA

Leo

------------------------------
[]

Leonardo Kenji Shikida

Original Message:
Sent: Mon January 23, 2023 11:43 AM
From: Liam Mahoney
Subject: Slow Playbooks

All,

I have a support case open for playbooks that are taking a long time to complete. However, once they learned that I created the playbook they're quick to say custom work in unsupported. I've been trying to convince them that the bottleneck seems to be on the IBM SOAR side.

I have matched up app/function logs that show the functions ran and returned data and then the playbook takes ~5-10 minutes before the next step of the playbook is completed. Obviously the 'next steps' of each playbook are different, but I've found examples where the next steps are a condition (did the function fail or not based on the 'success' flag of the response) and then a script to post notes and potentially close a task. 

We have a situation where the same playbook is called three times on the same IBM SOAR case. Two of the instances took 12 seconds to complete. The third took 15 minutes. I verified that the same paths within the playbook were followed by all three instances.

Does anyone have ideas about what could be causing playbooks to be running slowly or any investigative steps to take to try and find out?

Any help would be greatly appreciated.

------------------------------
Liam Mahoney
------------------------------

Currently experiencing the playbook slowness and submitted the payload

{
 "filters": [
  {
   "conditions": [
    {
     "method": "equals",
     "field_name": "status",
     "value": "running"
    }
   ]
  }
 ]
}

to the API endpoint POST /orgs/{org_id}/playbooks/execution/query_paged and it returned that there were 567 recordsTotal that matched the filter. So that's telling me we currently have 567 playbook instances running. Could this be an issue? I would think most of these playbooks are dormant waiting for tasks to be closed, but I wouldn't be that surprised if ~50-100 of those playbook instances / executions that are actively doing function calls, waiting for scripts or script conditions to evaluate, etc.

------------------------------
Liam Mahoney
------------------------------

Original Message:
Sent: Wed January 25, 2023 09:45 AM
From: Liam Mahoney
Subject: Slow Playbooks

Leonardo,

Hard to say for sure as this number fluctuates throughout the day. I would say it's pretty common for there to be multiple instances of the same playbook running simultaneously on the same incident. In this scenario these playbooks are configured to either run on task or artifact objects. So multiple tasks / artifacts are added to the incident, each one of them has a playbook that runs for it.

For example, one of the playbooks we've been experiencing this with is our artifact enrichment playbook. Each artifact that is created triggers an instance of one of the artifact enrichment playbooks. If we get a few incidents created within a span of a few seconds of each other and each incident has multiple artifacts we're looking at quite a few instances of that playbook running at the same time.

------------------------------
Liam Mahoney

Original Message:
Sent: Tue January 24, 2023 11:35 AM
From: Leonardo Kenji Shikida
Subject: Slow Playbooks

Hi

How many instances of the playbook are being running simultaneously?

TIA

Leo

------------------------------
[]

Leonardo Kenji Shikida

Original Message:
Sent: Mon January 23, 2023 11:43 AM
From: Liam Mahoney
Subject: Slow Playbooks

All,

I have a support case open for playbooks that are taking a long time to complete. However, once they learned that I created the playbook they're quick to say custom work in unsupported. I've been trying to convince them that the bottleneck seems to be on the IBM SOAR side.

I have matched up app/function logs that show the functions ran and returned data and then the playbook takes ~5-10 minutes before the next step of the playbook is completed. Obviously the 'next steps' of each playbook are different, but I've found examples where the next steps are a condition (did the function fail or not based on the 'success' flag of the response) and then a script to post notes and potentially close a task. 

We have a situation where the same playbook is called three times on the same IBM SOAR case. Two of the instances took 12 seconds to complete. The third took 15 minutes. I verified that the same paths within the playbook were followed by all three instances.

Does anyone have ideas about what could be causing playbooks to be running slowly or any investigative steps to take to try and find out?

Any help would be greatly appreciated.

------------------------------
Liam Mahoney
------------------------------

Hi Liam,
Could you point me to the customer case that you created so I can take a look? Thank you.


------------------------------
Eric Yee
------------------------------

Original Message:
Sent: Mon January 23, 2023 11:43 AM
From: Liam Mahoney
Subject: Slow Playbooks

All,

I have a support case open for playbooks that are taking a long time to complete. However, once they learned that I created the playbook they're quick to say custom work in unsupported. I've been trying to convince them that the bottleneck seems to be on the IBM SOAR side.

I have matched up app/function logs that show the functions ran and returned data and then the playbook takes ~5-10 minutes before the next step of the playbook is completed. Obviously the 'next steps' of each playbook are different, but I've found examples where the next steps are a condition (did the function fail or not based on the 'success' flag of the response) and then a script to post notes and potentially close a task. 

We have a situation where the same playbook is called three times on the same IBM SOAR case. Two of the instances took 12 seconds to complete. The third took 15 minutes. I verified that the same paths within the playbook were followed by all three instances.

Does anyone have ideas about what could be causing playbooks to be running slowly or any investigative steps to take to try and find out?

Any help would be greatly appreciated.

------------------------------
Liam Mahoney
------------------------------

Hi Liam,
Would you point me to the support case so I can take a look please? Please feel free to contact me directly regarding this issue. Thank you.

------------------------------
Eric Yee
IBM QRadar SOAR
Software Development Manager (Level 3 Support, Performance)
------------------------------

Original Message:
Sent: Mon January 23, 2023 11:43 AM
From: Liam Mahoney
Subject: Slow Playbooks

All,

I have a support case open for playbooks that are taking a long time to complete. However, once they learned that I created the playbook they're quick to say custom work in unsupported. I've been trying to convince them that the bottleneck seems to be on the IBM SOAR side.

I have matched up app/function logs that show the functions ran and returned data and then the playbook takes ~5-10 minutes before the next step of the playbook is completed. Obviously the 'next steps' of each playbook are different, but I've found examples where the next steps are a condition (did the function fail or not based on the 'success' flag of the response) and then a script to post notes and potentially close a task. 

We have a situation where the same playbook is called three times on the same IBM SOAR case. Two of the instances took 12 seconds to complete. The third took 15 minutes. I verified that the same paths within the playbook were followed by all three instances.

Does anyone have ideas about what could be causing playbooks to be running slowly or any investigative steps to take to try and find out?

Any help would be greatly appreciated.

------------------------------
Liam Mahoney
------------------------------

Eric,

It is support case TS011863435. The support engineer has started to triage if there are any signs of bottlenecks on the IBM SOAR system since posting here.

------------------------------
Liam Mahoney
------------------------------

Original Message:
Sent: Tue January 24, 2023 03:56 PM
From: Eric Yee
Subject: Slow Playbooks

Hi Liam,
Would you point me to the support case so I can take a look please? Please feel free to contact me directly regarding this issue. Thank you.

------------------------------
Eric Yee
IBM QRadar SOAR
Software Development Manager (Level 3 Support, Performance)

Original Message:
Sent: Mon January 23, 2023 11:43 AM
From: Liam Mahoney
Subject: Slow Playbooks

All,

I have a support case open for playbooks that are taking a long time to complete. However, once they learned that I created the playbook they're quick to say custom work in unsupported. I've been trying to convince them that the bottleneck seems to be on the IBM SOAR side.

I have matched up app/function logs that show the functions ran and returned data and then the playbook takes ~5-10 minutes before the next step of the playbook is completed. Obviously the 'next steps' of each playbook are different, but I've found examples where the next steps are a condition (did the function fail or not based on the 'success' flag of the response) and then a script to post notes and potentially close a task. 

We have a situation where the same playbook is called three times on the same IBM SOAR case. Two of the instances took 12 seconds to complete. The third took 15 minutes. I verified that the same paths within the playbook were followed by all three instances.

Does anyone have ideas about what could be causing playbooks to be running slowly or any investigative steps to take to try and find out?

Any help would be greatly appreciated.

------------------------------
Liam Mahoney
------------------------------

hello community

is there is playbook log files i could see?

------------------------------
kamal ghanem
------------------------------

Original Message:
Sent: Wed January 25, 2023 09:49 AM
From: Liam Mahoney
Subject: Slow Playbooks

Eric,

It is support case TS011863435. The support engineer has started to triage if there are any signs of bottlenecks on the IBM SOAR system since posting here.

------------------------------
Liam Mahoney

Original Message:
Sent: Tue January 24, 2023 03:56 PM
From: Eric Yee
Subject: Slow Playbooks

Hi Liam,
Would you point me to the support case so I can take a look please? Please feel free to contact me directly regarding this issue. Thank you.

------------------------------
Eric Yee
IBM QRadar SOAR
Software Development Manager (Level 3 Support, Performance)

Original Message:
Sent: Mon January 23, 2023 11:43 AM
From: Liam Mahoney
Subject: Slow Playbooks

All,

I have a support case open for playbooks that are taking a long time to complete. However, once they learned that I created the playbook they're quick to say custom work in unsupported. I've been trying to convince them that the bottleneck seems to be on the IBM SOAR side.

I have matched up app/function logs that show the functions ran and returned data and then the playbook takes ~5-10 minutes before the next step of the playbook is completed. Obviously the 'next steps' of each playbook are different, but I've found examples where the next steps are a condition (did the function fail or not based on the 'success' flag of the response) and then a script to post notes and potentially close a task. 

We have a situation where the same playbook is called three times on the same IBM SOAR case. Two of the instances took 12 seconds to complete. The third took 15 minutes. I verified that the same paths within the playbook were followed by all three instances.

Does anyone have ideas about what could be causing playbooks to be running slowly or any investigative steps to take to try and find out?

Any help would be greatly appreciated.

------------------------------
Liam Mahoney
------------------------------