IBM Security QRadar SOAR

 View Only
Expand all | Collapse all

com.resilient.scripting.exception.TimeoutScriptingException: Script timeout error

  • 1.  com.resilient.scripting.exception.TimeoutScriptingException: Script timeout error

    Posted Fri February 04, 2022 07:41 AM
    Hi

    I was getting this error and I've figured out that 

    /usr/share/co3/conf/scripting-server.properties

    Allows to change the value of resScriptTimeout from 5 to something else (I've set to 60)

    I am curious if this is the right way to fix this and what's the maximum value allowed for this parameter.

    I understand that it must be responsive, but at the same time, 5 may be too little for some situations.

    It's in seconds, right?


    ------------------------------
    []

    Leonardo Kenji Shikida
    ------------------------------


  • 2.  RE: com.resilient.scripting.exception.TimeoutScriptingException: Script timeout error

    Posted Fri February 04, 2022 11:13 AM
    Hi,
      Yes, this is the right file, and the right setting, and the right denominator.
      However, I am concerned that scripts, which are meant to be short in execution time, are taking longer than five seconds. Rule (and hence script) execution ties up at least one database connection and these are finite resources.
      Are the scripts that were failing with the timeout error now passing? Can you see from the logs how long they are taking? What is your script doing?
    -P.J.

    ------------------------------
    Patrick (PJ) McKenna
    Resilient Development
    ------------------------------



  • 3.  RE: com.resilient.scripting.exception.TimeoutScriptingException: Script timeout error

    Posted Fri February 04, 2022 01:39 PM
    Edited by Leonardo Kenji Shikida Fri February 04, 2022 01:51 PM
    Hi Patrick
     
    The script was not supposed to take long to complete. They are the python scripts before and after the call of fn_utilities REST call.
     
    Maybe, the delay is not in the script itself, I am still investigating if the delay is in the execution of the REST call, or if there are too many simultaneous REST calls and they are competing for the same resources in the resilient-scripting service. 
     
    From what I've noticed from the resilient-scripting.service logs, the python scripts are completed pretty fast (0.1 secs avg), but since each request to the scripting service seems to have an ETA, I wonder if the issue is just too many python scripts requests that are filling the service scripting queue.
     
    If that's the case, is there any setting I can change in order to allow the resilient-scripting service to use more processes to handle the throughput (which seems better than increasing the timeout)?
     
    TIA
    Leonardo Kenji Shikida
    Security Delivery Specialist
     
    Phone: 55 31 99994 0875







  • 4.  RE: com.resilient.scripting.exception.TimeoutScriptingException: Script timeout error

    Posted Mon February 07, 2022 08:22 AM
    Are you using Python2 or Python3? Are you using CP4S or Appliance based SOAR? Are these scripts that used to run in under 5 seconds or new scripts?

    The setting should only applies to how long the actual python code takes.

    Ben

    ------------------------------
    Ben Lurie
    ------------------------------



  • 5.  RE: com.resilient.scripting.exception.TimeoutScriptingException: Script timeout error

    Posted Mon February 07, 2022 10:07 AM
    Hi

    Python 3
    Appliance
    Same scripts, higher request load

    ------------------------------
    []

    Leonardo Kenji Shikida
    ------------------------------



  • 6.  RE: com.resilient.scripting.exception.TimeoutScriptingException: Script timeout error

    Posted Mon February 07, 2022 01:22 PM
    Edited by PATRICK MCKENNA Mon February 07, 2022 01:45 PM
    Hello,
      An optional value can be set in /usr/share/co3/conf/scripting-server.properties to control how many simultaneous Python 3 scripts can be run.
      The value to set is resPython3MaxProcesses. The default value is 20. If circumstances dictate that more script executions are executed than this value, then the latecomers are queued until previously started scripts finish. In order for a change to the scripting-server.properties file to take effect, you must restart the scripting service.

      I would have thought that the throughput for 20 simultaneous scripts that each take 0.1 seconds to execute, over 5 seconds, would be sufficient for most scenarios. Do you know what the demand on the system is?

      Is there the line number where the script timed out mentioned in the logs? It's presence or absence could be informative. If the Python 3 process crashes, it might look like a timeout from the perspective of the Resilient appliance, but no line number would be recorded. There could be information in the scripts service log file that would indicate what's happening.

      Is your script performing an incident lookup using helper.findIncidents()? In theory, on a heavily loaded system, that search could take a long time.

      Which version of SOAR/Resilient are you running?

    -P.J.


    ------------------------------
    Patrick (PJ) McKenna
    SOAR Development
    ------------------------------



  • 7.  RE: com.resilient.scripting.exception.TimeoutScriptingException: Script timeout error

    Posted Fri November 04, 2022 04:42 AM
      |   view attached
    Hi,
    I also edited /usr/share/co3/conf/scripting-server.properties and set the Timeout to 10 for testing since i got a script that always runs into a timeout error. After saving i restarted the server and since then everytime i start a script i get a window with an error message. i attached a screenshot of it here.
    Even after setting the config back to resScriptTimeout=5 and restarting resilient-scripting, the error persists.

    And of course i didn't do a backup beforehand.... 

    Is there a way to fix this? Did i forget something?

    ------------------------------
    Benjamin Walden
    ------------------------------



  • 8.  RE: com.resilient.scripting.exception.TimeoutScriptingException: Script timeout error

    Posted Tue November 08, 2022 01:48 AM
    this is a long shoot , but did you try to also restart the following :

    sudo systemctl restart resilient-scripting
    sudo systemctl restart resilient
    sudo systemctl restart resilient-messaging
    sudo systemctl restart resilient-email

    If you solved it please let me know how

    ------------------------------
    mohamad islam hamadieh
    ------------------------------



  • 9.  RE: com.resilient.scripting.exception.TimeoutScriptingException: Script timeout error

    Posted Tue November 08, 2022 02:23 AM
    Hi Mohamad,

    Ben Williams from IBM Support helped me resolving the issue. In fact we tried the restarts as well with no positive result.
    The problem was with defective data in the queue of the resilient-messaging service. Ben checked the database with this command:

    sudo -u postgres -i psql co3 -c "select container, count(*) from monapp.activemq_msgs group by container order by container"

    And in the queue action feed_data were about 800 items that were waiting for the defective data to process. With a script found here https://www.ibm.com/support/pages/node/6235716 we deleted the queue and at least the GUI worked again. But i had to restart all my apps on the app host for them to work again as well.


    ------------------------------
    Benjamin Walden
    ------------------------------



  • 10.  RE: com.resilient.scripting.exception.TimeoutScriptingException: Script timeout error

    Posted Tue November 08, 2022 07:38 AM
    thanks a lot for the detailed reply Benjamin.

    ------------------------------
    mohamad islam hamadieh
    ------------------------------



  • 11.  RE: com.resilient.scripting.exception.TimeoutScriptingException: Script timeout error

    Posted Wed November 09, 2022 02:24 AM
    It is worth adding that a problem was caused when a parameter was accidentally set to resScriptTimeout=202020202020132646584 or a number similar to this but equally large. This necessitated the messages in two message destinations to be cleared before SOAR could successfully connect to all message destinations.

    Changing resScriptTimeout to a sensible number has never caused the problem seen by Benjamin.

    There are numerous posts in the community about "com.resilient.scripting.exception.TimeoutScriptingException: Script timeout error" and ways to resolve it. In most cases it requires changes to scripts to remove inefficiencies ensuring that the code processing incident or email data runs quickly before the default timeout of five seconds passes.

    ------------------------------
    BEN WILLIAMS
    ------------------------------