IBM QRadar SOAR

IBM QRadar

Join this online topic group to communicate across Security product users and IBM experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.


#Security
#QRadar
#SecuringhybridcloudandAI
 View Only

  • 1.  Sometimes functions are not invoked

    Posted 09/23/19 10:38 AM
    Hi,

    I see that sometimes function (py function implementation) are not invoked in Action Processor.
    For example, If I invoke function "fn_bigfix_artifact" from Resilient. I could see below print in the Action Module.
    2019-09-23 19:53:50,299 INFO [actions_component] Event: <fn_bigfix_artifact[] (id=7, workflow=bigfix_query_for_artifact, user=anbalagan@secbounty.com) 2019-09-23 14:23:50.505000> Channel: functions.fn_bigfix_artifact
    But the actual python function is not getting invoked.

    But if I rerun the "resilient-circuit run" program, it is working properly (the python function is invoked).

    I have more function in my Resilient deployment which will be called frequently from workflows.

    Calling more function causes any issue?
    Any procedure to debug the issue?

    Regards,
    Anbalagan R

    ------------------------------
    Anbalagan Rajagopal
    ------------------------------


  • 2.  RE: Sometimes functions are not invoked

    Posted 10/16/19 12:32 PM
    Hi Anbalagan,

    Some questions on this issue.
    Are you still seeing the issue?
    Is is something which happens often and is it repeatable?
    What state is the 'Workflow status' is when  the function is invoked?
    Does  the 'Action status' or 'Workflow status' show any errors.

    To get more  logging  for a Resilient Function execution you can run Resilient circuits in DEBUG mode.
    There are 2 ways to run  in debug mode .

    From command-line:

    resilient-circuits run --loglevel=DEBUG

    Set option in the [resilient] section of the app.config file:

    loglevel=DEBUG

    This should add extra debug logging in Resilient circuits log and hopefully help to debug the issue.

    Regards,
    John



    ------------------------------
    JOHN PRENDERGAST
    ------------------------------



  • 3.  RE: Sometimes functions are not invoked

    Posted 10/16/19 02:09 PM
    Hi John,

    Thanks for your reply.

    Now the issue is not reproducible, it looks like resolved.

    Below is my observation:
    • I have configured Resilient to trigger chain of workflows (one triggers another) as soon as an artifact is added to an incident.
    • Some of the workflows wait for hours in a timer function.
    • Customer accidentally added around 1000 artifacts at one-shot by copy pasting into "Add Artifact" dialogue.
      • Customer didn't intimate us (partner) regarding this.
    • So this caused many workflow to trigger simultaneously and none of them finished immediately due to wait timer in the workflows.
    • This situation leads to not letting other action to run.
    • Investigating further, I identified that, in resilient_circuits module message is received but new thread is not spanned to execute the function implementation.
    • So I suspected that the max thread limit would have reached in linux/Python. And leads to not invoking function implementation.
    • This problem persisted even after rebooting Resilient and Action Processor machines.

    Solution
    • After identifying and deleting the incident which has huge number of artifacts, the issue is not reproducible.
    • All the workflows and functions started working properly.
     
    I am not very sure my suspect, correct me if I am wrong.

    Regards,
    Anbalagan R

    ------------------------------
    Anbalagan Rajagopal
    ------------------------------

    Original Message


  • 4.  RE: Sometimes functions are not invoked

    Posted 10/19/19 10:09 AM
    Hi John,

    I have identified the root cause for the problem.

    • When more then 10 timer function (fn_utilities timer function with long wait period) happened to run simultaneously, resilient circuits framework doesn't invoke another function until any one of the timer expires.
    • Resilient circuit framework can execute only 10 function simultaneously, because of max number of worker thread restriction circuits package.
      • In circuits\core\workers.py module, DEFAULT_WORKERS = 10

    Can I increase this value?
    Is there any way to increase this limit without hardcoding in py file?

    Regards,
    Anbalagan R

    ------------------------------
    Anbalagan Rajagopal
    ------------------------------

    Original Message


Related Content