IBM Security QRadar SOAR

 View Only

  • 1.  IOC Parser failed rows more then 25000 IOC

    Posted Thu November 12, 2020 07:33 PM
    Edited by Sunil I B Thu November 12, 2020 07:36 PM
      |   view attached
    Hi All, 

    When we do IOC parser for a text file more then 29718 IOC, it failed to attached to artifacts, When we check app.log, we noticed it completed, however failed on Action status

    APP.log2020-11-12 19:33:12,170 INFO [iocparser] Found 29718 IOCs.
    2020-11-12 19:33:30,478 DEBUG [client] Received heart-beat
    2020-11-12 19:33:45,442 DEBUG [client] Received heart-beat
    2020-11-12 19:33:52,622 INFO [decorators] [func_ioc_parser_v2] StatusMessage: Completed IOC Parsing on artifact/attachment data
    2020-11-12 19:33:52,623 DEBUG [decorators] [func_ioc_parser_v2] FunctionResult: <resilient_circuits.action_message.FunctionResult object at 0x7f3443837450>
    2020-11-12 19:33:52,654 DEBUG [stomp_component] send()
    2020-11-12 19:33:52,655 DEBUG [client] Sending SEND frame [headers={'destination': u'/queue/acks.201.fn_ioc_parser_v2', 'correlation-id': u'invid:163'}, body='{"message": "Complet...', version=1.2]
    2020-11-12 19:33:52,656 DEBUG [stomp_component] Message sent
    2020-11-12 19:33:52,758 DEBUG [actions_component] success! [<resilient_circuits.action_message.FunctionResult object at 0x7f3443837450>], <func_ioc_parser_v2[functions.func_ioc_parser_v2] (id=17, workflow=example_parse_iocs_attachment, user=resilient@xxx.com) 2020-11-12 11:33:02.229000>
    2020-11-12 19:33:52,758 DEBUG [actions_component] Message: Completed
    2020-11-12 19:33:52,758 DEBUG [actions_component] Ack ID:XXXXXXX-39628-1603703973382-3:2:828:1:1[Less]


    1) Please refer the attached error status from resilient action
    2) IOC parser is successful and posted to resilient when we check using app.log, if so why the artifacts not added to list and action status shows as failed
    3) How many artifacts we can add for per incident to resilient artifacts tab.



    ------------------------------
    Sunil I B
    ------------------------------

    Attachment(s)

    py
    IOC_Parser_script.py   4 KB 1 version


  • 2.  RE: IOC Parser failed rows more then 25000 IOC

    IBM Champion
    Posted Thu November 12, 2020 10:06 PM
    Hey @Sunil I B,

    1. This is a scripting limit, a script/processor can only execute 50,000 lines of code before it forcibly exits with a "timeout" error. This is a fail-safe to prevent problematic scripts from causing issues, such as a script that gets stuck in a loop. I believe this can be modified, but I'm not sure on the steps for this. We see here that the post-processor is running over this limit, causing the error.

    2. ​The pre-processor is working, the function is working, but the post-processor is not. This is why you see an error despite the function doing it's job.

    3. I'm not sure what the system limit on artifacts per incident is, if one exists.

    ------------------------------
    Jared Fagel
    Cyber Security Analyst I
    Public Utility
    ------------------------------

    Original Message


  • 3.  RE: IOC Parser failed rows more then 25000 IOC

    Posted Fri November 13, 2020 12:21 AM
    Thanks Jared Fagel,  50,000 lines of code refers to Number of lines in attachment for ioc extractions 

    ------------------------------
    Sunil I B
    ------------------------------

    Original Message


  • 4.  RE: IOC Parser failed rows more then 25000 IOC

    Posted Mon November 23, 2020 02:53 AM
    I will note create a incident with 25000 Artifacts, because:
    1) this will launch Threat Intelligence for 25000 per Threat
    - It is not design to support that
    - there is a limit at 100,000 request per month (see license file)
    2) It will launch all your Enrichment 25000 times ==> 25000 x Actions 1+2+3+4+5...
    - It is not design to support that
    - there is an action limit on you action license
    3) NO HUMAN CAN READ 25000 Results on Threat Intelligence or actions
    and even if we could have automation there, thinks MUST BE HUMANLY Workable, automation are here to speed up the process, not to do it all at once.

    You better look at SIEMS for those volumes !

    ------------------------------
    BENOIT ROSTAGNI
    ------------------------------

    Original Message