IBM Security QRadar SOAR

 View Only
Expand all | Collapse all

Resilient Action Always Pending Status

  • 1.  Resilient Action Always Pending Status

    Posted Tue August 04, 2020 10:21 AM

    Hi All, 

    We created resilient actions by following below document and using IBM Resilient IRP - Fortinet FortiGate Firewall integration DEMO guide.pdf, however actions always pending status, please refer the attached script we used and also app.log file with debug and info mode 

    https://github.com/ibmresilient/resilient-reference/blob/master/developer_guides/Resilient%20IRP%20Function%20Developer%20Guide.pdf

    The logs does not showing much information's why it has pending and could you please help us to resolve the issue, we need to create custom firewall policies for the artifacts. 


    ------------------------------
    Regards,
    Sunil I B,
    014-3213219
    ------------------------------

    Attachment(s)

    log
    app.log   406 KB 1 version
    py
    fortinet.py   2 KB 1 version


  • 2.  RE: Resilient Action Always Pending Status

    Posted Thu August 06, 2020 12:26 AM
    Anyone could you please help us on this asap

    ------------------------------
    Sunil I B
    ------------------------------



  • 3.  RE: Resilient Action Always Pending Status

    Posted Thu August 06, 2020 08:24 AM
    The reason it is in Pending status is because the resilient circuits service did not pick up the request from the action. There is no indication in the circuits log that is recognized there is an action message being proccessed. No errors either.

    I think for some reason maybe circuits is not configured to get data from the appropriate resilient server.

    Hard to say without additional troubleshooting.

    Ben

    ------------------------------
    Ben Lurie
    ------------------------------



  • 4.  RE: Resilient Action Always Pending Status

    Posted Thu August 06, 2020 08:37 AM
    Hi Ben Lurie, 

    Thanks a lot for the response, could you please share troubleshooting steps to investigate the issue further, Yes we noticed that during stop and start circuits able to process the script, but when we execute the actions no message returns to the logs. 
    FYI, we raised a case with IBM Support team, they requested to post here to resolve the issue. 




    ------------------------------
    Sunil I B
    ------------------------------



  • 5.  RE: Resilient Action Always Pending Status

    IBM Champion
    Posted Mon August 10, 2020 11:26 AM
    When you start circuits do you notice any log messages stating it's registered to the message destinations of the integration you are trying to use? If not, is it possible the integration wasn't pip installed properly?

    ------------------------------
    Liam Mahoney
    ------------------------------



  • 6.  RE: Resilient Action Always Pending Status

    Posted Mon August 10, 2020 12:18 PM
    Edited by Sunil I B Mon August 10, 2020 12:25 PM
    We downloaded some of the functions from XForce App Exchange for LDAP etc, those message we able to see in app.log files and also In our app logs we noticed that no errors for fortigate action script, however no message like destinations of the integration for Fortigate destination queue,  please refer the attached pip and latest app.log. 


    [root@resilient tmp]# systemctl status resilient_circuits
    â resilient_circuits.service - Resilient-Circuits Service
    Loaded: loaded (/etc/systemd/system/resilient_circuits.service; enabled; vendor preset: disabled)
    Active: active (running) since Tue 2020-08-11 00:08:15 +08; 3s ago
    Main PID: 24171 (resilient-circu)
    CGroup: /system.slice/resilient_circuits.service
    ââ24171 /usr/local/bin/python2.7 /usr/local/bin/resilient-circuits run

    Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: 2020-08-11 00:08:17,732 INFO [stomp_component] No Client heartbeats will be sent
    Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: 2020-08-11 00:08:17,733 INFO [stomp_component] Requested heartbeats from server.
    Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: 2020-08-11 00:08:17,734 DEBUG [client] Received heart-beat
    Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: 2020-08-11 00:08:17,734 INFO [actions_component] resilient-circuits has started successfully and is now running...
    Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: <Connected[stomp] ()>
    Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: 2020-08-11 00:08:17,735 INFO [actions_component] STOMP connected.
    Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: <registered[*] (<Timer/* 24171:MainThread (queued=0) [S]>, <StompClient/stomp 24171:MainThread (queued=0) [S]> )>
    Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: <Connect_success[*] (<Connect[*] ()>, 'success' )>
    Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: 2020-08-11 00:08:17,736 DEBUG [actions_component] Connected successfully. Resubscribe? False
    Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: <Connected_success[stomp] (<Connected[stomp] ()>, None )>
    [root@resilient tmp]#


    2020-08-11 00:08:16,991 DEBUG [connectionpool] https://resilient.localdomain:443 "GET /rest/orgs/201 HTTP/1.1" 200 None
    2020-08-11 00:08:17,077 DEBUG [connectionpool] https://resilient.localdomain:443 "GET /rest/orgs/201/types/incident/fields HTTP/1.1" 200 None
    2020-08-11 00:08:17,115 DEBUG [connectionpool] https://resilient.localdomain:443 "GET /rest/orgs/201/types/actioninvocation/fields HTTP/1.1" 200 None
    2020-08-11 00:08:17,139 DEBUG [connectionpool] https://resilient.localdomain:443 "GET /rest/orgs/201/message_destinations HTTP/1.1" 200 None
    2020-08-11 00:08:17,166 DEBUG [connectionpool] https://resilient.localdomain:443 "GET /rest/orgs/201/functions HTTP/1.1" 200 None
    2020-08-11 00:08:17,196 DEBUG [connectionpool] https://resilient.localdomain:443 "GET /rest/orgs/201/functions/ldap_utilities_toggle_access HTTP/1.1" 200 None
    2020-08-11 00:08:17,227 DEBUG [connectionpool] https://resilient.localdomain:443 "GET /rest/orgs/201/types/__function/fields HTTP/1.1" 200 None
    2020-08-11 00:08:17,232 DEBUG [actions_component] Reset idle timer
    2020-08-11 00:08:17,293 DEBUG [connectionpool] https://resilient.localdomain:443 "GET /rest/orgs/201/actions HTTP/1.1" 200 None
    2020-08-11 00:08:17,300 INFO [app] Components auto-load directory: /root/venv/resilient-circuits/components
    2020-08-11 00:08:17,301 INFO [component_loader] Loading 'fortinet' from /root/venv/resilient-circuits/components/fortinet.py
    2020-08-11 00:08:17,325 DEBUG [actions_component] Reset idle timer
    2020-08-11 00:08:17,326 DEBUG [actions_component] Reset idle timer
    2020-08-11 00:08:17,326 WARNING [actions_component] Unverified STOMP TLS certificate (cafile=false)
    2020-08-11 00:08:17,333 INFO [stomp_component] Connect to resilient.localdomain:65001
    2020-08-11 00:08:17,335 INFO [app] App Started
    2020-08-11 00:08:17,336 INFO [component_loader] Loaded and registered component 'fortinet'
    2020-08-11 00:08:17,337 INFO [actions_component] STOMP attempting to connect
    2020-08-11 00:08:17,337 INFO [app] Components loaded
    2020-08-11 00:08:17,339 DEBUG [app] Components:


    ------------------------------
    Sunil I B
    ------------------------------

    Attachment(s)

    log
    app.log   9 KB 1 version
    txt
    PIP_Output.txt   7 KB 1 version


  • 7.  RE: Resilient Action Always Pending Status

    IBM Champion
    Posted Mon August 10, 2020 02:18 PM

    Sunil,

    Apologies, I didn't see that you attached your log output to your initial message. I noticed it looks like the fortinet integration is getting loaded through the component auto-load directory. Unfortunately I don't have much experience using that feature. It does look like something for fortinet is getting loaded in your logs though:

    2020-08-11 00:08:17,336 INFO [component_loader] Loaded and registered component 'fortinet'

    I noticed it looks like you have the fortinet integration installed in a virtual environment based on the output of your pip list command, is this correct? I'm wondering if this might do anything:

    1. stop the resilient_circuits process (sudo systemctl stop resilient_circuits)

    2. activate the virtual environment you pip installed the integration in (looks like it's /root/venv - if this is correct, the command would be 'source /root/venv/bin/activate')

    3. start resilient circuits with the command resilient-circuits run

    4. once circuits is started, see if any fortinet messages are being written to the log, or if they're still hung up in the action status

    Don't forget to kill the resilient_circuits process you manually started (ctrl+C) and restart your resilient circuits process (sudo systemctl start resilient_circuits) once your test is done.

    I'm hoping it has something to do with the integration being installed in a virtual environment but your circuits process isn't leveraging that virtual environment. Again, I haven't used the auto load component directory, so I'm not sure if that should bypass what I'm thinking is wrong or not.



    ------------------------------
    Liam Mahoney
    ------------------------------



  • 8.  RE: Resilient Action Always Pending Status

    Posted Mon August 10, 2020 10:26 PM
    Edited by Sunil I B Mon August 10, 2020 10:57 PM
    HI Liam Mahoney, 

    Thanks a lot for the response, still facing same issues, we are unable to see fortigate queue destination and no any actions messages for the manual actions, please refer the attached logs. 

     Regarding with packaging of the functions, with help of developer guide using the functions, when Package the integration code & function, we received as depreciated error. 


    [root@resilient ~]# resilient-circuits codegen
    DEPRECATED: The 'codegen' command has been deprecated for resilient-circuits. This functionality has been moved to the resilient-sdk tool.
    [root@resilient ~]#

    ------------------------------
    Sunil I B
    ------------------------------

    Attachment(s)

    txt
    Manual Run.txt   18 KB 1 version
    log
    app.log   61 KB 1 version


  • 9.  RE: Resilient Action Always Pending Status

    Posted Tue August 11, 2020 05:06 AM
    Sunil,

    The following suggests you have not run resilient-circuits customize to import the exchange function into Resilient

    2020-08-11 09:12:57,693 WARNING [actions_component] Function 'exchange_online_delete_messages_from_query_results' is not defined in this Resilient platform!2020-08-11 09:12:57,693 ERROR [component_loader] Failed to load 'fn_exchange_online.components.exchange_online_delete_messages_from_query_results.FunctionComponent'Traceback (most recent call last): File "/root/venv/resilient-circuits/lib/python2.7/site-packages/resilient_circuits/component_loader.py", line 102, in _register_components component_class(opts=self.opts).register(self) File "/root/venv/resilient-circuits/lib/python2.7/site-packages/fn_exchange_online/components/exchange_online_delete_messages_from_query_results.py", line 29, in __init__ self.load_options(opts) File "/root/venv/resilient-circuits/lib/python2.7/site-packages/fn_exchange_online/components/exchange_online_delete_messages_from_query_results.py", line 24, in load_options validate_fields(required_fields, self.options) File "/root/venv/resilient-circuits/lib/python2.7/site-packages/resilient_lib/components/resilient_common.py", line 130, in validate_fields raise ValueError(mandatory_err_msg.format(field))ValueError: 'microsoft_graph_token_url' is mandatory and is not set. You must set this value to run this function

    The fortinet code looks to have imported successfully now.

    2020-08-11 09:12:58,183 INFO [component_loader] Loaded and registered component 'fortinet'



    ------------------------------
    BEN WILLIAMS
    ------------------------------



  • 10.  RE: Resilient Action Always Pending Status

    Posted Tue August 11, 2020 06:45 AM
    Edited by Sunil I B Tue August 11, 2020 06:45 AM
    Thanks for the response, still Fortinet action always in pending state after we receiving the message loaded and registered component 'fortinet'


    Our focus to automate the actions for fortinet firewall, Mca Fee IPS etc, yes we are aware of exchange issue we not customize yet, Initial exchange module loaded for other testing purpose. 

    app,log not showing anything after we execute actions, we able to see only below messages on app.log

    2020-08-11 18:42:48,068 DEBUG [client] Received heart-beat
    2020-08-11 18:43:03,121 DEBUG [client] Received heart-beat
    2020-08-11 18:43:18,085 DEBUG [client] Received heart-beat
    2020-08-11 18:43:33,115 DEBUG [client] Received heart-beat
    2020-08-11 18:43:47,680 DEBUG [actions_component] Idle reset
    2020-08-11 18:43:48,068 DEBUG [client] Received heart-beat
    2020-08-11 18:44:03,113 DEBUG [client] Received heart-beat
    2020-08-11 18:44:18,074 DEBUG [client] Received heart-beat
    2020-08-11 18:44:33,121 DEBUG [client] Received heart-beat
    2020-08-11 18:44:48,069 DEBUG [client] Received heart-beat
    2020-08-11 18:45:03,116 DEBUG [client] Received heart-beat


    ------------------------------
    Sunil I B
    ------------------------------



  • 11.  RE: Resilient Action Always Pending Status

    Posted Tue August 11, 2020 10:45 AM
    Edited by BEN WILLIAMS Tue August 11, 2020 10:48 AM
    Hi Sunil,

    Do you have this problem with any other functions? For example, if you install an IBM supported function/app into Resilient Circuits does it work?

    If you run the following command, do you see messages being written to the message destination that fortinet is configured to use?

    sudo -u postgres -i psql -c "select container, count(*) from monapp.activemq_msgs group by container order by container" co3

    ------------------------------
    BEN WILLIAMS
    ------------------------------



  • 12.  RE: Resilient Action Always Pending Status

    Posted Tue August 11, 2020 12:03 PM
      |   view attached
    Previously we tested few functions like ioc praser, email header analysis, however we noticed that ioc also not working now, refer the attached log file and also below error message. 

    Now we able to see subscribe to message destination actions for ioc

    2020-08-11 23:45:17,670 DEBUG [client] Received CONNECTED frame [headers={u'session': u'ID:resilient.localdomain-34795-1597158989131-4:5', u'version': u'1.2', u'server': u'ActiveMQ/5.15.9',
    u'heart-beat': u'15000,0'}, version=1.2]
    2020-08-11 23:45:17,671 INFO [client] Connected to stomp broker [session=ID:resilient.localdomain-34795-1597158989131-4:5, version=1.2]
    2020-08-11 23:45:17,671 DEBUG [stomp_component] State after Connection Attempt: connected
    2020-08-11 23:45:17,672 INFO [stomp_component] Connected to failover:(ssl://resilient.localdomain:65001)?maxReconnectAttempts=1,startupMaxReconnectAttempts=1
    2020-08-11 23:45:17,672 INFO [stomp_component] Client HB: 0 Server HB: 15000
    2020-08-11 23:45:17,672 INFO [stomp_component] No Client heartbeats will be sent
    2020-08-11 23:45:17,673 INFO [stomp_component] Requested heartbeats from server.
    2020-08-11 23:45:17,674 DEBUG [client] Received heart-beat
    2020-08-11 23:45:17,674 INFO [actions_component] resilient-circuits has started successfully and is now running...
    2020-08-11 23:45:17,674 INFO [actions_component] Subscribe to message destination 'fn_ioc_parser_v2'
    2020-08-11 23:45:17,675 INFO [actions_component] STOMP connected.
    2020-08-11 23:45:17,676 INFO [stomp_component] Subscribe to message destination actions.201.fn_ioc_parser_v2
    2020-08-11 23:45:17,676 DEBUG [client] Sending SUBSCRIBE frame [headers={'ack': 'client-individual', 'destination': 'actions.201.fn_ioc_parser_v2', 'id': 'actions.201.fn_ioc_parser_v2', 'ac
    tivemq.prefetchSize': 20}, version=1.2]
    2020-08-11 23:45:17,677 DEBUG [actions_component] Connected successfully. Resubscribe? False

    2020-08-11 23:45:18,094 WARNING [actions_component] Action 35 is unknown.
    2020-08-11 23:45:18,095 DEBUG [actions_component] Reset idle timer
    2020-08-11 23:45:18,304 DEBUG [connectionpool] https://resilient.localdomain:443 "GET /rest/orgs/201/actions HTTP/1.1" 200 None
    2020-08-11 23:45:18,306 ERROR [actions_component] Action 35 is not defined.
    Traceback (most recent call last):
    File "/usr/local/lib/python2.7/site-packages/resilient_circuits/actions_component.py", line 334, in action_name
    defn = self.action_defs[action_id]
    KeyError: 35
    2020-08-11 23:45:18,307 ERROR [actions_component] 35
    Traceback (most recent call last):
    File "/usr/local/lib/python2.7/site-packages/resilient_circuits/actions_component.py", line 426, in on_stomp_message
    log_dir=self.logging_directory)
    File "/usr/local/lib/python2.7/site-packages/resilient_circuits/action_message.py", line 180, in __init__
    self.displayname = source.action_name(self.action_id)
    File "/usr/local/lib/python2.7/site-packages/resilient_circuits/actions_component.py", line 334, in action_name
    defn = self.action_defs[action_id]
    KeyError: 35

    ------------------------------
    Sunil I B
    ------------------------------

    Attachment(s)

    log
    app.log   570 KB 1 version


  • 13.  RE: Resilient Action Always Pending Status

    Posted Wed August 12, 2020 06:18 AM
    Hi Sunil,

    Please see https://www.ibm.com/support/pages/node/1159492 relating to "Action 35 is not defined."

    So, a community app (IOC Parser Function v2 for IBM Resilient ) is able to read from the message destination successfully but your code, fortinet.py is not. I see that your colleague has a support case open which I will respond to. I hope that someone in the community will be able to assist you further with your code.

    If you find that no one is able to assist, you may get in touch with your sales person or Customer Success Manager and talk with them about other options such as engaging with Security Expert labs to help you overcome the problems you have with your code.

    ------------------------------
    BEN WILLIAMS
    ------------------------------



  • 14.  RE: Resilient Action Always Pending Status

    Posted Wed August 12, 2020 07:55 AM
    We resolved IOC function issue, however we cant see still message destination successfully for Fortinet Fortigate and always pending status and no error message has been returned. 

    Moreover packaging the code and functions by following developer guide approached has been changed to SDK version now. 

    Noticed that functions to message destination working fine for the functions downloaded from app exchange for ioc parser, however actions to message destination not working when following developer guide. 

    We able to see only Fortinet Fortigate action has been loaded and registered only, no message related to message destinations. 

    Without knowing the issue with code or components actions to message destination 

    So could you please let us know alternative approach to troubleshoot further whether it is related to developed custom code or components actions to message destination issue. 



    ------------------------------
    Sunil I B
    ------------------------------



  • 15.  RE: Resilient Action Always Pending Status

    Posted Wed August 19, 2020 09:28 AM
    Hi All.

    I have the same problem wherein it always pending status. I tried to check if the python script is running, I notice that it always meet this condition:
    @handler()
    def run_phantom_action(self, event, *args, **kwargs):
    """
    The string passed to @handler must match the action name in Resilient
    """
    if not isinstance(event, ActionMessage):
        print('return checking')
        return

    That cause to not to run the whole script. Please let us know what we need to do.
    Thanks.

    ------------------------------
    Marc Lainez
    ------------------------------



  • 16.  RE: Resilient Action Always Pending Status

    IBM Champion
    Posted Wed August 19, 2020 12:13 PM
    @Sev Cu

    What app is that?​

    ------------------------------
    Jared Fagel
    Cyber Security Analyst I
    Public Utility
    ------------------------------



  • 17.  RE: Resilient Action Always Pending Status

    Posted Thu August 20, 2020 02:11 AM
    Hi @Jared Fagel.

    Thanks for your reply the name of app is rc-phantomcyber what it does is it will send resilient incident to other automation tools, it was created by previous developer. I think my problem were now solved by changing the authentication type in app config from API key to basic authentication.

    In terms of using API key I still encountered the issue, is there's any configuration that I need to add?

    thanks​

    ------------------------------
    Marc Lainez
    ------------------------------