Previously we tested few functions like ioc praser, email header analysis, however we noticed that ioc also not working now, refer the attached log file and also below error message.
Now we able to see subscribe to message destination actions for ioc
2020-08-11 23:45:17,670 DEBUG [client] Received CONNECTED frame [headers={u'session': u'ID:resilient.localdomain-34795-1597158989131-4:5', u'version': u'1.2', u'server': u'ActiveMQ/5.15.9',
u'heart-beat': u'15000,0'}, version=1.2]
2020-08-11 23:45:17,671 INFO [client] Connected to stomp broker [session=ID:resilient.localdomain-34795-1597158989131-4:5, version=1.2]
2020-08-11 23:45:17,671 DEBUG [stomp_component] State after Connection Attempt: connected
2020-08-11 23:45:17,672 INFO [stomp_component] Connected to failover:(ssl://resilient.localdomain:65001)?maxReconnectAttempts=1,startupMaxReconnectAttempts=1
2020-08-11 23:45:17,672 INFO [stomp_component] Client HB: 0 Server HB: 15000
2020-08-11 23:45:17,672 INFO [stomp_component] No Client heartbeats will be sent
2020-08-11 23:45:17,673 INFO [stomp_component] Requested heartbeats from server.
2020-08-11 23:45:17,674 DEBUG [client] Received heart-beat
2020-08-11 23:45:17,674 INFO [actions_component] resilient-circuits has started successfully and is now running...
2020-08-11 23:45:17,674 INFO [actions_component] Subscribe to message destination 'fn_ioc_parser_v2'
2020-08-11 23:45:17,675 INFO [actions_component] STOMP connected.
2020-08-11 23:45:17,676 INFO [stomp_component] Subscribe to message destination actions.201.fn_ioc_parser_v2
2020-08-11 23:45:17,676 DEBUG [client] Sending SUBSCRIBE frame [headers={'ack': 'client-individual', 'destination': 'actions.201.fn_ioc_parser_v2', 'id': 'actions.201.fn_ioc_parser_v2', 'ac
tivemq.prefetchSize': 20}, version=1.2]
2020-08-11 23:45:17,677 DEBUG [actions_component] Connected successfully. Resubscribe? False
2020-08-11 23:45:18,094 WARNING [actions_component] Action 35 is unknown.
2020-08-11 23:45:18,095 DEBUG [actions_component] Reset idle timer
2020-08-11 23:45:18,304 DEBUG [connectionpool]
https://resilient.localdomain:443 "GET /rest/orgs/201/actions HTTP/1.1" 200 None
2020-08-11 23:45:18,306 ERROR [actions_component] Action 35 is not defined.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/resilient_circuits/actions_component.py", line 334, in action_name
defn = self.action_defs[action_id]
KeyError: 35
2020-08-11 23:45:18,307 ERROR [actions_component] 35
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/resilient_circuits/actions_component.py", line 426, in on_stomp_message
log_dir=self.logging_directory)
File "/usr/local/lib/python2.7/site-packages/resilient_circuits/action_message.py", line 180, in __init__
self.displayname = source.action_name(self.action_id)
File "/usr/local/lib/python2.7/site-packages/resilient_circuits/actions_component.py", line 334, in action_name
defn = self.action_defs[action_id]
KeyError: 35
------------------------------
Sunil I B
------------------------------
Original Message:
Sent: Tue August 11, 2020 10:44 AM
From: BEN WILLIAMS
Subject: Resilient Action Always Pending Status
Hi Sunil,
Do you have this problem with any other functions? For example, if you install an IBM supported function/app into Resilient Circuits does it work?
If you run the following command, do you see messages being written to the message destination that fortinet is configured to use?
sudo -u postgres -i psql -c "select container, count(*) from monapp.activemq_msgs group by container order by container" co3
------------------------------
BEN WILLIAMS
Original Message:
Sent: Tue August 11, 2020 06:44 AM
From: Sunil I B
Subject: Resilient Action Always Pending Status
Thanks for the response, still Fortinet action always in pending state after we receiving the message loaded and registered component 'fortinet'
Our focus to automate the actions for fortinet firewall, Mca Fee IPS etc, yes we are aware of exchange issue we not customize yet, Initial exchange module loaded for other testing purpose.
app,log not showing anything after we execute actions, we able to see only below messages on app.log
2020-08-11 18:42:48,068 DEBUG [client] Received heart-beat
2020-08-11 18:43:03,121 DEBUG [client] Received heart-beat
2020-08-11 18:43:18,085 DEBUG [client] Received heart-beat
2020-08-11 18:43:33,115 DEBUG [client] Received heart-beat
2020-08-11 18:43:47,680 DEBUG [actions_component] Idle reset
2020-08-11 18:43:48,068 DEBUG [client] Received heart-beat
2020-08-11 18:44:03,113 DEBUG [client] Received heart-beat
2020-08-11 18:44:18,074 DEBUG [client] Received heart-beat
2020-08-11 18:44:33,121 DEBUG [client] Received heart-beat
2020-08-11 18:44:48,069 DEBUG [client] Received heart-beat
2020-08-11 18:45:03,116 DEBUG [client] Received heart-beat
------------------------------
Sunil I B
Original Message:
Sent: Mon August 10, 2020 10:26 PM
From: Sunil I B
Subject: Resilient Action Always Pending Status
HI Liam Mahoney,
Thanks a lot for the response, still facing same issues, we are unable to see fortigate queue destination and no any actions messages for the manual actions, please refer the attached logs.
Regarding with packaging of the functions, with help of developer guide using the functions, when Package the integration code & function, we received as depreciated error.
[root@resilient ~]# resilient-circuits codegen
DEPRECATED: The 'codegen' command has been deprecated for resilient-circuits. This functionality has been moved to the resilient-sdk tool.
[root@resilient ~]#
------------------------------
Sunil I B
Original Message:
Sent: Mon August 10, 2020 02:17 PM
From: Liam Mahoney
Subject: Resilient Action Always Pending Status
Sunil,
Apologies, I didn't see that you attached your log output to your initial message. I noticed it looks like the fortinet integration is getting loaded through the component auto-load directory. Unfortunately I don't have much experience using that feature. It does look like something for fortinet is getting loaded in your logs though:
2020-08-11 00:08:17,336 INFO [component_loader] Loaded and registered component 'fortinet'
I noticed it looks like you have the fortinet integration installed in a virtual environment based on the output of your pip list command, is this correct? I'm wondering if this might do anything:
1. stop the resilient_circuits process (sudo systemctl stop resilient_circuits)
2. activate the virtual environment you pip installed the integration in (looks like it's /root/venv - if this is correct, the command would be 'source /root/venv/bin/activate')
3. start resilient circuits with the command resilient-circuits run
4. once circuits is started, see if any fortinet messages are being written to the log, or if they're still hung up in the action status
Don't forget to kill the resilient_circuits process you manually started (ctrl+C) and restart your resilient circuits process (sudo systemctl start resilient_circuits) once your test is done.
I'm hoping it has something to do with the integration being installed in a virtual environment but your circuits process isn't leveraging that virtual environment. Again, I haven't used the auto load component directory, so I'm not sure if that should bypass what I'm thinking is wrong or not.
------------------------------
Liam Mahoney
Original Message:
Sent: Mon August 10, 2020 12:17 PM
From: Sunil I B
Subject: Resilient Action Always Pending Status
We downloaded some of the functions from XForce App Exchange for LDAP etc, those message we able to see in app.log files and also In our app logs we noticed that no errors for fortigate action script, however no message like destinations of the integration for Fortigate destination queue, please refer the attached pip and latest app.log.
[root@resilient tmp]# systemctl status resilient_circuits
â resilient_circuits.service - Resilient-Circuits Service
Loaded: loaded (/etc/systemd/system/resilient_circuits.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2020-08-11 00:08:15 +08; 3s ago
Main PID: 24171 (resilient-circu)
CGroup: /system.slice/resilient_circuits.service
ââ24171 /usr/local/bin/python2.7 /usr/local/bin/resilient-circuits run
Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: 2020-08-11 00:08:17,732 INFO [stomp_component] No Client heartbeats will be sent
Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: 2020-08-11 00:08:17,733 INFO [stomp_component] Requested heartbeats from server.
Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: 2020-08-11 00:08:17,734 DEBUG [client] Received heart-beat
Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: 2020-08-11 00:08:17,734 INFO [actions_component] resilient-circuits has started successfully and is now running...
Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: <Connected[stomp] ()>
Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: 2020-08-11 00:08:17,735 INFO [actions_component] STOMP connected.
Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: <registered[*] (<Timer/* 24171:MainThread (queued=0) [S]>, <StompClient/stomp 24171:MainThread (queued=0) [S]> )>
Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: <Connect_success[*] (<Connect[*] ()>, 'success' )>
Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: 2020-08-11 00:08:17,736 DEBUG [actions_component] Connected successfully. Resubscribe? False
Aug 11 00:08:17 resilient.localdomain resilient-circuits[24171]: <Connected_success[stomp] (<Connected[stomp] ()>, None )>
[root@resilient tmp]#
2020-08-11 00:08:16,991 DEBUG [connectionpool] https://resilient.localdomain:443 "GET /rest/orgs/201 HTTP/1.1" 200 None
2020-08-11 00:08:17,077 DEBUG [connectionpool] https://resilient.localdomain:443 "GET /rest/orgs/201/types/incident/fields HTTP/1.1" 200 None
2020-08-11 00:08:17,115 DEBUG [connectionpool] https://resilient.localdomain:443 "GET /rest/orgs/201/types/actioninvocation/fields HTTP/1.1" 200 None
2020-08-11 00:08:17,139 DEBUG [connectionpool] https://resilient.localdomain:443 "GET /rest/orgs/201/message_destinations HTTP/1.1" 200 None
2020-08-11 00:08:17,166 DEBUG [connectionpool] https://resilient.localdomain:443 "GET /rest/orgs/201/functions HTTP/1.1" 200 None
2020-08-11 00:08:17,196 DEBUG [connectionpool] https://resilient.localdomain:443 "GET /rest/orgs/201/functions/ldap_utilities_toggle_access HTTP/1.1" 200 None
2020-08-11 00:08:17,227 DEBUG [connectionpool] https://resilient.localdomain:443 "GET /rest/orgs/201/types/__function/fields HTTP/1.1" 200 None
2020-08-11 00:08:17,232 DEBUG [actions_component] Reset idle timer
2020-08-11 00:08:17,293 DEBUG [connectionpool] https://resilient.localdomain:443 "GET /rest/orgs/201/actions HTTP/1.1" 200 None
2020-08-11 00:08:17,300 INFO [app] Components auto-load directory: /root/venv/resilient-circuits/components
2020-08-11 00:08:17,301 INFO [component_loader] Loading 'fortinet' from /root/venv/resilient-circuits/components/fortinet.py
2020-08-11 00:08:17,325 DEBUG [actions_component] Reset idle timer
2020-08-11 00:08:17,326 DEBUG [actions_component] Reset idle timer
2020-08-11 00:08:17,326 WARNING [actions_component] Unverified STOMP TLS certificate (cafile=false)
2020-08-11 00:08:17,333 INFO [stomp_component] Connect to resilient.localdomain:65001
2020-08-11 00:08:17,335 INFO [app] App Started
2020-08-11 00:08:17,336 INFO [component_loader] Loaded and registered component 'fortinet'
2020-08-11 00:08:17,337 INFO [actions_component] STOMP attempting to connect
2020-08-11 00:08:17,337 INFO [app] Components loaded
2020-08-11 00:08:17,339 DEBUG [app] Components:
------------------------------
Sunil I B
Original Message:
Sent: Mon August 10, 2020 11:25 AM
From: Liam Mahoney
Subject: Resilient Action Always Pending Status
When you start circuits do you notice any log messages stating it's registered to the message destinations of the integration you are trying to use? If not, is it possible the integration wasn't pip installed properly?
------------------------------
Liam Mahoney
Original Message:
Sent: Thu August 06, 2020 08:37 AM
From: Sunil I B
Subject: Resilient Action Always Pending Status
Hi Ben Lurie,
Thanks a lot for the response, could you please share troubleshooting steps to investigate the issue further, Yes we noticed that during stop and start circuits able to process the script, but when we execute the actions no message returns to the logs.
FYI, we raised a case with IBM Support team, they requested to post here to resolve the issue.
------------------------------
Sunil I B
Original Message:
Sent: Thu August 06, 2020 08:23 AM
From: Ben Lurie
Subject: Resilient Action Always Pending Status
The reason it is in Pending status is because the resilient circuits service did not pick up the request from the action. There is no indication in the circuits log that is recognized there is an action message being proccessed. No errors either.
I think for some reason maybe circuits is not configured to get data from the appropriate resilient server.
Hard to say without additional troubleshooting.
Ben
------------------------------
Ben Lurie
Original Message:
Sent: Thu August 06, 2020 12:26 AM
From: Sunil I B
Subject: Resilient Action Always Pending Status
Anyone could you please help us on this asap
------------------------------
Sunil I B
Original Message:
Sent: Tue August 04, 2020 10:20 AM
From: Sunil I B
Subject: Resilient Action Always Pending Status
------------------------------
Regards,
Sunil I B,
014-3213219
------------------------------