IBM Security Resilient

Expand all | Collapse all

How to avoid Resilient REST API conflict while uploading attachments from two different workflows?

Jump to Best Answer
  • 1.  How to avoid Resilient REST API conflict while uploading attachments from two different workflows?

    Posted Wed March 04, 2020 02:16 PM
    Hi all,

    I am facing a very core problem with Resilient platform's Python SDK. I have a rule which gets triggered when a new artifact of type "DNS Name" is created. The rule invokes a workflow which has my function that connects to an external API, gets the screenshot of the latest URL under that "DNS Name" artifact and uploads it to the incident as an attachment. Everything works fine when a single artifact is created in the incident.

    The problem occurs when I create multiple artifacts of type "DNS Name" simultaneously from a script. The rule is triggered for each artifact and two separate workflows are launched. When both of these workflows try to upload the attachment, one of them fails and my function in that workflow raises the exception below:

    Traceback (most recent call last):
      File "/home/integration/fn_slashnext/fn_slashnext/components/funct_slashnext_host_report.py", line 76, in _slashnext_host_report_function
        attachment_utils.attach_screenshot(response_list[2], scan_id)
      File "/home/integration/fn_slashnext/fn_slashnext/util/attachment_utils.py", line 17, in attach_screenshot
        self._attach_file(json_response.get('scData').get('scBase64'), file_name, 'image/jpeg', self.incident_id)
      File "/home/integration/fn_slashnext/fn_slashnext/util/attachment_utils.py", line 39, in _attach_file
        incident_id, content_type=content_type)
      File "/home/integration/ibm/lib/python2.7/site-packages/resilient_lib/components/resilient_common.py", line 277, in write_file_attachment
        mimetype=content_type)
      File "/home/integration/ibm/lib/python2.7/site-packages/resilient/co3.py", line 505, in post_attachment
        _raise_if_error(ex.get_response())
      File "/home/integration/ibm/lib/python2.7/site-packages/resilient/co3.py", line 211, in _raise_if_error
        raise SimpleHTTPException(response)
    SimpleHTTPException: Conflict:  {"error_code":"generic","hints":[],"message":"Conflict","success":false,"title":null}
    


    I am using the write_file_attachment function available in resilient_lib package. The class that I have implemented for this is given below (file: attachment_utils.py):

    from io import BytesIO
    import base64
    from resilient_lib import write_file_attachment
    
    
    class ResilientAttachments:
    
        def __init__(self, incident_id, rest_client, logger):
            self.incident_id = incident_id
            self.rest_client = rest_client
            self.logger = logger
    
        def attach_screenshot(self, json_response, name):
            if json_response.get('errorNo') == 0:
                file_name = name + '.jpeg'
                self.logger.info('Uploading Screenshot Attachment')
                self._attach_file(json_response.get('scData').get('scBase64'), file_name, 'image/jpeg', self.incident_id)
                self.logger.info('Uploading Screenshot Attachment Complete')
    
        def _attach_file(self, base64_data, file_name, content_type, incident_id):
            datastream = BytesIO(base64.b64decode(base64_data))
            new_attachment = write_file_attachment(self.rest_client, file_name, datastream,
                                                   incident_id, content_type=content_type)
            # self.logger.info(json.dumps(new_attachment))
    

    and then in my function, I use this class in my function code as (only relevant code shown):

    from attachment_utils import ResilientAttachments
    
    attachment_utils = ResilientAttachments(incident_id, self.rest_client(), self.logger)
    attachment_utils.attach_screenshot(response_list[2], scan_id)

    My guess is that the Resilient REST API, that the resilient_lib is using at back-end is not thread-safe and when two workflows simultaneously try to upload an attachment, one of them faces a conflict and has to abort. Can someone please point out any workaround for this issue as it is disturbing the core-functionality of Resilient and there is no way I can create my playbook without multiple workflows running simultaneously on different artifacts. Any help will be highly appreciated. 

    Thanks!



    ------------------------------
    Umair Ahmed
    ------------------------------


  • 2.  RE: How to avoid Resilient REST API conflict while uploading attachments from two different workflows?

    Posted Thu March 05, 2020 07:40 AM
    Edited by Mark Scherfling Thu March 05, 2020 07:40 AM
    Hi Umair,

    Your analysis may be correct. Can you open a support case with the Resilient log file so we can review? 

    Regards,
    Mark

    ------------------------------
    Mark Scherfling
    ------------------------------



  • 3.  RE: How to avoid Resilient REST API conflict while uploading attachments from two different workflows?
    Best Answer

    Posted Fri March 06, 2020 08:15 AM
    Edited by Carlos Ortigoza Fri March 06, 2020 01:12 PM
    Hi Umair,

    This is in response to the message you've sent to me on a different post (which I was unable to find, but it's not important).

    The problem you are facing is not related to thread-safety. You are getting this:
    SimpleHTTPException: Conflict:  {"error_code":"generic","hints":[],"message":"Conflict","success":false,"title":null}


    This is basically a race condition between the two invocations to your workflow: by the time the second action wants to update the incident, the first one has performed an update already.

    I'm not using resilient_lib but I have implemented similar functions to add attachments to Resilient and what I've done is handle the "resilient.SimpleHTTPException" (which is just a subclass of a "requests" exception) and check the status code. If it's Conflict 429 (or any other you decide you want to retry the operation), we raise the exception to let a retry decorator catch this and retry the operation as many times as we want and with the delay we want and so on. It looks similar to this:

    @retry(exceptions=resilient.co3.SimpleHTTPException, tries=5, delay=10, backoff=2, logger=logging.getLogger(__name__))
    def _add_attachment(self, attachment_filepath):
        try:
            self.rest_client.post_attachment(attachment_uri, attachment_filepath)
            return True
        except resilient.co3.SimpleHTTPException as e:
            if e.response.status_code in [409]:
                # Raise the exception so the retry decorator can see it
                raise e
            else:
                return False

    I hope this helps but feel free to reply to this message if you have troubles.
    ------------------------------

    Regards,
    Carlos Ortigoza
    ------------------------------



  • 4.  RE: How to avoid Resilient REST API conflict while uploading attachments from two different workflows?

    Posted Fri March 06, 2020 10:55 AM
    Hey Carlos,

    I tried your solution and it worked like a charm. It's really a very simple solution to this problem but I have observed that the "tries", "delay" and "backoff" parameters of the retry decorator need to be tuned according to the number of parallel workflows running. For some values of these parameters and a large number of workflows, I still saw some conflicts going on.

    Can you offer any insights on how should I tweak these parameters?

    Finally, I believe this approach should not have any impact on the performance of the platform since we are only increasing the number of requests to the REST server and nothing else. Please correct me if otherwise. 

    Thanks a lot for your valuable support!

    ------------------------------
    Umair Ahmed
    ------------------------------



  • 5.  RE: How to avoid Resilient REST API conflict while uploading attachments from two different workflows?

    Posted Fri March 06, 2020 11:10 AM
    Hi Umair,

    Indeed, you have to set those parameters according to your needs. There will always be the chance to keep having collisions and the more parallel actions running, the more likely the situation will be. The key in these cases is to also make use of a random jitter to improve the chances that these "workers" get away from each others:
    :param jitter: extra seconds added to delay between attempts. default: 0.
                   fixed if a number, random if a range tuple (min, max)

    By passing a tuple you will introduce randomness but you will still have to find the optimal values depending on the conditions of your environment.

    ------------------------------
    Regards,
    Carlos Ortigoza
    ------------------------------



  • 6.  RE: How to avoid Resilient REST API conflict while uploading attachments from two different workflows?

    Posted Fri March 06, 2020 01:12 PM
    Carlos,

    Can you share the retry decorator code that you are using? I found one online that does not have the jitter code and I am currently short on time to write my own. Thanks!

    ------------------------------
    Umair Ahmed
    ------------------------------



  • 7.  RE: How to avoid Resilient REST API conflict while uploading attachments from two different workflows?

    Posted Fri March 06, 2020 02:18 PM
    Hi Umair,

    It's this one: https://pypi.org/project/retry/

    ------------------------------
    Regards,
    Carlos Ortigoza
    ------------------------------