IBM Security QRadar SOAR

 View Only
Expand all | Collapse all

Send Artifacts via Outbound Email

  • 1.  Send Artifacts via Outbound Email

    Posted Fri September 03, 2021 08:51 AM
    Hi Team,

    We are trying to send artifacts scan data(threat score etc ) via fn_email_outbound. We need to fetch Source and destination IP and their reputation from artifacts and send as Outbound email text to networking team.  we are using the following extension IBM Security App Exchange - Outbound Email for SOAR
    Ibmcloud remove preview
    IBM Security App Exchange - Outbound Email for SOAR
    IBM X-Force Exchange is a threat intelligence sharing platform enabling research on security threats, aggregation of intelligence, and collaboration with peers
    View this on Ibmcloud >

       
    The current flow is based on Incident and we cannot directly call the artifact. 

    If you have any idea that how can we send the artifacts scan result(X-force) to the respective team via outbound email text. do let us know.

    Regards,
    Asad Aftab

    ------------------------------
    Asad Aftab
    ------------------------------


  • 2.  RE: Send Artifacts via Outbound Email

    IBM Champion
    Posted Fri September 03, 2021 05:19 PM
    Edited by Liam Mahoney Fri September 03, 2021 05:19 PM

    Asad,

    I would recommend creating a function to get all of the artifacts on an incident. This function would use the Resilient API, specifically the GET /orgs/{org_id}/incidents/{inc_id}/artifacts endpoint (it's possible this function already exists on the community, I'm not sure).

    This endpoint will return a list of dictionaries where each dictionary is an artifact on the incident. Within each artifact's dictionary will be a hits key. Here's an example with most of the other artifact data removed:

    {
    ...
    "hits": [
          {
            "value": "https://exmaple.com",
            "threat_source_id": 122,
            "artifact_type_id": 3,
            "properties": {
              "Analysis Start Time": "2021-09-03T19:53:45+00:00",
              "Full Report": "https://sandbox-analysis.com/urls/573e2bc054fd1dea202989be35b18abc5e64ef38d18a3dca13e71c2f57882d93",
              "Threat Score": "72",
              "Verdict": "malicious"
            },
            "active": true
          }
        ],
    ...
    }


    If you pass in the query parameter handle_format=objects with your API request, you'll get the CTS name along with the ID. This would also be helpful for the key type, which will contain the ID of the artifact type and the name of the artifact type (URL, DNS Name, etc.).

    Once this function is created I would call it in the workflow before you call the outbound email function. Within the pre-process script of the outbound email function you could iterate over the results of the first function, look for any applicable artifact types, and then look for the scan data you want included in the email (threat score, etc.). You could then use that data as the input to the outbound email function.

    Best of luck!



    ------------------------------
    Liam Mahoney
    ------------------------------



  • 3.  RE: Send Artifacts via Outbound Email

    Posted Mon September 06, 2021 11:53 AM
    hello liam,

    hope you are doing fine. Kindly tell me about how to creating a function to get all of the artifacts on an incident. like you said that use restAPI so kindly can you tell us a little more description and where to add that function in Resilient.

    Regards
    Asad Aftab


    ------------------------------
    Asad Aftab
    ------------------------------



  • 4.  RE: Send Artifacts via Outbound Email

    IBM Champion
    Posted Mon September 06, 2021 12:03 PM
    Asad,

    I think the App Developer's Guide documentation will explain it better than I can. If you have any questions I'd be happy to try and answer them.

    Once you start developing the function I'd take a look at the way this function uses self.rest_client to make requests to the IBM SOAR API. There is a helper object provided with the output of the codegen that makes interacting with the IBM SOAR API very easy from within a function.

    Let us know if you run into any problems

    ------------------------------
    Liam Mahoney
    ------------------------------



  • 5.  RE: Send Artifacts via Outbound Email

    Posted Mon September 06, 2021 06:18 PM
    Hello Liam, 

    I have exactly the same problem and same confusion, what does creating  a function mean in IBM SOAR? Does it mean  we have to install resilient-sdk on our environment through pip install resilient-sdk and write an extension/function? 
    OR 
    Creating a function means creating a function from soar web UI 
    Customize>function> and create a function

    I have gone through App Developer's Guide documentation multiple times, there is no guide/example to create custom function. I don't know where to write the function and where to place the .py file on my soar system.

    ------------------------------
    Mohsin Ali
    ------------------------------



  • 6.  RE: Send Artifacts via Outbound Email

    IBM Champion
    Posted Tue September 07, 2021 09:53 AM
    Mohsin,

    I haven't watched this one, but I'm thinking this video will probably explain everything I'm about to talk about a bit more clearly.

    Maybe this guide will help too, it is a little outdated, but it's what I used when I was first starting to use the product and it helped me understand the process. Any mention of any resilient-circuits commands other then resilient-circuits run have been replaced by resilient-sdk. The commands should be nearly identical other than that.

    A function (really a package, there can be many functions in a single package) is a separate process from the IBM SOAR console process(es) that registers with IBM SOAR and enables IBM SOAR to send that package messages when it encounters a function within a workflow. So you define the inputs to a function within a workflow, IBM SOAR then sends those inputs to the code you create, your code executes and returns data back to IBM SOAR and then the post-process script in the workflow can operate on the results of that function.

    This can get a little confusing at first, but once you've made a couple of functions you'll get the hang of it pretty quick. 

    The first think you will need to create is a message destination and then a function that uses that message destination within the GUI. Once you do that you will then start writing python code.

    resilient-sdk is a pypi library that provides some functionality useful for developers, you would need to pip install it into whatever environment you plan on doing your development in (I highly recommend using a python virtual environment for each package you create). When creating a function you can use the resilient-sdk codegen command to generate a boilerplate package (there can be multiple functions per a package - generally I make a package for a single security tool and include many functions within it). When you use the command you need to pass in the function / message destination that you have created within the IBM SOAR GUI. Within the files of the boilerplate project, the code that runs when the function gets called is under the package_name/package_name/components directory, where package_name is the name you specify when using the resilient-sdk codegen command. These are the files you would want to edit with whatever business logic / processes you need your function to do.

    Now that you have the files for a function you need a way to connect the function to Resilient and run it. While you are still developing the function(s) I would use an Integration Server (see this guide for more information). This is essentially a computer that has the pypi library resilient-circuits installed and a process created from using the resilient-circuits run command. Here's a diagram that may illustrate it better than my words:



    So you will need to create an integration server to actually run your function. I generally only use integration servers while I'm developing a function. Once I feel the function is complete I then move it to our apphost, but that's another topic. I do all of my development on my computer, so I turn my computer into an integration server by running the command resilient-circuits run and while the process that command creates is running my computer is registered to IBM SOAR and can listen for messages from it. Another note is that you will need to install the package (the python files you have been working on) before running resilient-circuits run in order to tell IBM SOAR that those functions are available on your computer.

    I would focus on first creating a connection to IBM SOAR with an integration server. Once you are able to connect successfully I would then start working on creating functions.

    Let us know if you have any questions

    ------------------------------
    Liam Mahoney
    ------------------------------



  • 7.  RE: Send Artifacts via Outbound Email

    Posted Tue September 07, 2021 10:07 PM
    Hello Liam, 

    Thankyou for you support, I was able to understand the functions and their lifecycle. I am reproducing the steps with screen shots I have performed till now, we do not have apphosts configured in our environment,  we are running apps on resilient circuits (which is installed on the SOAR machine)
    1. Installed resilient-SDK
    2. created following message destination

    3. Created the following function

     3.a) Organization ID is 
     
    3.b) Incident ID

    3.c) - artifact ID


    4.  Created workflow as 

    4.a )  input


    4. b )  pre process script

    4.c) post process script

    5. Rule 



    6.) generated boilerplate package using following command
      resilient-sdk codegen -p fn_artifacts_hits -m email_outbound1 -f attach_artifact_hits --rule "artifacts_hits" --workflow artifactshits tbound1 -f attach_artifact_hits --rule "artifacts_hits" --workflow artifactshits


    7. )  boilerplate was successfully crreated at following path 

    8.) our code to fetch artifact using RESTAPI

    # -*- coding: utf-8 -*-
    
    """AppFunction implementation"""
    
    import json
    import requests
    from requests.api import get
    from requests.models import Response
    from resilient_circuits import AppFunctionComponent, app_function, FunctionResult
    from resilient_lib import IntegrationError, validate_fields
    from resilient_lib.components.resilient_common import LOG
    
    PACKAGE_NAME = "fn_artifacts_hits"
    FN_NAME = "attach_artifact_hits"
    
    
    class FunctionComponent(AppFunctionComponent):
        """Component that implements function 'attach_artifact_hits'"""
    
        def __init__(self, opts):
            super(FunctionComponent, self).__init__(opts, PACKAGE_NAME)
    
        @app_function(FN_NAME)
        def _app_function(self, fn_inputs):
            """
            Function: None
            Inputs:
                -   fn_inputs.artifact_hits_xforce
            """
    
            yield self.status_message("Starting App Function: '{0}'".format(FN_NAME))
    
            # Example validating app_configs
            # validate_fields([
            #     {"name": "api_key", "placeholder": "<your-api-key>"},
            #     {"name": "base_url", "placeholder": "<api-base-url>"}],
            #     self.options)
    
            # Example getting access to self.get_fn_msg()
            # fn_msg = self.get_fn_msg()
            # self.LOG.info("fn_msg: %s", fn_msg)
    
            # Example raising an exception
            # raise IntegrationError("Example raising custom error")
    
            ##############################################
            # PUT YOUR FUNCTION IMPLEMENTATION CODE HERE #
            ##############################################
    
            # Call API implemtation example:
    
            def run(self, params={}):
                org_id = params.get("organization_id")
                inc_id = params.get("incident_id")
                artifact_id = params.get("artifact_id")
    
                url = self.connection.API_BASE + "/orgs/{org_id}/incidents/{inc_id}/artifacts/{artifact_id}".format(
                    org_id=org_id,
                    inc_id=inc_id,
                    artifact_id=artifact_id)
    
                self.logger.info("Retrieving artifact for incident %s..." % inc_id)
                try:
                    response = self.connection.SESSION.get(url)
                    artifact = response.json()
                    artifact = komand.helper.clean(artifact)
                except (requests.ConnectionError, requests.HTTPError, KeyError, ValueError) as error:
                    self.logger.error("Error: %s" % error)
                    raise
    
                return {"artifact": artifact}
    
                # params = {
            #     "api_key": self.app_configs.api_key,
            #     "ip_address": fn_inputs.artifact_value
            # }
            #
            # response = self.rc.execute(
            #     method="get",
            #     url=self.app_configs.api_base_url,
            #     params=params
            # )
            #
            # results = response.json()
    
            ##############################################
            yield self.status_message("Finished running App Function: '{0}'".format(FN_NAME))
    
            results = {"mock_key": "Mock Value!"}
    
            yield FunctionResult(results)
    # yield FunctionResult({}, success=False, reason="Bad call")
    ​

    9.) STOMP Component


    10.) Triggering the Rule

    11.) Response on resilient-circuit console


    Note: it hung at "returning results"

    12.) Error on Action Status




    This is the detail of our problem and I am stuck at this point. I really appreciate your help. Once this Use case is in running form I will be able to create other function. I am very thankful to you for your support and effort you put in our case. I believe i have attached enough information but if you need anything else please let me know.

    ------------------------------
    Mohsin Ali
    ------------------------------



  • 8.  RE: Send Artifacts via Outbound Email

    Posted Tue September 07, 2021 11:28 PM
    Thankyou Mohsin for providing the details. I really appreciate the effort that LIAM is putting in this case. Hope that I will solve that problem with the help of you guys.

    ------------------------------
    Asad Aftab
    ------------------------------



  • 9.  RE: Send Artifacts via Outbound Email

    IBM Champion
    Posted Wed September 08, 2021 10:12 AM

    Moshin,

    Is your end goal to get the data about a single artifact, or all of the artifacts on the incident?

    Right now you are using the function `Attach Artifact Hits` within the workflow, but in the post-process script of that function you are using artifacts.hits (the data on the single artifact the workflow was triggered on) instead of the data coming back from the function, which would be in the object results. Depending on your use case you may be able to delete the function and create a script with the Object Type of Artifact to get the required data. However, if you need more than one artifact's data within the workflow then you'll need the function.

    I don't see why your post-process script is causing that exception. It looks like you are passing `incident.addNote()` a value properly. I'm curious if you can reload all of your tabs / windows and double check the post-process script of the function still has incident.addNote(artifact.hits) and then run the workflow again? 

    One thing that I find very helpful when creating workflows is to have a tab  with the new script editor (Customization Settings -> Scripts -> new script) open. You can test scripts out on incidents / different object types to get a better idea of the data available. For example if I have the following in the new script editor:



    When I click Run it prompts for an incident ID:


    And since I have the object type 'Artifact' selected for the script it will then ask me for a specific artifact on that incident to run the script against:


    Then when I select one of the artifacts on this screen, that is the artifact that will be the artifact object within my test script. 

    Since the script uses log.info(..), a window pops up with the data you pass into that method. This way you can see the data you would be working with in your post-process script in your workflow, since it's using the artifact object. In your workflow's context, the artifact object would be whatever artifact the rule was called on.

    These test scripts aren't limited to only using log.info(..), you can do any operations available to scripts to test the functionality this way.

    Let me know if any of that was unclear!



    ------------------------------
    Liam Mahoney
    ------------------------------



  • 10.  RE: Send Artifacts via Outbound Email

    Posted Thu September 09, 2021 02:49 AM
    Edited by BENOIT ROSTAGNI Thu September 09, 2021 02:50 AM
      |   view attached

    Hi Asad, I have been reviewing this problem and will suggest another approach, more simple, using directly just script, fields and rules.

    1) Create a Field named Artifact TI Detected Text Area + Rich Text
    2) Create a Script Artifact TI Detected type Artifact that will analyze the TI field and update the new Artifact TI Detectedfield with new values
    The script example attached is looking at X-Force, VirusTotal & Threat Grid to extract values and present them:If the script is run multiple times (each time a new threat is detected), it will update the relevant section of the target field Artifact TI Detectedfield.
    A prepared rich_text colored values exist on each section if you want to modify it.
    You can also use classic json, dict and list walk python code to walk the global output of the Threat Intel and write those values in the field.
    3) Create a Artifact TI Detected Rule at artifact level, to launch the script in -2-, each time artifact.hits is changed
    the Artifact TI DetectedField will always be updated with the latest values.
    4) add the field in an email sending template, and you will send the Threat Intelligence enrichment extract (as seen here, or all using walk) by email when you want.



    ------------------------------
    BENOIT ROSTAGNI
    ------------------------------

    Attachment(s)

    py
    Artifact_TI_Detected.py   8 KB 1 version


  • 11.  RE: Send Artifacts via Outbound Email

    Posted Mon September 13, 2021 08:07 AM
    Hello BENOIT,

    As you send us the script of Artifact_TI_Detected.py. I work great as manual but when we need to run the script automatically to send the Artifact.hits data to outbound email. Currently we have to manually run the script once an incident is created it then populates the Artifact TI Detected text area, which we are using in the "send email outbound " template. Any work around for this ?   I am attaching screen shots for better understanding.

    how we can automatically run the script as our workflow is incident based and the script is artifact based.

    regards,
    Asad Aftab


    ------------------------------
    Asad Aftab
    ------------------------------



  • 12.  RE: Send Artifacts via Outbound Email

    Posted Wed September 15, 2021 04:05 AM
    Hi, Asad,

    The idea to move from the Artifact level to the Incident level is to use the Artifact TI Detected field populated by the script.
    Each time the Artifact gets a Hit - Artifact Level - it changes the Field Artifact TI Detected at Incident Level.

    You can use an automatic incident rule that says: Each time the field Artifact TI Detected send the mail using outbound email with the content of Artifact TI Detected. But in this case, many emails will be sent (each time a hit match)
    I would suggest using a combination of the scheduler app and a boolean test field Artifact TI Send Waiting State :
    • Create a Menu Item Incident Level rule Artifact TI Send by email that is sending the email using outbound email with the content of Artifact TI Detected and setting the field Artifact TI Send Waiting State to FALSE. This rule could be used to sent "manually" by analyst if needed. But the head is to use the scheduler to run it.
    • Create an Automatic Incident Level rule Artifact TI create Scheduler: Each time the field Artifact TI Detected and Artifact TI Send Waiting State is not TRUE, set the Artifact TI Send Waiting State to TRUE and run the workflow Artifact TI Scheduler rule
    • The workflow Artifact TI Scheduler rule is a workflow base on the example from the App Scheduler, that will allow you to run a rule at a specific time, frequency, etc... I suggest you set the parameters to: run after 5 minutes the rule Artifact TI Send by email. Using the 5 minutes window will allow, after the first HIT, to collect the majority of all TI information, and will normally send only ONE email.



    ------------------------------
    BENOIT ROSTAGNI
    ------------------------------



  • 13.  RE: Send Artifacts via Outbound Email

    Posted Thu September 16, 2021 10:11 AM

    Hello 

    I want to thanks @Liam Mahoney and @BENOIT ROSTAGNI for the help. You both guys helped us a lot.

    ​​​​​​

    ------------------------------
    Asad Aftab
    ------------------------------