IBM QRadar SOAR

IBM QRadar

Join this online user group to communicate across Security product users and IBM experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only
  • 1.  Email Header Validation output Change request

    Posted Wed July 17, 2019 11:30 AM
    Hi,

    The current output « results » does not contain the necessary information for our usage. We would like to get from a similar or an upgraded function the following:
    • Sender
    • Receivers
    • Ip sender
    • Ip receiver
    • Hostname sender
    • Hostname receiver
    • Reply-to
    • Full Text Header to do further regexp check in other process ;
    Could it be possible ?

    The current Output is:
                results = {
                    "dkim_verify": dkim_results,
                    "arc_verify": arc_results[0] == 'pass',
                    "dkim_message": dkim_message,
                    "arc_message": arc_results[2]
                }


    ------------------------------
    BENOIT ROSTAGNI - IBM Resilient
    ------------------------------


  • 2.  RE: Email Header Validation output Change request

    Posted Wed July 17, 2019 01:56 PM
    If you are talking about email header extraction, we built our own .eml parser for this purpose (among others), assuming you have access to the .eml file (attachment).

    It quite literally gets every element from the header into the results['header'] for the post-processor and subsequent processors.

    Take a look:
    https://github.com/jjfallete/resilient/blob/master/functions/utilities/utility_email_file_parser.py


    You can then use like this in the processor:

    for item, content in results.header:  # Get each header element and it's value
    if 'return-path' in item.lower():
       incident.addNote('The return-path was: ' + str(content))


    ------------------------------
    Jared Fagel
    Cyber Security Analyst Intern
    Public Utility
    ------------------------------



  • 3.  RE: Email Header Validation output Change request

    Posted Thu July 18, 2019 05:10 AM
    Thanks Jared, I need an additional help as when I tried to work you code, I get an error in the App trying to load it
    2019-07-18 11:02:51,524 ERROR [component_loader] Failed to load component 'utility_email_file_parser'
    2019-07-18 11:02:51,529 ERROR [component_loader] No module named utilities.util.selftest
    Traceback (most recent call last):
    File "/usr/local/lib/python2.7/site-packages/resilient_circuits/component_loader.py", line 40, in safe_but_noisy_import
    return __import__(name, globals(), locals(), [""])
    File "/home/resadmin/resilient_integration/components/utility_email_file_parser.py", line 24, in <module>
    import utilities.util.selftest as selftest
    ImportError: No module named utilities.util.selftest
    2019-07-18 11:02:51,530 ERROR [app] A component failed to load. The application cannot start.
    2019-07-18 11:02:51,531 INFO [app] App Stopped

    I have resolved some pre-requisits  in installing addition package,
    (sudo) pip install email
    (sudo) pip install BeautifulSoup

    but for this one  utilities.util.selftest, I can find the source

    Thanks for help

    ------------------------------
    BENOIT ROSTAGNI
    ------------------------------



  • 4.  RE: Email Header Validation output Change request

    Posted Thu July 18, 2019 10:18 AM
    You can remove the self test. This was added automatically when building the function, and does not serve a purpose.

    So, delete these two lines to fix:
    Line 24: import utilities.util.selftest as selftest
    Line 327: selftest.selftest_function(opts)

    @Shane Curtin my guess is that it would be tough to say if that would be useful, without seeing the actual email header those results are from. As in, is that including every header element (it looks to be missing some)?

    From what I can tell, the only items missing in my function's output results that are included in the results you posted are the metrics (not really useful) and the elements of the attachment(s). The function I have created just lists attachment names, adding the other elements would not be difficult if desired by looking at lines 215-232, and following guidance from Python's email.message documentation. I chose to directly upload the attachments to the incident, but you could add the payload into the attachments list as well (making a list of lists), although I do not see use in doing so.

    ------------------------------
    Jared Fagel
    Cyber Security Analyst Intern
    Public Utility
    ------------------------------



  • 5.  RE: Email Header Validation output Change request

    Posted Thu July 18, 2019 08:51 AM

    Hi Benoit,

    We currently have a new version of fn_utilities under development and it will include updates to the Utilities: Email Parse function.

    This function takes a .eml attachment and parses it.

    Here is a sample output of the updated function:

    results = {
        'inputs': {
            'incident_id': 2095,
            'attachment_id': 8,
            'utilities_parse_email_attachments': True
        },
        'metrics': {
            'package': 'unknown',
            'timestamp': '2019-07-17 16:38:18',
            'package_version': 'unknown',
            'host': 'xxx.ie.ibm.com',
            'version': '1.0',
            'execution_time_ms': 2132
        },
        'success': True,
        'content': {
            'body': 'After reviewing your profile with our management, we are pleased to...',
            'received': [{
                'from': 'info@example.com',
                'by': 'exmaple.com Postfix',
                'delay': 0,
                'date_utc': '2017-11-16T05:48:58',
                'hop': 1,
                'date': 'Thu, 16 Nov 2017 13:48:58 +0800 SGT',
                'with': 'ESMTPSA',
                'id': '80EABD601B2'
            }, {
                'from': 'example.com unknown x.x.x.x',
                'by': 'secure1.example.com Postfix',
                'delay': 78.0,
                'date_utc': '2017-11-16T05:50:16',
                'hop': 2,
                'date': 'Thu, 16 Nov 2017 13:50:16 +0800 SGT',
                'with': 'ESMTP',
                'id': '4A2A41561A55'
            }, {
                'from': '0.0.0.0',
                'delay': 474.0,
                'date_utc': '2017-11-16T05:58:10',
                'hop': 3,
                'date': 'Thu, 16 Nov 2017 05:58:10 +0000',
                'by': 'removed'
            }],
            'from': [
                ['Monika Bakun', 'admin@example.com']
            ],
            'attachments': [{
                'binary': True,
                'mail_content_type': 'application/msword',
                'charset': None,
                'filename': 'SKMBT_C201711xxxx.doc',
                'content-id': '',
                'content_transfer_encoding': 'base64',
                'payload': 'e1xydGZIMigxMS0xMyksSHVhbm...'
            }],
            'plain_body': '["After reviewing your profile with our management, we are pleased to..."]',
            'content-type': 'multipart/mixed; boundary="===============1876313359=="',
            'x-sf-rx-return-path': '<admin@example.com>',
            'to_domains': ['ibm.com'],
            'to': [
                ['Recipients', 'joe.bloggs@ibm.com']
            ],
            'html_body': '["<HTML><head><meta charset=iso-8859-1\\"/></head><BODY><P>After reviewing your profile with our management, we are pleased to ...</BODY></HTML>"]',
            'x-sf-helo-domain': 'secure1.example.com',
            'x-sf-originating-ip': '0.0.0.0',
            'date': '2017-11-16T05:49:54',
            'timezone': '+0',
            'mime-version': '1.0',
            'has_defects': False,
            'subject': 'Request For Quotation'
        },
        'reason': None,
        'version': '1.0'
    }


    Would the above help with your use case?



    ------------------------------
    Shane Curtin
    Integrations Engineer - IBM Resilient
    ------------------------------



  • 6.  RE: Email Header Validation output Change request

    Posted Wed July 31, 2019 12:30 PM

    Hi team

    How can I download and integrate this utility on my environment ? Where can I download it ?


    Also, is it possible to add the support for .msg file ?

    Thanks for your help.



    ------------------------------
    Jérôme PAQUET
    ------------------------------



  • 7.  RE: Email Header Validation output Change request

    Posted Wed July 31, 2019 01:14 PM
    Check with your sales engineer for guidance on using integrations. I linked mine above. I'm unsure when Resilient will update their version.

    As for msg to eml, this is possible if you build a function to do this. We have this planned in our Resilient development roadmap, and will publish it when we do (unless Resilient devs beat us to the punch!).

    These resources may be useful if you plan to do it youself, and are what we will use when the time comes:


    ------------------------------
    Jared Fagel
    Cyber Security Analyst Intern
    Public Utility
    ------------------------------



  • 8.  RE: Email Header Validation output Change request

    Posted Tue August 06, 2019 05:22 AM
    Hi @Jérôme PAQUET,

    Our Utility Functions package has be updated to v1.0.10 and you can download it from our App Exchange here: https://exchange.xforce.ibmcloud.com/hub/extension/2b6699ac8a3976b67dfbddee26dbe3a5

    We have updated the Email Parse Function and you can see the latest documentation here: https://github.com/ibmresilient/resilient-community-apps/tree/master/fn_utilities/doc#user-guide-fn_utilities_v1010

    The Email Parse Function now uses https://github.com/SpamScope/mail-parser to parse both .eml and .msg email files. In order to support parsing .msg files on your Integration Server, you will need to install the msgconvert tool. See the documentation linked above.

    ------------------------------
    Shane Curtin
    Integrations Engineer - IBM Resilient
    ------------------------------