IBM QRadar SOAR

I am already using the Utilities:Email Parser function, but I am wondering if anyone has a good way to parse the email attachment of an .EML file (base64) and add it as an attached file to the incident. I am able to parse the other headers I need to create Artifacts, but am stuck on the attachment part.

------------------------------
Ryan Terry
------------------------------

Hi

I have the same requirement . What would be really useful would be to take the e-mail attachment and add it as an artefact to the incident  and create the hash files as well all within the email parser or workflow  . i.e. mimic the add malware sample function that is available under add artefact

Regards

Nigel

------------------------------
nigel sykes
------------------------------

Original Message:
Sent: 02-25-2019 05:13 PM
From: Ryan Terry
Subject: Email Parser for EML attachments

I am already using the Utilities:Email Parser function, but I am wondering if anyone has a good way to parse the email attachment of an .EML file (base64) and add it as an attached file to the incident. I am able to parse the other headers I need to create Artifacts, but am stuck on the attachment part.

------------------------------
Ryan Terry
------------------------------

Hi Ryan,
              If the email comes into resilient with the original email as an attachment then from a script you can add it as an attachment to the incident. This allows you to then kick off an automated workflow to take further actions and parse your artifacts out of it. This video shows a very similar use case to what you described Ryan & I think it's a good starting point for you: https://www.youtube.com/watch?v=jacDmwf-C3g

Hopefully it helps.

------------------------------
PATRICK DIVILLY
------------------------------

Original Message:
Sent: 02-26-2019 06:06 AM
From: nigel sykes
Subject: Email Parser for EML attachments

Hi

I have the same requirement . What would be really useful would be to take the e-mail attachment and add it as an artefact to the incident  and create the hash files as well all within the email parser or workflow  . i.e. mimic the add malware sample function that is available under add artefact

Regards

Nigel

------------------------------
nigel sykes

Original Message:
Sent: 02-25-2019 05:13 PM
From: Ryan Terry
Subject: Email Parser for EML attachments

I am already using the Utilities:Email Parser function, but I am wondering if anyone has a good way to parse the email attachment of an .EML file (base64) and add it as an attached file to the incident. I am able to parse the other headers I need to create Artifacts, but am stuck on the attachment part.

------------------------------
Ryan Terry
------------------------------

Hi Nigel,
          Currently from the scripts you are limited in the type of artifacts you can add to the incident. They can only be of type strings e.g. incident.addArtifact("Email Attachment Name", "ImBad.pdf", "It's bad don't open it"). This means you won't be able to attache a file as a malware sample directly from a script. However you can add the attachment as an attachment on the incident and then fire off a workflow which would be able to add it to the artifacts as a malware sample. You might also find the youtube link I posted just above to Ryan useful to show how this can be done.

------------------------------
PATRICK DIVILLY
------------------------------

Original Message:
Sent: 02-26-2019 06:06 AM
From: nigel sykes
Subject: Email Parser for EML attachments

Hi

I have the same requirement . What would be really useful would be to take the e-mail attachment and add it as an artefact to the incident  and create the hash files as well all within the email parser or workflow  . i.e. mimic the add malware sample function that is available under add artefact

Regards

Nigel

------------------------------
nigel sykes

Original Message:
Sent: 02-25-2019 05:13 PM
From: Ryan Terry
Subject: Email Parser for EML attachments

I am already using the Utilities:Email Parser function, but I am wondering if anyone has a good way to parse the email attachment of an .EML file (base64) and add it as an attached file to the incident. I am able to parse the other headers I need to create Artifacts, but am stuck on the attachment part.

------------------------------
Ryan Terry
------------------------------

Hi all,

Check out my implementation here to be used on the .eml mail file attachment (if available for you):

https://github.com/jjfallete/resilient/blob/master/functions/utilities/utility_email_file_parser.py


It works very well. In your post processor here is some content to get started:

# Get each header element and it's value
for item, content in results.mail_items:
if item.lower() == 'to':
 for to in content.split(','):
 if '@' not in content and '<' in content and '>' in content: to_address = content.split('>')[0].split('<')[-1]
 else: to_address = ((to.split('@')[0].split('<')[-1] + '@' + to.split('@')[-1].split('>')[0]).replace('"', '').replace("'", ''))
 try: to_name = to.split('"')[1].split('"')[-1]
 except: to_name = to_address
 to_list.append([to_address, to_name])

elif item.lower() == 'from':
 if '@' not in content and '<' in content and '>' in content: from_address = content.split('>')[0].split('<')[-1]
 else: from_address = (content.split('@')[0].split('<')[-1] + '@' + content.split('@')[-1].split('>')[0]).replace('"', '').replace("'", '')
 try: from_name = content.replace("'", '"').split('"')[1].split('"')[-1] # content.rsplit(' ')[-1].replace('<', '').replace('>', '')
 except: from_name = from_address

# ... 

# Clean returned HTML for rich-text description, note, etc.
email_body_cleaned = ''
email_body = (results.body).strip()
for line in email_body.splitlines():
        line = line.replace('<a ', ' <a ')
        while ' ' in line: line = line.replace(' ', ' ')  # Replace two spaces with one space
        email_body_cleaned += (line)
email_body_cleaned = helper.createRichText(email_body_cleaned.strip())

# ...

for url in results.urls: 
       pattern = re.compile("(http|https|file|gopher|ftp):\\/\\/[^\\s]+")
        if pattern.match(url): incident.addArtifact('URL', url, 'URL from email body')
        else: incident.addArtifact('DNS Name', url, 'URL from email body') # Use a different artifact type. Or... should we append http:// to start and do as 'URL' ??? TBD

​​

------------------------------
Jared Fagel
Cyber Security Analyst Intern
Public Utility
------------------------------

Original Message:
Sent: 02-25-2019 05:13 PM
From: Ryan Terry
Subject: Email Parser for EML attachments

I am already using the Utilities:Email Parser function, but I am wondering if anyone has a good way to parse the email attachment of an .EML file (base64) and add it as an attached file to the incident. I am able to parse the other headers I need to create Artifacts, but am stuck on the attachment part.

------------------------------
Ryan Terry
------------------------------​

Jared,

Thank you for sharing. Are you requiring Attachment to Base64 input or what would you use for Input and Pre-Process Script within the Workflow?


Thanks,
Ryan

------------------------------
Ryan Terry
------------------------------

Original Message:
Sent: 02-27-2019 07:16 PM
From: Jared Fagel
Subject: Email Parser for EML attachments

Hi all,

Check out my implementation here to be used on the .eml mail file attachment (if available for you):

https://github.com/jjfallete/resilient/blob/master/functions/utility_email_file_parser.py

I still need to clean the code up (commenting, etc.), but it should work well.


In your post processor here is some content to get started:

# Get each header element and it's value
for item, content in results.mail_items:
if item.lower() == 'to':
       to_address = (content.split('@')[0].split('<')[-1] + '@' + content.split('@')[-1].split('>')[0]).replace('"', '').replace("'", '')
       try: to_name = content.replace("'", '"').split('"')[1].split('"')[-1]
       except: to_name = to_address

# ... 

# Clean returned HTML for rich-text description, note, etc.
email_body_cleaned = ''
email_body = (results.body).strip()
for line in email_body.splitlines():
       line = line.replace('<a ', ' <a ')
       while ' ' in line: line = line.replace(' ', ' ')  # Replace two spaces with one space
       email_body_cleaned += (line)
email_body_cleaned = helper.createRichText(email_body_cleaned.strip())

# ...

for url in results.urls: 
       pattern = re.compile("(http|https|file|gopher|ftp):\\/\\/[^\\s]+")
       if pattern.match(url): incident.addArtifact('URL', url, 'URL from email body')
       else: incident.addArtifact('DNS Name', url, 'URL from email body') # Use a different artifact type. Or... should we append http:// to start and do as 'URL' ??? TBD



​​

------------------------------
Jared Fagel
Cyber Security Analyst Intern
Public Utility

Original Message:
Sent: 02-25-2019 05:13 PM
From: Ryan Terry
Subject: Email Parser for EML attachments

I am already using the Utilities:Email Parser function, but I am wondering if anyone has a good way to parse the email attachment of an .EML file (base64) and add it as an attached file to the incident. I am able to parse the other headers I need to create Artifacts, but am stuck on the attachment part.

------------------------------
Ryan Terry
------------------------------​

Hey Ryan,

It does not require that utility; I tried to make it as simple as possible. It uses incident.id, attachment.id, and attachment.name (lines 328-330 in the function).


So, for pre-processor:
inputs.incident_id = incident.id
inputs.attachment_id = attachment.id
inputs.attachment_name = attachment.name

​The only requirement is that the function uses BeautifulSoup, so this must be pip installed onto the integration server. This requirement could be removed by using RegEx to look for img tag URLs, but I thought it better and less error-prone to use BeautifulSoup. See the use of this on lines 266-267. We do this to ensure that the URLs returned are not from images.


Let me know if you have any other questions. I spent some time working on this, and the more that benefit from it, the better!

------------------------------
Jared Fagel
Cyber Security Analyst Intern
Public Utility
------------------------------

Original Message:
Sent: 02-28-2019 04:56 PM
From: Ryan Terry
Subject: Email Parser for EML attachments

Jared,

Thank you for sharing. Are you requiring Attachment to Base64 input or what would you use for Input and Pre-Process Script within the Workflow?


Thanks,
Ryan

------------------------------
Ryan Terry

Original Message:
Sent: 02-27-2019 07:16 PM
From: Jared Fagel
Subject: Email Parser for EML attachments

Hi all,

Check out my implementation here to be used on the .eml mail file attachment (if available for you):

https://github.com/jjfallete/resilient/blob/master/functions/utility_email_file_parser.py

I still need to clean the code up (commenting, etc.), but it should work well.


In your post processor here is some content to get started:

# Get each header element and it's value
for item, content in results.mail_items:
if item.lower() == 'to':
       to_address = (content.split('@')[0].split('<')[-1] + '@' + content.split('@')[-1].split('>')[0]).replace('"', '').replace("'", '')
       try: to_name = content.replace("'", '"').split('"')[1].split('"')[-1]
       except: to_name = to_address

# ... 

# Clean returned HTML for rich-text description, note, etc.
email_body_cleaned = ''
email_body = (results.body).strip()
for line in email_body.splitlines():
       line = line.replace('<a ', ' <a ')
       while ' ' in line: line = line.replace(' ', ' ')  # Replace two spaces with one space
       email_body_cleaned += (line)
email_body_cleaned = helper.createRichText(email_body_cleaned.strip())

# ...

for url in results.urls: 
       pattern = re.compile("(http|https|file|gopher|ftp):\\/\\/[^\\s]+")
       if pattern.match(url): incident.addArtifact('URL', url, 'URL from email body')
       else: incident.addArtifact('DNS Name', url, 'URL from email body') # Use a different artifact type. Or... should we append http:// to start and do as 'URL' ??? TBD



​​

------------------------------
Jared Fagel
Cyber Security Analyst Intern
Public Utility

Original Message:
Sent: 02-25-2019 05:13 PM
From: Ryan Terry
Subject: Email Parser for EML attachments

I am already using the Utilities:Email Parser function, but I am wondering if anyone has a good way to parse the email attachment of an .EML file (base64) and add it as an attached file to the incident. I am able to parse the other headers I need to create Artifacts, but am stuck on the attachment part.

------------------------------
Ryan Terry
------------------------------​

Even after installing the BeautifulSoup, its giving error 
BeautifulSoup (3.2.1)
beautifulsoup4 (4.7.1)
bs4 (0.0.1)




------------------------------
Zeeshan Ahmad
------------------------------

Original Message:
Sent: Thu February 28, 2019 06:40 PM
From: Jared Fagel
Subject: Email Parser for EML attachments

Hey Ryan,

It does not require that utility; I tried to make it as simple as possible. It uses incident.id, attachment.id, and attachment.name (lines 328-330 in the function).


So, for pre-processor:
inputs.incident_id = incident.id
inputs.attachment_id = attachment.id
inputs.attachment_name = attachment.name

​The only requirement is that the function uses BeautifulSoup, so this must be pip installed onto the integration server. This requirement could be removed by using RegEx to look for img tag URLs, but I thought it better and less error-prone to use BeautifulSoup. See the use of this on lines 266-267. We do this to ensure that the URLs returned are not from images.


Let me know if you have any other questions. I spent some time working on this, and the more that benefit from it, the better!

------------------------------
Jared Fagel
Cyber Security Analyst Intern
Public Utility

Original Message:
Sent: 02-28-2019 04:56 PM
From: Ryan Terry
Subject: Email Parser for EML attachments

Jared,

Thank you for sharing. Are you requiring Attachment to Base64 input or what would you use for Input and Pre-Process Script within the Workflow?


Thanks,
Ryan

------------------------------
Ryan Terry

Original Message:
Sent: 02-27-2019 07:16 PM
From: Jared Fagel
Subject: Email Parser for EML attachments

Hi all,

Check out my implementation here to be used on the .eml mail file attachment (if available for you):

https://github.com/jjfallete/resilient/blob/master/functions/utility_email_file_parser.py

I still need to clean the code up (commenting, etc.), but it should work well.


In your post processor here is some content to get started:

# Get each header element and it's value
for item, content in results.mail_items:
if item.lower() == 'to':
       to_address = (content.split('@')[0].split('<')[-1] + '@' + content.split('@')[-1].split('>')[0]).replace('"', '').replace("'", '')
       try: to_name = content.replace("'", '"').split('"')[1].split('"')[-1]
       except: to_name = to_address

# ... 

# Clean returned HTML for rich-text description, note, etc.
email_body_cleaned = ''
email_body = (results.body).strip()
for line in email_body.splitlines():
       line = line.replace('<a ', ' <a ')
       while ' ' in line: line = line.replace(' ', ' ')  # Replace two spaces with one space
       email_body_cleaned += (line)
email_body_cleaned = helper.createRichText(email_body_cleaned.strip())

# ...

for url in results.urls: 
       pattern = re.compile("(http|https|file|gopher|ftp):\\/\\/[^\\s]+")
       if pattern.match(url): incident.addArtifact('URL', url, 'URL from email body')
       else: incident.addArtifact('DNS Name', url, 'URL from email body') # Use a different artifact type. Or... should we append http:// to start and do as 'URL' ??? TBD



​​

------------------------------
Jared Fagel
Cyber Security Analyst Intern
Public Utility

Original Message:
Sent: 02-25-2019 05:13 PM
From: Ryan Terry
Subject: Email Parser for EML attachments

I am already using the Utilities:Email Parser function, but I am wondering if anyone has a good way to parse the email attachment of an .EML file (base64) and add it as an attached file to the incident. I am able to parse the other headers I need to create Artifacts, but am stuck on the attachment part.

------------------------------
Ryan Terry
------------------------------​

Hey @Zeeshan Ahmad

​​That is because this is a function, not a script. You must create this as a function for use in a workflow. Contact your sales engineer for assistance with doing this.☺


Jared

------------------------------
Jared Fagel
Cyber Security Analyst Intern
Public Utility
------------------------------

Original Message:
Sent: Wed April 24, 2019 08:39 AM
From: Zeeshan Ahmad
Subject: Email Parser for EML attachments

Even after installing the BeautifulSoup, its giving error 
BeautifulSoup (3.2.1)
beautifulsoup4 (4.7.1)
bs4 (0.0.1)




------------------------------
Zeeshan Ahmad

Original Message:
Sent: Thu February 28, 2019 06:40 PM
From: Jared Fagel
Subject: Email Parser for EML attachments

Hey Ryan,

It does not require that utility; I tried to make it as simple as possible. It uses incident.id, attachment.id, and attachment.name (lines 328-330 in the function).


So, for pre-processor:
inputs.incident_id = incident.id
inputs.attachment_id = attachment.id
inputs.attachment_name = attachment.name

​The only requirement is that the function uses BeautifulSoup, so this must be pip installed onto the integration server. This requirement could be removed by using RegEx to look for img tag URLs, but I thought it better and less error-prone to use BeautifulSoup. See the use of this on lines 266-267. We do this to ensure that the URLs returned are not from images.


Let me know if you have any other questions. I spent some time working on this, and the more that benefit from it, the better!

------------------------------
Jared Fagel
Cyber Security Analyst Intern
Public Utility

Original Message:
Sent: 02-28-2019 04:56 PM
From: Ryan Terry
Subject: Email Parser for EML attachments

Jared,

Thank you for sharing. Are you requiring Attachment to Base64 input or what would you use for Input and Pre-Process Script within the Workflow?


Thanks,
Ryan

------------------------------
Ryan Terry

Original Message:
Sent: 02-27-2019 07:16 PM
From: Jared Fagel
Subject: Email Parser for EML attachments

Hi all,

Check out my implementation here to be used on the .eml mail file attachment (if available for you):

https://github.com/jjfallete/resilient/blob/master/functions/utility_email_file_parser.py

I still need to clean the code up (commenting, etc.), but it should work well.


In your post processor here is some content to get started:

# Get each header element and it's value
for item, content in results.mail_items:
if item.lower() == 'to':
       to_address = (content.split('@')[0].split('<')[-1] + '@' + content.split('@')[-1].split('>')[0]).replace('"', '').replace("'", '')
       try: to_name = content.replace("'", '"').split('"')[1].split('"')[-1]
       except: to_name = to_address

# ... 

# Clean returned HTML for rich-text description, note, etc.
email_body_cleaned = ''
email_body = (results.body).strip()
for line in email_body.splitlines():
       line = line.replace('<a ', ' <a ')
       while ' ' in line: line = line.replace(' ', ' ')  # Replace two spaces with one space
       email_body_cleaned += (line)
email_body_cleaned = helper.createRichText(email_body_cleaned.strip())

# ...

for url in results.urls: 
       pattern = re.compile("(http|https|file|gopher|ftp):\\/\\/[^\\s]+")
       if pattern.match(url): incident.addArtifact('URL', url, 'URL from email body')
       else: incident.addArtifact('DNS Name', url, 'URL from email body') # Use a different artifact type. Or... should we append http:// to start and do as 'URL' ??? TBD



​​

------------------------------
Jared Fagel
Cyber Security Analyst Intern
Public Utility

Original Message:
Sent: 02-25-2019 05:13 PM
From: Ryan Terry
Subject: Email Parser for EML attachments

I am already using the Utilities:Email Parser function, but I am wondering if anyone has a good way to parse the email attachment of an .EML file (base64) and add it as an attached file to the incident. I am able to parse the other headers I need to create Artifacts, but am stuck on the attachment part.

------------------------------
Ryan Terry
------------------------------​