IBM Security QRadar SOAR

Hi,

I have the URLScan.io integration installed, everything is configured right and url's that come in emails we receive are automatically sent to URLScan.io to be scanned. 

However, I learned that URLScan.io has rate limiting in place so their resources arent overwhelmed by misconfigured scripts. So they only accept 1 new scan every 2 seconds. The way i have it configured, Resilient pulls all URLs out of the email at once and pushes them out to get scanned (details below). When this happens, the first 1 completes fine and the rest error out. Has anyone else come across the same issue? Any tips on getting around this?

We still use IRHub instead of the new email function, and IRHub is set to add the email body as the Incident.Description. I use in-product scripting with regex to find all URL's in the description and add them as artifacts, then a separate rule with the conditionto call urlscan function with the condition that an artifact is created.

------------------------------
Wub a lub a dub dub!
------------------------------

Hello,

We had the same issue and the solution is to decorate the function with a retry with exponential back-off.

Unfortunately, we didn't open a PR to the original repository so these changes aren't publicly available but the key part is this:

@retry(exceptions=requests.HTTPError, tries=5, backoff=1.5, delay=2, jitter=(0, 1), logger=logging.getLogger(__name__))
def submit_url_for_analysis(options, url):
    try:
        response = requests.post(<<<FIXME>>>)
        response.raise_for_status()
    except requests.HTTPError as e:
        # HTTP 429 is returned when rate limit of submissions allowed on API is reached (2s between requests)
        if e.response.status_code == 429:
            raise e
        else:
            raise Exception("Report for incident could not be generated")
    else:
        return response

Bear in mind that you have to play with the parameters of the decorator and adjust them to your needs and workload .

Regards,
Carlos

------------------------------
Carlos Ortigoza
------------------------------

Original Message:
Sent: Fri September 13, 2019 04:39 PM
From: William Chen
Subject: URLScan.io rate limiting

Hi,

I have the URLScan.io integration installed, everything is configured right and url's that come in emails we receive are automatically sent to URLScan.io to be scanned.

However, I learned that URLScan.io has rate limiting in place so their resources arent overwhelmed by misconfigured scripts. So they only accept 1 new scan every 2 seconds. The way i have it configured, Resilient pulls all URLs out of the email at once and pushes them out to get scanned (details below). When this happens, the first 1 completes fine and the rest error out. Has anyone else come across the same issue? Any tips on getting around this?

We still use IRHub instead of the new email function, and IRHub is set to add the email body as the Incident.Description. I use in-product scripting with regex to find all URL's in the description and add them as artifacts, then a separate rule with the conditionto call urlscan function with the condition that an artifact is created.

------------------------------
Wub a lub a dub dub!
------------------------------

Hi William, 

Thank you for raising this on the community forum. I can see where this might be a problem as if your ingested email has many URLs, you will more than likely be challenged by this. 

When you are submitting all these URLs, because the Action Module processes them so fast, you are getting rate limited. You may be able to get the functionality you want by making a Workflow which combines both the URLScan.io integration and also one of the fn_utilities components Utilities: Timer

This is an excerpt from the source code detailing what it does :
This function implements a simple timer. A workflow using this function will sleep for the
specified amount of time. The function takes as input utilities_time or utilities_epoch as input.
The function periodically checks the status of the calling workflow and will end
function execution if the workflow has been terminated.



I wanted to try this out before posting but I got it to work and here is a screenshot showing the Workflow.



In the above example, I am using the Utilities Timer to delay each submission by 5 seconds followed by invoking the default UrlScanIO workflow unmodified. This is a great example of how you can encapsulate a workflow within another to resolve an issue like this. If you try this out, keep an eye on your resilient_circuits log and you should see the sleep timer working and then the URL submission right after.

Let us know if this works for you or any questions.

------------------------------
Ryan Gordon
Security Software Engineer
IBM
------------------------------

Original Message:
Sent: Fri September 13, 2019 04:39 PM
From: William Chen
Subject: URLScan.io rate limiting

Hi,

I have the URLScan.io integration installed, everything is configured right and url's that come in emails we receive are automatically sent to URLScan.io to be scanned.

However, I learned that URLScan.io has rate limiting in place so their resources arent overwhelmed by misconfigured scripts. So they only accept 1 new scan every 2 seconds. The way i have it configured, Resilient pulls all URLs out of the email at once and pushes them out to get scanned (details below). When this happens, the first 1 completes fine and the rest error out. Has anyone else come across the same issue? Any tips on getting around this?

We still use IRHub instead of the new email function, and IRHub is set to add the email body as the Incident.Description. I use in-product scripting with regex to find all URL's in the description and add them as artifacts, then a separate rule with the conditionto call urlscan function with the condition that an artifact is created.

------------------------------
Wub a lub a dub dub!
------------------------------

Thanks for the responses.

@Carlos Ortigoza; Im not too familiar with this so im not sure how/where/what to change. Where would this code go?

@Ryan Gordon; This might not work, at least with my config, since all the artifacts call the workflow at the same time, the timer would just delay them all the same amount and it would hit the same limitation, just 5 seconds later. lol. appreciate the idea though. it gave me the below idea, that u maybe could use if yours is similar to mine:

Adding a pause to the artifact creation in my 'in product' script. The script regex matches all the URLs first and puts it into an array. then i use a loop to put each url into an artifact. Im thinking of putting a "time.sleep" in the middle of that loop, so the script pauses between each artifact creation, since artifact creation automatically pushes it out to URLScan.

That way, the script creates a url artifact, and pauses; while it's paused there, Res is already sending the first url out to URLScan.io and retrieving the results. then the script continues through the loop.

Putting it all down on paper (or screen, whatever) really helped.

------------------------------
Wub a lub a dub dub!
------------------------------

Original Message:
Sent: Tue September 17, 2019 04:15 AM
From: Ryan Gordon
Subject: URLScan.io rate limiting

Hi William,

Thank you for raising this on the community forum. I can see where this might be a problem as if your ingested email has many URLs, you will more than likely be challenged by this.

When you are submitting all these URLs, because the Action Module processes them so fast, you are getting rate limited. You may be able to get the functionality you want by making a Workflow which combines both the URLScan.io integration and also one of the fn_utilities components Utilities: Timer

This is an excerpt from the source code detailing what it does :
This function implements a simple timer. A workflow using this function will sleep for the
specified amount of time. The function takes as input utilities_time or utilities_epoch as input.
The function periodically checks the status of the calling workflow and will end
function execution if the workflow has been terminated.



I wanted to try this out before posting but I got it to work and here is a screenshot showing the Workflow.

In the above example, I am using the Utilities Timer to delay each submission by 5 seconds followed by invoking the default UrlScanIO workflow unmodified. This is a great example of how you can encapsulate a workflow within another to resolve an issue like this. If you try this out, keep an eye on your resilient_circuits log and you should see the sleep timer working and then the URL submission right after.

Let us know if this works for you or any questions.

------------------------------
Ryan Gordon
Security Software Engineer
IBM

Original Message:
Sent: Fri September 13, 2019 04:39 PM
From: William Chen
Subject: URLScan.io rate limiting

Hi,

I have the URLScan.io integration installed, everything is configured right and url's that come in emails we receive are automatically sent to URLScan.io to be scanned.

However, I learned that URLScan.io has rate limiting in place so their resources arent overwhelmed by misconfigured scripts. So they only accept 1 new scan every 2 seconds. The way i have it configured, Resilient pulls all URLs out of the email at once and pushes them out to get scanned (details below). When this happens, the first 1 completes fine and the rest error out. Has anyone else come across the same issue? Any tips on getting around this?

We still use IRHub instead of the new email function, and IRHub is set to add the email body as the Incident.Description. I use in-product scripting with regex to find all URL's in the description and add them as artifacts, then a separate rule with the conditionto call urlscan function with the condition that an artifact is created.

------------------------------
Wub a lub a dub dub!
------------------------------

Nevermind. Resilient doesnt allow importing the time module in the in product scripting. back to square 1

------------------------------
Wub a lub a dub dub!
------------------------------

Original Message:
Sent: Tue September 17, 2019 04:58 PM
From: William Chen
Subject: URLScan.io rate limiting

Thanks for the responses.

@Carlos Ortigoza; Im not too familiar with this so im not sure how/where/what to change. Where would this code go?

@Ryan Gordon; This might not work, at least with my config, since all the artifacts call the workflow at the same time, the timer would just delay them all the same amount and it would hit the same limitation, just 5 seconds later. lol. appreciate the idea though. it gave me the below idea, that u maybe could use if yours is similar to mine:

Adding a pause to the artifact creation in my 'in product' script. The script regex matches all the URLs first and puts it into an array. then i use a loop to put each url into an artifact. Im thinking of putting a "time.sleep" in the middle of that loop, so the script pauses between each artifact creation, since artifact creation automatically pushes it out to URLScan.

That way, the script creates a url artifact, and pauses; while it's paused there, Res is already sending the first url out to URLScan.io and retrieving the results. then the script continues through the loop.

Putting it all down on paper (or screen, whatever) really helped.

------------------------------
Wub a lub a dub dub!

Original Message:
Sent: Tue September 17, 2019 04:15 AM
From: Ryan Gordon
Subject: URLScan.io rate limiting

Hi William,

Thank you for raising this on the community forum. I can see where this might be a problem as if your ingested email has many URLs, you will more than likely be challenged by this.

When you are submitting all these URLs, because the Action Module processes them so fast, you are getting rate limited. You may be able to get the functionality you want by making a Workflow which combines both the URLScan.io integration and also one of the fn_utilities components Utilities: Timer

This is an excerpt from the source code detailing what it does :
This function implements a simple timer. A workflow using this function will sleep for the
specified amount of time. The function takes as input utilities_time or utilities_epoch as input.
The function periodically checks the status of the calling workflow and will end
function execution if the workflow has been terminated.



I wanted to try this out before posting but I got it to work and here is a screenshot showing the Workflow.

In the above example, I am using the Utilities Timer to delay each submission by 5 seconds followed by invoking the default UrlScanIO workflow unmodified. This is a great example of how you can encapsulate a workflow within another to resolve an issue like this. If you try this out, keep an eye on your resilient_circuits log and you should see the sleep timer working and then the URL submission right after.

Let us know if this works for you or any questions.

------------------------------
Ryan Gordon
Security Software Engineer
IBM

Original Message:
Sent: Fri September 13, 2019 04:39 PM
From: William Chen
Subject: URLScan.io rate limiting

Hi,

I have the URLScan.io integration installed, everything is configured right and url's that come in emails we receive are automatically sent to URLScan.io to be scanned.

However, I learned that URLScan.io has rate limiting in place so their resources arent overwhelmed by misconfigured scripts. So they only accept 1 new scan every 2 seconds. The way i have it configured, Resilient pulls all URLs out of the email at once and pushes them out to get scanned (details below). When this happens, the first 1 completes fine and the rest error out. Has anyone else come across the same issue? Any tips on getting around this?

We still use IRHub instead of the new email function, and IRHub is set to add the email body as the Incident.Description. I use in-product scripting with regex to find all URL's in the description and add them as artifacts, then a separate rule with the conditionto call urlscan function with the condition that an artifact is created.

------------------------------
Wub a lub a dub dub!
------------------------------

Hi William,

We changed this in the Python code (the one in the Python package itself).

I will try to open a PR but I can't tell you when and, more importantly, it's not sure that the maintainer of the repository will accept my changes.

If that happens, I will fork the repository and leave it publicly available for you.

Regards,
Carlos

------------------------------
Carlos Ortigoza
------------------------------

Original Message:
Sent: Tue September 17, 2019 04:58 PM
From: William Chen
Subject: URLScan.io rate limiting

Thanks for the responses.

@Carlos Ortigoza; Im not too familiar with this so im not sure how/where/what to change. Where would this code go?

@Ryan Gordon; This might not work, at least with my config, since all the artifacts call the workflow at the same time, the timer would just delay them all the same amount and it would hit the same limitation, just 5 seconds later. lol. appreciate the idea though. it gave me the below idea, that u maybe could use if yours is similar to mine:

Adding a pause to the artifact creation in my 'in product' script. The script regex matches all the URLs first and puts it into an array. then i use a loop to put each url into an artifact. Im thinking of putting a "time.sleep" in the middle of that loop, so the script pauses between each artifact creation, since artifact creation automatically pushes it out to URLScan.

That way, the script creates a url artifact, and pauses; while it's paused there, Res is already sending the first url out to URLScan.io and retrieving the results. then the script continues through the loop.

Putting it all down on paper (or screen, whatever) really helped.

------------------------------
Wub a lub a dub dub!

Original Message:
Sent: Tue September 17, 2019 04:15 AM
From: Ryan Gordon
Subject: URLScan.io rate limiting

Hi William,

Thank you for raising this on the community forum. I can see where this might be a problem as if your ingested email has many URLs, you will more than likely be challenged by this.

When you are submitting all these URLs, because the Action Module processes them so fast, you are getting rate limited. You may be able to get the functionality you want by making a Workflow which combines both the URLScan.io integration and also one of the fn_utilities components Utilities: Timer

This is an excerpt from the source code detailing what it does :
This function implements a simple timer. A workflow using this function will sleep for the
specified amount of time. The function takes as input utilities_time or utilities_epoch as input.
The function periodically checks the status of the calling workflow and will end
function execution if the workflow has been terminated.



I wanted to try this out before posting but I got it to work and here is a screenshot showing the Workflow.

In the above example, I am using the Utilities Timer to delay each submission by 5 seconds followed by invoking the default UrlScanIO workflow unmodified. This is a great example of how you can encapsulate a workflow within another to resolve an issue like this. If you try this out, keep an eye on your resilient_circuits log and you should see the sleep timer working and then the URL submission right after.

Let us know if this works for you or any questions.

------------------------------
Ryan Gordon
Security Software Engineer
IBM

Original Message:
Sent: Fri September 13, 2019 04:39 PM
From: William Chen
Subject: URLScan.io rate limiting

Hi,

I have the URLScan.io integration installed, everything is configured right and url's that come in emails we receive are automatically sent to URLScan.io to be scanned.

However, I learned that URLScan.io has rate limiting in place so their resources arent overwhelmed by misconfigured scripts. So they only accept 1 new scan every 2 seconds. The way i have it configured, Resilient pulls all URLs out of the email at once and pushes them out to get scanned (details below). When this happens, the first 1 completes fine and the rest error out. Has anyone else come across the same issue? Any tips on getting around this?

We still use IRHub instead of the new email function, and IRHub is set to add the email body as the Incident.Description. I use in-product scripting with regex to find all URL's in the description and add them as artifacts, then a separate rule with the conditionto call urlscan function with the condition that an artifact is created.

------------------------------
Wub a lub a dub dub!
------------------------------

Thanks Carlos. I'll be keeping an eye out for changes in the app exchange

------------------------------
Wub a lub a dub dub!
------------------------------

Original Message:
Sent: Thu September 19, 2019 03:58 AM
From: Carlos Ortigoza
Subject: URLScan.io rate limiting

Hi William,

We changed this in the Python code (the one in the Python package itself).

I will try to open a PR but I can't tell you when and, more importantly, it's not sure that the maintainer of the repository will accept my changes.

If that happens, I will fork the repository and leave it publicly available for you.

Regards,
Carlos

------------------------------
Carlos Ortigoza

Original Message:
Sent: Tue September 17, 2019 04:58 PM
From: William Chen
Subject: URLScan.io rate limiting

Thanks for the responses.

@Carlos Ortigoza; Im not too familiar with this so im not sure how/where/what to change. Where would this code go?

@Ryan Gordon; This might not work, at least with my config, since all the artifacts call the workflow at the same time, the timer would just delay them all the same amount and it would hit the same limitation, just 5 seconds later. lol. appreciate the idea though. it gave me the below idea, that u maybe could use if yours is similar to mine:

Adding a pause to the artifact creation in my 'in product' script. The script regex matches all the URLs first and puts it into an array. then i use a loop to put each url into an artifact. Im thinking of putting a "time.sleep" in the middle of that loop, so the script pauses between each artifact creation, since artifact creation automatically pushes it out to URLScan.

That way, the script creates a url artifact, and pauses; while it's paused there, Res is already sending the first url out to URLScan.io and retrieving the results. then the script continues through the loop.

Putting it all down on paper (or screen, whatever) really helped.

------------------------------
Wub a lub a dub dub!

Original Message:
Sent: Tue September 17, 2019 04:15 AM
From: Ryan Gordon
Subject: URLScan.io rate limiting

Hi William,

Thank you for raising this on the community forum. I can see where this might be a problem as if your ingested email has many URLs, you will more than likely be challenged by this.

When you are submitting all these URLs, because the Action Module processes them so fast, you are getting rate limited. You may be able to get the functionality you want by making a Workflow which combines both the URLScan.io integration and also one of the fn_utilities components Utilities: Timer

This is an excerpt from the source code detailing what it does :
This function implements a simple timer. A workflow using this function will sleep for the
specified amount of time. The function takes as input utilities_time or utilities_epoch as input.
The function periodically checks the status of the calling workflow and will end
function execution if the workflow has been terminated.



I wanted to try this out before posting but I got it to work and here is a screenshot showing the Workflow.

In the above example, I am using the Utilities Timer to delay each submission by 5 seconds followed by invoking the default UrlScanIO workflow unmodified. This is a great example of how you can encapsulate a workflow within another to resolve an issue like this. If you try this out, keep an eye on your resilient_circuits log and you should see the sleep timer working and then the URL submission right after.

Let us know if this works for you or any questions.

------------------------------
Ryan Gordon
Security Software Engineer
IBM

Original Message:
Sent: Fri September 13, 2019 04:39 PM
From: William Chen
Subject: URLScan.io rate limiting

Hi,

I have the URLScan.io integration installed, everything is configured right and url's that come in emails we receive are automatically sent to URLScan.io to be scanned.

However, I learned that URLScan.io has rate limiting in place so their resources arent overwhelmed by misconfigured scripts. So they only accept 1 new scan every 2 seconds. The way i have it configured, Resilient pulls all URLs out of the email at once and pushes them out to get scanned (details below). When this happens, the first 1 completes fine and the rest error out. Has anyone else come across the same issue? Any tips on getting around this?

We still use IRHub instead of the new email function, and IRHub is set to add the email body as the Incident.Description. I use in-product scripting with regex to find all URL's in the description and add them as artifacts, then a separate rule with the conditionto call urlscan function with the condition that an artifact is created.

------------------------------
Wub a lub a dub dub!
------------------------------