IBM QRadar SOAR

 View Only

  • 1.  URLScan.io is failing to work on some conditions, error message not being implemented in the App

    Posted Thu March 25, 2021 04:37 AM

    1) "We could not scan this website" - 404 error

    when the target website is down, Resilient fail in downloading the image generated link :
    and failed in downloading the API report: https://urlscan.io/api/v1/result/3b42b797-6a48-4f32-ae17-6090c9ab4b25/

    Urlscan remove preview
    View this on Urlscan >


    Results is an integration application log error, and losing valid report link of website, report that exist at ScanIO, and delivering an error in the Action & Workflow status, not mentioned to Analyst.

    Log details:

    2021-03-24 23:06:56,083 INFO [decorators] [urlscanio] StatusMessage: Downloaded report from https://urlscan.io/api/v1/result/3b42b797-6a48-4f32-ae17-6090c9ab4b25/
2021-03-24 23:06:56,264 ERROR [requests_common] 404 Client Error: Not Found for url: https://urlscan.io/screenshots/3b42b797-6a48-4f32-ae17-6090c9ab4b25.png
2021-03-24 23:06:56,289 ERROR [actions_component] <task[functionworker] (<function function.__call__.<locals>.decorated.<locals>._call_the_task at 0x7f650c65f1e0>, <urlscanio[functions.urlscanio] (id=133, workflow=enrichment_scanio, user=analyst@local.io) 2021-03-24 23:06:23.744000> urlscanio_url='https://telefonfiyatlari.org', incident_id=2135)> (<class 'resilient_circuits.action_message.FunctionException_'>): 
Traceback (most recent call last):
File "/opt/app-root/lib/python3.6/site-packages/resilient_lib/components/requests_common.py", line 130, in execute_call_v2
response.raise_for_status()
File "/opt/app-root/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://urlscan.io/screenshots/3b42b797-6a48-4f32-ae17-6090c9ab4b25.png

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/app-root/lib/python3.6/site-packages/fn_urlscanio/components/urlscanio.py", line 103, in _urlscanio_function
urlscanio_png_get = req_common.execute_call_v2("GET", urlscanio_png_url, self.timeout)
File "/opt/app-root/lib/python3.6/site-packages/resilient_lib/components/requests_common.py", line 138, in execute_call_v2
raise IntegrationError(msg)
resilient_lib.components.integration_errors.IntegrationError: '404 Client Error: Not Found for url: https://urlscan.io/screenshots/3b42b797-6a48-4f32-ae17-6090c9ab4b25.png'

    2) Error: 429 Client Error: Too Many Requests for url

    When getting multiple URL artifact in a new incident, SCanIO refuse to deliver the result, as too many requests are made.

    Looking at the error and looping with a pause before re-submitting should avoid those errors, only visible in the Action & Workflow status, not mentioned to Analyst, and can't be use in a workflow process design (Pause - resubmit)

    Traceback (most recent call last): File "/opt/app-root/lib/python3.6/site-packages/resilient_lib/components/requests_common.py", line 130, in execute_call_v2 response.raise_for_status() File "/opt/app-root/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://urlscan.io/api/v1/scan/ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/app-root/lib/python3.6/site-packages/fn_urlscanio/components/urlscanio.py", line 70, in _urlscanio_function data=json.dumps(urlscanio_data)) File "/opt/app-root/lib/python3.6/site-packages/resilient_lib/components/requests_common.py", line 138, in execute_call_v2 raise IntegrationError(msg) resilient_lib.components.integration_errors.IntegrationError: '429 Client Error: Too Many Requests for url: https://urlscan.io/api/v1/scan/'

    Log:

    2021-03-25 08:25:35,386 INFO [urlscanio] urlscanio_url: https://kindlink.global
2021-03-25 08:25:35,487 ERROR [requests_common] 429 Client Error: Too Many Requests for url: https://urlscan.io/api/v1/scan/
2021-03-25 08:25:35,488 ERROR [actions_component] <task[functionworker] (<function function.__call__.<locals>.decorated.<locals>._call_the_task at 0x7f6850634840>, <urlscanio[functions.urlscanio] (id=133, workflow=enrichment_scanio, user=analyst@local.io) 2021-03-25 08:25:34.090000> urlscanio_url='https://kindlink.global', incident_id=2135)> (<class 'resilient_circuits.action_message.FunctionException_'>): 
Traceback (most recent call last):
File "/opt/app-root/lib/python3.6/site-packages/resilient_lib/components/requests_common.py", line 130, in execute_call_v2
response.raise_for_status()
File "/opt/app-root/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://urlscan.io/api/v1/scan/

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/app-root/lib/python3.6/site-packages/fn_urlscanio/components/urlscanio.py", line 70, in _urlscanio_function
data=json.dumps(urlscanio_data))
File "/opt/app-root/lib/python3.6/site-packages/resilient_lib/components/requests_common.py", line 138, in execute_call_v2
raise IntegrationError(msg)
resilient_lib.components.integration_errors.IntegrationError: '429 Client Error: Too Many Requests for url: https://urlscan.io/api/v1/scan/'

    3) 400 error

    On some Websites we get a 400 error - See log below

    Results is an integration application log error, and losing a valid report link of website, report that exist at ScanIO, and delivering an error in the Action & Workflow status, not mentioned to Analyst.

    021-03-24 23:06:32,775 INFO [urlscanio] urlscanio_url: https://www.antibasic.ga
2021-03-24 23:06:34,120 ERROR [requests_common] 400 Client Error: Bad Request for url: https://urlscan.io/api/v1/scan/
2021-03-24 23:06:34,180 ERROR [actions_component] <task[functionworker] (<function function.__call__.<locals>.decorated.<locals>._call_the_task at 0x7f650c4e38c8>, <urlscanio[functions.urlscanio] (id=133, workflow=enrichment_scanio, user=analyst@local.io) 2021-03-24 23:06:32.269000> urlscanio_url='https://www.antibasic.ga', incident_id=2135)> (<class 'resilient_circuits.action_message.FunctionException_'>): 
Traceback (most recent call last):
File "/opt/app-root/lib/python3.6/site-packages/resilient_lib/components/requests_common.py", line 130, in execute_call_v2
response.raise_for_status()
File "/opt/app-root/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://urlscan.io/api/v1/scan/

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/app-root/lib/python3.6/site-packages/fn_urlscanio/components/urlscanio.py", line 70, in _urlscanio_function
data=json.dumps(urlscanio_data))
File "/opt/app-root/lib/python3.6/site-packages/resilient_lib/components/requests_common.py", line 138, in execute_call_v2
raise IntegrationError(msg)
resilient_lib.components.integration_errors.IntegrationError: '400 Client Error: Bad Request for url: https://urlscan.io/api/v1/scan/'


    How can the code be modified to push back the error when there is one (to be processed by the workflow), and not just break the action and the workflow?



    ------------------------------
    BENOIT ROSTAGNI
    ------------------------------


  • 2.  RE: URLScan.io is failing to work on some conditions, error message not being implemented in the App

    Posted Thu March 25, 2021 10:45 AM
    Hi Benoit,

    Unfortunately the Function is designed to Fail when it gets an error from urlscan.io

    It will have to be a code change in the FunctionComponent code to address this

    We can make this change on our next release of it

    ------------------------------
    Shane Curtin
    Apps Engineer - IBM Resilient
    ------------------------------

    Original Message


  • 3.  RE: URLScan.io is failing to work on some conditions, error message not being implemented in the App

    Posted Wed June 29, 2022 04:18 PM
    Hi Shane,

    Would it be possible to also handle 409 status error code and any other error code gracefully (catch all) so that function doesn't break no matter what new condition occur in the future? 

    I think if a graceful error message is passed to the post-processing script, it can be handled that way but an exception within function completely breaks playbook.

    Thanks,
    Alex

    ------------------------------
    Alex Trylysenko
    ------------------------------

    Original Message


  • 4.  RE: URLScan.io is failing to work on some conditions, error message not being implemented in the App

    Posted Mon July 04, 2022 07:55 AM
    Hi Alex,

    Since my last post, we have actually updated resilient-circuits to handle a new tap_exception config - see https://community.ibm.com/community/user/security/blogs/shane-curtin1/2022/05/23/release-of-v45i-python-libraries-to-pypi for more details

    In your app's section in the app.config file, just add:
    [my_app]
trap_exception=True​


    Hopefully this supports your use case

    ------------------------------
    Shane Curtin
    Apps Engineer - IBM Security SOAR
    ------------------------------

    Original Message


  • 5.  RE: URLScan.io is failing to work on some conditions, error message not being implemented in the App

    Posted Mon August 22, 2022 03:45 PM
    Hi Shane,

    The trap exception feature is not working for us - anything we can do? We are running version 45.2.37 and latest error we received for URLScan.IO is:
    An error occurred while processing the action acknowledgement.  Additional information: Function result 'playbook.functions.results.fn_urlscanio_output' exceeds the maximum size of 5 MB.
Downloaded PNG screenshot from https://urlscan.io/screenshots/e36ddd91-89f9-49bb-aaf0-952527d9ad19.png
Downloaded report from https://urlscan.io/api/v1/result/e36ddd91-89f9-49bb-aaf0-952527d9ad19/
Submitted URL successfully as e36ddd91-89f9-49bb-aaf0-952527d9ad19​

    Any help is appreciated. 

    This is our app.config just in case:
    [urlscanio]
# API key for urlscan.io
urlscanio_api_key = $API_Key
# Base URL for the urlscanio API
urlscanio_report_url = https://urlscan.io/api/v1
# Base URL to access screenshots in urlscanio
urlscanio_screenshot_url = https://urlscan.io/screenshots
# Catch all exceptions
trap_exception=True​

    Thank you,
    Alex

    ------------------------------
    Alex Trylysenko
    ------------------------------

    Original Message


  • 6.  RE: URLScan.io is failing to work on some conditions, error message not being implemented in the App

    Posted Mon August 29, 2022 07:36 AM
    Hi Alex,

    My sincere apologies. I miss spoke.

    trap_exception must be set at a "global" level under the [resilient] heading!

    [resilient]
trap_exception=True​​

    Now however, I would need to see the full error here to debug further. Is that error message taken from resilient-circuits? It seems the error may have occurred on SOAR itself and if that is the case, the config will be no use here - we would need to modify how your Playbooks are "chained" together or see how you are using this App - maybe the correct solution is to modify it so it does not return the screenshot as bytes as well as attaching it to the Incident?

    Shane



    ------------------------------
    Shane Curtin
    Apps Engineer - IBM Security SOAR
    ------------------------------

    Original Message