Hi there, 

We recently came across the URLScan.io app on the app exchange that we were excited to use. In order for it to work in our environment we had to make a few modifications that I wanted to share in case anyone else runs into these same issues.

Our environment utilizes a proxy. The first thing we noticed after installation and running the workflow was a proxy error. We downloaded the code from Github to make the following modifications. 

In our app.config we added two keys for the proxy. We also had to change the config section name to urlscanio for some reason. 

#[fn_urlscanio] -> original name when installed

[urlscanio]
urlscanio_api_key=OUR-API-KEY
https_proxy = https://proxy:8080
http_proxy = http://proxy:8080

In the code in the __init__ function we added the following: 

self.proxies = {
 "https": self.options.get("https_proxy"),
 "http": self.options.get("http_proxy")
 }

​​​
And then anywhere in the code we saw requests.post or a requests.get we added: 

proxies=self.proxies

So it ended up looking like this (had to do it 4 times):  

 urlscanio_post = requests.post('https://urlscan.io/api/v1/scan/', headers=urlscanio_headers, data=json.dumps(urlscanio_data), proxies=self.proxies)
urlscanio_get = requests.get('https://urlscan.io/api/v1/result/{}/'.format(uuid), proxies=self.proxies)

urlscanio_report_get = requests.get(urlscanio_report_url, proxies=self.proxies)
urlscanio_png_get = requests.get(urlscanio_png_url, proxies=self.proxies)

In order to get the screenshot to be attached as an attachment using the base64_to_attachment function, the documentation said to add the following to the pre-process script. 
inputs.base64content = workflow.properties.urlscanio.png_file

However we received an error after trying that so we modified it to this in order for it to work:
inputs.base64content = workflow.properties.urlscanio.png_base64content

I hope that explanation was clear and helps someone else if they need it. 
 
Thanks, 

Adina 

------------------------------
Adina Bodkins
------------------------------

Hi Adina,

Thanks very much for sharing your proxy improvement journey, we handle this capability using the RequestsCommon object that is located in the resilient_lib package in newer integrations:

requestcommon_object = RequestsCommon(opts=self.opts, function_opts=self.options)
self.proxies = requestcommon_object.get_proxies()

This handles seamlessly for when no proxies are used as we had identified the limitation in some of our older integrations. Thanks again for sharing this on the forum, we really value the input and your contribution to making this a vibrant community forum.

Kind regards,

------------------------------
Sean OGorman
------------------------------

Hi Sean, 

Thanks for the information. We just starting running into this problem again with other apps that we downloaded from the app exchange. Where is the requestcommon_object getting the proxies from?

Are there key value sets we can add in the app.config that are global where we can add the proxy information to?

Thanks,

------------------------------
Adina Bodkins
------------------------------

Original Message:
Sent: Mon January 06, 2020 07:41 AM
From: Sean OGorman
Subject: URLScan.io App Modifications - Proxy Aware

Hi Adina,

Thanks very much for sharing your proxy improvement journey, we handle this capability using the RequestsCommon object that is located in the resilient_lib package in newer integrations:

requestcommon_object = RequestsCommon(opts=self.opts, function_opts=self.options)
self.proxies = requestcommon_object.get_proxies()

This handles seamlessly for when no proxies are used as we had identified the limitation in some of our older integrations. Thanks again for sharing this on the forum, we really value the input and your contribution to making this a vibrant community forum.

Kind regards,

------------------------------
Sean OGorman
------------------------------

Hi again,

Take our have I been pwned integration, we can specify normally as you did in your app.config section and gather any proxies with this pattern:

import logging
import requests
import time
from resilient_circuits import ResilientComponent, function, handler, StatusMessage, FunctionResult, FunctionError
from resilient_lib import ResultPayload, RequestsCommon


class FunctionComponent(ResilientComponent):
    """Component that implements Resilient function 'have_i_been_pwned_get_breaches"""

    def __init__(self, opts):
        """constructor provides access to the configuration options"""
        super(FunctionComponent, self).__init__(opts)
        self.options = opts.get("fn_hibp", {})

        # Get proxies
        requestcommon_object = RequestsCommon(opts=self.opts, function_opts=self.options)
        self.proxies_data = requestcommon_object.get_proxies()​

This is possible due to this handy bit of code in our resilient_lib library:

class RequestsCommon:
    """
    This class represents common functions around the use of the requests package for REST based APIs.
    It incorporates the app.config section "integrations" which can be used to define a common set of proxies
    for use by all functions using this library:
    [integrations]
    http_proxy=
    https_proxy=
    Similar properties may exist in the function's section which would override the [integrations] properties.
    """
    def __init__(self, opts=None, function_opts=None):
        # capture the properties for the integration as well as the global settings for all integrations for proxy urls
        self.integration_options = opts.get('integrations', None) if opts else None
        self.function_opts = function_opts

    def get_proxies(self):
        """ proxies can be specified globally for all integrations or specifically per function """
        proxies = None
        if self.integration_options and (self.integration_options.get("http_proxy") or self.integration_options.get("https_proxy")):
            proxies = {'http': self.integration_options.get("http_proxy"), 'https': self.integration_options.get("https_proxy")}

        if self.function_opts and (self.function_opts.get("http_proxy") or self.function_opts.get("https_proxy")):
            proxies = {'http': self.function_opts.get("http_proxy"), 'https': self.function_opts.get("https_proxy")}

        return proxies

This just saves a little manual work for incorporating proxies into your development or production environment.

Hope that is helpful,
Kind regards,

------------------------------
Sean OGorman
------------------------------

Original Message:
Sent: Tue January 07, 2020 11:36 AM
From: Adina Bodkins
Subject: URLScan.io App Modifications - Proxy Aware

Hi Sean, 

Thanks for the information. We just starting running into this problem again with other apps that we downloaded from the app exchange. Where is the requestcommon_object getting the proxies from?

Are there key value sets we can add in the app.config that are global where we can add the proxy information to?

Thanks,

------------------------------
Adina Bodkins
------------------------------


Original Message:
Sent: Mon January 06, 2020 07:41 AM
From: Sean OGorman
Subject: URLScan.io App Modifications - Proxy Aware

Hi Adina,

Thanks very much for sharing your proxy improvement journey, we handle this capability using the RequestsCommon object that is located in the resilient_lib package in newer integrations:

requestcommon_object = RequestsCommon(opts=self.opts, function_opts=self.options)
self.proxies = requestcommon_object.get_proxies()

This handles seamlessly for when no proxies are used as we had identified the limitation in some of our older integrations. Thanks again for sharing this on the forum, we really value the input and your contribution to making this a vibrant community forum.

Kind regards,

------------------------------
Sean OGorman