Hi again,
Take our have I been pwned integration, we can specify normally as you did in your app.config section and gather any proxies with this pattern:
import logging
import requests
import time
from resilient_circuits import ResilientComponent, function, handler, StatusMessage, FunctionResult, FunctionError
from resilient_lib import ResultPayload, RequestsCommon
class FunctionComponent(ResilientComponent):
"""Component that implements Resilient function 'have_i_been_pwned_get_breaches"""
def __init__(self, opts):
"""constructor provides access to the configuration options"""
super(FunctionComponent, self).__init__(opts)
self.options = opts.get("fn_hibp", {})
requestcommon_object = RequestsCommon(opts=self.opts, function_opts=self.options)
self.proxies_data = requestcommon_object.get_proxies()
This is possible due to this handy bit of code in our resilient_lib library:
class RequestsCommon:
"""
This class represents common functions around the use of the requests package for REST based APIs.
It incorporates the app.config section "integrations" which can be used to define a common set of proxies
for use by all functions using this library:
[integrations]
http_proxy=
https_proxy=
Similar properties may exist in the function's section which would override the [integrations] properties.
"""
def __init__(self, opts=None, function_opts=None):
self.integration_options = opts.get('integrations', None) if opts else None
self.function_opts = function_opts
def get_proxies(self):
""" proxies can be specified globally for all integrations or specifically per function """
proxies = None
if self.integration_options and (self.integration_options.get("http_proxy") or self.integration_options.get("https_proxy")):
proxies = {'http': self.integration_options.get("http_proxy"), 'https': self.integration_options.get("https_proxy")}
if self.function_opts and (self.function_opts.get("http_proxy") or self.function_opts.get("https_proxy")):
proxies = {'http': self.function_opts.get("http_proxy"), 'https': self.function_opts.get("https_proxy")}
return proxies
This just saves a little manual work for incorporating proxies into your development or production environment.
Hope that is helpful,
Kind regards,
------------------------------
Sean OGorman
------------------------------
Original Message:
Sent: Tue January 07, 2020 11:36 AM
From: Adina Bodkins
Subject: URLScan.io App Modifications - Proxy Aware
Hi Sean,
Thanks for the information. We just starting running into this problem again with other apps that we downloaded from the app exchange. Where is the requestcommon_object getting the proxies from?
Are there key value sets we can add in the app.config that are global where we can add the proxy information to?
Thanks,
------------------------------
Adina Bodkins
Original Message:
Sent: Mon January 06, 2020 07:41 AM
From: Sean OGorman
Subject: URLScan.io App Modifications - Proxy Aware
Hi Adina,
Thanks very much for sharing your proxy improvement journey, we handle this capability using the RequestsCommon object that is located in the resilient_lib package in newer integrations:
requestcommon_object = RequestsCommon(opts=self.opts, function_opts=self.options)
self.proxies = requestcommon_object.get_proxies()
This handles seamlessly for when no proxies are used as we had identified the limitation in some of our older integrations. Thanks again for sharing this on the forum, we really value the input and your contribution to making this a vibrant community forum.
Kind regards,
------------------------------
Sean OGorman
Original Message:
Sent: Thu November 21, 2019 03:53 PM
From: Adina Bodkins
Subject: URLScan.io App Modifications - Proxy Aware
Hi there,
We recently came across the URLScan.io app on the app exchange that we were excited to use. In order for it to work in our environment we had to make a few modifications that I wanted to share in case anyone else runs into these same issues.
Our environment utilizes a proxy. The first thing we noticed after installation and running the workflow was a proxy error. We downloaded the code from Github to make the following modifications.
In our app.config we added two keys for the proxy. We also had to change the config section name to urlscanio for some reason. #[fn_urlscanio] -> original name when installed
[urlscanio]
urlscanio_api_key=OUR-API-KEY
https_proxy =
https://proxy:8080
http_proxy =
http://proxy:8080
In the code in the __init__ function we added the following: self.proxies = {
"https": self.options.get("https_proxy"),
"http": self.options.get("http_proxy")
}
And then anywhere in the code we saw requests.post or a requests.get we added: proxies=self.proxies
So it ended up looking like this (had to do it 4 times): urlscanio_post = requests.post('https://urlscan.io/api/v1/scan/', headers=urlscanio_headers, data=json.dumps(urlscanio_data), proxies=self.proxies)
urlscanio_get = requests.get('https://urlscan.io/api/v1/result/{}/'.format(uuid), proxies=self.proxies)
urlscanio_report_get = requests.get(urlscanio_report_url, proxies=self.proxies)
urlscanio_png_get = requests.get(urlscanio_png_url, proxies=self.proxies)
In order to get the screenshot to be attached as an attachment using the base64_to_attachment function, the documentation said to add the following to the pre-process script.
inputs.base64content = workflow.properties.urlscanio.png_file
However we received an error after trying that so we modified it to this in order for it to work:
inputs.base64content = workflow.properties.urlscanio.png_base64content
I hope that explanation was clear and helps someone else if they need it.
Thanks,
Adina
------------------------------
Adina Bodkins
------------------------------
