Thanks for the answers.
Using unicode() is not a viable option as long as I have a dictionary at code level. But I tried anyway, by first using json.dumps() and then applied unicode(str, "utf-8). This is the result now:
{"type": "email_subject", "value": "Win a \u00a3100 gift card by telling Sky what you think!"}
I would like to avoid to take care of each value of my dict to have the desired result because dictionary itself is big and, maybe, it's not a programmatic solution.
What I did after reading your answers in locale was this:
import json
my_dict = {u'related': {u'type': u'email_subject', u'value': u'Win a \xa3100 gift card by telling Sky what you think!'}}
# option 1
def safe_str(obj):
try:
return str(obj)
except UnicodeEncodeError:
return obj.encode('ascii', 'ignore').decode('ascii')
print(safe_str(my_dict))
# >>> {u'related': {u'type': u'email_subject', u'value': u'Win a \xa3100 gift card by telling Sky what you think!'}}
# option 2
dumped = json.dumps(my_dict)
print(dumped)
# >>> {"related": {"type": "email_subject", "value": "Win a \u00a3100 gift card by telling Sky what you think!"}}
loaded = json.loads(dumped)
print(loaded)
# >>> {u'related': {u'type': u'email_subject', u'value': u'Win a \xa3100 gift card by telling Sky what you think!'}}
# option 3
print(dumped.encode('ascii', 'ignore').decode('ascii'))
# >>> {"related": {"type": "email_subject", "value": "Win a \u00a3100 gift card by telling Sky what you think!"}}
# option 4
print(unicode(dumped, "utf-8"))
# >>> {"related": {"type": "email_subject", "value": "Win a \u00a3100 gift card by telling Sky what you think!"}}
# option 5
for k, v in my_dict.items():
print(k, v)
for g, h in v.items():
print(safe_str(g), safe_str(h))
# >>> (u'related', {u'type': u'email_subject', u'value': u'Win a \xa3100 gift card by telling Sky what you think!'})
# >>> ('type', 'email_subject')
# >>> ('value', u'Win a 100 gift card by telling Sky what you think!') ---> here the pound symbol disappeared...
I still did not tried anything new (apart of what already did before). I don't know, at this point, if is right to expect the pound symbol in offline code/at function level.
In the end, what is mandatory is to have a dictionary in input (so in output from the function) in the post-process because the whole code in post is based on a dict (and it's already working - :) - with normal characters).
To answer
@Ben Lurie, I tried to print my function response to app.log but still it go to error (master_response is my dictionary):
Traceback (most recent call last): File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/cisco_threat_response/components/funct_cisco_threat_response.py", line 128, in _cisco_threat_response_function log.info("CTR - MASTER RESPONSE: {}".format(u'' + str(master_response)))UnicodeEncodeError: 'ascii' codec can't encode character u'\xa3' in position 53331: ordinal not in range(128)Before the test I made above, the value is logged normally.
This is a major part of the code I use in my post-process script:
cisco_sightings = results["value"]["cisco_sightings"]
sightings_list = []
for d in cisco_sightings["data"]:
try:
if d["module"] == "SMA Email":
for s in d["data"]["sightings"]["docs"]:
cisco_sightings_response = {"artifact_value": None, "observed_time": None, "confidence": None,
"resolution": None,
"relations": None,
"description": None, "type": None}
cisco_sightings_response["artifact_value"] = artifact.value
cisco_sightings_response["observed_time"] = s["observed_time"]
cisco_sightings_response["confidence"] = s["confidence"]
cisco_sightings_response["resolution"] = s["resolution"]
cisco_sightings_response["relations"] = s["relations"]
cisco_sightings_response["description"] = s["description"]
cisco_sightings_response["type"] = s["type"]
sightings_list.append(cisco_sightings_response)
except KeyError as key_error:
continue
result = ""
relations_dict = {"Origin": None, "Source type": None, "Source value": None, "Relation": None, "Related type": None,
"Related value": None}
for i, s in enumerate(sightings_list):
result += "Sighting {}".format(i + 1)
result += "\nArtifact value: {}".format(s["artifact_value"])
result += "\nObserved time: {}".format(s["observed_time"]["start_time"])
result += "\nConfidence: {}".format(s["confidence"])
result += "\nResolution: {}".format(s["resolution"])
result += "\nDescription: {}".format(s["description"])
result += "\nType: {}".format(s["type"])
relations_list = []
for r in s["relations"]:
relations_list.append(r)
for e, rel in enumerate(relations_list):
relations_dict["Origin"] = rel["origin"]
relations_dict["Source type"] = rel["source"]["type"]
relations_dict["Source value"] = rel["source"]["value"]
relations_dict["Relation"] = rel["relation"]
relations_dict["Related type"] = rel["related"]["type"]
relations_dict["Related value"] = rel["related"]["value"]
result += "\n\nRelation {}:\n{}".format(e + 1, "\n".join(
[str("\t" + k + ": {}".format(v)) for k, v in relations_dict.items()]))
result += "\n\n"
result += "#" * 50
result += "\n\n"
I think the part where the code breaks is this (from my debugging, here the pound symbol should appear):
relations_dict[
"Related value"] = rel[
"related"][
"value"]
If I do a simple incident.addNote(str(results[
"value"])), I have the next output printed in Notes (I extracted just the related part):
{u'related': {u'type': u'email_subject', u'value': u'Win a \xa3100 gift card by telling Sky what you think!'} ... }Finally, I think last Jared message touched a point here about ascii encoding. I also though about this, but the question is why ? Because of Python 2 ?
If I print this in my IDE:
print(u"Win a \u00a3100 gift card by telling Sky what you think!")
the output is correct:
Win a £100 gift card by telling Sky what you think!Feel free to suggest any solution.
Thanks
------------------------------
Lucian Sipos
------------------------------
Original Message:
Sent: Mon April 13, 2020 11:46 AM
From: Jared Fagel
Subject: UnicodeEncodeError 'ascii' 'ordinal not in range(128)'
Hey Ben,
I believe the pre/post processor scripts essentially have this defined on the backend:
# -*- coding: ascii -*-
This error is reported within the action status view when it occurs. As a test, you can try to add the mentioned symbol into the incident name by directly modifying the field. Then, try to run a script or a pre/post processor containing a method that accesses the field. Simply doing a .replace() should produce this.
We've had this issue, and our solution has been to strip the characters or to handle them in the functions directly.
------------------------------
Jared Fagel
Cyber Security Analyst I
Public Utility
Original Message:
Sent: Mon April 13, 2020 08:53 AM
From: Ben Lurie
Subject: UnicodeEncodeError 'ascii' 'ordinal not in range(128)'
I would like to understand this use case better. Resilient should be able to handle unicode. It may be that some python code is assuming ASCII where a unicode string is present that may result in errors.
This link https://www.ibm.com/support/knowledgecenter/en/SSBRUQ_36.0.0/doc/playbook/resilient_playbook_configscripts_considerations.html only refers to hard coded strings within in-product scripts. If you have constructed the dictionary with unicode (which it looks like you have) and the python code executing handles unicode then it should work.
You mention this error:
UnicodeEncodeError('ascii', u'Win a \xa3100 gift card by telling Sky what you think!', 6, 7, 'ordinal not in range(128)')
Is this error showing up in the circuits logs? Or is this error showing up in the Resilient logs? I'm trying to figure out if the code causing this is circuits code or in-product script code. Can you post the code that is executing when this error occurs?
Ben
------------------------------
Ben Lurie
Original Message:
Sent: Fri April 10, 2020 07:00 AM
From: Lucian Sipos
Subject: UnicodeEncodeError 'ascii' 'ordinal not in range(128)'
In my post process results I have this as a dictionary value:
{u'type': u'email_subject', u'value': u'Win a \xa3100 gift card by telling Sky what you think!'}
It's a result passed directly from my function script to Resilient. If I try to execute my function, which does some other operations on the dictionary (as a json in this case), I obtain the below error:
UnicodeEncodeError('ascii', u'Win a \xa3100 gift card by telling Sky what you think!', 6, 7, 'ordinal not in range(128)')
I tried first to do some operations in post-process:
- instead of cast my code to str() I used unicode() - nothing changed
- use .encode() or .decode() function but I get a very annoying message: An error occurred while processing the action acknowledgement. Additional information: Post-processing script for Function 'Cisco Threat Response' from Workflow 'Cisco Sightings' was unable to complete because: access denied ("java.lang.RuntimePermission" "accessClassInPackage.encodings") - I don't know why encoding functions are disabled...
Then, in the function script (or my local script):
- encode() decode() functions - change the way it's encoded (from \xa to \ua for example and viceversa) but the problems remains: I don't get the pund symbol £
- in my "offline" script used json.dumps (keep in mind that we are starting from a json) with ensure_ascii=False - this, finally, showed my the £ symbol in console. I use PyCharm and I learned that it try to show you the result in console in utf-8 format so this solution is not a solution at all as long as I also learned that Resilient use ascii encoding.
What I have to do ? How can I solve this problem ?
My goal is to have well formatted the pound symbol £ in final Notes.
Thanks
------------------------------
Lucian Sipos
------------------------------