Hi Zohra,
This looks like it could be a Python thing. I don't have the files your testing against so it is hard to say for sure. Can you step through your code and verify if it is seeing the lines as equal to each other.
My guess is you are using
.strip() on the lines <g class="gr_ gr_427 gr-alert gr_gramm gr_inline_cards gr_run_anim Style multiReplace" id="427" data-gr-id="427">in </g>
contents<g class="gr_ gr_427 gr-alert gr_gramm gr_inline_cards gr_disable_anim_appear Style multiReplace" id="427" data-gr-id="427"> but</g> not <g class="gr_ gr_467 gr-alert gr_gramm gr_inline_cards gr_run_anim Style multiReplace" id="467" data-gr-id="467">in </g>
contents2<g class="gr_ gr_467 gr-alert gr_gramm gr_inline_cards gr_disable_anim_appear Style multiReplace" id="467" data-gr-id="467">,</g> so when comparing the lines it thinks they are different.
It could also be your for loop, where it looks like it will add the line from
contents <g class="gr_ gr_791 gr-alert gr_gramm gr_inline_cards gr_run_anim Style multiReplace" id="791" data-gr-id="791">to </g>
results<g class="gr_ gr_791 gr-alert gr_gramm gr_inline_cards gr_disable_anim_appear Style multiReplace" id="791" data-gr-id="791"> even</g> if that line exists later in <g class="gr_ gr_668 gr-alert gr_gramm gr_inline_cards gr_run_anim Style multiReplace" id="668" data-gr-id="668">the </g>
contents2<g class="gr_ gr_668 gr-alert gr_gramm gr_inline_cards gr_disable_anim_appear Style multiReplace" id="668" data-gr-id="668"> list</g>. I am not sure if that is the behavior you want. The below example will only add the line if it does not exist at all in
contents2:
...
results = []
input = [line.strip() for line in contents]
whitelist_input = [line.strip() for line in whitelist]
for line in input:
if line not in whitelist_input:
newline = ast.literal_eval(line)
results.append(line)
break------------------------------
Brian Walsh
------------------------------
Original Message:
Sent: Wed April 10, 2019 05:12 PM
From: Zohra SMAIL
Subject: Extracting delta of two files
Hi Brian,
Thanks for helping.
I had actually copied the wrong code. Here is what I had and that is supposed to work:
inputStream = open(tempiocoutput.name) inputStream2 = open('/usr/share/integration/components/whitelist.txt') contents = inputStream.readlines() contents2 = inputStream2.readlines() results = [] input = [line.strip() for line in contents] for line in input: for line2 in contents2: if line != line2: newline = ast.literal_eval(line) results.append(newline) break log.info("Return results to Resilient") yield FunctionResult({"value": results}) I still have the whole content of my IOCs file in the Artifacts tab, so no comparison was made with the whitelist.
------------------------------
Zohra SMAIL
Original Message:
Sent: Wed April 10, 2019 11:06 AM
From: Brian Walsh
Subject: Extracting delta of two files
Hi Zohra,
Without testing your code sample it looks like you are not reading lines from inputStream2; this is just opening the file which returns a file object.
Also when dealing with files it is better practice to handle the opening and closing of files when done using. This can be handled easily with something like below:
with open('/usr/share/integration/components/whitelist.txt') as inputStream2: contents2 = inputStream2.readlines()
------------------------------
Brian Walsh
Original Message:
Sent: Wed April 10, 2019 05:09 AM
From: Zohra SMAIL
Subject: Extracting delta of two files
Hello team,
I have created function allowing me to to push artifact value to a text file and I am using the IOC parser function to compare this file with an input of IOCs contained in a file attachment and extract the delta between them. Here is the code I added to the IOC parser function:
# Read self.filepath and return contents
inputStream = open(tempiocoutput.name)
inputStream2 = open('/usr/share/integration/components/whitelist.txt')
contents = inputStream.readlines()
results = []
input = [line.strip() for line in contents]
whitelist = [line.strip() for line in inputStream2]
for line in input:
if line not in whitelist:
newline = ast.literal_eval(line)
results.append(newline)
log.info("Return results to Resilient")
yield FunctionResult({"value": results})
I don't see what's wrong about this code as it is supposed to do its job and extract the delta between two files. So I suspect Resilient for preventing the code from working correctly. Or maybe I missed Something I shouldn't have.
Thanks in advance for your help.
------------------------------
Zohra SMAIL
------------------------------