IBM Security QRadar SOAR

 View Only

  • 1.  Error generating incident from script with name containing unicode data.

    Posted Thu February 06, 2020 04:15 AM

    Hello,

    We have a script that generates new incidents from incoming emails, using as incident name the subject of the email.

    We have noticed that with some emails, the script fails, and we have seen that it fails when the subject contains some special characters or tildes.

    For example, with the subjects:

                    "Ayúdanos a protegerte: consejos de seguridad de Google"  -> FAILS

     

    Here the code and the error seen in the log:

        Script Code:

    …

      # Create a suitable title for an incident based on the email

      newIncidentTitle = u"Incident generated from email \"{0}\" via mailbox {1}".format(emailmessage.subject, emailmessage.inbound_mailbox)

    …

    # Create an incident with a title based on the email subject, owned identified by variable newIncidentOwner

        emailmessage.createAssociatedIncident(newIncidentTitle, newIncidentOwner)

     

        # Add the subject to the incident as an artifact, and set the incident reporter.

     

     

    Error:

     

    18:53:55.277 [Camel (camel-1) thread #55 - JmsConsumer[email-service.save-email-data]] ERROR c.c.e.EmailMessageIngesterImpl - Rule 'Process email message v2.0.1' is unable to update the Email Message 'emailmessage-245' because: UnicodeE$

    com.co3.domain.exceptions.Co3IllegalStateException: Rule 'Process email message v2.0.1' is unable to update the Email Message 'emailmessage-245' because: UnicodeEncodeError('ascii', u'Ay\xfadanos a protegerte: consejos de seguridad de Google', 2, 3, 'ordinal not in range(128)')

            at com.co3.actions.ActionAutomationResultProcessor.processFailedActionResult(ActionAutomationResultProcessor.java:399)

            at com.co3.actions.ActionAutomationResultProcessor.add(ActionAutomationResultProcessor.java:115)

            at com.co3.actions.RequestLocalActionProcessorService.processTarget(RequestLocalActionProcessorService.java:403)

            at com.co3.actions.RequestLocalActionProcessorService.processActions(RequestLocalActionProcessorService.java:305)

            at com.co3.actions.RequestLocalActionProcessorService.processActions(RequestLocalActionProcessorService.java:293)

            at com.co3.actions.RequestLocalActionProcessorService.onObjectsCommitted(RequestLocalActionProcessorService.java:130)

            at com.co3.actions.ActionProcessorService.onObjectsCommitted(ActionProcessorService.java:140)

            at com.co3.tracking.EntityChangeTracker.processCommittedHandlers(EntityChangeTracker.java:716)

            at com.co3.tracking.EntityChangeTracker.processEvents(EntityChangeTracker.java:608)

            at com.co3.tracking.EntityChangeTracker.processEvents(EntityChangeTracker.java:578)

            at sun.reflect.GeneratedMethodAccessor1298.invoke(Unknown Source)

            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)

            at java.lang.reflect.Method.invoke(Method.java:508)

            at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)

            at com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:144)

            at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)

            at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)

            at com.google.common.eventbus.Subscriber.dispatchEvent(Subscriber.java:67)

            at com.google.common.eventbus.Dispatcher$PerThreadQueuedDispatcher.dispatch(Dispatcher.java:108)

            at com.google.common.eventbus.EventBus.post(EventBus.java:212)

            at com.ibm.resilient.common.event.Co3EventBus.post(Co3EventBus.java:30)

            at com.co3.tracking.ActionRunner.runActions(ActionRunner.java:20)



    ------------------------------
    Aitor Vivanco Sata Cruz
    ------------------------------


  • 2.  RE: Error generating incident from script with name containing unicode data.

    IBM Champion
    Posted Thu February 06, 2020 11:49 AM
    See if this helps you, we've had fairly good success with it: emailmessage.subject.encode('ascii', 'ignore').decode('ascii', 'ignore')

    In the future, I'd like to see Resilient allow importing unicodedata in scripts / processors, which would provide the ability to do this:
    unicodedata    .normalize("NFKD", emailmessage.subject.decode('utf-8', 'ignore'))


    ------------------------------
    Jared Fagel
    Cyber Security Analyst I
    Public Utility
    ------------------------------

    Original Message


  • 3.  RE: Error generating incident from script with name containing unicode data.

    Posted Fri February 07, 2020 09:43 AM
    This should work. Here you can see I created a script to create an incident from an email message the same as the one you were posting and created an incident from it:


    In general Resilient works with Unicode characters. It may be tricky to write the appropriate Python to manage unicode, but the example you showed should work. If you are having trouble you may want to open a support ticket.

    Ben

    ------------------------------
    Ben Lurie
    ------------------------------

    Original Message


  • 4.  RE: Error generating incident from script with name containing unicode data.

    IBM Champion
    Posted Fri February 07, 2020 10:02 AM
    Hey @Ben Lurie,

    Do you see the same success if you split that into two lines, as @Aitor Vivanco Santa Cruz​​ showed? Resilient is good at handling unicode internally, but the minute you start working with it in the scripts, it can be game-over... at least from my experience.

    Like:
    newIncidentTitle = u"Incident generated from email \"{0}\" via mailbox {1}".format(emailmessage.subject, emailmessage.inbound_mailbox)  # We try to assign non-UTF-8 to a variable first here....

    emailmessage.createAssociatedIncident(newIncidentTitle, "blurie@resilientsystems.com")

    ​

    ------------------------------
    Jared Fagel
    Cyber Security Analyst I
    Public Utility
    ------------------------------

    Original Message