IBM QRadar

 View Only

 Importing Log Source Extension XML fails: Null Pointer Exception

Vindler Borsson's profile image
Vindler Borsson posted 12/19/24 09:06 AM

Hello guys!

I wrote this log source extension for Unifi network devices, id=4002 is the empty log source type I added:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns2:device-extension xmlns:ns2="event_parsing/device_extension">
    <pattern id="EventTime-Pattern" type="JavaPattern" use-default-pattern="false">(\w{3}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2})</pattern>
    <pattern id="DeviceName-Pattern" type="JavaPattern" use-default-pattern="false">(\S+)</pattern>
    <pattern id="DeviceMAC-Pattern" type="JavaPattern" use-default-pattern="false">(\S+),(\S+):</pattern>
    <pattern id="ProcessName-Pattern" type="JavaPattern" use-default-pattern="false">(\S+):</pattern>
    <pattern id="SubProcessName-Pattern" type="JavaPattern" use-default-pattern="false">(\S+)(?:\[(\d+)\])?:</pattern>
    <pattern id="EventDescription-Pattern" type="JavaPattern" use-default-pattern="false">(.+)</pattern>
    <pattern id="LinkUpDown-Pattern" type="JavaPattern" use-default-pattern="false">(Link Up|Link Down): (\d+/\d+)</pattern>
    <pattern id="RoleChange-Pattern" type="JavaPattern" use-default-pattern="false">Port \((\d+)\) inst\((\d+)\) role changing from (\S+) to (\S+)</pattern>
    <pattern id="SSHAuth-Pattern" type="JavaPattern" use-default-pattern="false">Pubkey auth succeeded for '(\S+)' with (\S+) key</pattern>
    <pattern id="SessionStartEnd-Pattern" type="JavaPattern" use-default-pattern="false">Session (\d+) of type (\d+) (started|ended) for user</pattern>
    <pattern id="InformPacket-Pattern" type="JavaPattern" use-default-pattern="false">(immediately send user inform packet)</pattern>
    <pattern id="SSHDisconnect-Pattern" type="JavaPattern" use-default-pattern="false">Exit \((\S+)\) from [\u003c](\S+):(\d+)[\u003e]: Disconnect received</pattern>
    <pattern id="SSHConnection-Pattern" type="JavaPattern" use-default-pattern="false">Child connection from (\S+):(\d+)</pattern>
    <pattern id="AllEvents" type="JavaPattern">(.*?)</pattern>
    <match-group order="1" device-type-id-override="4002">
        <matcher order="1" field="EventTime" pattern-id="EventTime-Pattern" capture-group="\1" enable-substitutions="true"/>
        <matcher order="2" field="DeviceName" pattern-id="DeviceName-Pattern" capture-group="\1" enable-substitutions="true"/>
        <matcher order="3" field="DeviceMAC" pattern-id="DeviceMAC-Pattern" capture-group="\1" enable-substitutions="true"/>
        <matcher order="4" field="ProcessName" pattern-id="ProcessName-Pattern" capture-group="\1" enable-substitutions="true"/>
        <matcher order="5" field="SubProcessName" pattern-id="SubProcessName-Pattern" capture-group="\1" enable-substitutions="true"/>
        <matcher order="6" field="EventDescription" pattern-id="EventDescription-Pattern" capture-group="\1" enable-substitutions="true"/>
        <matcher order="7" field="LinkStatus" pattern-id="LinkUpDown-Pattern" capture-group="\1,\2" enable-substitutions="true"/>
        <matcher order="8" field="RoleChange" pattern-id="RoleChange-Pattern" capture-group="\1,\2,\3,\4" enable-substitutions="true"/>
        <matcher order="9" field="SSHAuth" pattern-id="SSHAuth-Pattern" capture-group="\1,\2" enable-substitutions="true"/>
        <matcher order="10" field="SessionInfo" pattern-id="SessionStartEnd-Pattern" capture-group="\1,\2,\3" enable-substitutions="true"/>
        <matcher order="11" field="InformPacket" pattern-id="InformPacket-Pattern" capture-group="\1" enable-substitutions="true"/>
        <matcher order="12" field="SSHDisconnect" pattern-id="SSHDisconnect-Pattern" capture-group="\1,\2,\3" enable-substitutions="true"/>
        <matcher order="13" field="SSHConnection" pattern-id="SSHConnection-Pattern" capture-group="\1,\2" enable-substitutions="true"/>
        <event-match-multiple pattern-id="AllEvents" send-identity="UseDSMResults" force-qidmap-lookup-on-fixup="true"/>
    </match-group>
</ns2:device-extension>

When I try to import it via Admin -> Log Source Extensions, I get:

Dec 19 14:42:12 ::ffff:127.0.0.1 [tomcat.tomcat] [admin@10.x.x.x (3653) /console/do/sem/uploadMaintainDeviceExtension] com.q1labs.sem.ui.action.struts2.MaintainDeviceExtensions: [ERROR] [NOT:0000003000][10.x.x.x/- -] [-/- -]uploaded parsing extension is invalid
Dec 19 14:42:12 ::ffff:127.0.0.1 [tomcat.tomcat] [admin@10.x.x.x (3653) /console/do/sem/uploadMaintainDeviceExtension] java.lang.NullPointerException
Dec 19 14:42:12 ::ffff:127.0.0.1 [tomcat.tomcat] [admin@10.x.x.x (3653) /console/do/sem/uploadMaintainDeviceExtension]    at com.q1labs.sem.dsm.extension.RegexPropertyMatcher.<init>(RegexPropertyMatcher.java:46)
Dec 19 14:42:12 ::ffff:127.0.0.1 [tomcat.tomcat] [admin@10.x.x.x (3653) /console/do/sem/uploadMaintainDeviceExtension]    at com.ibm.si.data_ingestion.validation.lsx.MatchGroupTag.setMatcherTags(MatchGroupTag.java:192)
Dec 19 14:42:12 ::ffff:127.0.0.1 [tomcat.tomcat] [admin@10.x.x.x (3653) /console/do/sem/uploadMaintainDeviceExtension]    at com.ibm.si.data_ingestion.validation.lsx.MatchGroupTag.setMatchGroup(MatchGroupTag.java:98)
Dec 19 14:42:12 ::ffff:127.0.0.1 [tomcat.tomcat] [admin@10.x.x.x (3653) /console/do/sem/uploadMaintainDeviceExtension]    at com.ibm.si.data_ingestion.validation.lsx.ExtensionXml.setMatchGroupTags(ExtensionXml.java:98)
Dec 19 14:42:12 ::ffff:127.0.0.1 [tomcat.tomcat] [admin@10.x.x.x (3653) /console/do/sem/uploadMaintainDeviceExtension]    at com.ibm.si.data_ingestion.validation.lsx.ExtensionXml.setDeviceExtension(ExtensionXml.java:52)
Dec 19 14:42:12 ::ffff:127.0.0.1 [tomcat.tomcat] [admin@10.x.x.x (3653) /console/do/sem/uploadMaintainDeviceExtension]    at com.q1labs.sem.ui.action.struts2.MaintainDeviceExtensions.validateExtensionXml(MaintainDeviceExtensions.java:569)
.....

Something to do with java regex capture groups perhaps?

I am at loss.

Vindler Borsson's profile image
Vindler Borsson

Bump. Anyone?

COLIN HAY's profile image
COLIN HAY

Hi Vindler, I think the issue is that you are referring to fields which do not exist. Log Source Extensions only work with the standard DSM properties, they can't be used for custom properties, but you've got field values like SSHConnection, SSHDisconnect, InformPacket, etc which are invalid fields for an LSX, those need to be handled with custom properties. I would suggest using the DSM Editor UI to do this work, it will generate the LSX and custom properties for you automatically based on what you specify in the UI.

Cheers

Colin