We have on console server, 1 event processor and 2 event collectors.

The Current version is 7.4.2 FixPack 3 (Build 20210323172312)

After applying the FixPack, we see that log sources switch from Error to OK. But after around one hours, these log sources turned to Error.

I had tried with tcpdump from the event processor to the log source and got the following output :

[rootSupport Member~]# tcpdump -A host hostip and port 514

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on tun0, link-type RAW (Raw IP), capture size 262144 bytes

^C

0 packets captured

1828 packets received by filter

0 packets dropped by kernel

I checked the EPS allocated from License Pool Management and found we have limited EPS, but I am not sure whether it is the cause of error state of log sources.

hostname(hostip)

Appliance Type: Software

Maximum event rate for appliance type: 100000

Host Status: Active

Current Allocation Values

Allocated EPS: 2100

Average EPS: 4410

Peak EPS: 150600

Anyone had the same problem ?

Any idea is appreciated.

Thanks.

Hi,

You can use commands in this technote to see if the events are being received from the logsource.

https://www.ibm.com/support/pages/qradar-using-command-line-troubleshoot-syslog-event-source

If the events are seen in tcpdump , then they are not hitting the DSM/logsources... needs to be checked in the logs.

Hi,

thanks for your reply.

I am not faimilar with tcpdump and here is the return that I got:

[rootSupport Member~]# tcpdump -A host hostip and port 514

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on tun0, link-type RAW (Raw IP), capture size 262144 bytes

^C

0 packets captured

1828 packets received by filter

0 packets dropped by kernel

And logs from QRadar /var/log/qradar.log with no error, no warning:

Jul 20 13:51:32 qradarcollector replication[23760]: Using 192.168.*.* as our local IP.

Jul 20 13:51:32 qradarcollector replication[23760]: Downloading and applying latest database dumps from the console.

Jul 20 13:51:32 qradarcollector replication[23760]: Applying transaction 0000000000001936908

Jul 20 13:51:32 qradarcollector replication[23760]: Replication download timing: Downloading: 356 ms Overall: 518 ms DB transaction: 83 ms Transaction verification: 78 ms

Jul 20 13:51:32 ::ffff:192.168.124.1 [hostcontext.hostcontext] [8bbec99f-f4ff-462d-b4d7-4a518b5eb749/SequentialEventDispatcher] com.q1labs.frameworks.core.JMSReplicationMessageSender: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]Reading and sending JMS messages from mailbox.

Jul 20 13:51:32 ::ffff:192.168.124.1 [hostcontext.hostcontext] [8bbec99f-f4ff-462d-b4d7-4a518b5eb749/SequentialEventDispatcher] com.q1labs.frameworks.core.JMSReplicationMessageSender: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]Sent 3 JMS messages.

Jul 20 13:51:32 ::ffff:192.168.124.1 [ecs-ep.ecs-ep] [Thread-62 - notification updates] com.q1labs.core.assetprofile.dao.light.HostProfileServices: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]HostProfile update processed: insert (0) + update (4) = 0.004000 s; delete (0) = 0.000000 s; total = 0.004000 s

Jul 20 13:51:32 ::ffff:192.168.124.1 [ecs-ep.ecs-ep] [Thread-62 - notification updates] com.q1labs.core.assetprofile.dao.light.HostProfileServices: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]HostProfile update processed: insert (0) + update (1) = 0.001000 s; delete (0) = 0.000000 s; total = 0.001000 s

Jul 20 13:51:41 ::ffff:192.168.124.1 [hostcontext.hostcontext] [Server Host Status Processor] com.q1labs.configservices.controller.ServerHostStatusUpdater: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]Sent update status of host 192.168.124.1 to ACTIVE

Jul 20 13:51:54 ::ffff:192.168.124.1 [ecs-ec.ecs-ec] [7e7e97b3-7cab-4022-9d46-42f070bbd530/SequentialEventDispatcher] com.q1labs.sem.monitors.SourceMonitor: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]Incoming raw event rate (5s: 76.60 eps), (10s: 76.60 eps), (15s: 76.60 eps), (30s: 80.50 eps), (60s: 83.58 eps), (300s: 77.85 eps), (900s: 77.85 eps). Peak in the last 60s: 129.80 eps. Max Seen 160.40 eps. EC Throttles/5s (60s: 0.00). Total EC Throttles in the last 60s: 0. Total EC Throttles: 0. Appliance Threshold: 2108.00

Jul 20 13:51:59 ::ffff:192.168.124.1 [ariel.ariel_query_server] [q1labs_worker_2] com.q1labs.ariel.searches.AccessManager: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]+++> Ariel Server is running 0 queries <+++

Jul 20 13:51:59 ::ffff:192.168.124.1 [ecs-ec.ecs-ec] [Thread-125] com.ibm.si.ec.filters.TenantQueuedEventThrottleFilter: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -][Tenant:2:TOTO Managed Services] Current tenant event rate limit 500. Processed at an average EPS rate of 0 events over last one minute. The current queue size is 0 events.

Jul 20 13:52:00 ::ffff:192.168.124.1 [ecs-ep.ecs-ep] [a58ffc97-64ac-4eb1-9d30-0327b7d5ce14/SequentialEventDispatcher] com.q1labs.sem.monitors.EPMonitor: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]Incoming Event Rate (60s: 208.17 eps), (300s: 207.23 eps), (900s: 205.85 eps). Max Seen 1190.65 eps.

Jul 20 13:52:04 ::ffff:192.168.124.1 [ecs-ec-ingress.ecs-ec-ingress] [8dc45ecc-e8d0-4cfd-884a-9954e0a3eb84/SequentialEventDispatcher] com.q1labs.sem.monitors.SourceMonitor: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]Incoming raw event rate (5s: 76.60 eps), (10s: 76.60 eps), (15s: 76.60 eps), (30s: 76.60 eps), (60s: 83.58 eps), (300s: 77.98 eps), (900s: 77.98 eps). Peak in the last 60s: 129.80 eps. Max Seen 160.40 eps. EC Throttles/5s (60s: 0.00). Total EC Throttles in the last 60s: 0. Total EC Throttles: 0. Appliance Threshold: 2108.00

Jul 20 13:52:09 ::ffff:192.168.124.1 [ecs-ec.ecs-ec] [7acf0e8f-2a25-408e-b232-e28df94fb3eb/SequentialEventDispatcher] com.q1labs.sem.dsm.extension.DSMExtensionFactory: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]DSM Extension Statistics - Least-performant by average parse time: VaronisDSMCustom_ext (0.000000 msec)

Jul 20 13:52:09 ::ffff:192.168.124.1 [ecs-ec.ecs-ec] [[type=com.ibm.si.ec.filters.ForwardingFilter][parent=qradarcollector.toto.com:ecs-ec/EC/Processor2]] com.ibm.si.ec.filters.ForwardingFilter: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -] (Total Events Seen: 45286138), (Total Events Dropped: 0), (Total Events Forwarded: 0), (Total Events Not Correlated: 0)

Jul 20 13:52:09 ::ffff:192.168.124.1 [ecs-ec.ecs-ec] [com.ibm.si.ec.filters.LicenseGivebackFilter] com.ibm.si.ec.filters.LicenseGivebackFilter: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]License giveback Event count (SensorDevices [60s: 84.18 eps]), (Events Dropped [60s: 0.00 eps]), (Events Log Only [60s: 0.00 eps]), (Total [60s: 84.18 eps])

Jul 20 13:52:09 ::ffff:192.168.124.1 [ecs-ec.ecs-ec] [[type=com.ibm.si.ec.filters.stat.StatFilter][parent=qradarcollector.toto.com:ecs-ec/EC/Processor2]] com.ibm.si.ec.filters.stat.StatFilter: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -] Events per second: 1s:419,419 (peak 1740,1740) (compression: 0%) 5s:83,83 (peak 373,373) (compression: 0%) 10s:80,80 (peak 199,199) (compression: 0%) 30s:77,77 (peak 94,94) (compression: 0%) 60s:84,84 (peak 85,85) (compression: 0%)

Jul 20 13:52:10 ::ffff:192.168.124.1 [accumulator.accumulator] [AccumulationService] com.q1labs.cve.accumulation.AccumulationService: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]Start processing interval: Tue Jul 20 13:51:00 CEST 2021

Jul 20 13:52:10 ::ffff:192.168.124.1 [accumulator.accumulator] [AccumulationService] com.q1labs.cve.accumulation.AccumulationService: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]Finished processing interval: Tue Jul 20 13:51:00 CEST 2021 in 734ms

Jul 20 13:52:12 ::ffff:192.168.124.1 [hostcontext.hostcontext] [Server Host Status Processor] com.q1labs.configservices.controller.ServerHostStatusUpdater: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]Sent update status of host 192.168.124.1 to ACTIVE

Jul 20 13:52:14 ::ffff:192.168.124.1 [ecs-ep.ecs-ep] [0ff57ea2-54ed-4913-a04c-cb405fea5b2b/SequentialEventDispatcher] com.ibm.si.ep.filters.EPEntryRoutingFilter: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]MPC Event Rate: 0 eps. Central Processing Event Rate: 0 eps.

Jul 20 13:52:17 ::ffff:192.168.124.1 [ecs-ec.ecs-ec] [SourceMonitorTimerTask] com.q1labs.sem.monitors.SourceMonitor: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]Incoming flow rate [5s: (0.00):(0.00) fps], [10s: (0.00):(0.00) fps], [15s: (0.00):(0.00) fps], [30s: (0.00):(0.00) fps], [60s: (0.00):(0.00) fps], [300s: (0.00):(0.00) fps], [900s: (0.00):(0.00) fps]. Peak in the last 60s: (0.00):(0.00) fps. Max Seen (0.00):(0.00) fps.License Threshold: 0.00

Jul 20 13:52:31 ::ffff:192.168.124.1 [hostcontext.hostcontext] [8bbec99f-f4ff-462d-b4d7-4a518b5eb749/SequentialEventDispatcher] com.q1labs.hostcontext.replication.MHReplication: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]Timer expired. Attempting to download updates

Jul 20 13:52:31 ::ffff:192.168.124.1 [hostcontext.hostcontext] [8bbec99f-f4ff-462d-b4d7-4a518b5eb749/SequentialEventDispatcher] com.q1labs.hostcontext.replication.MHReplication: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]Downloading updates request starting...

Jul 20 13:52:34 qradarcollector replication[27378]: Using 192.168.124.1 as our local IP.

Jul 20 13:52:34 qradarcollector replication[27378]: Downloading and applying latest database dumps from the console.

Jul 20 13:52:34 qradarcollector replication[27378]: Applying transaction 0000000000001936909

Jul 20 13:52:34 qradarcollector replication[27378]: Replication download timing: Downloading: 1826 ms Overall: 1863 ms DB transaction: 28 ms Transaction verification: 7 ms

Jul 20 13:52:34 ::ffff:192.168.124.1 [hostcontext.hostcontext] [8bbec99f-f4ff-462d-b4d7-4a518b5eb749/SequentialEventDispatcher] com.q1labs.frameworks.core.JMSReplicationMessageSender: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]Reading and sending JMS messages from mailbox.

Jul 20 13:52:34 ::ffff:192.168.124.1 [hostcontext.hostcontext] [8bbec99f-f4ff-462d-b4d7-4a518b5eb749/SequentialEventDispatcher] com.q1labs.frameworks.core.JMSReplicationMessageSender: [INFO] [NOT:0000006000][192.168.124.1/- -] [-/- -]Sent 0 JMS messages.

Any ideas ?

Thanks in advance.

Sorry I would like to reply to you but I forgot to click on Reply, you can find logs above.