IBM QRadar

IBM QRadar

Join this online topic group to communicate across Security product users and IBM experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.


#Security
#QRadar
#SecuringhybridcloudandAI
 View Only

  • 1.  Missing File at /store/ariel/events/payloads/yyyy/MM/dd/HH

    Posted 11/14/20 02:27 AM
    Hello everyone,

    We have detected some of the payload folders there are less then 60 files. 

    On the one of EP's /store/ariel/events/payloads/2020/11/10/6 path, there is no 20th minute file. When I search that Event Processor from the GUI, 11.10.2020 6:20 - 6:20 it also returned no results. However, one minute after (6:21) there are more than 300K events.

    I checked the Data Nodes' same paths connected to that EP from CLI, but 20th file is missing from them too.

    I am a bit suspicious from the cron jobs (like time sync process) however, it is said that there are some more issues at other EPs.

    Do you have any idea what can cause this issue? Or do you think time sync job can make that gap?

    Thank you in advance for your opinions.

    Regards.

    ------------------------------
    Halil BALIM

    ------------------------------


  • 2.  RE: Missing File at /store/ariel/events/payloads/yyyy/MM/dd/HH

    Posted 11/16/20 07:17 AM
    Hi
    Is this true for records? /store/ariel/events/records?
    You can check audit logs from qradar in case of manual deletion may be.

    ------------------------------
    Hasan Erhan AYDINOĞLU
    ------------------------------



  • 3.  RE: Missing File at /store/ariel/events/payloads/yyyy/MM/dd/HH

    Posted 11/16/20 08:31 AM
    Hi,

    Thank you for your reply.

    I have checked the records now. The file belongs to the same minute is missing there too.

    I am thinking the time sync job and backup job at crontab actually.

    Thank you for your opinion.

    Regards.

    ------------------------------
    Halil BALIM
    ------------------------------

    Original Message


  • 4.  RE: Missing File at /store/ariel/events/payloads/yyyy/MM/dd/HH

    Posted 11/16/20 09:43 AM
    Hi Halil,

    The ecs-ep process (which is responsible for writing events to disk) may have been down during this minute, due to a full deploy occurring. When this service is down, the "upstream" services buffer the event data until the service is back up. This would explain why there are so many events in the 6:21 file. If this is the case, you should see that many events in the 6:21 minute have a Log Source Time value in the 6:20 minute, meaning they are not lost, they were just delayed being written to disk because of the service restart.

    Cheers
    Colin

    ------------------------------
    COLIN HAY
    IBM Security
    ------------------------------



  • 5.  RE: Missing File at /store/ariel/events/payloads/yyyy/MM/dd/HH

    Posted 11/17/20 02:18 AM
    Hi Colin,

    Actually the event count is approximately the same before and after that minute. So, we are not seeing any peak after that minute. I have checked the Log Source Time column of the 6:21 events. Because there are up to almost 40K events are coming in seconds, the log source time column differs so much in time.

    How about time sync process or back up (netbackup) that started almost that time?

    Is it possible to make ecs-ec-ingress services to stop because of these processes?

    Thank you for your response.
    Regards.

    ------------------------------
    Halil BALIM
    ------------------------------

    Original Message


  • 6.  RE: Missing File at /store/ariel/events/payloads/yyyy/MM/dd/HH

    Posted 11/17/20 02:31 AM

    Ah ok, I thought you were pointing out the 300K events in the following minute because it was more than usual.

    Nothing in QRadar should restart the ecs-ec-ingress service automatically. It only needs to be restarted if its core codebase is changed in a patch, or for a few types of config changes, but in either case the user must specifically confirm the restart, or you can do it on-demand by going to the Advanced menu in the Admin tab and clicking the Restart Event Collection Services option.

    I'm not sure what you're referring to with the time sync process - is this something you setup yourself?

    Anyway a restart on ecs-ec-ingress would result in missing data, but a restart of ecs-ep (which writes data to disk) would not cause loss, just a gap (the events that would normally be written while it is down would be written once it's back up, which is what I thought was happening, though in retrospect it would be unusual for it to be down for a full minute. Did you check the audit logs as Hasan suggested? Maybe someone did delete the files, though I hope not.

    Cheers
    Colin



    ------------------------------
    COLIN HAY
    IBM Security
    ------------------------------

    Original Message


Related Content