IBM QRadar

IBM QRadar

Join this online user group to communicate across Security product users and IBM experts by sharing advice and best practices with peers and staying up to date regarding product enhancements.

 View Only

  • 1.  Error "Process monitor app failed to start multiple times" keeps showing up

    Posted Tue October 29, 2019 12:56 PM
    Edited by Derrick Nidar Tue October 29, 2019 02:52 PM
    I recently installed an All-in-One appliance in our VM. Version is v7.3.2 Build 20181119184207 patched to v7.3.2 Build 20190803012943. As soon as I started it up, the error "Process monitor app failed to start multiple times" keeps showing up every minute, even when I didn't have a log source configured. 
    Payload Info: 
    Oct 29 11:33:01 127.0.0.1 [ProcessMonitor] com.q1labs.hostcontext.processmonitor.ProcessManager: [ERROR] [NOT:0150114103][10.21.66.55/- -] [-/- -]Process ecs-ec-ingress has failed to start for 1519 intervals. Continuing to try to start...

    I checked the service ecs-ec-ingress, and it was running. I restarted Event Collection Service through the user interface, but the error keeps on popping. I restarted the service in the Console, and the error was still popping up. I added some log sources, and I was getting the logs and info into QRadar without a problem, but the error was still there. I tried to stop the ecs-ec-ingress, the error stopped, but there were no logs coming since I stopped it. 

    I was wondering if anyone experienced something similar before, and what can be done to fix it?

    ------------------------------
    Derrick Nidar
    ------------------------------


  • 2.  RE: Error "Process monitor app failed to start multiple times" keeps showing up

    Posted Wed October 30, 2019 03:45 PM
    Edited by Jonathan Pechta Wed October 30, 2019 03:46 PM

    @Derrick Nidar

    This is definitely an issue where you should open a case with QRadar Support. Flag it as Severity 1 (System Down) if you haven't already opened a case with us.

    It sounds like to me that either the service is going Out-of-Memory (OOM) or the ecs-ec-ingress service is stopping/restating on you. When this happens or ecs-ec-ingress has an issue, there are normally jheap files that are written to disk. You might look to see if you have files in: /store/jheap/ecs-ec-ingress.ecs-ec-ingress

    I would submit a get_logs.sh with a case to the support team. My guess is that ecs-ec-ingress is likely running out of memory. However, if you see any files in that jheap ecs-ec-ingress.ecs-ec-ingress folder, attach at least one of those dump files to your case too as it will point to the root problem if ecs-ec-ingress is indeed crashing for some reason.

    We'll need to dig in to the error logs in /var/log/qradar.error to determine what is going on. My guess is an Out-of-Memory (OOM) issue, but the logs will reveal more. If you wanted to, you could also try the following command: less /var/log/qradar.error | grep OutOfMemoryMonitor and see if any results jump out at you from the logs. My guess (if correct is that you'll see a lot of OutofMemoryMonitor errors related to ecs-ec-ingress).

    We've seen some issues where some jar files (specifically jtds-1.2.6.jar) can lock up processes as protocols are loaded by ecs-ec-ingress and the lock causes out of memory notices in the logs and sounds similar to what you are seeing.


    Get a case open and we can confirm with you.

    Hope this helps, 
    - Jonathan



    ------------------------------
    Jonathan Pechta
    QRadar Support Content Lead
    Support forums: ibm.biz/qradarforums
    jonathan.pechta1@ibm.com
    ------------------------------

    Original Message


  • 3.  RE: Error "Process monitor app failed to start multiple times" keeps showing up

    Posted Wed October 30, 2019 04:18 PM
    @Jonathan Pechta

    Thanks for the information. I followed your suggestions. I checked /store/jheap/ecs-ec-ingress.ecs-ec-ingress and there were no jheap files. I also check qradar.error and there were nothing specifically pointing to OOM for ecs-ec-ingress. It just stated "Starting out-of-memory monitoring (enabled: yes)". I will follow your suggestion to open a case with QRadar Support. The logs might have something that would help. Thank you very much.

    ------------------------------
    Derrick Nidar
    ------------------------------

    Original Message