IBM Security QRadar

 View Only
Expand all | Collapse all

Error "Process monitor app failed to start multiple times" keeps showing up

  • 1.  Error "Process monitor app failed to start multiple times" keeps showing up

    Posted Tue October 29, 2019 12:56 PM
    Edited by Derrick Nidar Tue October 29, 2019 02:52 PM
    I recently installed an All-in-One appliance in our VM. Version is v7.3.2 Build 20181119184207 patched to v7.3.2 Build 20190803012943As soon as I started it up, the error "Process monitor app failed to start multiple times" keeps showing up every minute, even when I didn't have a log source configured. 
    Payload Info: 
    Oct 29 11:33:01 127.0.0.1 [ProcessMonitor] com.q1labs.hostcontext.processmonitor.ProcessManager: [ERROR] [NOT:0150114103][10.21.66.55/- -] [-/- -]Process ecs-ec-ingress has failed to start for 1519 intervals. Continuing to try to start...

    I checked the service ecs-ec-ingress, and it was running. I restarted Event Collection Service through the user interface, but the error keeps on popping. I restarted the service in the Console, and the error was still popping up. I added some log sources, and I was getting the logs and info into QRadar without a problem, but the error was still there. I tried to stop the ecs-ec-ingress, the error stopped, but there were no logs coming since I stopped it. 

    I was wondering if anyone experienced something similar before, and what can be done to fix it?

    ------------------------------
    Derrick Nidar
    ------------------------------


  • 2.  RE: Error "Process monitor app failed to start multiple times" keeps showing up

    Posted Wed October 30, 2019 03:45 PM
    Edited by Jonathan Pechta Wed October 30, 2019 03:46 PM

    @Derrick Nidar

    This is definitely an issue where you should open a case with QRadar Support. Flag it as Severity 1 (System Down) if you haven't already opened a case with us.

    It sounds like to me that either the service is going Out-of-Memory (OOM) or the ecs-ec-ingress service is stopping/restating on you. When this happens or ecs-ec-ingress has an issue, there are normally jheap files that are written to disk. You might look to see if you have files in: /store/jheap/ecs-ec-ingress.ecs-ec-ingress

    I would submit a get_logs.sh with a case to the support team. My guess is that ecs-ec-ingress is likely running out of memory. However, if you see any files in that jheap ecs-ec-ingress.ecs-ec-ingress folder, attach at least one of those dump files to your case too as it will point to the root problem if ecs-ec-ingress is indeed crashing for some reason.

    We'll need to dig in to the error logs in /var/log/qradar.error to determine what is going on. My guess is an Out-of-Memory (OOM) issue, but the logs will reveal more. If you wanted to, you could also try the following command: less /var/log/qradar.error | grep OutOfMemoryMonitor and see if any results jump out at you from the logs. My guess (if correct is that you'll see a lot of OutofMemoryMonitor errors related to ecs-ec-ingress).

    We've seen some issues where some jar files (specifically jtds-1.2.6.jar) can lock up processes as protocols are loaded by ecs-ec-ingress and the lock causes out of memory notices in the logs and sounds similar to what you are seeing.


    Get a case open and we can confirm with you.

    Hope this helps, 
    - Jonathan



    ------------------------------
    Jonathan Pechta
    QRadar Support Content Lead
    Support forums: ibm.biz/qradarforums
    jonathan.pechta1@ibm.com
    ------------------------------



  • 3.  RE: Error "Process monitor app failed to start multiple times" keeps showing up

    Posted Wed October 30, 2019 04:18 PM
    @Jonathan Pechta

    Thanks for the information. I followed your suggestions. I checked /store/jheap/ecs-ec-ingress.ecs-ec-ingress and there were no jheap files. I also check qradar.error and there were nothing specifically pointing to OOM for ecs-ec-ingress. It just stated "Starting out-of-memory monitoring (enabled: yes)". I will follow your suggestion to open a case with QRadar Support. The logs might have something that would help. Thank you very much.

    ------------------------------
    Derrick Nidar
    ------------------------------