Our app appears to be crashing and it's happened twice in the last 2 weeks now. The buttons in Offenses simply disappear and we're not sure why. We opened a case with support and they couldn't determine the cause. Support has asked that we try asking here to see if anyone might have a fix.

Here are our initial questions:

• Are similar issues being reported or not with QRadar 7.3.3 FP2?
• Is there any debug that can be enabled within the container that would give us additional info when the app stops functioning?
• Are there any plans to release an updated version of the app?

Here's what we've done so far to troubleshoot:

• The app framework services have been checked, all services are up and running on the app host
• The app framework certs have been verified and all have valid expiration dates
• Issue seems to be specific to the Analyst Custom Searches App, all other containers are running:

43806ec7801d       console.localdeployment:5000/qapp/2251:1.2.1-20200203130103  "sh /start_container…"  4 hours ago        Up 4 hours         0.0.0.0:32784->5000/tcp  qapp-2251-wSdQgFX7

30b772cc4398       console.localdeployment:5000/qapp/2253:2.2.0-20200403105740  "sh /start_container…"  6 days ago         Up 6 days          0.0.0.0:32780->5000/tcp  qapp-2253-eKBgAPUV

0e008c442a87       console.localdeployment:5000/qapp/2258:3.5.0-20200403105415  "sh /start_container…"  6 days ago         Up 6 days          0.0.0.0:32779->5000/tcp  qapp-2258-nbLiWVxh

20a4498c7bfb       console.localdeployment:5000/qapp/2203:5.0.1-20200403105239  "sh /start_container…"  6 days ago         Up 6 days          0.0.0.0:32778->5000/tcp  qapp-2203-YULPvSsM

50199289fd90       console.localdeployment:5000/qapp/2254:1.2.5-20200403104947  "sh /start_container…"  6 days ago         Up 6 days          0.0.0.0:32776->5000/tcp  qapp-2254-W4eCJRpi

09d1dfde35ae       console.localdeployment:5000/qapp/2356:2.0.1-20200403103341  "sh /start_container…"  6 days ago         Up 6 days          0.0.0.0:32775->5000/tcp  qapp-2356-h4H0ULJS

7507d8f67e5d       console.localdeployment:5000/qapp/2353:1.4.4-20200403111158  "sh /start_container…"  6 days ago         Up 6 days          0.0.0.0:32774->5000/tcp  qapp-2353-LqxL07lw

b1f39b91030d       console.localdeployment:5000/qapp/2355:1.2.3-20200403110917  "sh /start_container…"  6 days ago         Up 6 days          0.0.0.0:32773->5000/tcp  qapp-2355-r1itSTHP

 

• There seem to be no errors in the log that would explain why the app stopped working. The only thing visible in the startup.log is the services starting up after restarting the container:

020-04-03 12:18:19,725 INFO spawned: 'startflask' with pid 36

2020-04-03 12:18:20,729 INFO success: startflask entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

2020-04-09 08:49:13,599 CRIT Supervisor running as root (no user in config file)

2020-04-09 08:49:13,685 INFO RPC interface 'supervisor' initialized

2020-04-09 08:49:13,685 CRIT Server 'unix_http_server' running without any HTTP authentication checking

2020-04-09 08:49:13,685 INFO supervisord started with pid 36

2020-04-09 08:49:14,688 INFO spawned: 'startflask' with pid 37

2020-04-09 08:49:15,689 INFO success: startflask entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

2020-04-09 08:59:17,045 CRIT Supervisor running as root (no user in config file)

------------------------------
Matt M
Sr. Cyber, CISSP, GCFE, GCFA, GREM
------------------------------

Hello Matt,

I did not hear from any similar issue about this app.

How many searches did you configure in total? Maybe this is a memory issue, but this should be highly unlikely.
You can check with docker stats how much memory apps consume. With around 30 searches, the app should take around 50 MB of 200 MB memory.

Do the buttons disappear for all offenses and for all users?
If the buttons disappear, does the container continue to run?
If the container still runs, can you try to load the offense in a private window (or different browser) to check, if it might be a browser caching issue?

Regards
Julian

------------------------------
Julian Sattelmair
------------------------------

Hey Julian,

We have 42 searches currently configured. All of the buttons disappear for all users when the issue occurs. Yes, the container still appears to be running.

If it happens again (It's only happened twice and we've since patched) I'll try clearing cache/private browsing and check resource usage for the docker.

Thank you,

------------------------------
Matt M
Sr. Cyber, CISSP, GCFE, GCFA, GREM
------------------------------

Original Message:
Sent: Tue April 21, 2020 04:44 PM
From: Julian Sattelmair
Subject: Analyst Custom Searches App Issues

Hello Matt,

I did not hear from any similar issue about this app.

How many searches did you configure in total? Maybe this is a memory issue, but this should be highly unlikely.
You can check with docker stats how much memory apps consume. With around 30 searches, the app should take around 50 MB of 200 MB memory.

Do the buttons disappear for all offenses and for all users?
If the buttons disappear, does the container continue to run?
If the container still runs, can you try to load the offense in a private window (or different browser) to check, if it might be a browser caching issue?

Regards
Julian

------------------------------
Julian Sattelmair
------------------------------

Hello Matt,

how many searches do you have configured in Analyst Custom Searches? It might be a memory issue, but this is very unlikely.

Do the buttons disappear for all users or only for some users?
Is the docker container still running during this time?

If it is still running, can one of the affected users try to open the offense in a private window or different window? Maybe it is a browser caching issue.

If this is not a caching issue, You could try to reinstall the app. Just do a backup of the searches using the Backup and Restore menu, uninstall the app, install it again, restore the searches in the Backup and restore menu by pasting the backed up JSON string.

Once I had an issue, that some apps did not work properly after an update, because the docker images was not build properly. This could be fixed, by backup, reinstall and restore of app data.

------------------------------
Julian Sattelmair
------------------------------

Hi Julian,

It has happened again. Here's the output of "docker stats"

CONTAINER ID        NAME                 CPU %               MEM USAGE / LIMIT    MEM %               NET I/O             BLOCK I/O           PIDS

c51a4391745d        qapp-2251-14oHFwl5   0.03%               31.09MiB / 200MiB    15.55%              37.4MB / 40.5MB     32.6MB / 1.56MB     5

We also tried 2 other browsers and the buttons are not showing.

The App appears to have fixed itself. Granted, we did do a regular deploy, but not sure if that is what may have resolved it.
------------------------------
Matt M
Sr. Cyber, CISSP, GCFE, GCFA, GREM
------------------------------

Original Message:
Sent: Wed April 22, 2020 06:23 AM
From: Julian Sattelmair
Subject: Analyst Custom Searches App Issues

Hello Matt,

how many searches do you have configured in Analyst Custom Searches? It might be a memory issue, but this is very unlikely.

Do the buttons disappear for all users or only for some users?
Is the docker container still running during this time?

If it is still running, can one of the affected users try to open the offense in a private window or different window? Maybe it is a browser caching issue.

If this is not a caching issue, You could try to reinstall the app. Just do a backup of the searches using the Backup and Restore menu, uninstall the app, install it again, restore the searches in the Backup and restore menu by pasting the backed up JSON string.

Once I had an issue, that some apps did not work properly after an update, because the docker images was not build properly. This could be fixed, by backup, reinstall and restore of app data.

------------------------------
Julian Sattelmair
------------------------------

------Original Message------

Hi Julian,

It has happened again. Here's the output of "docker stats"

CONTAINER ID        NAME                 CPU %               MEM USAGE / LIMIT    MEM %               NET I/O             BLOCK I/O           PIDS

c51a4391745d        qapp-2251-14oHFwl5   0.03%               31.09MiB / 200MiB    15.55%              37.4MB / 40.5MB     32.6MB / 1.56MB     5

We also tried 2 other browsers and the buttons are not showing.

The App appears to have fixed itself. Granted, we did do a regular deploy, but not sure if that is what may have resolved it.
------------------------------
Matt M
Sr. Cyber, CISSP, GCFE, GCFA, GREM
------------------------------