Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

6. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Ayappan P

Posted Thu September 23, 2021 11:30 AM

We didn't encounter these issues in our environment.
The permissions are properly set in the spec file -- > https://public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/SPECS/nagios-4.4.6-1.spec

The "Error:Cannot open .." is also not seen. Probably some older nagios processes are still running in the system.

------------------------------
Ayappan P
------------------------------

Original Message

7. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Erich Wolz

Posted Thu September 23, 2021 11:57 AM

> The permissions are properly set in the spec file

The only "chmod" commands I see in the above spec file (to set permissions) are the following (I do not see any chown commands):

chmod 0755 config.status

chmod 0755 ${RPM_BUILD_ROOT}/etc/rc.d/init.d/%{name}

> Probably some older nagios processes are still running in the system.

Nope:

[root@hl1axmon:/var/log/nagios] >ps -ef | grep nagios [root@hl1axmon:/var/log/nagios] >nohup /opt/freeware/bin/nagios -d /etc/nagios/nagios.cfg > /dev/console 2>&1 & [1] 24510746 [1] + Done nohup /opt/freeware/bin/nagios -d /etc/nagios/nagios.cfg > /dev/console 2>&1 & [root@hl1axmon:/var/log/nagios] >ps -ef| grep nagios nagios 8782330 19726594 0 0:00 <defunct> nagios 9699664 19726594 0 0:00 <defunct> nagios 10027378 19726594 0 0:00 <defunct> nagios 11141512 19726594 0 0:00 <defunct> nagios 11338124 19726594 0 0:00 <defunct> nagios 13762998 19726594 0 0:00 <defunct> nagios 14483880 19726594 0 0:00 <defunct> nagios 16777658 19726594 0 0:00 <defunct> nagios 19726594 1 0 10:43:19 - 0:00 /opt/freeware/bin/nagios -d /etc/nagios/nagios.cfg nagios 19923402 19726594 0 10:43:22 - 0:00 /opt/freeware/bin/nagios -d /etc/nagios/nagios.cfg nagios 20906282 19726594 0 0:00 <defunct> nagios 21889462 19726594 0 0:00 <defunct> nagios 24314338 19726594 0 0:00 <defunct> nagios 26280370 19726594 0 0:00 <defunct> [root@hl1axmon:/var/log/nagios] >cat nagios.log [1632411799] Nagios 4.4.6 starting... (PID=19726594) [1632411799] Local time is Thu Sep 23 10:43:19 CDT 2021 [1632411799] LOG VERSION: 2.0 [1632411799] qh: Socket '/var/nagios/rw/nagios.qh' successfully initialized [1632411799] qh: core query handler registered [1632411799] qh: echo service query handler registered [1632411799] qh: help for the query handler registered [1632411799] wproc: Successfully registered manager as @wproc with query handler [1632411802] Successfully launched command file worker with pid 19923402 [1632411802] Unable to send check for host <hostname1> to worker (ret=-2) [1632411803] Unable to send check for host <hostname2> to worker (ret=-2) [1632411804] Unable to send check for host <hostname3> to worker (ret=-2) (etc.)

and in the syslog:

Sep 23 10:43:19 hl1axmon user:info syslog: Nagios 4.4.6 starting... (PID=19726594)
Sep 23 10:43:19 hl1axmon user:info syslog: Local time is Thu Sep 23 10:43:19 CDT 2021
Sep 23 10:43:19 hl1axmon user:info syslog: LOG VERSION: 2.0
Sep 23 10:43:19 hl1axmon user:info syslog: qh: Socket '/var/nagios/rw/nagios.qh' successfully initialized
Sep 23 10:43:19 hl1axmon user:info syslog: qh: core query handler registered
Sep 23 10:43:19 hl1axmon user:info syslog: qh: echo service query handler registered
Sep 23 10:43:19 hl1axmon user:info syslog: qh: help for the query handler registered
Sep 23 10:43:19 hl1axmon user:info syslog: wproc: Successfully registered manager as @wproc with query handler
Sep 23 10:43:19 hl1axmon user:info syslog: Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!
Sep 23 10:43:19 hl1axmon user:info last message repeated 11 times
Sep 23 10:43:22 hl1axmon user:info syslog: Successfully launched command file worker with pid 19923402
Sep 23 10:43:22 hl1axmon user:info syslog: Unable to send check for host <hostname1> to worker (ret=-2)
(etc.)

On a working monitor still running Nagios 3.x there are no defunct processes. Instead there are a bunch of spawned "check_ping" commands and the like.

------------------------------
Erich Wolz
------------------------------

Original Message

8. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Erich Wolz

Posted Tue September 28, 2021 09:51 AM

> We didn't encounter these issues in our environment.

Are you actually using Nagios to monitor your environment, or are you just building Nagios without actually trying to use it?

If the former, then there must be some difference between your environment and mine, and it would help to know what that is :-)

What is your oslevel -s? Your lsdev | grep Available? Your rpm -qa | sort?

Up until last week, my older Nagios 3.x was running on IBM HTTP Server; I recently migrated to the AIX Toolbox version of apache to eliminate that as a variable (i.e. my older Nagios 3.x runs on Toolbox apache, but my Toolbox Nagios does not... so it would appear that apache itself is not the source of the problem).

------------------------------
Erich Wolz
------------------------------

Original Message

9. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Erich Wolz

Posted Mon October 04, 2021 09:48 AM

Ping

------------------------------
Erich Wolz
------------------------------

Original Message

10. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

RESHMA KUMAR

Posted Tue October 05, 2021 10:27 AM

| view attached

# oslevel -s
7200-04-02-2028

# lsdev | grep Available
L2cache0 Available L2 Cache
cluster0 Available Cluster Node
dac0 Available 17-T1-01 DS3/4/5K PCM User Interface
dac1 Available 17-T1-01 DS3/4/5K PCM User Interface
en0 Available Standard Ethernet Network Interface
ent0 Available Virtual I/O Ethernet Adapter (l-lan)
fcs0 Available 17-T1 Virtual Fibre Channel Client Adapter
fscsi0 Available 17-T1-01 FC SCSI I/O Controller Protocol Device
hdisk0 Available Virtual SCSI Disk Drive
hdisk1 Available 17-T1-01 MPIO DS5100/5300 Disk
hdisk2 Available 17-T1-01 MPIO DS5100/5300 Disk
inet0 Available Internet Network Extension
iscsi0 Available iSCSI Protocol Device
lo0 Available Loopback Network Interface
loop0 Available Loopback Device
lvdd Available LVM Device Driver
mem0 Available Memory
pkcs11 Available ACF/PKCS#11 Device
proc0 Available 00-00 Processor
proc4 Available 00-04 Processor
proc8 Available 00-08 Processor
proc12 Available 00-12 Processor
pty0 Available Asynchronous Pseudo-Terminal
sfw0 Available Storage Framework Module
sfwcomm0 Available 17-T1-01-FF Fibre Channel Storage Framework Comm
sys0 Available System Object
sysplanar0 Available System Planar
vio0 Available Virtual I/O Bus
vsa0 Available LPAR Virtual Serial Adapter
vscsi0 Available Virtual SCSI Client Adapter
vty0 Available Asynchronous Terminal

Please find the attached document for the output of rpm -qa | sort

------------------------------
RESHMA KUMAR
------------------------------

Attachment(s)

rpm-qa-output.txt 20 KB 1 version

Original Message

11. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Erich Wolz

Posted Tue October 05, 2021 12:03 PM

| view attached

Thanks for the above command outputs. However, I can't tell whether you are actually using Nagios to monitor your environment, or you are just building Nagios without actually trying to run it?

First difference between our environments: my oslevel (7200-03-05-2016)

Devices you have that I don't have (not sure if any of these would account for the issues I'm seeing):

< dac0 Available 17-T1-01 DS3/4/5K PCM User Interface
< dac1 Available 17-T1-01 DS3/4/5K PCM User Interface
< fcs0 Available 17-T1 Virtual Fibre Channel Client Adapter
< fscsi0 Available 17-T1-01 FC SCSI I/O Controller Protocol Device
< hdisk1 Available 17-T1-01 MPIO DS5100/5300 Disk
< hdisk2 Available 17-T1-01 MPIO DS5100/5300 Disk
< loop0 Available Loopback Device
< proc4 Available 00-04 Processor
< proc12 Available 00-12 Processor
< sfwcomm0 Available 17-T1-01-FF Fibre Channel Storage Framework Comm

Devices I have that you don't have (pretty sure none of these would account for the issues I'm seeing):

> cd0 Available Virtual SCSI Optical Served by VIO Server
> hdisk1 Available Virtual SCSI Disk Drive
> vscsi1 Available Virtual SCSI Client Adapter

RPMs you have that I don't have (see attached file "rpmdiff.txt")

RPMs I have that you don't have:

> curl-7.76.1-1.ppc
> mod_perl-2.0.11-2.ppc
> mod_ssl-2.4.48-1.ppc
> ncurses-6.2-2.ppc
> php-7.4.22-1.ppc
> python-pycurl-7.43.0-1.ppc
> python-urlgrabber-3.10.1-1.noarch
> sudo-1.9.5p2-1.ppc
> yum-3.4.3-8.noarch

As I am not familiar with many (if not most) of the RPMs you have installed, could one of them perhaps account for the fact that I am seeing "Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!" and "Unable to send check for host <hostname> to worker (ret=-2)" messages in my syslog? If so, it should probably be noted as a prereq.

------------------------------
Erich Wolz
------------------------------

Attachment(s)

rpmdiff.txt 19 KB 1 version

Original Message

12. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Erich Wolz

Posted Wed October 06, 2021 05:15 PM

Also still wondering why the syslog shows

syslog: Nagios 4.4.6 starting... (PID=13762976)
syslog: Local time is Wed Oct 06 16:00:54 CDT 2021
syslog: LOG VERSION: 2.0
syslog: qh: Socket '/var/nagios/rw/nagios.qh' successfully initialized
syslog: qh: core query handler registered
syslog: qh: echo service query handler registered
syslog: qh: help for the query handler registered
syslog: wproc: Successfully registered manager as @wproc with query handler
syslog: Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!

when the nagios config file is specified as follows in /etc/inittab:

/opt/freeware/bin/nagios -d /etc/nagios/nagios.cfg > /dev/console 2>&1

/etc/nagios/nagios.cfg is a file; /var/nagios/rw/nagios.qh is a socket. I don't know where the problem is, but clearly something's not being parsed correctly.

$ ls -l /etc/nagios/nagios.cfg /var/nagios/rw/nagios.qh
-rw-rw-r-- 1 nagios nagios 45626 Sep 22 20:31 /etc/nagios/nagios.cfg
srwxrwxrwx 1 nagios nagiscmd 0 Oct 06 16:00 /var/nagios/rw/nagios.qh

------------------------------
Erich Wolz
------------------------------

Original Message

13. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Erich Wolz

Posted Thu October 07, 2021 06:07 PM

Is there a particular debug_level I should set, to see why the AIX Toolbox nagios 4.4.6 thinks the main configuration file is the socket '/var/nagios/rw/nagios.qh' and not the file '/etc/nagios/nagios.cfg' that is being specified on the command line?

------------------------------
Erich Wolz
------------------------------

Original Message

14. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Erich Wolz

Posted Mon October 11, 2021 12:17 PM

Ping.

From the nagios log itself, it would appear that the config file is (correctly) found to be /etc/nagios/nagios.cfg -- given that the config file specifies the socket to use for the query handler interface, and this socket is successfully initialized:

# grep query_socket /etc/nagios/nagios.cfg
query_socket=/var/nagios/rw/nagios.qh

# head -14 /var/log/nagios/nagios.log
[1633650391] Nagios 4.4.6 starting... (PID=16777578)
[1633650391] Local time is Thu Oct 07 18:46:31 CDT 2021
[1633650391] LOG VERSION: 2.0
[1633650391] qh: Socket '/var/nagios/rw/nagios.qh' successfully initialized
[1633650391] qh: core query handler registered
[1633650391] qh: echo service query handler registered
[1633650391] qh: help for the query handler registered
[1633650391] wproc: Successfully registered manager as @wproc with query handler
[1633650394] Successfully launched command file worker with pid 13763004
[1633650394] Unable to send check for host <hostname> to worker (ret=-2)

However, once the query handler has been initialized/registered, the nagios server is trying to read from the socket -- and not the config file -- for further config info (I'm guessing this may be due to the config file handle not having been restored properly after the name of the socket was determined):

# grep "Oct 7 18:46:3" /syslogs/syslog
Oct 7 18:46:31 hl1axmon user:info syslog: Nagios 4.4.6 starting... (PID=16777578)
Oct 7 18:46:31 hl1axmon user:info syslog: Local time is Thu Oct 07 18:46:31 CDT 2021
Oct 7 18:46:31 hl1axmon user:info syslog: LOG VERSION: 2.0
Oct 7 18:46:31 hl1axmon user:info syslog: qh: Socket '/var/nagios/rw/nagios.qh' successfully initialized
Oct 7 18:46:31 hl1axmon user:info syslog: qh: core query handler registered
Oct 7 18:46:31 hl1axmon user:info syslog: qh: echo service query handler registered
Oct 7 18:46:31 hl1axmon user:info syslog: qh: help for the query handler registered
Oct 7 18:46:31 hl1axmon user:info syslog: wproc: Successfully registered manager as @wproc with query handler
Oct 7 18:46:31 hl1axmon user:info syslog: Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!
Oct 7 18:46:31 hl1axmon user:info last message repeated 11 times
Oct 7 18:46:34 hl1axmon user:info syslog: Successfully launched command file worker with pid 13763004
Oct 7 18:46:34 hl1axmon user:info syslog: Unable to send check for host <hostname> to worker (ret=-2)

------------------------------
Erich Wolz
------------------------------

Original Message

15. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Erich Wolz

Posted Thu October 14, 2021 10:37 PM

Ping.

From https://stackoverflow.com/questions/42777659/nagios-unable-to-send-check-for-host-or-run-check-for-service

...With NagiosSupport's help, we found out that it was SELinux that was in the "enforcing" mode (and/or a bad policy that was not updated in the EPEL package when updating Nagios) that was causing the issues....

Of course, this was on a Linux server not an AIX server... but sure enough, https://serverfault.com/questions/894349/nagios-on-rhel-6-epel-package-stopped-working-after-update seems to corroborate that a change to something in the EPEL package for Nagios on RedHat Enterprise Linux was to blame.

Since (in addition to "Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!") I am also seeing "Unable to send check for host" and "Unable to run check for service" messages, might there be an AIX equivalent to the EPEL package for Nagios on RHEL that is preventing these checks from running and/or causing the nagios daemon to "reset" the value of the main configuration file?

------------------------------
Erich Wolz
------------------------------

Original Message

16. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Erich Wolz

Posted Mon October 18, 2021 07:00 PM

Ping.

Also, for what it's worth... the Health Check policy we use wants each active inittab entry's file/command/script executable -- and all existing directories in its path -- to have settings for "group" and "other" of r-x or more stringent, if owned by a group not considered as privileged (which the "nagios" group is not, by default). However, changing the ownership of /opt/freeware/bin/nagios to root:system (and the permissions to 0700, 0750, or 0755) results in the following messages in the syslog:

Error: failed to access() /opt/freeware/bin/nagios: Permission denied
Error: Spawning workers will be impossible. Aborting.

My existing Nagios 3.1.2 executable (built on AIX 6.1 years ago) is owned by root:system and has permissions of 0700, and does not have this problem. Needless to say, I changed the ownership of /opt/freeware/bin/nagios back to nagios:nagios and the permissions back to 0774... but I am still left with the fact that there's something in the Nagios 4.4.6 source code that is causing the name of the main config file to be changed from from '/etc/nagios/nagios.cfg' to '/var/nagios/rw/nagios.qh' at some point after initialization -- and this is (clearly) the more important issue.

------------------------------
Erich Wolz
------------------------------

Original Message

17. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Erich Wolz

Posted Wed October 27, 2021 09:54 AM

Ping

------------------------------
Erich Wolz
------------------------------

Original Message

18. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

RESHMA KUMAR

Posted Fri October 29, 2021 01:30 AM

Thanks for sharing the details. We are looking into the issue.

------------------------------
RESHMA KUMAR
------------------------------

Original Message

19. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Erich Wolz

Posted Wed November 17, 2021 11:31 AM

Anything to report?

------------------------------
Erich Wolz
------------------------------

Original Message

20. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

RESHMA KUMAR

Posted Wed November 24, 2021 06:52 AM

We are working on it

------------------------------
RESHMA KUMAR
------------------------------

Original Message

21. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

RESHMA KUMAR

Posted Fri November 26, 2021 12:45 AM

We were able to fix the following error with a patch.
Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!

We are checking "Unable to run check for service " errors now.

------------------------------
RESHMA KUMAR
------------------------------

Original Message

22. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Erich Wolz

Posted Mon November 29, 2021 10:31 AM

That is indeed welcome news! Looking forward to the availability of the patch.

------------------------------
Erich Wolz
------------------------------

Original Message

23. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

RESHMA KUMAR

Posted Tue December 14, 2021 02:13 AM

We have fixed the issue and uploaded nagios-4.4.6-2.
https://public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/RPMS/ppc/nagios/

Before starting nagios server, please increase the ulimit value of nofiles.

------------------------------
RESHMA KUMAR
------------------------------

Original Message

24. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Erich Wolz

Posted Tue December 14, 2021 11:10 AM

That is indeed good news! What is the recommended value for nofiles?

------------------------------
Erich Wolz
------------------------------

Original Message

25. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Ayappan P

Posted Wed December 15, 2021 02:51 AM

Try "ulimit -n 20000"

------------------------------
Ayappan P
------------------------------

Original Message

26. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Erich Wolz

Posted Thu December 16, 2021 11:30 AM

After updating the nagios and nagios-gui packages and setting nofiles=20000 I no longer get the "Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!" messages in nagios.log... but I still get "Unable to send check for host" and "Unable to run check for service" messages:

[root@hl1axmon:/var/log/nagios] >cat nagios.log
[1639671588] Nagios 4.4.6 starting... (PID=14025042)
[1639671588] Local time is Thu Dec 16 10:19:48 CST 2021
[1639671588] LOG VERSION: 2.0
[1639671588] qh: Socket '/var/nagios/rw/nagios.qh' successfully initialized
[1639671588] qh: core query handler registered
[1639671588] qh: echo service query handler registered
[1639671588] qh: help for the query handler registered
[1639671588] wproc: Successfully registered manager as @wproc with query handler
[1639671588] wproc: Registry request: name=Core Worker 21168520;pid=21168520
[1639671588] wproc: Registry request: name=Core Worker 13566242;pid=13566242
[1639671588] wproc: Registry request: name=Core Worker 15663522;pid=15663522
[1639671588] wproc: Registry request: name=Core Worker 15794538;pid=15794538
[1639671588] wproc: Registry request: name=Core Worker 5964042;pid=5964042
[1639671588] wproc: Registry request: name=Core Worker 19399124;pid=19399124
[1639671588] wproc: Registry request: name=Core Worker 18350506;pid=18350506
[1639671588] wproc: Registry request: name=Core Worker 19661182;pid=19661182
[1639671588] wproc: Registry request: name=Core Worker 13828518;pid=13828518
[1639671588] wproc: Registry request: name=Core Worker 18022714;pid=18022714
[1639671588] wproc: Registry request: name=Core Worker 16843188;pid=16843188
[1639671588] wproc: Registry request: name=Core Worker 18481458;pid=18481458
[1639671588] Successfully launched command file worker with pid 22479158
[1639671590] wproc: Socket to worker Core Worker 21168520 broken, removing
[1639671590] wproc: 'Core Worker 15794538' seems to be choked. ret = -1; bufsize = 132: written = 0; errno = 32 (Broken pipe)
[1639671590] wproc: Socket to worker Core Worker 13566242 broken, removing
[1639671590] wproc: 'Core Worker 19399124' seems to be choked. ret = -1; bufsize = 132: written = 0; errno = 32 (Broken pipe)
[1639671590] wproc: Socket to worker Core Worker 15663522 broken, removing
[1639671590] wproc: Socket to worker Core Worker 15794538 broken, removing
[1639671590] wproc: Socket to worker Core Worker 5964042 broken, removing
[1639671590] wproc: Socket to worker Core Worker 19399124 broken, removing
[1639671590] wproc: Socket to worker Core Worker 18350506 broken, removing
[1639671590] wproc: Socket to worker Core Worker 19661182 broken, removing
[1639671590] wproc: Socket to worker Core Worker 13828518 broken, removing
[1639671590] wproc: Socket to worker Core Worker 18022714 broken, removing
[1639671590] wproc: Socket to worker Core Worker 16843188 broken, removing
[1639671590] wproc: Socket to worker Core Worker 18481458 broken, removing

At this point, the rest of the log consists of "Unable to send check for host" and "Unable to run check for service" messages.

------------------------------
Erich Wolz
------------------------------

Original Message

27. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Erich Wolz

Posted Thu December 16, 2021 12:06 PM

bumping nofiles to 65536 didn't help any

------------------------------
Erich Wolz
------------------------------

Original Message

28. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Ayappan P

Posted Fri December 17, 2021 02:02 AM

For some reason , the workers are getting choked and removed in your machine. Because of that the "unable to check/send" errors are shown in the log.
Can you share the output of "ulimit -a" and vmstat ?

------------------------------
Ayappan P
------------------------------

Original Message

29. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Erich Wolz

Posted Tue January 04, 2022 06:17 PM

Sorry, was on vacation and only just saw this request.

[root@hl1axmon:/] >ulimit -a
time(seconds) unlimited
file(blocks) 2097151
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2097151
nofiles(descriptors) 65536
threads(per process) unlimited
processes(per user) unlimited
[root@hl1axmon:/] >prtconf 2>/dev/null | head -15
System Model: IBM,8286-42A
Machine Serial Number: 219BDDV
Processor Type: PowerPC_POWER8
Processor Implementation Mode: POWER 8
Processor Version: PV_8_Compat
Number Of Processors: 2
Processor Clock Speed: 3525 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
LPAR Info: 13 hl1axmon
Memory Size: 8192 MB
Good Memory Size: 8192 MB
Platform Firmware level: SV860_215
Firmware Version: IBM,FW860.81 (SV860_215)
Console Login: enable
[root@hl1axmon:/] >vmstat

System configuration: lcpu=8 mem=8192MB ent=0.20

kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
2 1 593600 624946 0 0 0 0 0 0 2 1301 231 0 0 99 0 0.00 1.2

------------------------------
Erich Wolz
------------------------------

Original Message

30. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Ayappan P

Posted Fri January 07, 2022 05:09 AM

Can you try with two workers by setting "check_workers=2" in the nagios.cfg file ?

------------------------------
Ayappan P
------------------------------

Original Message

Original Message:
Sent: Tue January 04, 2022 06:16 PM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Sorry, was on vacation and only just saw this request.

[root@hl1axmon:/] >ulimit -a
time(seconds) unlimited
file(blocks) 2097151
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2097151
nofiles(descriptors) 65536
threads(per process) unlimited
processes(per user) unlimited
[root@hl1axmon:/] >prtconf 2>/dev/null | head -15
System Model: IBM,8286-42A
Machine Serial Number: 219BDDV
Processor Type: PowerPC_POWER8
Processor Implementation Mode: POWER 8
Processor Version: PV_8_Compat
Number Of Processors: 2
Processor Clock Speed: 3525 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
LPAR Info: 13 hl1axmon
Memory Size: 8192 MB
Good Memory Size: 8192 MB
Platform Firmware level: SV860_215
Firmware Version: IBM,FW860.81 (SV860_215)
Console Login: enable
[root@hl1axmon:/] >vmstat

System configuration: lcpu=8 mem=8192MB ent=0.20

kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
2 1 593600 624946 0 0 0 0 0 0 2 1301 231 0 0 99 0 0.00 1.2

------------------------------
Erich Wolz

Original Message:
Sent: Fri December 17, 2021 02:02 AM
From: Ayappan P
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

For some reason , the workers are getting choked and removed in your machine. Because of that the "unable to check/send" errors are shown in the log.
Can you share the output of "ulimit -a" and vmstat ?

------------------------------
Ayappan P

Original Message:
Sent: Thu December 16, 2021 12:06 PM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

bumping nofiles to 65536 didn't help any

------------------------------
Erich Wolz

Original Message:
Sent: Wed December 15, 2021 02:50 AM
From: Ayappan P
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Try "ulimit -n 20000"

------------------------------
Ayappan P

Original Message:
Sent: Tue December 14, 2021 11:09 AM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

That is indeed good news! What is the recommended value for nofiles?

------------------------------
Erich Wolz

Original Message:
Sent: Tue December 14, 2021 02:12 AM
From: RESHMA KUMAR
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

We have fixed the issue and uploaded nagios-4.4.6-2.
https://public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/RPMS/ppc/nagios/

Before starting nagios server, please increase the ulimit value of nofiles.

------------------------------
RESHMA KUMAR

Original Message:
Sent: Mon November 29, 2021 10:31 AM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

That is indeed welcome news! Looking forward to the availability of the patch.

------------------------------
Erich Wolz

Original Message:
Sent: Fri November 26, 2021 12:44 AM
From: RESHMA KUMAR
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

We were able to fix the following error with a patch.
Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!

We are checking "Unable to run check for service " errors now.

------------------------------
RESHMA KUMAR

Original Message:
Sent: Wed November 24, 2021 06:51 AM
From: RESHMA KUMAR
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

We are working on it

------------------------------
RESHMA KUMAR

Original Message:
Sent: Wed November 17, 2021 11:30 AM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Anything to report?

------------------------------
Erich Wolz

Original Message:
Sent: Fri October 29, 2021 01:30 AM
From: RESHMA KUMAR
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Thanks for sharing the details. We are looking into the issue.

------------------------------
RESHMA KUMAR

Original Message:
Sent: Wed October 27, 2021 09:54 AM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Ping

------------------------------
Erich Wolz

Original Message:
Sent: Mon October 18, 2021 07:00 PM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Ping.

Also, for what it's worth... the Health Check policy we use wants each active inittab entry's file/command/script executable -- and all existing directories in its path -- to have settings for "group" and "other" of r-x or more stringent, if owned by a group not considered as privileged (which the "nagios" group is not, by default). However, changing the ownership of /opt/freeware/bin/nagios to root:system (and the permissions to 0700, 0750, or 0755) results in the following messages in the syslog:

Error: failed to access() /opt/freeware/bin/nagios: Permission denied
Error: Spawning workers will be impossible. Aborting.

My existing Nagios 3.1.2 executable (built on AIX 6.1 years ago) is owned by root:system and has permissions of 0700, and does not have this problem. Needless to say, I changed the ownership of /opt/freeware/bin/nagios back to nagios:nagios and the permissions back to 0774... but I am still left with the fact that there's something in the Nagios 4.4.6 source code that is causing the name of the main config file to be changed from from '/etc/nagios/nagios.cfg' to '/var/nagios/rw/nagios.qh' at some point after initialization -- and this is (clearly) the more important issue.

------------------------------
Erich Wolz

Original Message:
Sent: Thu October 14, 2021 10:37 PM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Ping.

From https://stackoverflow.com/questions/42777659/nagios-unable-to-send-check-for-host-or-run-check-for-service

...With NagiosSupport's help, we found out that it was SELinux that was in the "enforcing" mode (and/or a bad policy that was not updated in the EPEL package when updating Nagios) that was causing the issues....

Of course, this was on a Linux server not an AIX server... but sure enough, https://serverfault.com/questions/894349/nagios-on-rhel-6-epel-package-stopped-working-after-update seems to corroborate that a change to something in the EPEL package for Nagios on RedHat Enterprise Linux was to blame.

Since (in addition to "Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!") I am also seeing "Unable to send check for host" and "Unable to run check for service" messages, might there be an AIX equivalent to the EPEL package for Nagios on RHEL that is preventing these checks from running and/or causing the nagios daemon to "reset" the value of the main configuration file?

------------------------------
Erich Wolz

Original Message:
Sent: Mon October 11, 2021 12:16 PM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Ping.

From the nagios log itself, it would appear that the config file is (correctly) found to be /etc/nagios/nagios.cfg -- given that the config file specifies the socket to use for the query handler interface, and this socket is successfully initialized:

# grep query_socket /etc/nagios/nagios.cfg
query_socket=/var/nagios/rw/nagios.qh

# head -14 /var/log/nagios/nagios.log
[1633650391] Nagios 4.4.6 starting... (PID=16777578)
[1633650391] Local time is Thu Oct 07 18:46:31 CDT 2021
[1633650391] LOG VERSION: 2.0
[1633650391] qh: Socket '/var/nagios/rw/nagios.qh' successfully initialized
[1633650391] qh: core query handler registered
[1633650391] qh: echo service query handler registered
[1633650391] qh: help for the query handler registered
[1633650391] wproc: Successfully registered manager as @wproc with query handler
[1633650394] Successfully launched command file worker with pid 13763004
[1633650394] Unable to send check for host <hostname> to worker (ret=-2)

However, once the query handler has been initialized/registered, the nagios server is trying to read from the socket -- and not the config file -- for further config info (I'm guessing this may be due to the config file handle not having been restored properly after the name of the socket was determined):

# grep "Oct 7 18:46:3" /syslogs/syslog
Oct 7 18:46:31 hl1axmon user:info syslog: Nagios 4.4.6 starting... (PID=16777578)
Oct 7 18:46:31 hl1axmon user:info syslog: Local time is Thu Oct 07 18:46:31 CDT 2021
Oct 7 18:46:31 hl1axmon user:info syslog: LOG VERSION: 2.0
Oct 7 18:46:31 hl1axmon user:info syslog: qh: Socket '/var/nagios/rw/nagios.qh' successfully initialized
Oct 7 18:46:31 hl1axmon user:info syslog: qh: core query handler registered
Oct 7 18:46:31 hl1axmon user:info syslog: qh: echo service query handler registered
Oct 7 18:46:31 hl1axmon user:info syslog: qh: help for the query handler registered
Oct 7 18:46:31 hl1axmon user:info syslog: wproc: Successfully registered manager as @wproc with query handler
Oct 7 18:46:31 hl1axmon user:info syslog: Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!
Oct 7 18:46:31 hl1axmon user:info last message repeated 11 times
Oct 7 18:46:34 hl1axmon user:info syslog: Successfully launched command file worker with pid 13763004
Oct 7 18:46:34 hl1axmon user:info syslog: Unable to send check for host <hostname> to worker (ret=-2)

------------------------------
Erich Wolz

Original Message:
Sent: Thu October 07, 2021 06:07 PM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Is there a particular debug_level I should set, to see why the AIX Toolbox nagios 4.4.6 thinks the main configuration file is the socket '/var/nagios/rw/nagios.qh' and not the file '/etc/nagios/nagios.cfg' that is being specified on the command line?

------------------------------
Erich Wolz

Original Message:
Sent: Wed October 06, 2021 05:14 PM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Also still wondering why the syslog shows

syslog: Nagios 4.4.6 starting... (PID=13762976)
syslog: Local time is Wed Oct 06 16:00:54 CDT 2021
syslog: LOG VERSION: 2.0
syslog: qh: Socket '/var/nagios/rw/nagios.qh' successfully initialized
syslog: qh: core query handler registered
syslog: qh: echo service query handler registered
syslog: qh: help for the query handler registered
syslog: wproc: Successfully registered manager as @wproc with query handler
syslog: Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!

when the nagios config file is specified as follows in /etc/inittab:

/opt/freeware/bin/nagios -d /etc/nagios/nagios.cfg > /dev/console 2>&1

/etc/nagios/nagios.cfg is a file; /var/nagios/rw/nagios.qh is a socket. I don't know where the problem is, but clearly something's not being parsed correctly.

$ ls -l /etc/nagios/nagios.cfg /var/nagios/rw/nagios.qh
-rw-rw-r-- 1 nagios nagios 45626 Sep 22 20:31 /etc/nagios/nagios.cfg
srwxrwxrwx 1 nagios nagiscmd 0 Oct 06 16:00 /var/nagios/rw/nagios.qh

------------------------------
Erich Wolz

Original Message:
Sent: Tue October 05, 2021 12:03 PM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Thanks for the above command outputs. However, I can't tell whether you are actually using Nagios to monitor your environment, or you are just building Nagios without actually trying to run it?

First difference between our environments: my oslevel (7200-03-05-2016)

Devices you have that I don't have (not sure if any of these would account for the issues I'm seeing):

< dac0 Available 17-T1-01 DS3/4/5K PCM User Interface
< dac1 Available 17-T1-01 DS3/4/5K PCM User Interface
< fcs0 Available 17-T1 Virtual Fibre Channel Client Adapter
< fscsi0 Available 17-T1-01 FC SCSI I/O Controller Protocol Device
< hdisk1 Available 17-T1-01 MPIO DS5100/5300 Disk
< hdisk2 Available 17-T1-01 MPIO DS5100/5300 Disk
< loop0 Available Loopback Device
< proc4 Available 00-04 Processor
< proc12 Available 00-12 Processor
< sfwcomm0 Available 17-T1-01-FF Fibre Channel Storage Framework Comm

Devices I have that you don't have (pretty sure none of these would account for the issues I'm seeing):

> cd0 Available Virtual SCSI Optical Served by VIO Server
> hdisk1 Available Virtual SCSI Disk Drive
> vscsi1 Available Virtual SCSI Client Adapter

RPMs you have that I don't have (see attached file "rpmdiff.txt")

RPMs I have that you don't have:

> curl-7.76.1-1.ppc
> mod_perl-2.0.11-2.ppc
> mod_ssl-2.4.48-1.ppc
> ncurses-6.2-2.ppc
> php-7.4.22-1.ppc
> python-pycurl-7.43.0-1.ppc
> python-urlgrabber-3.10.1-1.noarch
> sudo-1.9.5p2-1.ppc
> yum-3.4.3-8.noarch

As I am not familiar with many (if not most) of the RPMs you have installed, could one of them perhaps account for the fact that I am seeing "Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!" and "Unable to send check for host <hostname> to worker (ret=-2)" messages in my syslog? If so, it should probably be noted as a prereq.

------------------------------
Erich Wolz

Original Message:
Sent: Tue October 05, 2021 10:27 AM
From: RESHMA KUMAR
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

# oslevel -s
7200-04-02-2028

# lsdev | grep Available
L2cache0 Available L2 Cache
cluster0 Available Cluster Node
dac0 Available 17-T1-01 DS3/4/5K PCM User Interface
dac1 Available 17-T1-01 DS3/4/5K PCM User Interface
en0 Available Standard Ethernet Network Interface
ent0 Available Virtual I/O Ethernet Adapter (l-lan)
fcs0 Available 17-T1 Virtual Fibre Channel Client Adapter
fscsi0 Available 17-T1-01 FC SCSI I/O Controller Protocol Device
hdisk0 Available Virtual SCSI Disk Drive
hdisk1 Available 17-T1-01 MPIO DS5100/5300 Disk
hdisk2 Available 17-T1-01 MPIO DS5100/5300 Disk
inet0 Available Internet Network Extension
iscsi0 Available iSCSI Protocol Device
lo0 Available Loopback Network Interface
loop0 Available Loopback Device
lvdd Available LVM Device Driver
mem0 Available Memory
pkcs11 Available ACF/PKCS#11 Device
proc0 Available 00-00 Processor
proc4 Available 00-04 Processor
proc8 Available 00-08 Processor
proc12 Available 00-12 Processor
pty0 Available Asynchronous Pseudo-Terminal
sfw0 Available Storage Framework Module
sfwcomm0 Available 17-T1-01-FF Fibre Channel Storage Framework Comm
sys0 Available System Object
sysplanar0 Available System Planar
vio0 Available Virtual I/O Bus
vsa0 Available LPAR Virtual Serial Adapter
vscsi0 Available Virtual SCSI Client Adapter
vty0 Available Asynchronous Terminal

Please find the attached document for the output of rpm -qa | sort

------------------------------
RESHMA KUMAR

Original Message:
Sent: Mon October 04, 2021 09:48 AM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Ping

------------------------------
Erich Wolz

Original Message:
Sent: Tue September 28, 2021 09:51 AM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

> We didn't encounter these issues in our environment.

Are you actually using Nagios to monitor your environment, or are you just building Nagios without actually trying to use it?

If the former, then there must be some difference between your environment and mine, and it would help to know what that is :-)

What is your oslevel -s? Your lsdev | grep Available? Your rpm -qa | sort?

Up until last week, my older Nagios 3.x was running on IBM HTTP Server; I recently migrated to the AIX Toolbox version of apache to eliminate that as a variable (i.e. my older Nagios 3.x runs on Toolbox apache, but my Toolbox Nagios does not... so it would appear that apache itself is not the source of the problem).

------------------------------
Erich Wolz

Original Message:
Sent: Thu September 23, 2021 11:29 AM
From: Ayappan P
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

We didn't encounter these issues in our environment.
The permissions are properly set in the spec file -- > https://public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/SPECS/nagios-4.4.6-1.spec

The "Error:Cannot open .." is also not seen. Probably some older nagios processes are still running in the system.

------------------------------
Ayappan P

Original Message:
Sent: Wed September 22, 2021 11:13 PM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Getting the Nagios 4.4.6 RPMs from the AIX Toolbox up and running on AIX 7.2.3.5 has had to take a back seat for awhile, but I again found time to look at this. After copying my Nagios 3.x config files over and tweaking them so that /opt/freeware/bin/nagios -v /etc/nagios/nagios.cfg reports no warnings, no errors, and no serious problems detected during the pre-flight check, I saw the following in the nagios.log

[1632363052] Error: failed to access() /opt/freeware/bin/nagios: Permission denied
[1632363052] Error: Spawning workers will be impossible. Aborting.

The nagios command was owned by root:system and its permissions were 0750; by changing the ownership to nagios:nagios and restarting, I got to the point of seeing the following in the nagios.log

[1632363795] Nagios 4.4.6 starting... (PID=17301766)
[1632363795] Local time is Wed Sep 22 21:23:15 CDT 2021
[1632363795] LOG VERSION: 2.0
[1632363795] qh: Socket '/var/nagios/rw/nagios.qh' successfully initialized
[1632363795] qh: core query handler registered
[1632363795] qh: echo service query handler registered
[1632363795] qh: help for the query handler registered
[1632363795] wproc: Successfully registered manager as @wproc with query handler
[1632363797] Successfully launched command file worker with pid 25690398

However, the rest of the log file consists of "Unable to send check for host <hostname> to worker (ret=-2)" and "Unable to run check for service <servicename> on host <hostname>" messages.

I'm no longer seeing the "Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!" messages in the nagios.log like I did back in May... but unfortunately I'm still seeing them in the syslog. Ownership/permissions of this file are

$ ls -l /var/nagios/rw/nagios.qh
srwxrwxrwx 1 nagios nagiscmd 0 Sep 22 21:53 /var/nagios/rw/nagios.qh

A bit of googling turned up this URL (though the problem occurred on SELinux not AIX): https://stackoverflow.com/questions/42777659/nagios-unable-to-send-check-for-host-or-run-check-for-service?rq=1, specifically the following comment: "No, a bad policy is causing the issues, not selinux. The policy was apparently not updated in the EPEL package when updating to nagios-4.3.2"

Any chance something similar can be going on with AIX?

------------------------------
Erich Wolz

Original Message:
Sent: Wed May 12, 2021 12:08 PM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

re: "Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!" -- I saw that, too... but how can this be, if the Nagios config file was specified as follows:

/opt/freeware/bin/nagios -d /etc/nagios/nagios.cfg > /dev/console 2>&1

------------------------------
Erich Wolz

Original Message:
Sent: Tue May 11, 2021 06:47 AM
From: Zaki Jääskeläinen
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Hi,

Just an observation:

In /var/log/nagios/nagios.log it shows:
[1620308993] qh: Socket '/var/nagios/rw/nagios.qh' successfully initialized

But in /var/nagios/nagios.log it shows:
[1620308754] Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!

------------------------------
Zaki Jääskeläinen

Original Message:
Sent: Thu May 06, 2021 10:05 AM
From: Oliver Stadler
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

We have installed Nagios 4.4.6 using the following AIX Toolbox RPM

bash-5.0# rpm -qa | grep -i nagios
nagios-4.4.6-1.ppc
nagios-plugins-2.3.3-1.ppc

First question is, is there anybody on this forum who has managed to successfully start Nagios 4.4.6 on AIX? If yes it would be great if you could provide a quick feedback.

When we start the Nagios server we see the following errors.

bash-5.0# tail -f /var/log/nagios/nagios.log
[1620308968] Successfully shutdown... (PID=12059090)
[1620308993] Nagios 4.4.6 starting... (PID=13435208)
[1620308993] Local time is Thu May 06 15:49:53 MEsT 2021
[1620308993] LOG VERSION: 2.0
[1620308993] qh: Socket '/var/nagios/rw/nagios.qh' successfully initialized
[1620308993] qh: core query handler registered
[1620308993] qh: echo service query handler registered
[1620308993] qh: help for the query handler registered
[1620308993] wproc: Successfully registered manager as @wproc with query handler
[1620308996] Successfully launched command file worker with pid 14942638
[1620309018] Unable to run check for service 'Swap Usage' on host 'localhost'
[1620309033] Unable to run check for service 'SSH' on host 'localhost'

bash-5.0# tail -f /var/nagios/nagios.log
[1620308754] Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!
[1620308754] Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!
[1620308754] Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!

We can rule out a permission problem because we also tried to set directory as well as the file to global rwx. We also set all limits in /etc/security/limits to unlimited.

It looks like Nagios is not able to spawn any worker processes. That is something which was introduced in Nagios 4 (we tried Nagios 3 as well which worked fine).

We compared that to an x86 based installation and there we can see that Nagios is launching the worker processes using the follwing command
/opt/freeware/bin/nagios --worker /var/nagios/rw/nagios.qh

If we run the same command on AIX we get an:
/opt/freeware/bin/nagios: illegal option -- -
/opt/freeware/bin/nagios: illegal option -- w
/opt/freeware/bin/nagios: illegal option -- o
/opt/freeware/bin/nagios: illegal option -- r
/opt/freeware/bin/nagios: illegal option -- k
/opt/freeware/bin/nagios: illegal option -- e
/opt/freeware/bin/nagios: illegal option -- r

According the manpage the --woker option should be the same like -W option. But if we launch the command like this
/opt/freeware/bin/nagios -W /var/nagios/rw/nagios.qh

We see the error
Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading! that is also in the logfile

------------------------------
Oliver Stadler
------------------------------

31. RE: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Like

Erich Wolz

Posted Mon January 10, 2022 02:02 PM

# grep ^check_workers /etc/nagios/nagios.cfg
check_workers=2
# cat nagios.log
[1641841183] Nagios 4.4.6 starting... (PID=14746100)
[1641841183] Local time is Mon Jan 10 12:59:43 CST 2022
[1641841183] LOG VERSION: 2.0
[1641841183] qh: Socket '/var/nagios/rw/nagios.qh' successfully initialized
[1641841183] qh: core query handler registered
[1641841183] qh: echo service query handler registered
[1641841183] qh: help for the query handler registered
[1641841183] wproc: Successfully registered manager as @wproc with query handler
[1641841183] wproc: Registry request: name=Core Worker 20447494;pid=20447494
[1641841183] wproc: Registry request: name=Core Worker 12583238;pid=12583238
[1641841183] Successfully launched command file worker with pid 9765198
[1641841185] wproc: Socket to worker Core Worker 20447494 broken, removing
[1641841185] wproc: 'Core Worker 12583238' seems to be choked. ret = -1; bufsize = 132: written = 0; errno = 32 (Broken pipe)
[1641841185] wproc: Socket to worker Core Worker 12583238 broken, removing
[1641841185] wproc: Error: can't get_worker() in fo_reassign_wproc_job
[1641841185] Unable to send check for host ...

------------------------------
Erich Wolz
------------------------------

Original Message

Original Message:
Sent: Fri January 07, 2022 05:09 AM
From: Ayappan P
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Can you try with two workers by setting "check_workers=2" in the nagios.cfg file ?

------------------------------
Ayappan P

Original Message:
Sent: Tue January 04, 2022 06:16 PM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Sorry, was on vacation and only just saw this request.

[root@hl1axmon:/] >ulimit -a
time(seconds) unlimited
file(blocks) 2097151
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2097151
nofiles(descriptors) 65536
threads(per process) unlimited
processes(per user) unlimited
[root@hl1axmon:/] >prtconf 2>/dev/null | head -15
System Model: IBM,8286-42A
Machine Serial Number: 219BDDV
Processor Type: PowerPC_POWER8
Processor Implementation Mode: POWER 8
Processor Version: PV_8_Compat
Number Of Processors: 2
Processor Clock Speed: 3525 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
LPAR Info: 13 hl1axmon
Memory Size: 8192 MB
Good Memory Size: 8192 MB
Platform Firmware level: SV860_215
Firmware Version: IBM,FW860.81 (SV860_215)
Console Login: enable
[root@hl1axmon:/] >vmstat

System configuration: lcpu=8 mem=8192MB ent=0.20

kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
2 1 593600 624946 0 0 0 0 0 0 2 1301 231 0 0 99 0 0.00 1.2

------------------------------
Erich Wolz

Original Message:
Sent: Fri December 17, 2021 02:02 AM
From: Ayappan P
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

For some reason , the workers are getting choked and removed in your machine. Because of that the "unable to check/send" errors are shown in the log.
Can you share the output of "ulimit -a" and vmstat ?

------------------------------
Ayappan P

Original Message:
Sent: Thu December 16, 2021 12:06 PM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

bumping nofiles to 65536 didn't help any

------------------------------
Erich Wolz

Original Message:
Sent: Wed December 15, 2021 02:50 AM
From: Ayappan P
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Try "ulimit -n 20000"

------------------------------
Ayappan P

Original Message:
Sent: Tue December 14, 2021 11:09 AM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

That is indeed good news! What is the recommended value for nofiles?

------------------------------
Erich Wolz

Original Message:
Sent: Tue December 14, 2021 02:12 AM
From: RESHMA KUMAR
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

We have fixed the issue and uploaded nagios-4.4.6-2.
https://public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/RPMS/ppc/nagios/

Before starting nagios server, please increase the ulimit value of nofiles.

------------------------------
RESHMA KUMAR

Original Message:
Sent: Mon November 29, 2021 10:31 AM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

That is indeed welcome news! Looking forward to the availability of the patch.

------------------------------
Erich Wolz

Original Message:
Sent: Fri November 26, 2021 12:44 AM
From: RESHMA KUMAR
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

We were able to fix the following error with a patch.
Error: Cannot open main configuration file '/var/nagios/rw/nagios.qh' for reading!

We are checking "Unable to run check for service " errors now.

------------------------------
RESHMA KUMAR

Original Message:
Sent: Wed November 24, 2021 06:51 AM
From: RESHMA KUMAR
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

We are working on it

------------------------------
RESHMA KUMAR

Original Message:
Sent: Wed November 17, 2021 11:30 AM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Anything to report?

------------------------------
Erich Wolz

Original Message:
Sent: Fri October 29, 2021 01:30 AM
From: RESHMA KUMAR
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Thanks for sharing the details. We are looking into the issue.

------------------------------
RESHMA KUMAR

Original Message:
Sent: Wed October 27, 2021 09:54 AM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh

Ping

------------------------------
Erich Wolz

Original Message:
Sent: Mon October 18, 2021 07:00 PM
From: Erich Wolz
Subject: Nagios 4.6 server on AIX / Cannot open main configuration file '/var/nagios/rw/nagios.qh