Now that the main suspect seems to have been identified - you can check (lsof) if the library (libreplace-samba4.so) is opened by some process before running update, if not - you may not need to restart Informix, only ensure that connections using PAM/AD authentication are not happening during update.
Original Message:
Sent: Fri January 29, 2021 04:00 PM
From: TOM GIRSCH
Subject: Informix + PAM + AD + CentOS Samba Update Issue
I at least have a workaround. The main problem is that I have a couple of instances that I can't easily restart. So it looks like I'll have to schedule some maintenance windows. :(
------------------------------
TOM GIRSCH
Original Message:
Sent: Fri January 29, 2021 03:45 PM
From: Vladimir Kolobrodov
Subject: Informix + PAM + AD + CentOS Samba Update Issue
I guess the important thing is - PAM works for you now.
:-)
> whatever fixes went into [12.10.FC14] XO (which was built on top of XF) seem to have fixed this issue
Support probably knows what it was. I'll make a note to myself to follow up at some time.
Thank's for the tip!
------------------------------
Vladimir Kolobrodov
Original Message:
Sent: Fri January 29, 2021 03:37 PM
From: TOM GIRSCH
Subject: Informix + PAM + AD + CentOS Samba Update Issue
OK, I need to correct myself on some stuff that I wrote above. Apparently I had some bad ordering (or bad recollections) on my prior tests. I've just tested this in great detail.
First, perf_event_paranoid was a red herring. It has nothing to do with the issue.
What I've found is that after the winbind update from 4.10.4 to 4.10.16, PAM/winbind authentications will fail with the dlopen error until the engine is restarted. But once the engine is restarted, PAM/winbind authentications will work again. I've verified this behavior through multiple tests on 12.10.FC12, 12.10.FC14XF and 14.10.FC5.
But here's what's super interesting to me: We have a few instances that are running 12.10.FC14XO, and on that patch version, no engine restart is necessary. So whatever fixes went into XO (which was built on top of XF) seem to have fixed this issue. And they seem NOT to have made it into the 14.10 source tree.
------------------------------
TOM GIRSCH
Original Message:
Sent: Fri January 29, 2021 02:33 PM
From: Vladimir Kolobrodov
Subject: Informix + PAM + AD + CentOS Samba Update Issue
Yes, weird. Your experience seem to suggest process continuing to use library locked in memory rather than kernel paranoid settings, but I'll be very interested in your findings about the real cause of this issue.
------------------------------
Vladimir Kolobrodov
Original Message:
Sent: Fri January 29, 2021 01:47 PM
From: TOM GIRSCH
Subject: Informix + PAM + AD + CentOS Samba Update Issue
Vladimir:
OK, this is really weird. The value was indeed set to 2. I set it to 1 and bounced the engine and the PAM authentication started working as expected. Then I set it back to 2, bounced the engine again, and everything still works. I completely rebooted the system, verified that the value was still 2, and it still works. Is there perhaps some one-time operation it was choking on that, once it got one success, never choked on again?
This is fascinating and thoroughly confusing.
Now I have to try it all again on another host.
------------------------------
TOM GIRSCH
Original Message:
Sent: Fri January 29, 2021 01:02 PM
From: Vladimir Kolobrodov
Subject: Informix + PAM + AD + CentOS Samba Update Issue
> I would expect the engine to be looking for 4.10.4
that would depend which library was "locked" - if top library was not - it'll be loaded fresh and then could try to use dependency still in memory.
However that's clearly not the case, since after reboot you see same issue.
Another shot in the dark - can you check settings for "perf_event_paranoid":
# sysctl kernel.perf_event_paranoid
If it's "2" - try changing to "1"
# echo 1 > /proc/sys/kernel/perf_event_paranoid
or
# sysctl kernel.perf_event_paranoid=1
and see if it makes any difference.
(may affect behavior programs which start as root and then change effective user id to non privileged user)
------------------------------
Vladimir Kolobrodov
Original Message:
Sent: Fri January 29, 2021 12:50 PM
From: TOM GIRSCH
Subject: Informix + PAM + AD + CentOS Samba Update Issue
The problem persists even after an engine bounce; in fact, it persists even after a full reboot.
Also, while the cached library idea could have some merit, I would expect the engine to be looking for 4.10.4 (the version that was in place when the engine was started / the first time it made a PAM call) rather than 4.10.16 (the now-current version). That it's looking for 4.10.16 tells me it recognizes the new version.
------------------------------
TOM GIRSCH
Original Message:
Sent: Fri January 29, 2021 12:38 PM
From: Vladimir Kolobrodov
Subject: Informix + PAM + AD + CentOS Samba Update Issue
Yes, setuid could play a role, but then it will be the same with previous version of packages ....
Bizarre ...
Unfortunately I don't have environment with PAM to play with, not sure if you would get more info by looking at traces for original / updated package for the Informix VP which runs authentication.
strace -fp <pid-of-msc-vp>
And then try to connect. That would also show full path to objects.
Asking just in case - you did bounce Informix after updating samba packages ? (sometimes library in memory may be of previous version if it's not released)
The puzzling thing to me is
/usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found
when all indications are that library was updated.
UNIX process sometimes can use cached library image in memory even after it's been deleted / overwritten.
------------------------------
Vladimir Kolobrodov
Original Message:
Sent: Fri January 29, 2021 12:20 PM
From: TOM GIRSCH
Subject: Informix + PAM + AD + CentOS Samba Update Issue
Recall that oninit runs as setuid root. And there are no restricted permissions anywhere in the directory tree.
Note, too, that from /var/log/secure, the MariaDB authentication seems to be walking the exact same path, and is working. For what it's worth, the MariaDB process does not run setuid root.
[The CPU VPs and encrypt VPs -- the latter for at-rest disk encryption -- run as user informix; all other VPs run as root. I believe authentication like this is done by the msc VP, which runs as root.]
------------------------------
TOM GIRSCH
Original Message:
Sent: Fri January 29, 2021 11:37 AM
From: Andreas Legner
Subject: Informix + PAM + AD + CentOS Samba Update Issue
Maybe one difference is: shell authentication is performed by a privileged process (sshd?) while Informix PAM auth conducted by a non-privileged oninit process - had to check this?
Not sure about MariaDB...
------------------------------
Andreas Legner
Original Message:
Sent: Fri January 29, 2021 11:28 AM
From: TOM GIRSCH
Subject: Informix + PAM + AD + CentOS Samba Update Issue
I should note that your initial question, "How could the problem possibly be on the Informix end?" was also my initial question.
But given the results of my detailed testing below (and the outputs of the commands Vladimir suggested), my question shifted to "how could the problem possibly be anywhere other than Informix?" Shell authentications work fine. Other DBs using the same PAM stack work fine. Informix breaks.
------------------------------
TOM GIRSCH
Original Message:
Sent: Fri January 29, 2021 10:19 AM
From: Andreas Legner
Subject: Informix + PAM + AD + CentOS Samba Update Issue
Interesting.
Yet, in my understanding, Informix really only uses PAM, but wouldn't own or influence any of it, so having my doubts this anyhow is 'caused' by Informix.
What differences would exist in (again non-Informix) PAM configuration between shell login access, MariaDB access and Informix access?
And did you search for libreplace-samba4.so - on your system and elsewhere? This library missing looks to be the most immediate problem.
------------------------------
Andreas Legner
Original Message:
Sent: Fri December 18, 2020 11:20 AM
From: TOM GIRSCH
Subject: Informix + PAM + AD + CentOS Samba Update Issue
OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.
Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.
First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.
Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:
Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)
What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.
I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.
I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.
I'm inclined to open a ticket, but I'm not sure they'll have much to go on.
------------------------------
TOM GIRSCH
------------------------------
#Informix