Informix

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

```
Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)
```

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

```
Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)
```

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

```
Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)
```

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

[Technically, I don't think Red Hat *ever* supported CentOS. The whole point of CentOS is that it's the truly open-source equivalent of the decidedly non-open-source RedHat]

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

```
Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)
```

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

Yes, tough, could be many things that went wrong, agree with Andreas  - not likely Informix related  ...

I'd start by looking  at libreplace-samba:

/bin/ls -l /usr/lib64/samba/libreplace-samba4.so*

there maybe multiple copies of this library with few versions, and

readelf -V /usr/lib64/samba/libreplace-samba4.so

to confirm what is the actual version of the library that pam_winbind.so is trying to load

Considering that packages had been just updated to 4.10.16 it's not clear why libreplace-samba4.so is not the latest version, at least that's how I understand the error message.

you can also check with 

rpm -qf /usr/lib64/samba/libreplace-samba4.so

that library actually belongs to the package with new version.

ldd /usr/lib64/security/pam_winbind.so 

Is likely to give the same error? (version `SAMBA_4.10.16' not found)

Potential fix (if libreplace-samba4 library is indeed older) would be to make sure it's either updated (yum install  samba-client-libs) or linked to the right library if it's already there.

That's one direction, another would be to check what "ldconfig" is showing, could be messy too, since you may be picking libraries from unexpected places.

Keep the group posted on the findings!

------------------------------
Vladimir Kolobrodov

Original Message:
Sent: Fri January 29, 2021 10:19 AM
From: Andreas Legner
Subject: Informix + PAM + AD + CentOS Samba Update Issue

Interesting.

Yet, in my understanding, Informix really only uses PAM, but wouldn't own or influence any of it, so having my doubts this anyhow is 'caused' by Informix.

What differences would exist in (again non-Informix) PAM configuration between shell login access, MariaDB access and Informix access?

And did you search for libreplace-samba4.so - on your system and elsewhere? This library missing looks to be the most immediate problem.

------------------------------
Andreas Legner

Original Message:
Sent: Fri December 18, 2020 11:20 AM
From: TOM GIRSCH
Subject: Informix + PAM + AD + CentOS Samba Update Issue

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

I can think of no other explanation as to why shell access and MariaDB access would continue to work, but Informix would break. What seems to be happening here is that Informix is reporting the libarary as being missing even though it is not.

Here's the baseline system:

$ uname -aLinux [myhost].aemaine.com 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux$ sudo yum list installed | egrep "samba|winbind|libwb"libwbclient.x86_64              4.10.4-11.el7_8                @centos7-base    samba-client-libs.x86_64        4.10.4-11.el7_8                @centos7-base    samba-common.noarch             4.10.4-11.el7_8                @centos7-base    samba-common-libs.x86_64        4.10.4-11.el7_8                @centos7-base    samba-common-tools.x86_64       4.10.4-11.el7_8                @centos7-base    samba-libs.x86_64               4.10.4-11.el7_8                @centos7-base    samba-winbind.x86_64            4.10.4-11.el7_8                @centos7-base    samba-winbind-clients.x86_64    4.10.4-11.el7_8                @centos7-base    samba-winbind-modules.x86_64    4.10.4-11.el7_8                @centos7-base    $ ls -la /usr/lib64/samba/libreplace*                      -rwxr-xr-x. 1 root root 11128 May 12  2020 /usr/lib64/samba/libreplace-samba4.so$ strings /usr/lib64/samba/libreplace-samba4.so | grep 4.10SAMBA_4.10.4[attempt to connect as AD user 'tgirsch' succeeds.]$ sudo tail -9 /var/log/secureJan 29 15:48:22 [myhost] sudo: tgirsch : TTY=pts/1 ; PWD=/home/tgirsch ; USER=root ; COMMAND=/bin/yum list installedJan 29 15:48:22 [myhost] sudo: pam_unix(sudo:session): session opened for user root by (uid=0)Jan 29 15:48:22 [myhost] sudo: pam_unix(sudo:session): session closed for user rootJan 29 15:48:46 [myhost] oninit: pam_unix(ifmx-auth:auth): authentication failure; logname= uid=202 euid=0 tty= ruser=tgirsch rhost=[myhost].aemaine.com  user=tgirschJan 29 15:48:46 [myhost] oninit: pam_winbind(ifmx-auth:auth): getting password (0x00002050)Jan 29 15:48:46 [myhost] oninit: pam_winbind(ifmx-auth:auth): pam_get_item returned a passwordJan 29 15:48:46 [myhost] oninit: pam_winbind(ifmx-auth:auth): user 'tgirsch' granted accessJan 29 15:48:54 [myhost] sudo: tgirsch : TTY=pts/1 ; PWD=/home/tgirsch ; USER=root ; COMMAND=/bin/tail -9 /var/log/secureJan 29 15:48:54 [myhost] sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Everything works as expected.

Now I run sudo yum -y update samba-winbind, and it updates all the samba-related packages:

$ uname -aLinux [myhost].aemaine.com 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux$ sudo yum list installed | egrep "samba|winbind|libwb"libwbclient.x86_64              4.10.16-9.el7_9                @centos7-base    samba-client-libs.x86_64        4.10.16-9.el7_9                @centos7-base    samba-common.noarch             4.10.16-9.el7_9                @centos7-base    samba-common-libs.x86_64        4.10.16-9.el7_9                @centos7-base    samba-common-tools.x86_64       4.10.16-9.el7_9                @centos7-base    samba-libs.x86_64               4.10.16-9.el7_9                @centos7-base    samba-winbind.x86_64            4.10.16-9.el7_9                @centos7-base    samba-winbind-clients.x86_64    4.10.16-9.el7_9                @centos7-base    samba-winbind-modules.x86_64    4.10.16-9.el7_9                @centos7-base    $ ls -la /usr/lib64/samba/libreplace*-rwxr-xr-x. 1 root root 11136 Dec 15 16:43 /usr/lib64/samba/libreplace-samba4.so$ strings /usr/lib64/samba/libreplace-samba4.so | grep 4.10SAMBA_4.10.16[attempt to connect as AD user 'tgirsch' fails with error 1809]$ sudo tail -8 /var/log/secureJan 29 15:44:47 [myhost] sudo: tgirsch : TTY=pts/1 ; PWD=/home/tgirsch ; USER=root ; COMMAND=/bin/yum list installedJan 29 15:44:47 [myhost] sudo: pam_unix(sudo:session): session opened for user root by (uid=0)Jan 29 15:44:48 [myhost] sudo: pam_unix(sudo:session): session closed for user rootJan 29 15:45:23 [myhost] oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)Jan 29 15:45:23 [myhost] oninit: PAM adding faulty module: /usr/lib64/security/pam_winbind.soJan 29 15:45:23 [myhost] oninit: pam_unix(ifmx-auth:auth): authentication failure; logname= uid=202 euid=0 tty= ruser=tgirsch rhost=[myhost].aemaine.com  user=tgirschJan 29 15:45:34 [myhost] sudo: tgirsch : TTY=pts/1 ; PWD=/home/tgirsch ; USER=root ; COMMAND=/bin/tail -8 /var/log/secureJan 29 15:45:34 [myhost] sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

As you can see, the library file is there, the version is correct, yet oninit returns an "unable to dlopen" error.

If I run 'yum history undo' to revert back to 4.10.4, everything is fine again.

Here's the PAM stack I'm using for Informix:

$ cat /etc/pam.d/ifmx-authauth        required      pam_env.soauth        required      pam_faildelay.so delay=2000000auth        sufficient    pam_unix.so nullok try_first_passauth        requisite     pam_succeed_if.so uid >= 1000 quiet_successauth        sufficient    pam_winbind.so use_first_passauth        required      pam_deny.soaccount     required      pam_unix.so broken_shadowaccount     sufficient    pam_localuser.soaccount     sufficient    pam_succeed_if.so uid < 1000 quietaccount     required      pam_permit.so

As a sanity check, here's a MariaDB test to prove the point:

$ uname -aLinux [myhost].aemaine.com 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux$ sudo yum list installed | egrep "samba|winbind|libwb"libwbclient.x86_64              4.10.16-9.el7_9                @centos7-basesamba-client-libs.x86_64        4.10.16-9.el7_9                @centos7-basesamba-common.noarch             4.10.16-9.el7_9                @centos7-basesamba-common-libs.x86_64        4.10.16-9.el7_9                @centos7-basesamba-common-tools.x86_64       4.10.16-9.el7_9                @centos7-basesamba-libs.x86_64               4.10.16-9.el7_9                @centos7-basesamba-winbind.x86_64            4.10.16-9.el7_9                @centos7-basesamba-winbind-clients.x86_64    4.10.16-9.el7_9                @centos7-basesamba-winbind-modules.x86_64    4.10.16-9.el7_9                @centos7-base$ ls -la /usr/lib64/samba/libreplace*-rwxr-xr-x 1 root root 11136 Dec 15 16:43 /usr/lib64/samba/libreplace-samba4.so$ strings /usr/lib64/samba/libreplace* | grep "4.10"SAMBA_4.10.16$ mysql -h localhost -u tgirsch -pEnter password: Welcome to the MariaDB monitor.  Commands end with ; or \g.Your MariaDB connection id is 22111Server version: 10.5.8-MariaDB-log MariaDB ServerCopyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MariaDB [(none)]> SHOW GRANTS FOR 'tgirsch';+------------------------------------------------------------------------+| Grants for tgirsch@%                                                   |+------------------------------------------------------------------------+| GRANT USAGE ON *.* TO `tgirsch`@`%` IDENTIFIED VIA pam USING 'db-auth' || GRANT SELECT ON `supplier`.* TO `tgirsch`@`%`                          |+------------------------------------------------------------------------+2 rows in set (0.000 sec)MariaDB [(none)]> exitBye$ sudo tail -9 /var/log/secureJan 29 15:55:34 [myhost] sudo: mariadb : TTY=pts/1 ; PWD=/home/mariadb ; USER=root ; COMMAND=/bin/yum list installedJan 29 15:55:34 [myhost] sudo: pam_unix(sudo:session): session opened for user root by (uid=0)Jan 29 15:55:35 [myhost] sudo: pam_unix(sudo:session): session closed for user rootJan 29 15:55:55 [myhost] mariadbd: pam_unix(db-auth:auth): authentication failure; logname= uid=502 euid=502 tty= ruser= rhost=  user=tgirschJan 29 15:55:55 [myhost] mariadbd: pam_winbind(db-auth:auth): getting password (0x00000050)Jan 29 15:55:55 [myhost] mariadbd: pam_winbind(db-auth:auth): pam_get_item returned a passwordJan 29 15:55:55 [myhost] mariadbd: pam_winbind(db-auth:auth): user 'tgirsch' granted accessJan 29 15:56:17 [myhost] sudo: mariadb : TTY=pts/1 ; PWD=/home/mariadb ; USER=root ; COMMAND=/bin/tail -9 /var/log/secureJan 29 15:56:17 [myhost] sudo: pam_unix(sudo:session): session opened for user root by (uid=0)$ cat /etc/pam.d/db-authauth        required      pam_env.soauth        required      pam_faildelay.so delay=2000000auth        sufficient    pam_unix.so nullok try_first_passauth        requisite     pam_succeed_if.so uid >= 1000 quiet_successauth        sufficient    pam_winbind.so use_first_passauth        required      pam_deny.soaccount     required      pam_unix.so broken_shadowaccount     sufficient    pam_localuser.soaccount     sufficient    pam_succeed_if.so uid < 1000 quietaccount     required      pam_permit.so

Identical OS versions, identical Samba versions, identical PAM stacks. Shell login works. MariaDB works. Informix fails.

If the problem exists external to Informix, I fail to see where or how.

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

I should note that your initial question, "How could the problem possibly be on the Informix end?" was also my initial question.

But given the results of my detailed testing below (and the outputs of the commands Vladimir suggested), my question shifted to "how could the problem possibly be anywhere other than Informix?" Shell authentications work fine. Other DBs using the same PAM stack work fine. Informix breaks.

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

I should note that your initial question, "How could the problem possibly be on the Informix end?" was also my initial question.

But given the results of my detailed testing below (and the outputs of the commands Vladimir suggested), my question shifted to "how could the problem possibly be anywhere other than Informix?" Shell authentications work fine. Other DBs using the same PAM stack work fine. Informix breaks.

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

Recall that oninit runs as setuid root. And there are no restricted permissions anywhere in the directory tree.

Note, too, that from /var/log/secure, the MariaDB authentication seems to be walking the exact same path, and is working. For what it's worth, the MariaDB process does not run setuid root.

[The CPU VPs and encrypt VPs -- the latter for at-rest disk encryption -- run as user informix; all other VPs run as root. I believe authentication like this is done by the msc VP, which runs as root.]

I should note that your initial question, "How could the problem possibly be on the Informix end?" was also my initial question.

But given the results of my detailed testing below (and the outputs of the commands Vladimir suggested), my question shifted to "how could the problem possibly be anywhere other than Informix?" Shell authentications work fine. Other DBs using the same PAM stack work fine. Informix breaks.

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

Yes, setuid could play a role, but then it will be the same with previous version of packages ....

Bizarre ...

Unfortunately I don't have environment with PAM to play with, not sure if you would get more info by looking at traces for original / updated package for the Informix VP which runs authentication. 

strace -fp <pid-of-msc-vp>

And then try to connect. That would also show full path to objects.

Asking just in case - you did bounce Informix after updating samba packages ? (sometimes library in memory may be of previous version if it's not released)

The puzzling thing to me is

/usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found

when all indications are that library was updated.

UNIX process sometimes can use cached library image in memory even after it's been deleted / overwritten.

Recall that oninit runs as setuid root. And there are no restricted permissions anywhere in the directory tree.

Note, too, that from /var/log/secure, the MariaDB authentication seems to be walking the exact same path, and is working. For what it's worth, the MariaDB process does not run setuid root.

[The CPU VPs and encrypt VPs -- the latter for at-rest disk encryption -- run as user informix; all other VPs run as root. I believe authentication like this is done by the msc VP, which runs as root.]

I should note that your initial question, "How could the problem possibly be on the Informix end?" was also my initial question.

But given the results of my detailed testing below (and the outputs of the commands Vladimir suggested), my question shifted to "how could the problem possibly be anywhere other than Informix?" Shell authentications work fine. Other DBs using the same PAM stack work fine. Informix breaks.

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

Yes, setuid could play a role, but then it will be the same with previous version of packages ....

Bizarre ...

Unfortunately I don't have environment with PAM to play with, not sure if you would get more info by looking at traces for original / updated package for the Informix VP which runs authentication. 

strace -fp <pid-of-msc-vp>

And then try to connect. That would also show full path to objects.

Asking just in case - you did bounce Informix after updating samba packages ? (sometimes library in memory may be of previous version if it's not released)

The puzzling thing to me is

/usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found

when all indications are that library was updated.

UNIX process sometimes can use cached library image in memory even after it's been deleted / overwritten.

Recall that oninit runs as setuid root. And there are no restricted permissions anywhere in the directory tree.

Note, too, that from /var/log/secure, the MariaDB authentication seems to be walking the exact same path, and is working. For what it's worth, the MariaDB process does not run setuid root.

[The CPU VPs and encrypt VPs -- the latter for at-rest disk encryption -- run as user informix; all other VPs run as root. I believe authentication like this is done by the msc VP, which runs as root.]

I should note that your initial question, "How could the problem possibly be on the Informix end?" was also my initial question.

But given the results of my detailed testing below (and the outputs of the commands Vladimir suggested), my question shifted to "how could the problem possibly be anywhere other than Informix?" Shell authentications work fine. Other DBs using the same PAM stack work fine. Informix breaks.

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

> I would expect the engine to be looking for 4.10.4

that would depend which library was "locked" - if top library was not - it'll be loaded fresh and then could try to use dependency still in memory.

However that's clearly not the case, since after reboot you see same issue.

Another shot in the dark - can you check settings for "perf_event_paranoid":

# sysctl kernel.perf_event_paranoid

If it's "2" - try changing to "1"

# echo 1 > /proc/sys/kernel/perf_event_paranoid

or

# sysctl kernel.perf_event_paranoid=1

and see if it makes any difference.

(may affect behavior programs which start as root and then change effective user id to non privileged user)

Yes, setuid could play a role, but then it will be the same with previous version of packages ....

Bizarre ...

Unfortunately I don't have environment with PAM to play with, not sure if you would get more info by looking at traces for original / updated package for the Informix VP which runs authentication. 

strace -fp <pid-of-msc-vp>

And then try to connect. That would also show full path to objects.

Asking just in case - you did bounce Informix after updating samba packages ? (sometimes library in memory may be of previous version if it's not released)

The puzzling thing to me is

/usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found

when all indications are that library was updated.

UNIX process sometimes can use cached library image in memory even after it's been deleted / overwritten.

Recall that oninit runs as setuid root. And there are no restricted permissions anywhere in the directory tree.

Note, too, that from /var/log/secure, the MariaDB authentication seems to be walking the exact same path, and is working. For what it's worth, the MariaDB process does not run setuid root.

[The CPU VPs and encrypt VPs -- the latter for at-rest disk encryption -- run as user informix; all other VPs run as root. I believe authentication like this is done by the msc VP, which runs as root.]

I should note that your initial question, "How could the problem possibly be on the Informix end?" was also my initial question.

But given the results of my detailed testing below (and the outputs of the commands Vladimir suggested), my question shifted to "how could the problem possibly be anywhere other than Informix?" Shell authentications work fine. Other DBs using the same PAM stack work fine. Informix breaks.

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

Vladimir:

OK, this is really weird. The value was indeed set to 2. I set it to 1 and bounced the engine and the PAM authentication started working as expected. Then I set it back to 2, bounced the engine again, and everything still works. I completely rebooted the system, verified that the value was still 2, and it still  works. Is there perhaps some one-time operation it was choking on that, once it got one success, never choked on again?

This is fascinating and thoroughly confusing.

Now I have to try it all again on another host.

> I would expect the engine to be looking for 4.10.4

that would depend which library was "locked" - if top library was not - it'll be loaded fresh and then could try to use dependency still in memory.

However that's clearly not the case, since after reboot you see same issue.

Another shot in the dark - can you check settings for "perf_event_paranoid":

# sysctl kernel.perf_event_paranoid

If it's "2" - try changing to "1"

# echo 1 > /proc/sys/kernel/perf_event_paranoid

or

# sysctl kernel.perf_event_paranoid=1

and see if it makes any difference.

(may affect behavior programs which start as root and then change effective user id to non privileged user)

Yes, setuid could play a role, but then it will be the same with previous version of packages ....

Bizarre ...

Unfortunately I don't have environment with PAM to play with, not sure if you would get more info by looking at traces for original / updated package for the Informix VP which runs authentication. 

strace -fp <pid-of-msc-vp>

And then try to connect. That would also show full path to objects.

Asking just in case - you did bounce Informix after updating samba packages ? (sometimes library in memory may be of previous version if it's not released)

The puzzling thing to me is

/usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found

when all indications are that library was updated.

UNIX process sometimes can use cached library image in memory even after it's been deleted / overwritten.

Recall that oninit runs as setuid root. And there are no restricted permissions anywhere in the directory tree.

Note, too, that from /var/log/secure, the MariaDB authentication seems to be walking the exact same path, and is working. For what it's worth, the MariaDB process does not run setuid root.

[The CPU VPs and encrypt VPs -- the latter for at-rest disk encryption -- run as user informix; all other VPs run as root. I believe authentication like this is done by the msc VP, which runs as root.]

I should note that your initial question, "How could the problem possibly be on the Informix end?" was also my initial question.

But given the results of my detailed testing below (and the outputs of the commands Vladimir suggested), my question shifted to "how could the problem possibly be anywhere other than Informix?" Shell authentications work fine. Other DBs using the same PAM stack work fine. Informix breaks.

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

Vladimir:

OK, this is really weird. The value was indeed set to 2. I set it to 1 and bounced the engine and the PAM authentication started working as expected. Then I set it back to 2, bounced the engine again, and everything still works. I completely rebooted the system, verified that the value was still 2, and it still  works. Is there perhaps some one-time operation it was choking on that, once it got one success, never choked on again?

This is fascinating and thoroughly confusing.

Now I have to try it all again on another host.

> I would expect the engine to be looking for 4.10.4

that would depend which library was "locked" - if top library was not - it'll be loaded fresh and then could try to use dependency still in memory.

However that's clearly not the case, since after reboot you see same issue.

Another shot in the dark - can you check settings for "perf_event_paranoid":

# sysctl kernel.perf_event_paranoid

If it's "2" - try changing to "1"

# echo 1 > /proc/sys/kernel/perf_event_paranoid

or

# sysctl kernel.perf_event_paranoid=1

and see if it makes any difference.

(may affect behavior programs which start as root and then change effective user id to non privileged user)

Yes, setuid could play a role, but then it will be the same with previous version of packages ....

Bizarre ...

Unfortunately I don't have environment with PAM to play with, not sure if you would get more info by looking at traces for original / updated package for the Informix VP which runs authentication. 

strace -fp <pid-of-msc-vp>

And then try to connect. That would also show full path to objects.

Asking just in case - you did bounce Informix after updating samba packages ? (sometimes library in memory may be of previous version if it's not released)

The puzzling thing to me is

/usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found

when all indications are that library was updated.

UNIX process sometimes can use cached library image in memory even after it's been deleted / overwritten.

Recall that oninit runs as setuid root. And there are no restricted permissions anywhere in the directory tree.

Note, too, that from /var/log/secure, the MariaDB authentication seems to be walking the exact same path, and is working. For what it's worth, the MariaDB process does not run setuid root.

[The CPU VPs and encrypt VPs -- the latter for at-rest disk encryption -- run as user informix; all other VPs run as root. I believe authentication like this is done by the msc VP, which runs as root.]

I should note that your initial question, "How could the problem possibly be on the Informix end?" was also my initial question.

But given the results of my detailed testing below (and the outputs of the commands Vladimir suggested), my question shifted to "how could the problem possibly be anywhere other than Informix?" Shell authentications work fine. Other DBs using the same PAM stack work fine. Informix breaks.

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

Vladimir:

OK, this is really weird. The value was indeed set to 2. I set it to 1 and bounced the engine and the PAM authentication started working as expected. Then I set it back to 2, bounced the engine again, and everything still works. I completely rebooted the system, verified that the value was still 2, and it still  works. Is there perhaps some one-time operation it was choking on that, once it got one success, never choked on again?

This is fascinating and thoroughly confusing.

Now I have to try it all again on another host.

> I would expect the engine to be looking for 4.10.4

that would depend which library was "locked" - if top library was not - it'll be loaded fresh and then could try to use dependency still in memory.

However that's clearly not the case, since after reboot you see same issue.

Another shot in the dark - can you check settings for "perf_event_paranoid":

# sysctl kernel.perf_event_paranoid

If it's "2" - try changing to "1"

# echo 1 > /proc/sys/kernel/perf_event_paranoid

or

# sysctl kernel.perf_event_paranoid=1

and see if it makes any difference.

(may affect behavior programs which start as root and then change effective user id to non privileged user)

Yes, setuid could play a role, but then it will be the same with previous version of packages ....

Bizarre ...

Unfortunately I don't have environment with PAM to play with, not sure if you would get more info by looking at traces for original / updated package for the Informix VP which runs authentication. 

strace -fp <pid-of-msc-vp>

And then try to connect. That would also show full path to objects.

Asking just in case - you did bounce Informix after updating samba packages ? (sometimes library in memory may be of previous version if it's not released)

The puzzling thing to me is

/usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found

when all indications are that library was updated.

UNIX process sometimes can use cached library image in memory even after it's been deleted / overwritten.

Recall that oninit runs as setuid root. And there are no restricted permissions anywhere in the directory tree.

Note, too, that from /var/log/secure, the MariaDB authentication seems to be walking the exact same path, and is working. For what it's worth, the MariaDB process does not run setuid root.

[The CPU VPs and encrypt VPs -- the latter for at-rest disk encryption -- run as user informix; all other VPs run as root. I believe authentication like this is done by the msc VP, which runs as root.]

I should note that your initial question, "How could the problem possibly be on the Informix end?" was also my initial question.

But given the results of my detailed testing below (and the outputs of the commands Vladimir suggested), my question shifted to "how could the problem possibly be anywhere other than Informix?" Shell authentications work fine. Other DBs using the same PAM stack work fine. Informix breaks.

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

I guess the important thing is - PAM works for you now.

:-)

> whatever fixes went into [12.10.FC14] XO (which was built on top of XF) seem to have fixed this issue

Support probably knows what it was. I'll make a note to myself to follow up at some time.

Thank's for the tip!

Vladimir:

OK, this is really weird. The value was indeed set to 2. I set it to 1 and bounced the engine and the PAM authentication started working as expected. Then I set it back to 2, bounced the engine again, and everything still works. I completely rebooted the system, verified that the value was still 2, and it still  works. Is there perhaps some one-time operation it was choking on that, once it got one success, never choked on again?

This is fascinating and thoroughly confusing.

Now I have to try it all again on another host.

> I would expect the engine to be looking for 4.10.4

that would depend which library was "locked" - if top library was not - it'll be loaded fresh and then could try to use dependency still in memory.

However that's clearly not the case, since after reboot you see same issue.

Another shot in the dark - can you check settings for "perf_event_paranoid":

# sysctl kernel.perf_event_paranoid

If it's "2" - try changing to "1"

# echo 1 > /proc/sys/kernel/perf_event_paranoid

or

# sysctl kernel.perf_event_paranoid=1

and see if it makes any difference.

(may affect behavior programs which start as root and then change effective user id to non privileged user)

Yes, setuid could play a role, but then it will be the same with previous version of packages ....

Bizarre ...

Unfortunately I don't have environment with PAM to play with, not sure if you would get more info by looking at traces for original / updated package for the Informix VP which runs authentication. 

strace -fp <pid-of-msc-vp>

And then try to connect. That would also show full path to objects.

Asking just in case - you did bounce Informix after updating samba packages ? (sometimes library in memory may be of previous version if it's not released)

The puzzling thing to me is

/usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found

when all indications are that library was updated.

UNIX process sometimes can use cached library image in memory even after it's been deleted / overwritten.

Recall that oninit runs as setuid root. And there are no restricted permissions anywhere in the directory tree.

Note, too, that from /var/log/secure, the MariaDB authentication seems to be walking the exact same path, and is working. For what it's worth, the MariaDB process does not run setuid root.

[The CPU VPs and encrypt VPs -- the latter for at-rest disk encryption -- run as user informix; all other VPs run as root. I believe authentication like this is done by the msc VP, which runs as root.]

I should note that your initial question, "How could the problem possibly be on the Informix end?" was also my initial question.

But given the results of my detailed testing below (and the outputs of the commands Vladimir suggested), my question shifted to "how could the problem possibly be anywhere other than Informix?" Shell authentications work fine. Other DBs using the same PAM stack work fine. Informix breaks.

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------

I guess the important thing is - PAM works for you now.

:-)

> whatever fixes went into [12.10.FC14] XO (which was built on top of XF) seem to have fixed this issue

Support probably knows what it was. I'll make a note to myself to follow up at some time.

Thank's for the tip!

Vladimir:

OK, this is really weird. The value was indeed set to 2. I set it to 1 and bounced the engine and the PAM authentication started working as expected. Then I set it back to 2, bounced the engine again, and everything still works. I completely rebooted the system, verified that the value was still 2, and it still  works. Is there perhaps some one-time operation it was choking on that, once it got one success, never choked on again?

This is fascinating and thoroughly confusing.

Now I have to try it all again on another host.

> I would expect the engine to be looking for 4.10.4

that would depend which library was "locked" - if top library was not - it'll be loaded fresh and then could try to use dependency still in memory.

However that's clearly not the case, since after reboot you see same issue.

Another shot in the dark - can you check settings for "perf_event_paranoid":

# sysctl kernel.perf_event_paranoid

If it's "2" - try changing to "1"

# echo 1 > /proc/sys/kernel/perf_event_paranoid

or

# sysctl kernel.perf_event_paranoid=1

and see if it makes any difference.

(may affect behavior programs which start as root and then change effective user id to non privileged user)

Yes, setuid could play a role, but then it will be the same with previous version of packages ....

Bizarre ...

Unfortunately I don't have environment with PAM to play with, not sure if you would get more info by looking at traces for original / updated package for the Informix VP which runs authentication. 

strace -fp <pid-of-msc-vp>

And then try to connect. That would also show full path to objects.

Asking just in case - you did bounce Informix after updating samba packages ? (sometimes library in memory may be of previous version if it's not released)

The puzzling thing to me is

/usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found

when all indications are that library was updated.

UNIX process sometimes can use cached library image in memory even after it's been deleted / overwritten.

Recall that oninit runs as setuid root. And there are no restricted permissions anywhere in the directory tree.

Note, too, that from /var/log/secure, the MariaDB authentication seems to be walking the exact same path, and is working. For what it's worth, the MariaDB process does not run setuid root.

[The CPU VPs and encrypt VPs -- the latter for at-rest disk encryption -- run as user informix; all other VPs run as root. I believe authentication like this is done by the msc VP, which runs as root.]

I should note that your initial question, "How could the problem possibly be on the Informix end?" was also my initial question.

But given the results of my detailed testing below (and the outputs of the commands Vladimir suggested), my question shifted to "how could the problem possibly be anywhere other than Informix?" Shell authentications work fine. Other DBs using the same PAM stack work fine. Informix breaks.

OK, kids, here's an esoteric one for you, which I've recently run across. For years now, we've been using PAM authentication to allow our domain users (in our case, MS Active Directory) to access certain databases with their domain credentials, saving us the trouble of separate account and password management. This has generally worked quite well.

Two somewhat recent updates to the Samba packages caused problems that you might wish to be aware of, one easily fixed, and one for which I do not yet have a resolution.

First, the easy one: In /etc/security/pam_winbind.conf, there's a parameter called require_membership_of, which allows us to specify a group or groups that are allowed to access. If this is set, anyone not a member of the listed groups is rejected even if they provide proper credentials. We've used this as a filter. With a recent update (sorry, I forget which), this stopped working. After some internet sleuthing, I figured out that they changed the parameter to require a domain prefix, which was previously not required. I also learned that they didn't update the documentation or release notes accordingly although, to my discredit, I hadn't read them anyway. (The update came in as part of a simple "yum update".) So instead of listing "mygroup" we needed to list "mydomain\mygroup"; implementing that simple fix worked.

Now, the tough one, which is quite recent: Another "yum update" command updated the Samba family of packages from 4.10.4 to 4.10.16. Following this update, Informix PAM/AD authentication stopped working. Authentication to the database would fail, and I'd see this error in /var/log/secure:

Dec 18 15:40:30 myhost oninit: PAM unable to dlopen(/usr/lib64/security/pam_winbind.so): /usr/lib64/samba/libreplace-samba4.so: version `SAMBA_4.10.16' not found (required by /usr/lib64/security/pam_winbind.so)

What's interesting about this is that the behavior seems to be exclusively limited to Informix. PAM-based shell access is granted for AD users so authorized. And PAM-based access to other databases like MariaDB continues to operate as expected, too. Whatever is going on here seems to be unique to Informix.

I've replicated this behavior in 12.10.FC14 and 14.10.FC4W1.

I'm trying to get my hands on the intermediate versions of Samba to see exactly where the problem was introduced. But curious if anyone else is running the CentOS + PAM + Informix + AD combination, and could verify the same problem.

I'm inclined to open a ticket, but I'm not sure they'll have much to go on.

------------------------------
TOM GIRSCH
------------------------------