Since a few months, and after, but not necessarily linked to the upgrade to oslevel 7200-05-03-2136, we've had multiple servers end up in a state where almost all commands, including basic stuff such as vi, sudo, dnf, lslpp, rpm and almost all others will fail to execute on all WPARs and sometimes, but not always the LPAR as well. The error returned is consistently in the following form (Here for python_32/DNF for example) :

Could not load program /opt/freeware/bin/python3_32:
Dependent module /opt/freeware/lib/libpython3.7m.so could not be loaded.
System error: Not enough space

After opening an official call @ our AIX supportcenter and some initial investigation, we have already received an intermediate reply that states the following :

-------------------
Indeed it shows that when the public library heap (Segment D) was full then you will need to run applications in a shared library as shown below.
# export LDR_CNTRL=NAMEDSHLIB=ABC
This is true for 32 bit applications. For 32 bit applications only Segment D is available for loading libraries. 
When it's full then you need to use a shared library to run more 32 bit applications. 
--------------------

This has now occurred on at least four different servers, impacting over 20 WPARs and a few LPARs and more will undoubtedly follow... Since the issue is related to the D segment, used by 32-bit applications, filling up, and several/many (?) of the open source-tools delivered without official support by IBM are in fact still delivered as 32-bit only, the question becomes, what can be done to avoid this problem in the future as well as correct it when in case it occurs again ?

AIX release 7.3 seems to have at least provided some improvement on this matter but we are a long way off before we can update our servers to the latest release.
This is having a serious impact on our environment since almost all the effect hereof is that almost the entire CLI becomes completely unusable.
I know that, for example, sudo can be switched to the 64_bit version delivered by the DNF toolbox, but what about DNF itself and what about all other software, delivered by the Toolbox in 32-bit only ?
Sorry to say so, but the workaround provided by IBM support to set the variable in an application's environment to define additional shared library segments to grow the available space is just incredibly shortsighted and not an acceptable solution for such an impactful issue.

Any sort of assistance or better insights on this matter would be greatly appreciated.

Davy Keppens
Senior AIX administrator @Infrabel Belgium


​​​

------------------------------
Davy Keppens
------------------------------

#AIXOpenSource

Hi Davy,

Thank you for reporting the issue.
This is first time I am hearing issue related to dnf/python because of shared library segment is full.
We have not encounter this issue ever. Is there any specific scenario when this occur or are there any environment setting ?
We would like to understand more why in your case this is happening ?

Also you have mentioned this is happening for lslpp command which is not an opensource command but base AIX. 

Anyway from AIX toolbox perspective we are moving towards 64bit and most of the packages delivers 64bit by default.
As rpm command is 32bit in AIX 7.2/7.1 we could not make change in dnf to be 64bit on those platform but with AIX 7.3 we made that change.
We pushed AIX to move to 64bit rpm command and we moved dnf/python also to 64bit on AIX 7.3
I am not sure how can we address this issue because AIX rpm is 32bit and we can not use 64bit dnf/python on AIX 7.2 because of this reason.

But we would really like to understand more why in your case you are seeing this issue so often where we (or I guess no one else as no one reported) never saw.

------------------------------
SANKET RATHI
------------------------------

Original Message:
Sent: Fri February 11, 2022 09:29 AM
From: Davy Keppens
Subject: 32-bit shared library segment consumption

Since a few months, and after, but not necessarily linked to the upgrade to oslevel 7200-05-03-2136, we've had multiple servers end up in a state where almost all commands, including basic stuff such as vi, sudo, dnf, lslpp, rpm and almost all others will fail to execute on all WPARs and sometimes, but not always the LPAR as well. The error returned is consistently in the following form (Here for python_32/DNF for example) :

Could not load program /opt/freeware/bin/python3_32:
Dependent module /opt/freeware/lib/libpython3.7m.so could not be loaded.
System error: Not enough space

After opening an official call @ our AIX supportcenter and some initial investigation, we have already received an intermediate reply that states the following :

-------------------
Indeed it shows that when the public library heap (Segment D) was full then you will need to run applications in a shared library as shown below.
# export LDR_CNTRL=NAMEDSHLIB=ABC
This is true for 32 bit applications. For 32 bit applications only Segment D is available for loading libraries. 
When it's full then you need to use a shared library to run more 32 bit applications. 
--------------------

This has now occurred on at least four different servers, impacting over 20 WPARs and a few LPARs and more will undoubtedly follow... Since the issue is related to the D segment, used by 32-bit applications, filling up, and several/many (?) of the open source-tools delivered without official support by IBM are in fact still delivered as 32-bit only, the question becomes, what can be done to avoid this problem in the future as well as correct it when in case it occurs again ?

AIX release 7.3 seems to have at least provided some improvement on this matter but we are a long way off before we can update our servers to the latest release.
This is having a serious impact on our environment since almost all the effect hereof is that almost the entire CLI becomes completely unusable.
I know that, for example, sudo can be switched to the 64_bit version delivered by the DNF toolbox, but what about DNF itself and what about all other software, delivered by the Toolbox in 32-bit only ?
Sorry to say so, but the workaround provided by IBM support to set the variable in an application's environment to define additional shared library segments to grow the available space is just incredibly shortsighted and not an acceptable solution for such an impactful issue.

Any sort of assistance or better insights on this matter would be greatly appreciated.

Davy Keppens
Senior AIX administrator @Infrabel Belgium


​​​

------------------------------
Davy Keppens
------------------------------

#AIXOpenSource

Original Message:
Sent: 2/15/2022 12:17:00 PM
From: SANKET RATHI
Subject: RE: 32-bit shared library segment consumption

Hi Davy,

Thank you for reporting the issue.
This is first time I am hearing issue related to dnf/python because of shared library segment is full.
We have not encounter this issue ever. Is there any specific scenario when this occur or are there any environment setting ?
We would like to understand more why in your case this is happening ?

Also you have mentioned this is happening for lslpp command which is not an opensource command but base AIX. 

Anyway from AIX toolbox perspective we are moving towards 64bit and most of the packages delivers 64bit by default.
As rpm command is 32bit in AIX 7.2/7.1 we could not make change in dnf to be 64bit on those platform but with AIX 7.3 we made that change.
We pushed AIX to move to 64bit rpm command and we moved dnf/python also to 64bit on AIX 7.3
I am not sure how can we address this issue because AIX rpm is 32bit and we can not use 64bit dnf/python on AIX 7.2 because of this reason.

But we would really like to understand more why in your case you are seeing this issue so often where we (or I guess no one else as no one reported) never saw.

------------------------------
SANKET RATHI
------------------------------

Original Message:
Sent: Fri February 11, 2022 09:29 AM
From: Davy Keppens
Subject: 32-bit shared library segment consumption

Since a few months, and after, but not necessarily linked to the upgrade to oslevel 7200-05-03-2136, we've had multiple servers end up in a state where almost all commands, including basic stuff such as vi, sudo, dnf, lslpp, rpm and almost all others will fail to execute on all WPARs and sometimes, but not always the LPAR as well. The error returned is consistently in the following form (Here for python_32/DNF for example) :

Could not load program /opt/freeware/bin/python3_32:
Dependent module /opt/freeware/lib/libpython3.7m.so could not be loaded.
System error: Not enough space

After opening an official call @ our AIX supportcenter and some initial investigation, we have already received an intermediate reply that states the following :

-------------------
Indeed it shows that when the public library heap (Segment D) was full then you will need to run applications in a shared library as shown below.
# export LDR_CNTRL=NAMEDSHLIB=ABC
This is true for 32 bit applications. For 32 bit applications only Segment D is available for loading libraries. 
When it's full then you need to use a shared library to run more 32 bit applications. 
--------------------

This has now occurred on at least four different servers, impacting over 20 WPARs and a few LPARs and more will undoubtedly follow... Since the issue is related to the D segment, used by 32-bit applications, filling up, and several/many (?) of the open source-tools delivered without official support by IBM are in fact still delivered as 32-bit only, the question becomes, what can be done to avoid this problem in the future as well as correct it when in case it occurs again ?

AIX release 7.3 seems to have at least provided some improvement on this matter but we are a long way off before we can update our servers to the latest release.
This is having a serious impact on our environment since almost all the effect hereof is that almost the entire CLI becomes completely unusable.
I know that, for example, sudo can be switched to the 64_bit version delivered by the DNF toolbox, but what about DNF itself and what about all other software, delivered by the Toolbox in 32-bit only ?
Sorry to say so, but the workaround provided by IBM support to set the variable in an application's environment to define additional shared library segments to grow the available space is just incredibly shortsighted and not an acceptable solution for such an impactful issue.

Any sort of assistance or better insights on this matter would be greatly appreciated.

Davy Keppens
Senior AIX administrator @Infrabel Belgium


​​​

------------------------------
Davy Keppens
------------------------------

#AIXOpenSource