AWX fails on AIX 7200-05-05-2246 after yum update

6. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Steve Munday

Posted Mon January 30, 2023 11:25 AM

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM
------------------------------

Original Message

7. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Andrey Klyachkin

Posted Mon January 30, 2023 11:39 AM

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com
------------------------------

Original Message

8. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Steve Munday

Posted Mon January 30, 2023 11:53 AM
Edited by Steve Munday Mon January 30, 2023 12:36 PM

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openl
openldap-2.4.58-4.ppc
# ls -la /opt/freeware/lib/libldap_r.a
lrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a
# ls -la /opt/freeware/lib | grep -i libldap
-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a
-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2
lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a
-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a
-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2
lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a
#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.log
ImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:
        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from
                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].
        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from
                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].
        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from
                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].
        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from
                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].
        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from
                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].
        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from
                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].
# dnf downgrade libxml2-2.9.14-1
Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.
Dependencies resolved.
=============================================================================================================================================================================================
 Package                                            Architecture                           Version                                      Repository                                      Size
=============================================================================================================================================================================================
Downgrading:
 libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M
 libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M
 libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 M

Transaction Summary
=============================================================================================================================================================================================
Downgrade  3 Packages

. . . .

# rpm -qa | grep -i libxml2
libxml2-2.9.14-1.ppc
libxml2-python3-2.9.14-1.ppc
libxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM
------------------------------

Original Message

9. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

Stanley Speegle

Posted Wed February 01, 2023 01:25 PM

Hi Team

I rolled back the OpenLDAP and the libxml2. I can log in but no jobs run with errors that the hosts are unreachable.

Here is the debug output from the AWX AD-HOC job.

ansible 2.9.14
config file = /etc/ansible/ansible.cfg
configured module search path = ['/var/lib/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /opt/freeware/lib/python3.7/site-packages/ansible
executable location = /opt/freeware/bin/ansible
python version = 3.7.10 (default, Jun 1 2021, 05:23:20) [GCC 8.3.0]
Using /etc/ansible/ansible.cfg as config file
SSH password:
host_list declined parsing /tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/tmpqaunbs57 as it did not pass its verify_file() method
Parsed /tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/tmpqaunbs57 inventory source with script plugin
META: ran handlers
<nimon> ESTABLISH SSH CONNECTION FOR USER: root
<nimon> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp nimon '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
<nimon> (0, b'/\\n', b"Control socket connect(/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp): Socket operation on non-socket\\r\\nWarning: the ECDSA host key for 'nimon' differs from the key for the IP address '10.220.3.59'\\nOffending key for IP in /var/lib/awx/.ssh/known_hosts:1\\nMatching host key in /var/lib/awx/.ssh/known_hosts:4\\r\\nControlSocket /tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp already exists, disabling multiplexing\\r\\n")
<nimon> ESTABLISH SSH CONNECTION FOR USER: root
<nimon> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp nimon '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /.ansible/tmp `"&& mkdir "` echo /.ansible/tmp/ansible-tmp-1675275150.058933-12845340-211798537736734 `" && echo ansible-tmp-1675275150.058933-12845340-211798537736734="` echo /.ansible/tmp/ansible-tmp-1675275150.058933-12845340-211798537736734 `" ) && sleep 0'"'"''
<nimon> (0, b'ansible-tmp-1675275150.058933-12845340-211798537736734=/.ansible/tmp/ansible-tmp-1675275150.058933-12845340-211798537736734\\n', b"Control socket connect(/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp): Socket operation on non-socket\\r\\nWarning: the ECDSA host key for 'nimon' differs from the key for the IP address '10.220.3.59'\\nOffending key for IP in /var/lib/awx/.ssh/known_hosts:1\\nMatching host key in /var/lib/awx/.ssh/known_hosts:4\\r\\nControlSocket /tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp already exists, disabling multiplexing\\r\\n")
<nimon> Attempting python interpreter discovery
<nimon> ESTABLISH SSH CONNECTION FOR USER: root
<nimon> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp nimon '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.5'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"''
<nimon> (255, b'', b"Control socket connect(/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp): Socket operation on non-socket\\r\\nWarning: the ECDSA host key for 'nimon' differs from the key for the IP address '10.220.3.59'\\nOffending key for IP in /var/lib/awx/.ssh/known_hosts:1\\nMatching host key in /var/lib/awx/.ssh/known_hosts:4\\r\\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\\r\\nPermission denied, please try again.\\r\\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\\r\\nPermission denied, please try again.\\r\\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\\r\\nroot@nimon: Permission denied (publickey,password,keyboard-interactive).\\r\\n")
[WARNING]: Unhandled error in Python interpreter discovery for host nimon:
Failed to connect to the host via ssh: Control socket
connect(/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp): Socket operation on non-
socket Warning: the ECDSA host key for 'nimon' differs from the key for the IP
address '10.220.3.59' Offending key for IP in /var/lib/awx/.ssh/known_hosts:1
Matching host key in /var/lib/awx/.ssh/known_hosts:4 ssh_askpass:
exec(/usr/sbin/ssh-askpass): No such file or directory Permission denied,
please try again. ssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or
directory Permission denied, please try again. ssh_askpass:
exec(/usr/sbin/ssh-askpass): No such file or directory root@nimon: Permission
denied (publickey,password,keyboard-interactive).
Using module file /opt/freeware/lib/python3.7/site-packages/ansible/modules/commands/command.py
<nimon> PUT /var/lib/awx/.ansible/tmp/ansible-local-13828606j2e4lsca/tmpz4mjws8a TO /.ansible/tmp/ansible-tmp-1675275150.058933-12845340-211798537736734/AnsiballZ_command.py
<nimon> SSH: EXEC sshpass -d12 sftp -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp '[nimon]'
<nimon> (0, b'sftp> put /var/lib/awx/.ansible/tmp/ansible-local-13828606j2e4lsca/tmpz4mjws8a /.ansible/tmp/ansible-tmp-1675275150.058933-12845340-211798537736734/AnsiballZ_command.py\\nUploading /var/lib/awx/.ansible/tmp/ansible-local-13828606j2e4lsca/tmpz4mjws8a to /.ansible/tmp/ansible-tmp-1675275150.058933-12845340-211798537736734/AnsiballZ_command.py\\n', b"Control socket connect(/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp): Socket operation on non-socket\\r\\nWarning: the ECDSA host key for 'nimon' differs from the key for the IP address '10.220.3.59'\\nOffending key for IP in /var/lib/awx/.ssh/known_hosts:1\\nMatching host key in /var/lib/awx/.ssh/known_hosts:4\\r\\nControlSocket /tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp already exists, disabling multiplexing\\r\\nConnected to nimon.\\n")
<nimon> ESTABLISH SSH CONNECTION FOR USER: root
<nimon> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp nimon '/bin/sh -c '"'"'chmod u+x /.ansible/tmp/ansible-tmp-1675275150.058933-12845340-211798537736734/ /.ansible/tmp/ansible-tmp-1675275150.058933-12845340-211798537736734/AnsiballZ_command.py && sleep 0'"'"''
<nimon> (0, b'', b"Control socket connect(/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp): Socket operation on non-socket\\r\\nWarning: the ECDSA host key for 'nimon' differs from the key for the IP address '10.220.3.59'\\nOffending key for IP in /var/lib/awx/.ssh/known_hosts:1\\nMatching host key in /var/lib/awx/.ssh/known_hosts:4\\r\\nControlSocket /tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp already exists, disabling multiplexing\\r\\n")
<nimon> ESTABLISH SSH CONNECTION FOR USER: root
<nimon> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp -tt nimon '/bin/sh -c '"'"'/usr/bin/python /.ansible/tmp/ansible-tmp-1675275150.058933-12845340-211798537736734/AnsiballZ_command.py && sleep 0'"'"''
<nimon> (255, b'', b"Control socket connect(/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp): Socket operation on non-socket\\r\\nWarning: the ECDSA host key for 'nimon' differs from the key for the IP address '10.220.3.59'\\nOffending key for IP in /var/lib/awx/.ssh/known_hosts:1\\nMatching host key in /var/lib/awx/.ssh/known_hosts:4\\r\\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\\r\\nPermission denied, please try again.\\r\\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\\r\\nPermission denied, please try again.\\r\\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\\r\\nroot@nimon: Permission denied (publickey,password,keyboard-interactive).\\r\\n")
nimon | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: Control socket connect(/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp): Socket operation on non-socket\\r\\nWarning: the ECDSA host key for 'nimon' differs from the key for the IP address '10.220.3.59'\\nOffending key for IP in /var/lib/awx/.ssh/known_hosts:1\\nMatching host key in /var/lib/awx/.ssh/known_hosts:4\\r\\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\\r\\nPermission denied, please try again.\\r\\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\\r\\nPermission denied, please try again.\\r\\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\\r\\nroot@nimon: Permission denied (publickey,password,keyboard-interactive).",
"unreachable": true
}

aix-dispatcher.log has this error.
2023-02-01 18:05:02,070 WARNING awx.main.dispatch scaling up worker pid:14745880
2023-02-01 18:05:02,109 WARNING awx.main.dispatch scaling up worker pid:13107710
2023-02-01 18:05:02,133 WARNING awx.main.dispatch scaling up worker pid:13369794
2023-02-01 18:05:02,145 WARNING awx.main.dispatch Running worker dispatcher listening to queues ['tower_broadcast_all', 'powervc-test2']
2023-02-01 18:05:02,154 DEBUG awx.main.tasks Syncing Schedules
2023-02-01 18:05:02,186 DEBUG awx.main.tasks Waited 0.0008640289306640625 seconds to obtain lock name: cluster_policy_lock
2023-02-01 18:05:02,200 DEBUG awx.main.tasks Policy List, adding Instances [1] to Group tower
2023-02-01 18:05:02,201 DEBUG awx.main.tasks Total non-isolated instances:1 available for policy: 1
2023-02-01 18:05:02,201 DEBUG awx.main.tasks Cluster policy no-op finished in 0.014945745468139648 seconds
2023-02-01 18:05:02,202 DEBUG awx.main.tasks Cluster node heartbeat task.
2023-02-01 18:05:02,258 DEBUG awx.main.utils.reload Issuing command to restart services, args=['supervisorctl', 'restart', 'tower-processes:awx-rsyslogd']
2023-02-01 18:05:02,693 ERROR awx.main.utils.reload supervisorctl restart awx-rsyslogd errored with exit code `1`, stdout:
b"error: <class 'PermissionError'>, [Errno 13] Permission denied: file: /opt/freeware/lib/python3.7/site-packages/supervisor/xmlrpc.py line: 560"stderr:
b''
2023-02-01 18:05:21,976 DEBUG awx.main.dispatch task c5a30495-8c77-4bf4-8512-15356f81fc72 starting awx.main.scheduler.tasks.run_task_manager(*[])
2023-02-01 18:05:21,999 DEBUG awx.main.scheduler Running Tower task manager.
2023-02-01 18:05:22,019 DEBUG awx.main.scheduler Starting Scheduler
2023-02-01 18:05:22,113 DEBUG awx.main.scheduler Finishing Scheduler
2023-02-01 18:05:31,992 DEBUG awx.main.dispatch task 439b0610-fb9b-4545-bbca-f39be8dde62f starting awx.main.tasks.awx_periodic_scheduler(*[])
2023-02-01 18:05:32,020 DEBUG awx.main.tasks Starting periodic scheduler
2023-02-01 18:05:32,023 DEBUG awx.main.tasks Last scheduler run was: 2023-02-01 18:03:48.684884+00:00
2023-02-01 18:05:42,013 DEBUG awx.main.dispatch task 0aa2ebbf-10a4-47f6-9d76-1333f9d6ae41 starting awx.main.scheduler.tasks.run_task_manager(*[])
2023-02-01 18:05:42,018 DEBUG awx.main.scheduler Running Tower task manager.
2023-02-01 18:05:42,035 DEBUG awx.main.scheduler Starting Scheduler

Another error is in the awx-daphne.log
awx:/var/log/supervisor #head -n 100 awx-daphne.log
2023-02-01 18:05:00,113 DEBUG awx.main.routing cleaning up Redis key b'asgi:group:jobs-summary'
2023-02-01 18:05:00,113 DEBUG cleaning up Redis key b'asgi:group:jobs-summary'
2023-02-01 18:05:00,118 DEBUG awx.main.routing cleaning up Redis key b'asgi:group:ad_hoc_command_events-231'
2023-02-01 18:05:00,118 DEBUG cleaning up Redis key b'asgi:group:ad_hoc_command_events-231'
2023-02-01 18:05:00,119 DEBUG awx.main.routing cleaning up Redis key b'asgi:group:control-limit_reached_1'
2023-02-01 18:05:00,119 DEBUG cleaning up Redis key b'asgi:group:control-limit_reached_1'
2023-02-01 18:05:00,120 DEBUG awx.main.routing cleaning up Redis key b'asgi:group:inventories-status_changed'
2023-02-01 18:05:00,120 DEBUG cleaning up Redis key b'asgi:group:inventories-status_changed'
2023-02-01 18:05:00,121 DEBUG awx.main.routing cleaning up Redis key b'asgi:group:jobs-status_changed'
2023-02-01 18:05:00,121 DEBUG cleaning up Redis key b'asgi:group:jobs-status_changed'
2023-02-01 18:05:00,122 DEBUG awx.main.routing cleaning up Redis key b'asgi:group:ad_hoc_command_events-230'
2023-02-01 18:05:00,122 DEBUG cleaning up Redis key b'asgi:group:ad_hoc_command_events-230'
2023-02-01 18:05:00,122 DEBUG awx.main.routing cleaning up Redis key b'asgi:group:ad_hoc_command_events-233'
2023-02-01 18:05:00,122 DEBUG cleaning up Redis key b'asgi:group:ad_hoc_command_events-233'
2023-02-01 18:05:00,123 DEBUG awx.main.routing cleaning up Redis key b'asgi:group:ad_hoc_command_events-205'
2023-02-01 18:05:00,123 DEBUG cleaning up Redis key b'asgi:group:ad_hoc_command_events-205'
2023-02-01 18:05:00,124 DEBUG awx.main.routing cleaning up Redis key b'asgi:group:ad_hoc_command_events-206'
2023-02-01 18:05:00,124 DEBUG cleaning up Redis key b'asgi:group:ad_hoc_command_events-206'
2023-02-01 18:05:00,125 INFO daphne.cli Starting server at unix:/var/run/tower/daphne.sock
2023-02-01 18:05:00,125 INFO Starting server at unix:/var/run/tower/daphne.sock
2023-02-01 18:05:00,126 INFO daphne.server HTTP/2 support not enabled (install the http2 and tls Twisted extras)
2023-02-01 18:05:00,126 INFO HTTP/2 support not enabled (install the http2 and tls Twisted extras)
2023-02-01 18:05:00,126 INFO daphne.server Configuring endpoint unix:/var/run/tower/daphne.sock
2023-02-01 18:05:00,126 INFO Configuring endpoint unix:/var/run/tower/daphne.sock
2023-02-01 18:05:00,286 ERROR awx.main.consumers Request user is not authenticated to use websocket.
2023-02-01 18:05:00,286 ERROR awx.main.consumers Request user is not authenticated to use websocket.
2023-02-01 18:05:00,286 ERROR Request user is not authenticated to use websocket.
2023-02-01 18:05:00,326 ERROR awx.main.consumers Request user is not authenticated to use websocket.
2023-02-01 18:05:00,326 ERROR awx.main.consumers Request user is not authenticated to use websocket.
2023-02-01 18:05:00,326 ERROR Request user is not authenticated to use websocket.

Here is the list of installed packages.
dnf list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.14-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.2-1 @AIX_Toolbox_noarch
bash.ppc 5.1.4-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2020.06.01-2 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.4 @System
curl.ppc 7.76.1-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.7-1 @AIX_Toolbox
dnf.noarch 4.2.17-1 @System
dnf-automatic.noarch 4.2.17-1 @System
dnf-data.noarch 4.2.17-1 @System
expat.ppc 2.2.9-2 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.8-1 @AIX_Toolbox
gcc.ppc 1:8-1 @AIX_Toolbox_72
gcc-c++.ppc 1:8-1 @AIX_Toolbox_72
gcc-cpp.ppc 1:8-1 @AIX_Toolbox_72
gcc8.ppc 8.3.0-5 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-5 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-5 @AIX_Toolbox_72
gdbm.ppc 1.18.1-1 @AIX_Toolbox
gettext.ppc 0.20.2-1 @AIX_Toolbox
git.ppc 2.20.4-1 @AIX_Toolbox
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.1.2-1 @AIX_Toolbox
golang.ppc 1.18.5-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.5-1 @AIX_Toolbox_72
golang-src.noarch 1.18.5-1 @AIX_Toolbox_noarch
grep.ppc 3.4-1 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.3-1 @AIX_Toolbox
less.ppc 557-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libcomps.ppc 0.1.11-1 @System
libdnf.ppc 0.39.1-1 @System
libffi.ppc 3.2.1-3 @AIX_Toolbox
libffi-devel.ppc 3.2.1-3 @AIX_Toolbox
libgcc.ppc 1:8-1 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-5 @AIX_Toolbox_72
libgcrypt.ppc 1.8.5-1 @AIX_Toolbox
libgd.ppc 2.2.5-3 @AIX_Toolbox
libgomp.ppc 1:8-1 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-5 @AIX_Toolbox_72
libgpg-error.ppc 1.36-1 @AIX_Toolbox
libiconv.ppc 1.16-1 @AIX_Toolbox
libjpeg.ppc 9d-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-1 @System
libmpc.ppc 1.1.0-1 @AIX_Toolbox
libnghttp2.ppc 1.41.0-1 @AIX_Toolbox
libpng.ppc 1.6.27-3 @AIX_Toolbox
librepo.ppc 1.11.0-1 @System
libsmartcols.ppc 2.34-1 @System
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-1 @System
libssh2.ppc 1.9.0-1 @AIX_Toolbox
libstdc++.ppc 1:8-1 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-5 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-5 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.20.2-1 @AIX_Toolbox
libtiff.ppc 4.2.0-1 @AIX_Toolbox
libtool.ppc 2.4.6-2 @AIX_Toolbox
libtool-ltdl.ppc 2.4.6-2 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.9.14-1 @AIX_Toolbox
libxslt.ppc 1.1.32-1 @AIX_Toolbox
libyaml.ppc 0.2.2-1 @AIX_Toolbox
libzstd.ppc 1.4.4-1 @System
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.2-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.0.2-2 @AIX_Toolbox
ncurses.ppc 6.2-2 @AIX_Toolbox
nginx.ppc 1.17.8-1 @AIX_Toolbox
openldap.ppc 2.4.58-4 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.16-2 @AIX_Toolbox
p11-kit-tools.ppc 0.23.16-2 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.34-1 @AIX_Toolbox
perl.ppc 1:5.30.3-2 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.11-1 @AIX_Toolbox
postgresql-libs.ppc 11.11-1 @AIX_Toolbox
postgresql-server.ppc 11.11-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-3 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-3 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-3 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.10-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.10-2 @AIX_Toolbox
python3-dnf.noarch 4.2.17-1 @System
python3-dnf-plugin-migrate.noarch 4.0.16-1 @System
python3-dnf-plugins-core.noarch 4.0.16-1 @System
python3-gpgme.ppc 1.13.1-1 @System
python3-hawkey.ppc 0.39.1-1 @System
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-1 @System
python3-libdnf.ppc 0.39.1-1 @System
python3-librepo.ppc 1.11.0-1 @System
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.3-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.35.5-1 @AIX_Toolbox
sshpass.ppc 1.06-2 @AIX_Toolbox
subversion.ppc 1.10.7-2 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum.noarch 4.2.17-1 @System
yum-metadata-parser.ppc 1.1.4-2 @System
yum-utils.noarch 1.1.31-2 @AIX_Toolbox_noarch
zchunk-libs.ppc 1.1.4-1 @System
zeromq.ppc 4.3.3-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.3-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.11-2 @AIX_Toolbox

Any ideas what could be causing these errors or how to get AWX to connect.

An ssh connection to these same systems from the command line works with no issues.

Thank you for your assistance.

Stan

------------------------------
Stanley
------------------------------

Original Message

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

10. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Andrey Klyachkin

Posted Fri February 03, 2023 08:52 AM

Hi Stanley,

it looks like something went wrong during the installation or with the permissions. You have too many error messages "permissions denied". Can you check that the user awx has access to the files?

------------------------------
Andrey Klyachkin

https://www.power-devops.com
------------------------------

Original Message

Original Message:
Sent: Wed February 01, 2023 01:25 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team

I rolled back the OpenLDAP and the libxml2. I can log in but no jobs run with errors that the hosts are unreachable.

Here is the debug output from the AWX AD-HOC job.

ansible 2.9.14
config file = /etc/ansible/ansible.cfg
configured module search path = ['/var/lib/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /opt/freeware/lib/python3.7/site-packages/ansible
executable location = /opt/freeware/bin/ansible
python version = 3.7.10 (default, Jun 1 2021, 05:23:20) [GCC 8.3.0]
Using /etc/ansible/ansible.cfg as config file
SSH password:
host_list declined parsing /tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/tmpqaunbs57 as it did not pass its verify_file() method
Parsed /tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/tmpqaunbs57 inventory source with script plugin
META: ran handlers
<nimon> ESTABLISH SSH CONNECTION FOR USER: root
<nimon> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp nimon '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
<nimon> (0, b'/\\n', b"Control socket connect(/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp): Socket operation on non-socket\\r\\nWarning: the ECDSA host key for 'nimon' differs from the key for the IP address '10.220.3.59'\\nOffending key for IP in /var/lib/awx/.ssh/known_hosts:1\\nMatching host key in /var/lib/awx/.ssh/known_hosts:4\\r\\nControlSocket /tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp already exists, disabling multiplexing\\r\\n")
<nimon> ESTABLISH SSH CONNECTION FOR USER: root
<nimon> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp nimon '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /.ansible/tmp `"&& mkdir "` echo /.ansible/tmp/ansible-tmp-1675275150.058933-12845340-211798537736734 `" && echo ansible-tmp-1675275150.058933-12845340-211798537736734="` echo /.ansible/tmp/ansible-tmp-1675275150.058933-12845340-211798537736734 `" ) && sleep 0'"'"''
<nimon> (0, b'ansible-tmp-1675275150.058933-12845340-211798537736734=/.ansible/tmp/ansible-tmp-1675275150.058933-12845340-211798537736734\\n', b"Control socket connect(/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp): Socket operation on non-socket\\r\\nWarning: the ECDSA host key for 'nimon' differs from the key for the IP address '10.220.3.59'\\nOffending key for IP in /var/lib/awx/.ssh/known_hosts:1\\nMatching host key in /var/lib/awx/.ssh/known_hosts:4\\r\\nControlSocket /tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp already exists, disabling multiplexing\\r\\n")
<nimon> Attempting python interpreter discovery
<nimon> ESTABLISH SSH CONNECTION FOR USER: root
<nimon> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp nimon '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.5'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"''
<nimon> (255, b'', b"Control socket connect(/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp): Socket operation on non-socket\\r\\nWarning: the ECDSA host key for 'nimon' differs from the key for the IP address '10.220.3.59'\\nOffending key for IP in /var/lib/awx/.ssh/known_hosts:1\\nMatching host key in /var/lib/awx/.ssh/known_hosts:4\\r\\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\\r\\nPermission denied, please try again.\\r\\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\\r\\nPermission denied, please try again.\\r\\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\\r\\nroot@nimon: Permission denied (publickey,password,keyboard-interactive).\\r\\n")
[WARNING]: Unhandled error in Python interpreter discovery for host nimon:
Failed to connect to the host via ssh: Control socket
connect(/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp): Socket operation on non-
socket Warning: the ECDSA host key for 'nimon' differs from the key for the IP
address '10.220.3.59' Offending key for IP in /var/lib/awx/.ssh/known_hosts:1
Matching host key in /var/lib/awx/.ssh/known_hosts:4 ssh_askpass:
exec(/usr/sbin/ssh-askpass): No such file or directory Permission denied,
please try again. ssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or
directory Permission denied, please try again. ssh_askpass:
exec(/usr/sbin/ssh-askpass): No such file or directory root@nimon: Permission
denied (publickey,password,keyboard-interactive).
Using module file /opt/freeware/lib/python3.7/site-packages/ansible/modules/commands/command.py
<nimon> PUT /var/lib/awx/.ansible/tmp/ansible-local-13828606j2e4lsca/tmpz4mjws8a TO /.ansible/tmp/ansible-tmp-1675275150.058933-12845340-211798537736734/AnsiballZ_command.py
<nimon> SSH: EXEC sshpass -d12 sftp -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp '[nimon]'
<nimon> (0, b'sftp> put /var/lib/awx/.ansible/tmp/ansible-local-13828606j2e4lsca/tmpz4mjws8a /.ansible/tmp/ansible-tmp-1675275150.058933-12845340-211798537736734/AnsiballZ_command.py\\nUploading /var/lib/awx/.ansible/tmp/ansible-local-13828606j2e4lsca/tmpz4mjws8a to /.ansible/tmp/ansible-tmp-1675275150.058933-12845340-211798537736734/AnsiballZ_command.py\\n', b"Control socket connect(/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp): Socket operation on non-socket\\r\\nWarning: the ECDSA host key for 'nimon' differs from the key for the IP address '10.220.3.59'\\nOffending key for IP in /var/lib/awx/.ssh/known_hosts:1\\nMatching host key in /var/lib/awx/.ssh/known_hosts:4\\r\\nControlSocket /tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp already exists, disabling multiplexing\\r\\nConnected to nimon.\\n")
<nimon> ESTABLISH SSH CONNECTION FOR USER: root
<nimon> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp nimon '/bin/sh -c '"'"'chmod u+x /.ansible/tmp/ansible-tmp-1675275150.058933-12845340-211798537736734/ /.ansible/tmp/ansible-tmp-1675275150.058933-12845340-211798537736734/AnsiballZ_command.py && sleep 0'"'"''
<nimon> (0, b'', b"Control socket connect(/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp): Socket operation on non-socket\\r\\nWarning: the ECDSA host key for 'nimon' differs from the key for the IP address '10.220.3.59'\\nOffending key for IP in /var/lib/awx/.ssh/known_hosts:1\\nMatching host key in /var/lib/awx/.ssh/known_hosts:4\\r\\nControlSocket /tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp already exists, disabling multiplexing\\r\\n")
<nimon> ESTABLISH SSH CONNECTION FOR USER: root
<nimon> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp -tt nimon '/bin/sh -c '"'"'/usr/bin/python /.ansible/tmp/ansible-tmp-1675275150.058933-12845340-211798537736734/AnsiballZ_command.py && sleep 0'"'"''
<nimon> (255, b'', b"Control socket connect(/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp): Socket operation on non-socket\\r\\nWarning: the ECDSA host key for 'nimon' differs from the key for the IP address '10.220.3.59'\\nOffending key for IP in /var/lib/awx/.ssh/known_hosts:1\\nMatching host key in /var/lib/awx/.ssh/known_hosts:4\\r\\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\\r\\nPermission denied, please try again.\\r\\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\\r\\nPermission denied, please try again.\\r\\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\\r\\nroot@nimon: Permission denied (publickey,password,keyboard-interactive).\\r\\n")
nimon | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: Control socket connect(/tmp/bwrap_234_xqei15iy/awx_234_xhn79j2f/cp): Socket operation on non-socket\\r\\nWarning: the ECDSA host key for 'nimon' differs from the key for the IP address '10.220.3.59'\\nOffending key for IP in /var/lib/awx/.ssh/known_hosts:1\\nMatching host key in /var/lib/awx/.ssh/known_hosts:4\\r\\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\\r\\nPermission denied, please try again.\\r\\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\\r\\nPermission denied, please try again.\\r\\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\\r\\nroot@nimon: Permission denied (publickey,password,keyboard-interactive).",
"unreachable": true
}

aix-dispatcher.log has this error.
2023-02-01 18:05:02,070 WARNING awx.main.dispatch scaling up worker pid:14745880
2023-02-01 18:05:02,109 WARNING awx.main.dispatch scaling up worker pid:13107710
2023-02-01 18:05:02,133 WARNING awx.main.dispatch scaling up worker pid:13369794
2023-02-01 18:05:02,145 WARNING awx.main.dispatch Running worker dispatcher listening to queues ['tower_broadcast_all', 'powervc-test2']
2023-02-01 18:05:02,154 DEBUG awx.main.tasks Syncing Schedules
2023-02-01 18:05:02,186 DEBUG awx.main.tasks Waited 0.0008640289306640625 seconds to obtain lock name: cluster_policy_lock
2023-02-01 18:05:02,200 DEBUG awx.main.tasks Policy List, adding Instances [1] to Group tower
2023-02-01 18:05:02,201 DEBUG awx.main.tasks Total non-isolated instances:1 available for policy: 1
2023-02-01 18:05:02,201 DEBUG awx.main.tasks Cluster policy no-op finished in 0.014945745468139648 seconds
2023-02-01 18:05:02,202 DEBUG awx.main.tasks Cluster node heartbeat task.
2023-02-01 18:05:02,258 DEBUG awx.main.utils.reload Issuing command to restart services, args=['supervisorctl', 'restart', 'tower-processes:awx-rsyslogd']
2023-02-01 18:05:02,693 ERROR awx.main.utils.reload supervisorctl restart awx-rsyslogd errored with exit code `1`, stdout:
b"error: <class 'PermissionError'>, [Errno 13] Permission denied: file: /opt/freeware/lib/python3.7/site-packages/supervisor/xmlrpc.py line: 560"stderr:
b''
2023-02-01 18:05:21,976 DEBUG awx.main.dispatch task c5a30495-8c77-4bf4-8512-15356f81fc72 starting awx.main.scheduler.tasks.run_task_manager(*[])
2023-02-01 18:05:21,999 DEBUG awx.main.scheduler Running Tower task manager.
2023-02-01 18:05:22,019 DEBUG awx.main.scheduler Starting Scheduler
2023-02-01 18:05:22,113 DEBUG awx.main.scheduler Finishing Scheduler
2023-02-01 18:05:31,992 DEBUG awx.main.dispatch task 439b0610-fb9b-4545-bbca-f39be8dde62f starting awx.main.tasks.awx_periodic_scheduler(*[])
2023-02-01 18:05:32,020 DEBUG awx.main.tasks Starting periodic scheduler
2023-02-01 18:05:32,023 DEBUG awx.main.tasks Last scheduler run was: 2023-02-01 18:03:48.684884+00:00
2023-02-01 18:05:42,013 DEBUG awx.main.dispatch task 0aa2ebbf-10a4-47f6-9d76-1333f9d6ae41 starting awx.main.scheduler.tasks.run_task_manager(*[])
2023-02-01 18:05:42,018 DEBUG awx.main.scheduler Running Tower task manager.
2023-02-01 18:05:42,035 DEBUG awx.main.scheduler Starting Scheduler

Another error is in the awx-daphne.log
awx:/var/log/supervisor #head -n 100 awx-daphne.log
2023-02-01 18:05:00,113 DEBUG awx.main.routing cleaning up Redis key b'asgi:group:jobs-summary'
2023-02-01 18:05:00,113 DEBUG cleaning up Redis key b'asgi:group:jobs-summary'
2023-02-01 18:05:00,118 DEBUG awx.main.routing cleaning up Redis key b'asgi:group:ad_hoc_command_events-231'
2023-02-01 18:05:00,118 DEBUG cleaning up Redis key b'asgi:group:ad_hoc_command_events-231'
2023-02-01 18:05:00,119 DEBUG awx.main.routing cleaning up Redis key b'asgi:group:control-limit_reached_1'
2023-02-01 18:05:00,119 DEBUG cleaning up Redis key b'asgi:group:control-limit_reached_1'
2023-02-01 18:05:00,120 DEBUG awx.main.routing cleaning up Redis key b'asgi:group:inventories-status_changed'
2023-02-01 18:05:00,120 DEBUG cleaning up Redis key b'asgi:group:inventories-status_changed'
2023-02-01 18:05:00,121 DEBUG awx.main.routing cleaning up Redis key b'asgi:group:jobs-status_changed'
2023-02-01 18:05:00,121 DEBUG cleaning up Redis key b'asgi:group:jobs-status_changed'
2023-02-01 18:05:00,122 DEBUG awx.main.routing cleaning up Redis key b'asgi:group:ad_hoc_command_events-230'
2023-02-01 18:05:00,122 DEBUG cleaning up Redis key b'asgi:group:ad_hoc_command_events-230'
2023-02-01 18:05:00,122 DEBUG awx.main.routing cleaning up Redis key b'asgi:group:ad_hoc_command_events-233'
2023-02-01 18:05:00,122 DEBUG cleaning up Redis key b'asgi:group:ad_hoc_command_events-233'
2023-02-01 18:05:00,123 DEBUG awx.main.routing cleaning up Redis key b'asgi:group:ad_hoc_command_events-205'
2023-02-01 18:05:00,123 DEBUG cleaning up Redis key b'asgi:group:ad_hoc_command_events-205'
2023-02-01 18:05:00,124 DEBUG awx.main.routing cleaning up Redis key b'asgi:group:ad_hoc_command_events-206'
2023-02-01 18:05:00,124 DEBUG cleaning up Redis key b'asgi:group:ad_hoc_command_events-206'
2023-02-01 18:05:00,125 INFO daphne.cli Starting server at unix:/var/run/tower/daphne.sock
2023-02-01 18:05:00,125 INFO Starting server at unix:/var/run/tower/daphne.sock
2023-02-01 18:05:00,126 INFO daphne.server HTTP/2 support not enabled (install the http2 and tls Twisted extras)
2023-02-01 18:05:00,126 INFO HTTP/2 support not enabled (install the http2 and tls Twisted extras)
2023-02-01 18:05:00,126 INFO daphne.server Configuring endpoint unix:/var/run/tower/daphne.sock
2023-02-01 18:05:00,126 INFO Configuring endpoint unix:/var/run/tower/daphne.sock
2023-02-01 18:05:00,286 ERROR awx.main.consumers Request user is not authenticated to use websocket.
2023-02-01 18:05:00,286 ERROR awx.main.consumers Request user is not authenticated to use websocket.
2023-02-01 18:05:00,286 ERROR Request user is not authenticated to use websocket.
2023-02-01 18:05:00,326 ERROR awx.main.consumers Request user is not authenticated to use websocket.
2023-02-01 18:05:00,326 ERROR awx.main.consumers Request user is not authenticated to use websocket.
2023-02-01 18:05:00,326 ERROR Request user is not authenticated to use websocket.

Here is the list of installed packages.
dnf list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.14-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.2-1 @AIX_Toolbox_noarch
bash.ppc 5.1.4-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2020.06.01-2 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.4 @System
curl.ppc 7.76.1-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.7-1 @AIX_Toolbox
dnf.noarch 4.2.17-1 @System
dnf-automatic.noarch 4.2.17-1 @System
dnf-data.noarch 4.2.17-1 @System
expat.ppc 2.2.9-2 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.8-1 @AIX_Toolbox
gcc.ppc 1:8-1 @AIX_Toolbox_72
gcc-c++.ppc 1:8-1 @AIX_Toolbox_72
gcc-cpp.ppc 1:8-1 @AIX_Toolbox_72
gcc8.ppc 8.3.0-5 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-5 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-5 @AIX_Toolbox_72
gdbm.ppc 1.18.1-1 @AIX_Toolbox
gettext.ppc 0.20.2-1 @AIX_Toolbox
git.ppc 2.20.4-1 @AIX_Toolbox
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.1.2-1 @AIX_Toolbox
golang.ppc 1.18.5-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.5-1 @AIX_Toolbox_72
golang-src.noarch 1.18.5-1 @AIX_Toolbox_noarch
grep.ppc 3.4-1 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.3-1 @AIX_Toolbox
less.ppc 557-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libcomps.ppc 0.1.11-1 @System
libdnf.ppc 0.39.1-1 @System
libffi.ppc 3.2.1-3 @AIX_Toolbox
libffi-devel.ppc 3.2.1-3 @AIX_Toolbox
libgcc.ppc 1:8-1 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-5 @AIX_Toolbox_72
libgcrypt.ppc 1.8.5-1 @AIX_Toolbox
libgd.ppc 2.2.5-3 @AIX_Toolbox
libgomp.ppc 1:8-1 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-5 @AIX_Toolbox_72
libgpg-error.ppc 1.36-1 @AIX_Toolbox
libiconv.ppc 1.16-1 @AIX_Toolbox
libjpeg.ppc 9d-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-1 @System
libmpc.ppc 1.1.0-1 @AIX_Toolbox
libnghttp2.ppc 1.41.0-1 @AIX_Toolbox
libpng.ppc 1.6.27-3 @AIX_Toolbox
librepo.ppc 1.11.0-1 @System
libsmartcols.ppc 2.34-1 @System
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-1 @System
libssh2.ppc 1.9.0-1 @AIX_Toolbox
libstdc++.ppc 1:8-1 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-5 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-5 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.20.2-1 @AIX_Toolbox
libtiff.ppc 4.2.0-1 @AIX_Toolbox
libtool.ppc 2.4.6-2 @AIX_Toolbox
libtool-ltdl.ppc 2.4.6-2 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.9.14-1 @AIX_Toolbox
libxslt.ppc 1.1.32-1 @AIX_Toolbox
libyaml.ppc 0.2.2-1 @AIX_Toolbox
libzstd.ppc 1.4.4-1 @System
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.2-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.0.2-2 @AIX_Toolbox
ncurses.ppc 6.2-2 @AIX_Toolbox
nginx.ppc 1.17.8-1 @AIX_Toolbox
openldap.ppc 2.4.58-4 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.16-2 @AIX_Toolbox
p11-kit-tools.ppc 0.23.16-2 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.34-1 @AIX_Toolbox
perl.ppc 1:5.30.3-2 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.11-1 @AIX_Toolbox
postgresql-libs.ppc 11.11-1 @AIX_Toolbox
postgresql-server.ppc 11.11-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-3 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-3 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-3 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.10-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.10-2 @AIX_Toolbox
python3-dnf.noarch 4.2.17-1 @System
python3-dnf-plugin-migrate.noarch 4.0.16-1 @System
python3-dnf-plugins-core.noarch 4.0.16-1 @System
python3-gpgme.ppc 1.13.1-1 @System
python3-hawkey.ppc 0.39.1-1 @System
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-1 @System
python3-libdnf.ppc 0.39.1-1 @System
python3-librepo.ppc 1.11.0-1 @System
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.3-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.35.5-1 @AIX_Toolbox
sshpass.ppc 1.06-2 @AIX_Toolbox
subversion.ppc 1.10.7-2 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum.noarch 4.2.17-1 @System
yum-metadata-parser.ppc 1.1.4-2 @System
yum-utils.noarch 1.1.31-2 @AIX_Toolbox_noarch
zchunk-libs.ppc 1.1.4-1 @System
zeromq.ppc 4.3.3-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.3-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.11-2 @AIX_Toolbox

Any ideas what could be causing these errors or how to get AWX to connect.

An ssh connection to these same systems from the command line works with no issues.

Thank you for your assistance.

Stan

------------------------------
Stanley

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

11. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Andrey Klyachkin

Posted Fri February 03, 2023 08:49 AM

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com
------------------------------

Original Message

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

12. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Steve Munday

Posted Fri February 03, 2023 09:29 AM

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM
------------------------------

Original Message

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

13. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Andrey Klyachkin

Posted Fri February 03, 2023 09:55 AM

The new AWX architecture uses so called execution environments which are special prepared containers indeed. It makes porting a way more difficult...

------------------------------
Andrey Klyachkin

https://www.power-devops.com
------------------------------

Original Message

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

14. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Steve Munday

Posted Fri February 03, 2023 11:12 AM

Andrey,

Ah yes, so is v17 the "newest" we can use without the "execution environments" then?

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM
------------------------------

Original Message

Original Message:
Sent: Fri February 03, 2023 09:55 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

The new AWX architecture uses so called execution environments which are special prepared containers indeed. It makes porting a way more difficult...

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

15. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Andrey Klyachkin

Posted Mon February 06, 2023 03:16 AM

Steve,

as for now - yes. I think it is still possible to use the AWX host system instead of an EE on newer AWX versions, but as for now v17 is the latest ported to AIX.

------------------------------
Andrey Klyachkin

https://www.power-devops.com
------------------------------

Original Message

Original Message:
Sent: Fri February 03, 2023 11:11 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Ah yes, so is v17 the "newest" we can use without the "execution environments" then?

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 09:55 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

The new AWX architecture uses so called execution environments which are special prepared containers indeed. It makes porting a way more difficult...

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

16. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Steve Munday

Posted Fri February 03, 2023 11:10 AM

Andrey,

I had to play around with yum vs. dnf (I only have dnf loaded) and then hit the below issues:

TASK [copy our required packages] ***********************************************************************************************************************************************************
changed: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)
changed: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)
changed: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)
changed: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)
changed: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)

TASK [install our packages] *****************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for linux_compat-0.1.2-1.ppc"}

PLAY RECAP **********************************************************************************************************************************************************************************
localhost                  : ok=10   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM
------------------------------

Original Message

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

17. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Andrey Klyachkin

Posted Mon February 06, 2023 03:19 AM

Steve,

can you please add:

disable_gpg_check: true

into dnf call in the playbook? The packages are not signed.

It should something like:

- name: install our packages
  dnf:
    name: "{{ copied | json_query('results[].dest') }}"
    disable_gpg_check: true

in 02_packages.yml

------------------------------
Andrey Klyachkin

https://www.power-devops.com
------------------------------

Original Message

Original Message:
Sent: Fri February 03, 2023 11:10 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I had to play around with yum vs. dnf (I only have dnf loaded) and then hit the below issues:

TASK [copy our required packages] ***********************************************************************************************************************************************************changed: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)changed: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)changed: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)changed: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)changed: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)TASK [install our packages] *****************************************************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for linux_compat-0.1.2-1.ppc"}PLAY RECAP **********************************************************************************************************************************************************************************localhost                  : ok=10   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

18. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Steve Munday

Posted Mon February 06, 2023 07:09 AM

Andrey,

Thanks for that, the install was able to run through without any further issues.

Once I completed the install and started the browser session to AWX I then saw the below (so the same as "before" the updated 17.1) it seems.

I re-checked openldap and libxml2.

# dnf list openldap libxml2
Last metadata expiration check: 1:37:58 ago on Mon Feb  6 05:18:01 EST 2023.
Installed Packages
libxml2.ppc                                                                               2.10.3-1                                                                               @AIX_Toolbox
openldap.ppc                                                                              2.5.12-1                                                                               @AIX_Toolbox
#

Checking the /var/log/nginx/access.log I see the following (2nd line == screenshot above, 3rd line == once I clicked the "X" in screenshot above)

10.1.1.1 - - [06/Feb/2023:06:52:11 -0500] "GET / HTTP/1.1" 200 1082 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"
10.1.1.1 - - [06/Feb/2023:06:52:12 -0500] "GET /api/v2/auth/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"
10.1.1.1 - - [06/Feb/2023:06:52:13 -0500] "GET /api/ HTTP/1.1" 200 186 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"
#

If I then enter "admin" and the password I get the following on the screen.

And this in the /var/log/nginx/access.log

10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "GET /api/login/ HTTP/1.1" 200 5720 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"
10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "POST /api/login/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"
#

Any thoughts?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM
------------------------------

Original Message

Original Message:
Sent: Mon February 06, 2023 03:19 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

can you please add:

disable_gpg_check: true

into dnf call in the playbook? The packages are not signed.

It should something like:

- name: install our packages  dnf:    name: "{{ copied | json_query('results[].dest') }}"    disable_gpg_check: true

in 02_packages.yml

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Fri February 03, 2023 11:10 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I had to play around with yum vs. dnf (I only have dnf loaded) and then hit the below issues:

TASK [copy our required packages] ***********************************************************************************************************************************************************changed: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)changed: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)changed: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)changed: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)changed: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)TASK [install our packages] *****************************************************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for linux_compat-0.1.2-1.ppc"}PLAY RECAP **********************************************************************************************************************************************************************************localhost                  : ok=10   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

19. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Steve Munday

Posted Mon February 06, 2023 12:34 PM

| view attached (0)

Andrey,

I tried to upload the /var/log/supervisor/awx-uwsgi.log however I get "Access denied" so have pasted it below.

# tail /var/log/supervisor/awx-uwsgi.log
[pid: 13631776|app: 0|req: 4/15] 10.207.114.45 () {58 vars in 1037 bytes} [Mon Feb  6 11:57:54 2023] POST /api/login/ => generated 41 bytes in 83 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)

##########################################################################################################
<<< "Refreshed" the browser to re-create the initial "Error!  Failed to fetch custom login ..." message >>
##########################################################################################################

[pid: 2752938|app: 0|req: 5/16] 10.207.114.45 () {52 vars in 860 bytes} [Mon Feb  6 17:18:53 2023] GET / => generated 1082 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 168 bytes (1 switches on core 0)
[pid: 13631776|app: 0|req: 5/17] 10.207.114.45 () {52 vars in 876 bytes} [Mon Feb  6 17:18:53 2023] GET /api/ => generated 186 bytes in 79 msecs (HTTP/1.1 200) 11 headers in 402 bytes (1 switches on core 0)
2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner
    return func(*args, **kwds)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch
    return super(APIView, self).dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get
    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends
    backend = module_member(auth_backend)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member
    module = import_module(mod)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module
    __import__(name)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>
    from social_core.backends.saml import OID_USERID
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>
    from onelogin.saml2.auth import OneLogin_Saml2_Auth
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>
    import xmlsec
ImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:
        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from
                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].
        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from
                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].
        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from
                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].
        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from
                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].
        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from
                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].
        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from
                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].
        0509-021 Additional errors occurred but are not reported.
        0509-192 Examine .loader section symbols with the
                 'dump -Tv' command.
2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner
    return func(*args, **kwds)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch
    return super(APIView, self).dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get
    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends
    backend = module_member(auth_backend)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member
    module = import_module(mod)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module
    __import__(name)
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>
    from social_core.backends.saml import OID_USERID
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>
    from onelogin.saml2.auth import OneLogin_Saml2_Auth
  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>
    import xmlsec
ImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:
        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from
                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].
        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from
                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].
        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from
                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].
        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from
                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].
        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from
                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].
        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from
                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].
        0509-021 Additional errors occurred but are not reported.
        0509-192 Examine .loader section symbols with the
                 'dump -Tv' command.
[pid: 11272526|app: 0|req: 2/18] 10.207.114.45 () {52 vars in 892 bytes} [Mon Feb  6 17:18:53 2023] GET /api/v2/auth/ => generated 41 bytes in 175 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)
#

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM
------------------------------

Original Message

Original Message:
Sent: Mon February 06, 2023 07:08 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for that, the install was able to run through without any further issues.

Once I completed the install and started the browser session to AWX I then saw the below (so the same as "before" the updated 17.1) it seems.

I re-checked openldap and libxml2.

# dnf list openldap libxml2Last metadata expiration check: 1:37:58 ago on Mon Feb  6 05:18:01 EST 2023.Installed Packageslibxml2.ppc                                                                               2.10.3-1                                                                               @AIX_Toolboxopenldap.ppc                                                                              2.5.12-1                                                                               @AIX_Toolbox#

Checking the /var/log/nginx/access.log I see the following (2nd line == screenshot above, 3rd line == once I clicked the "X" in screenshot above)

10.1.1.1 - - [06/Feb/2023:06:52:11 -0500] "GET / HTTP/1.1" 200 1082 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:12 -0500] "GET /api/v2/auth/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:13 -0500] "GET /api/ HTTP/1.1" 200 186 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

If I then enter "admin" and the password I get the following on the screen.

And this in the /var/log/nginx/access.log

10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "GET /api/login/ HTTP/1.1" 200 5720 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "POST /api/login/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

Any thoughts?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 03:19 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

can you please add:

disable_gpg_check: true

into dnf call in the playbook? The packages are not signed.

It should something like:

- name: install our packages  dnf:    name: "{{ copied | json_query('results[].dest') }}"    disable_gpg_check: true

in 02_packages.yml

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Fri February 03, 2023 11:10 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I had to play around with yum vs. dnf (I only have dnf loaded) and then hit the below issues:

TASK [copy our required packages] ***********************************************************************************************************************************************************changed: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)changed: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)changed: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)changed: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)changed: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)TASK [install our packages] *****************************************************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for linux_compat-0.1.2-1.ppc"}PLAY RECAP **********************************************************************************************************************************************************************************localhost                  : ok=10   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

20. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Andrey Klyachkin

Posted Tue February 07, 2023 09:42 AM

Steve,

I've got "Permission denied" when I wanted to reply :-) Thank you for the log. It means I have to do a little bit more.

------------------------------
Andrey Klyachkin

https://www.power-devops.com
------------------------------

Original Message

Original Message:
Sent: Mon February 06, 2023 12:33 PM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I tried to upload the /var/log/supervisor/awx-uwsgi.log however I get "Access denied" so have pasted it below.

# tail /var/log/supervisor/awx-uwsgi.log[pid: 13631776|app: 0|req: 4/15] 10.207.114.45 () {58 vars in 1037 bytes} [Mon Feb  6 11:57:54 2023] POST /api/login/ => generated 41 bytes in 83 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)##########################################################################################################<<< "Refreshed" the browser to re-create the initial "Error!  Failed to fetch custom login ..." message >>##########################################################################################################[pid: 2752938|app: 0|req: 5/16] 10.207.114.45 () {52 vars in 860 bytes} [Mon Feb  6 17:18:53 2023] GET / => generated 1082 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 168 bytes (1 switches on core 0)[pid: 13631776|app: 0|req: 5/17] 10.207.114.45 () {52 vars in 876 bytes} [Mon Feb  6 17:18:53 2023] GET /api/ => generated 186 bytes in 79 msecs (HTTP/1.1 200) 11 headers in 402 bytes (1 switches on core 0)2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.[pid: 11272526|app: 0|req: 2/18] 10.207.114.45 () {52 vars in 892 bytes} [Mon Feb  6 17:18:53 2023] GET /api/v2/auth/ => generated 41 bytes in 175 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)#

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 07:08 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for that, the install was able to run through without any further issues.

Once I completed the install and started the browser session to AWX I then saw the below (so the same as "before" the updated 17.1) it seems.

I re-checked openldap and libxml2.

# dnf list openldap libxml2Last metadata expiration check: 1:37:58 ago on Mon Feb  6 05:18:01 EST 2023.Installed Packageslibxml2.ppc                                                                               2.10.3-1                                                                               @AIX_Toolboxopenldap.ppc                                                                              2.5.12-1                                                                               @AIX_Toolbox#

Checking the /var/log/nginx/access.log I see the following (2nd line == screenshot above, 3rd line == once I clicked the "X" in screenshot above)

10.1.1.1 - - [06/Feb/2023:06:52:11 -0500] "GET / HTTP/1.1" 200 1082 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:12 -0500] "GET /api/v2/auth/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:13 -0500] "GET /api/ HTTP/1.1" 200 186 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

If I then enter "admin" and the password I get the following on the screen.

And this in the /var/log/nginx/access.log

10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "GET /api/login/ HTTP/1.1" 200 5720 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "POST /api/login/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

Any thoughts?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 03:19 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

can you please add:

disable_gpg_check: true

into dnf call in the playbook? The packages are not signed.

It should something like:

- name: install our packages  dnf:    name: "{{ copied | json_query('results[].dest') }}"    disable_gpg_check: true

in 02_packages.yml

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Fri February 03, 2023 11:10 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I had to play around with yum vs. dnf (I only have dnf loaded) and then hit the below issues:

TASK [copy our required packages] ***********************************************************************************************************************************************************changed: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)changed: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)changed: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)changed: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)changed: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)TASK [install our packages] *****************************************************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for linux_compat-0.1.2-1.ppc"}PLAY RECAP **********************************************************************************************************************************************************************************localhost                  : ok=10   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

21. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Andrey Klyachkin

Posted Wed February 08, 2023 10:56 AM

Steve,

I re-uploaded the file with new xmlssec RPMs and new playbook.

------------------------------
Andrey Klyachkin

https://www.power-devops.com
------------------------------

Original Message

Original Message:
Sent: Mon February 06, 2023 12:33 PM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I tried to upload the /var/log/supervisor/awx-uwsgi.log however I get "Access denied" so have pasted it below.

# tail /var/log/supervisor/awx-uwsgi.log[pid: 13631776|app: 0|req: 4/15] 10.207.114.45 () {58 vars in 1037 bytes} [Mon Feb  6 11:57:54 2023] POST /api/login/ => generated 41 bytes in 83 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)##########################################################################################################<<< "Refreshed" the browser to re-create the initial "Error!  Failed to fetch custom login ..." message >>##########################################################################################################[pid: 2752938|app: 0|req: 5/16] 10.207.114.45 () {52 vars in 860 bytes} [Mon Feb  6 17:18:53 2023] GET / => generated 1082 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 168 bytes (1 switches on core 0)[pid: 13631776|app: 0|req: 5/17] 10.207.114.45 () {52 vars in 876 bytes} [Mon Feb  6 17:18:53 2023] GET /api/ => generated 186 bytes in 79 msecs (HTTP/1.1 200) 11 headers in 402 bytes (1 switches on core 0)2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.[pid: 11272526|app: 0|req: 2/18] 10.207.114.45 () {52 vars in 892 bytes} [Mon Feb  6 17:18:53 2023] GET /api/v2/auth/ => generated 41 bytes in 175 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)#

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 07:08 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for that, the install was able to run through without any further issues.

Once I completed the install and started the browser session to AWX I then saw the below (so the same as "before" the updated 17.1) it seems.

I re-checked openldap and libxml2.

# dnf list openldap libxml2Last metadata expiration check: 1:37:58 ago on Mon Feb  6 05:18:01 EST 2023.Installed Packageslibxml2.ppc                                                                               2.10.3-1                                                                               @AIX_Toolboxopenldap.ppc                                                                              2.5.12-1                                                                               @AIX_Toolbox#

Checking the /var/log/nginx/access.log I see the following (2nd line == screenshot above, 3rd line == once I clicked the "X" in screenshot above)

10.1.1.1 - - [06/Feb/2023:06:52:11 -0500] "GET / HTTP/1.1" 200 1082 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:12 -0500] "GET /api/v2/auth/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:13 -0500] "GET /api/ HTTP/1.1" 200 186 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

If I then enter "admin" and the password I get the following on the screen.

And this in the /var/log/nginx/access.log

10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "GET /api/login/ HTTP/1.1" 200 5720 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "POST /api/login/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

Any thoughts?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 03:19 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

can you please add:

disable_gpg_check: true

into dnf call in the playbook? The packages are not signed.

It should something like:

- name: install our packages  dnf:    name: "{{ copied | json_query('results[].dest') }}"    disable_gpg_check: true

in 02_packages.yml

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Fri February 03, 2023 11:10 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I had to play around with yum vs. dnf (I only have dnf loaded) and then hit the below issues:

TASK [copy our required packages] ***********************************************************************************************************************************************************changed: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)changed: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)changed: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)changed: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)changed: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)TASK [install our packages] *****************************************************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for linux_compat-0.1.2-1.ppc"}PLAY RECAP **********************************************************************************************************************************************************************************localhost                  : ok=10   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

22. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Steve Munday

Posted Wed February 08, 2023 01:08 PM

Andrey,

Thanks for uploading the revised files.

I "just" installed the "new" xmlssec rpm(s) and re-installed ansible-awx. I then configured LDAP and was able to successfully connect using my AD account.

Question: Am I correct in saying I only (really) needed to install xmlssec & re-install ansible-awx as I already had everything else loaded?

Based upon my presumption my next task will be to (a) install the xmlssec and (b) re-install ansible-awx over the top of my "Live" environment as the LDAP authentication is no-longer working (local accounts can still connect though).

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM
------------------------------

Original Message

Original Message:
Sent: Wed February 08, 2023 10:56 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

I re-uploaded the file with new xmlssec RPMs and new playbook.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon February 06, 2023 12:33 PM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I tried to upload the /var/log/supervisor/awx-uwsgi.log however I get "Access denied" so have pasted it below.

# tail /var/log/supervisor/awx-uwsgi.log[pid: 13631776|app: 0|req: 4/15] 10.207.114.45 () {58 vars in 1037 bytes} [Mon Feb  6 11:57:54 2023] POST /api/login/ => generated 41 bytes in 83 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)##########################################################################################################<<< "Refreshed" the browser to re-create the initial "Error!  Failed to fetch custom login ..." message >>##########################################################################################################[pid: 2752938|app: 0|req: 5/16] 10.207.114.45 () {52 vars in 860 bytes} [Mon Feb  6 17:18:53 2023] GET / => generated 1082 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 168 bytes (1 switches on core 0)[pid: 13631776|app: 0|req: 5/17] 10.207.114.45 () {52 vars in 876 bytes} [Mon Feb  6 17:18:53 2023] GET /api/ => generated 186 bytes in 79 msecs (HTTP/1.1 200) 11 headers in 402 bytes (1 switches on core 0)2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.[pid: 11272526|app: 0|req: 2/18] 10.207.114.45 () {52 vars in 892 bytes} [Mon Feb  6 17:18:53 2023] GET /api/v2/auth/ => generated 41 bytes in 175 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)#

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 07:08 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for that, the install was able to run through without any further issues.

Once I completed the install and started the browser session to AWX I then saw the below (so the same as "before" the updated 17.1) it seems.

I re-checked openldap and libxml2.

# dnf list openldap libxml2Last metadata expiration check: 1:37:58 ago on Mon Feb  6 05:18:01 EST 2023.Installed Packageslibxml2.ppc                                                                               2.10.3-1                                                                               @AIX_Toolboxopenldap.ppc                                                                              2.5.12-1                                                                               @AIX_Toolbox#

Checking the /var/log/nginx/access.log I see the following (2nd line == screenshot above, 3rd line == once I clicked the "X" in screenshot above)

10.1.1.1 - - [06/Feb/2023:06:52:11 -0500] "GET / HTTP/1.1" 200 1082 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:12 -0500] "GET /api/v2/auth/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:13 -0500] "GET /api/ HTTP/1.1" 200 186 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

If I then enter "admin" and the password I get the following on the screen.

And this in the /var/log/nginx/access.log

10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "GET /api/login/ HTTP/1.1" 200 5720 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "POST /api/login/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

Any thoughts?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 03:19 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

can you please add:

disable_gpg_check: true

into dnf call in the playbook? The packages are not signed.

It should something like:

- name: install our packages  dnf:    name: "{{ copied | json_query('results[].dest') }}"    disable_gpg_check: true

in 02_packages.yml

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Fri February 03, 2023 11:10 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I had to play around with yum vs. dnf (I only have dnf loaded) and then hit the below issues:

TASK [copy our required packages] ***********************************************************************************************************************************************************changed: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)changed: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)changed: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)changed: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)changed: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)TASK [install our packages] *****************************************************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for linux_compat-0.1.2-1.ppc"}PLAY RECAP **********************************************************************************************************************************************************************************localhost                  : ok=10   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

23. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Andrey Klyachkin

Posted Thu February 09, 2023 04:06 AM

Steve,

thank you for the update!

Yes, I think it is enough to update 3 RPMs (xmlsec1-1.2.28-2.aix7.2.ppc.rpm, xmlsec1-openssl-1.2.28-2.aix7.2.ppc.rpm, and ansible-awx-17.1.0-2aix.aix7.2.ppc.rpm), but please bear in mind that AWX-related services such as nginx and supervisord must be stopped during the update.

------------------------------
Andrey Klyachkin

https://www.power-devops.com
------------------------------

Original Message

Original Message:
Sent: Wed February 08, 2023 01:07 PM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for uploading the revised files.

I "just" installed the "new" xmlssec rpm(s) and re-installed ansible-awx. I then configured LDAP and was able to successfully connect using my AD account.

Question: Am I correct in saying I only (really) needed to install xmlssec & re-install ansible-awx as I already had everything else loaded?

Based upon my presumption my next task will be to (a) install the xmlssec and (b) re-install ansible-awx over the top of my "Live" environment as the LDAP authentication is no-longer working (local accounts can still connect though).

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Wed February 08, 2023 10:56 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

I re-uploaded the file with new xmlssec RPMs and new playbook.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon February 06, 2023 12:33 PM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I tried to upload the /var/log/supervisor/awx-uwsgi.log however I get "Access denied" so have pasted it below.

# tail /var/log/supervisor/awx-uwsgi.log[pid: 13631776|app: 0|req: 4/15] 10.207.114.45 () {58 vars in 1037 bytes} [Mon Feb  6 11:57:54 2023] POST /api/login/ => generated 41 bytes in 83 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)##########################################################################################################<<< "Refreshed" the browser to re-create the initial "Error!  Failed to fetch custom login ..." message >>##########################################################################################################[pid: 2752938|app: 0|req: 5/16] 10.207.114.45 () {52 vars in 860 bytes} [Mon Feb  6 17:18:53 2023] GET / => generated 1082 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 168 bytes (1 switches on core 0)[pid: 13631776|app: 0|req: 5/17] 10.207.114.45 () {52 vars in 876 bytes} [Mon Feb  6 17:18:53 2023] GET /api/ => generated 186 bytes in 79 msecs (HTTP/1.1 200) 11 headers in 402 bytes (1 switches on core 0)2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.[pid: 11272526|app: 0|req: 2/18] 10.207.114.45 () {52 vars in 892 bytes} [Mon Feb  6 17:18:53 2023] GET /api/v2/auth/ => generated 41 bytes in 175 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)#

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 07:08 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for that, the install was able to run through without any further issues.

Once I completed the install and started the browser session to AWX I then saw the below (so the same as "before" the updated 17.1) it seems.

I re-checked openldap and libxml2.

# dnf list openldap libxml2Last metadata expiration check: 1:37:58 ago on Mon Feb  6 05:18:01 EST 2023.Installed Packageslibxml2.ppc                                                                               2.10.3-1                                                                               @AIX_Toolboxopenldap.ppc                                                                              2.5.12-1                                                                               @AIX_Toolbox#

Checking the /var/log/nginx/access.log I see the following (2nd line == screenshot above, 3rd line == once I clicked the "X" in screenshot above)

10.1.1.1 - - [06/Feb/2023:06:52:11 -0500] "GET / HTTP/1.1" 200 1082 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:12 -0500] "GET /api/v2/auth/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:13 -0500] "GET /api/ HTTP/1.1" 200 186 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

If I then enter "admin" and the password I get the following on the screen.

And this in the /var/log/nginx/access.log

10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "GET /api/login/ HTTP/1.1" 200 5720 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "POST /api/login/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

Any thoughts?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 03:19 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

can you please add:

disable_gpg_check: true

into dnf call in the playbook? The packages are not signed.

It should something like:

- name: install our packages  dnf:    name: "{{ copied | json_query('results[].dest') }}"    disable_gpg_check: true

in 02_packages.yml

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Fri February 03, 2023 11:10 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I had to play around with yum vs. dnf (I only have dnf loaded) and then hit the below issues:

TASK [copy our required packages] ***********************************************************************************************************************************************************changed: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)changed: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)changed: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)changed: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)changed: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)TASK [install our packages] *****************************************************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for linux_compat-0.1.2-1.ppc"}PLAY RECAP **********************************************************************************************************************************************************************************localhost                  : ok=10   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

24. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

Stanley Speegle

Posted Wed February 08, 2023 02:15 PM

Where are the new AWX install files available?

The error I am getting is that the host is unreachable but ssh & ansible work from the command line with on errors.

mh-p9-nim | UNREACHABLE! => {

10

"changed": false,

11

"msg": "Failed to connect to the host via ssh: Control socket connect(/tmp/bwrap_244_vgwmryb1/awx_244_7s37x9b0/cp): Socket operation on non-socket\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nroot@mh-p9-nim: Permission denied (publickey,password,keyboard-interactive).",

12

"unreachable": true

I have searched and can't find where the file ssh-askpass should be installed from?

Can I reinstall the new AWX version over the existing AWX to refresh the install?

Thank you for your help!

Stan

------------------------------
Stanley
------------------------------

Original Message

Original Message:
Sent: Wed February 08, 2023 10:56 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

I re-uploaded the file with new xmlssec RPMs and new playbook.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon February 06, 2023 12:33 PM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I tried to upload the /var/log/supervisor/awx-uwsgi.log however I get "Access denied" so have pasted it below.

# tail /var/log/supervisor/awx-uwsgi.log[pid: 13631776|app: 0|req: 4/15] 10.207.114.45 () {58 vars in 1037 bytes} [Mon Feb  6 11:57:54 2023] POST /api/login/ => generated 41 bytes in 83 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)##########################################################################################################<<< "Refreshed" the browser to re-create the initial "Error!  Failed to fetch custom login ..." message >>##########################################################################################################[pid: 2752938|app: 0|req: 5/16] 10.207.114.45 () {52 vars in 860 bytes} [Mon Feb  6 17:18:53 2023] GET / => generated 1082 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 168 bytes (1 switches on core 0)[pid: 13631776|app: 0|req: 5/17] 10.207.114.45 () {52 vars in 876 bytes} [Mon Feb  6 17:18:53 2023] GET /api/ => generated 186 bytes in 79 msecs (HTTP/1.1 200) 11 headers in 402 bytes (1 switches on core 0)2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.[pid: 11272526|app: 0|req: 2/18] 10.207.114.45 () {52 vars in 892 bytes} [Mon Feb  6 17:18:53 2023] GET /api/v2/auth/ => generated 41 bytes in 175 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)#

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 07:08 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for that, the install was able to run through without any further issues.

Once I completed the install and started the browser session to AWX I then saw the below (so the same as "before" the updated 17.1) it seems.

I re-checked openldap and libxml2.

# dnf list openldap libxml2Last metadata expiration check: 1:37:58 ago on Mon Feb  6 05:18:01 EST 2023.Installed Packageslibxml2.ppc                                                                               2.10.3-1                                                                               @AIX_Toolboxopenldap.ppc                                                                              2.5.12-1                                                                               @AIX_Toolbox#

Checking the /var/log/nginx/access.log I see the following (2nd line == screenshot above, 3rd line == once I clicked the "X" in screenshot above)

10.1.1.1 - - [06/Feb/2023:06:52:11 -0500] "GET / HTTP/1.1" 200 1082 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:12 -0500] "GET /api/v2/auth/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:13 -0500] "GET /api/ HTTP/1.1" 200 186 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

If I then enter "admin" and the password I get the following on the screen.

And this in the /var/log/nginx/access.log

10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "GET /api/login/ HTTP/1.1" 200 5720 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "POST /api/login/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

Any thoughts?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 03:19 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

can you please add:

disable_gpg_check: true

into dnf call in the playbook? The packages are not signed.

It should something like:

- name: install our packages  dnf:    name: "{{ copied | json_query('results[].dest') }}"    disable_gpg_check: true

in 02_packages.yml

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Fri February 03, 2023 11:10 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I had to play around with yum vs. dnf (I only have dnf loaded) and then hit the below issues:

TASK [copy our required packages] ***********************************************************************************************************************************************************changed: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)changed: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)changed: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)changed: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)changed: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)TASK [install our packages] *****************************************************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for linux_compat-0.1.2-1.ppc"}PLAY RECAP **********************************************************************************************************************************************************************************localhost                  : ok=10   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

25. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Andrey Klyachkin

Posted Thu February 09, 2023 04:13 AM

Stanley,

the new files are available from https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz

If you do a new install, just unpack them and run ansible-playbook -i localhost, -c local awx_install.yml.

If you already have AWX 17.1.0 and want to update it, unpack the archive, stop nginx and supervisord, and update the 3 RPMs, I wrote above in the answer to Steve. Then start supervisord and nginx again. It should work.

Regarding your question about ssh-askpass, I have no idea why is it asking for it.

------------------------------
Andrey Klyachkin

https://www.power-devops.com
------------------------------

Original Message

Original Message:
Sent: Wed February 08, 2023 02:15 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Where are the new AWX install files available?

The error I am getting is that the host is unreachable but ssh & ansible work from the command line with on errors.

mh-p9-nim | UNREACHABLE! => {

10

"changed": false,

11

"msg": "Failed to connect to the host via ssh: Control socket connect(/tmp/bwrap_244_vgwmryb1/awx_244_7s37x9b0/cp): Socket operation on non-socket\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nroot@mh-p9-nim: Permission denied (publickey,password,keyboard-interactive).",

12

"unreachable": true

I have searched and can't find where the file ssh-askpass should be installed from?

Can I reinstall the new AWX version over the existing AWX to refresh the install?

Thank you for your help!

Stan

------------------------------
Stanley

Original Message:
Sent: Wed February 08, 2023 10:56 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

I re-uploaded the file with new xmlssec RPMs and new playbook.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon February 06, 2023 12:33 PM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I tried to upload the /var/log/supervisor/awx-uwsgi.log however I get "Access denied" so have pasted it below.

# tail /var/log/supervisor/awx-uwsgi.log[pid: 13631776|app: 0|req: 4/15] 10.207.114.45 () {58 vars in 1037 bytes} [Mon Feb  6 11:57:54 2023] POST /api/login/ => generated 41 bytes in 83 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)##########################################################################################################<<< "Refreshed" the browser to re-create the initial "Error!  Failed to fetch custom login ..." message >>##########################################################################################################[pid: 2752938|app: 0|req: 5/16] 10.207.114.45 () {52 vars in 860 bytes} [Mon Feb  6 17:18:53 2023] GET / => generated 1082 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 168 bytes (1 switches on core 0)[pid: 13631776|app: 0|req: 5/17] 10.207.114.45 () {52 vars in 876 bytes} [Mon Feb  6 17:18:53 2023] GET /api/ => generated 186 bytes in 79 msecs (HTTP/1.1 200) 11 headers in 402 bytes (1 switches on core 0)2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.[pid: 11272526|app: 0|req: 2/18] 10.207.114.45 () {52 vars in 892 bytes} [Mon Feb  6 17:18:53 2023] GET /api/v2/auth/ => generated 41 bytes in 175 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)#

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 07:08 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for that, the install was able to run through without any further issues.

Once I completed the install and started the browser session to AWX I then saw the below (so the same as "before" the updated 17.1) it seems.

I re-checked openldap and libxml2.

# dnf list openldap libxml2Last metadata expiration check: 1:37:58 ago on Mon Feb  6 05:18:01 EST 2023.Installed Packageslibxml2.ppc                                                                               2.10.3-1                                                                               @AIX_Toolboxopenldap.ppc                                                                              2.5.12-1                                                                               @AIX_Toolbox#

Checking the /var/log/nginx/access.log I see the following (2nd line == screenshot above, 3rd line == once I clicked the "X" in screenshot above)

10.1.1.1 - - [06/Feb/2023:06:52:11 -0500] "GET / HTTP/1.1" 200 1082 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:12 -0500] "GET /api/v2/auth/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:13 -0500] "GET /api/ HTTP/1.1" 200 186 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

If I then enter "admin" and the password I get the following on the screen.

And this in the /var/log/nginx/access.log

10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "GET /api/login/ HTTP/1.1" 200 5720 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "POST /api/login/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

Any thoughts?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 03:19 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

can you please add:

disable_gpg_check: true

into dnf call in the playbook? The packages are not signed.

It should something like:

- name: install our packages  dnf:    name: "{{ copied | json_query('results[].dest') }}"    disable_gpg_check: true

in 02_packages.yml

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Fri February 03, 2023 11:10 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I had to play around with yum vs. dnf (I only have dnf loaded) and then hit the below issues:

TASK [copy our required packages] ***********************************************************************************************************************************************************changed: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)changed: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)changed: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)changed: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)changed: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)TASK [install our packages] *****************************************************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for linux_compat-0.1.2-1.ppc"}PLAY RECAP **********************************************************************************************************************************************************************************localhost                  : ok=10   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

26. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

Stanley Speegle

Posted Thu February 09, 2023 08:57 AM

Andrey,

I received an error when using the updated install package.

TASK [update all packages] *******************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "failures": [], "msg": "Depsolve Error occured: \n Problem 1: cannot install both dnf-4.2.17-32_50.ppc and dnf-4.2.17-1.noarch\n - package yum-4.2.17-1.noarch requires dnf = 4.2.17-1, but none of the providers can be installed\n - cannot install the best update candidate for package dnf-4.2.17-1.noarch\n - problem with installed package yum-4.2.17-1.noarch\n Problem 2: package yum-4.2.17-1.noarch requires dnf-data = 4.2.17-1, but none of the providers can be installed\n - cannot install both dnf-data-4.2.17-32_50.ppc and dnf-data-4.2.17-1.noarch\n - cannot install the best update candidate for package yum-4.2.17-1.noarch\n - cannot install the best update candidate for package dnf-data-4.2.17-1.noarch", "rc": 1, "results": []}

Tried dfn update but got a similar error.

root@awx:/home/awx-install #dnf update
Last metadata expiration check: 0:07:58 ago on Thu Feb 9 07:44:29 CST 2023.
Error:
Problem 1: cannot install both dnf-4.2.17-32_50.ppc and dnf-4.2.17-1.noarch
- package yum-4.2.17-1.noarch requires dnf = 4.2.17-1, but none of the providers can be installed
- cannot install the best update candidate for package dnf-4.2.17-1.noarch
- problem with installed package yum-4.2.17-1.noarch
Problem 2: package yum-4.2.17-1.noarch requires dnf-data = 4.2.17-1, but none of the providers can be installed
- cannot install both dnf-data-4.2.17-32_50.ppc and dnf-data-4.2.17-1.noarch
- cannot install the best update candidate for package yum-4.2.17-1.noarch
- cannot install the best update candidate for package dnf-data-4.2.17-1.noarch
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Do I need to use the dnf update --allowerasing then attempt the AWX reinstall?

Thank you for your help!

Stan

------------------------------
Stanley
------------------------------

Original Message

Original Message:
Sent: Thu February 09, 2023 04:13 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Stanley,

the new files are available from https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz

If you do a new install, just unpack them and run ansible-playbook -i localhost, -c local awx_install.yml.

If you already have AWX 17.1.0 and want to update it, unpack the archive, stop nginx and supervisord, and update the 3 RPMs, I wrote above in the answer to Steve. Then start supervisord and nginx again. It should work.

Regarding your question about ssh-askpass, I have no idea why is it asking for it.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed February 08, 2023 02:15 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Where are the new AWX install files available?

The error I am getting is that the host is unreachable but ssh & ansible work from the command line with on errors.

mh-p9-nim | UNREACHABLE! => {

10

"changed": false,

11

"msg": "Failed to connect to the host via ssh: Control socket connect(/tmp/bwrap_244_vgwmryb1/awx_244_7s37x9b0/cp): Socket operation on non-socket\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nroot@mh-p9-nim: Permission denied (publickey,password,keyboard-interactive).",

12

"unreachable": true

I have searched and can't find where the file ssh-askpass should be installed from?

Can I reinstall the new AWX version over the existing AWX to refresh the install?

Thank you for your help!

Stan

------------------------------
Stanley

Original Message:
Sent: Wed February 08, 2023 10:56 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

I re-uploaded the file with new xmlssec RPMs and new playbook.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon February 06, 2023 12:33 PM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I tried to upload the /var/log/supervisor/awx-uwsgi.log however I get "Access denied" so have pasted it below.

# tail /var/log/supervisor/awx-uwsgi.log[pid: 13631776|app: 0|req: 4/15] 10.207.114.45 () {58 vars in 1037 bytes} [Mon Feb  6 11:57:54 2023] POST /api/login/ => generated 41 bytes in 83 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)##########################################################################################################<<< "Refreshed" the browser to re-create the initial "Error!  Failed to fetch custom login ..." message >>##########################################################################################################[pid: 2752938|app: 0|req: 5/16] 10.207.114.45 () {52 vars in 860 bytes} [Mon Feb  6 17:18:53 2023] GET / => generated 1082 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 168 bytes (1 switches on core 0)[pid: 13631776|app: 0|req: 5/17] 10.207.114.45 () {52 vars in 876 bytes} [Mon Feb  6 17:18:53 2023] GET /api/ => generated 186 bytes in 79 msecs (HTTP/1.1 200) 11 headers in 402 bytes (1 switches on core 0)2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.[pid: 11272526|app: 0|req: 2/18] 10.207.114.45 () {52 vars in 892 bytes} [Mon Feb  6 17:18:53 2023] GET /api/v2/auth/ => generated 41 bytes in 175 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)#

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 07:08 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for that, the install was able to run through without any further issues.

Once I completed the install and started the browser session to AWX I then saw the below (so the same as "before" the updated 17.1) it seems.

I re-checked openldap and libxml2.

# dnf list openldap libxml2Last metadata expiration check: 1:37:58 ago on Mon Feb  6 05:18:01 EST 2023.Installed Packageslibxml2.ppc                                                                               2.10.3-1                                                                               @AIX_Toolboxopenldap.ppc                                                                              2.5.12-1                                                                               @AIX_Toolbox#

Checking the /var/log/nginx/access.log I see the following (2nd line == screenshot above, 3rd line == once I clicked the "X" in screenshot above)

10.1.1.1 - - [06/Feb/2023:06:52:11 -0500] "GET / HTTP/1.1" 200 1082 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:12 -0500] "GET /api/v2/auth/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:13 -0500] "GET /api/ HTTP/1.1" 200 186 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

If I then enter "admin" and the password I get the following on the screen.

And this in the /var/log/nginx/access.log

10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "GET /api/login/ HTTP/1.1" 200 5720 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "POST /api/login/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

Any thoughts?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 03:19 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

can you please add:

disable_gpg_check: true

into dnf call in the playbook? The packages are not signed.

It should something like:

- name: install our packages  dnf:    name: "{{ copied | json_query('results[].dest') }}"    disable_gpg_check: true

in 02_packages.yml

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Fri February 03, 2023 11:10 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I had to play around with yum vs. dnf (I only have dnf loaded) and then hit the below issues:

TASK [copy our required packages] ***********************************************************************************************************************************************************changed: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)changed: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)changed: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)changed: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)changed: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)TASK [install our packages] *****************************************************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for linux_compat-0.1.2-1.ppc"}PLAY RECAP **********************************************************************************************************************************************************************************localhost                  : ok=10   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

27. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Steve Munday

Posted Thu February 09, 2023 09:07 AM
Edited by Steve Munday Thu February 09, 2023 11:33 AM

Stan,

Hi, I only did the following as everything else was already set (I'm on 7.3.1.1 and had done a dnf update before running the steps below):

Stop AWX
dnf reinstall /path/to/rpm/ansible-awx-17.1.0-2aix.aix7.2.ppc.rpm
dnf install /path/to/rpm/xmlsec1-1.2.28-2.aix7.2.ppc.rpm
dnf install /path/to/rpm/xmlsec1-openssl-1.2.28-2.aix7.2.ppc.rpm
Start AWX

Hope that helps.

Regards, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM
------------------------------

Original Message

Original Message:
Sent: Thu February 09, 2023 08:57 AM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I received an error when using the updated install package.

TASK [update all packages] *******************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "failures": [], "msg": "Depsolve Error occured: \n Problem 1: cannot install both dnf-4.2.17-32_50.ppc and dnf-4.2.17-1.noarch\n - package yum-4.2.17-1.noarch requires dnf = 4.2.17-1, but none of the providers can be installed\n - cannot install the best update candidate for package dnf-4.2.17-1.noarch\n - problem with installed package yum-4.2.17-1.noarch\n Problem 2: package yum-4.2.17-1.noarch requires dnf-data = 4.2.17-1, but none of the providers can be installed\n - cannot install both dnf-data-4.2.17-32_50.ppc and dnf-data-4.2.17-1.noarch\n - cannot install the best update candidate for package yum-4.2.17-1.noarch\n - cannot install the best update candidate for package dnf-data-4.2.17-1.noarch", "rc": 1, "results": []}

Tried dfn update but got a similar error.

root@awx:/home/awx-install #dnf update
Last metadata expiration check: 0:07:58 ago on Thu Feb 9 07:44:29 CST 2023.
Error:
Problem 1: cannot install both dnf-4.2.17-32_50.ppc and dnf-4.2.17-1.noarch
- package yum-4.2.17-1.noarch requires dnf = 4.2.17-1, but none of the providers can be installed
- cannot install the best update candidate for package dnf-4.2.17-1.noarch
- problem with installed package yum-4.2.17-1.noarch
Problem 2: package yum-4.2.17-1.noarch requires dnf-data = 4.2.17-1, but none of the providers can be installed
- cannot install both dnf-data-4.2.17-32_50.ppc and dnf-data-4.2.17-1.noarch
- cannot install the best update candidate for package yum-4.2.17-1.noarch
- cannot install the best update candidate for package dnf-data-4.2.17-1.noarch
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Do I need to use the dnf update --allowerasing then attempt the AWX reinstall?

Thank you for your help!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu February 09, 2023 04:13 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Stanley,

the new files are available from https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz

If you do a new install, just unpack them and run ansible-playbook -i localhost, -c local awx_install.yml.

If you already have AWX 17.1.0 and want to update it, unpack the archive, stop nginx and supervisord, and update the 3 RPMs, I wrote above in the answer to Steve. Then start supervisord and nginx again. It should work.

Regarding your question about ssh-askpass, I have no idea why is it asking for it.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed February 08, 2023 02:15 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Where are the new AWX install files available?

The error I am getting is that the host is unreachable but ssh & ansible work from the command line with on errors.

mh-p9-nim | UNREACHABLE! => {

10

"changed": false,

11

"msg": "Failed to connect to the host via ssh: Control socket connect(/tmp/bwrap_244_vgwmryb1/awx_244_7s37x9b0/cp): Socket operation on non-socket\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nroot@mh-p9-nim: Permission denied (publickey,password,keyboard-interactive).",

12

"unreachable": true

I have searched and can't find where the file ssh-askpass should be installed from?

Can I reinstall the new AWX version over the existing AWX to refresh the install?

Thank you for your help!

Stan

------------------------------
Stanley

Original Message:
Sent: Wed February 08, 2023 10:56 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

I re-uploaded the file with new xmlssec RPMs and new playbook.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon February 06, 2023 12:33 PM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I tried to upload the /var/log/supervisor/awx-uwsgi.log however I get "Access denied" so have pasted it below.

# tail /var/log/supervisor/awx-uwsgi.log[pid: 13631776|app: 0|req: 4/15] 10.207.114.45 () {58 vars in 1037 bytes} [Mon Feb  6 11:57:54 2023] POST /api/login/ => generated 41 bytes in 83 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)##########################################################################################################<<< "Refreshed" the browser to re-create the initial "Error!  Failed to fetch custom login ..." message >>##########################################################################################################[pid: 2752938|app: 0|req: 5/16] 10.207.114.45 () {52 vars in 860 bytes} [Mon Feb  6 17:18:53 2023] GET / => generated 1082 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 168 bytes (1 switches on core 0)[pid: 13631776|app: 0|req: 5/17] 10.207.114.45 () {52 vars in 876 bytes} [Mon Feb  6 17:18:53 2023] GET /api/ => generated 186 bytes in 79 msecs (HTTP/1.1 200) 11 headers in 402 bytes (1 switches on core 0)2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.[pid: 11272526|app: 0|req: 2/18] 10.207.114.45 () {52 vars in 892 bytes} [Mon Feb  6 17:18:53 2023] GET /api/v2/auth/ => generated 41 bytes in 175 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)#

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 07:08 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for that, the install was able to run through without any further issues.

Once I completed the install and started the browser session to AWX I then saw the below (so the same as "before" the updated 17.1) it seems.

I re-checked openldap and libxml2.

# dnf list openldap libxml2Last metadata expiration check: 1:37:58 ago on Mon Feb  6 05:18:01 EST 2023.Installed Packageslibxml2.ppc                                                                               2.10.3-1                                                                               @AIX_Toolboxopenldap.ppc                                                                              2.5.12-1                                                                               @AIX_Toolbox#

Checking the /var/log/nginx/access.log I see the following (2nd line == screenshot above, 3rd line == once I clicked the "X" in screenshot above)

10.1.1.1 - - [06/Feb/2023:06:52:11 -0500] "GET / HTTP/1.1" 200 1082 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:12 -0500] "GET /api/v2/auth/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:13 -0500] "GET /api/ HTTP/1.1" 200 186 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

If I then enter "admin" and the password I get the following on the screen.

And this in the /var/log/nginx/access.log

10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "GET /api/login/ HTTP/1.1" 200 5720 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "POST /api/login/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

Any thoughts?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 03:19 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

can you please add:

disable_gpg_check: true

into dnf call in the playbook? The packages are not signed.

It should something like:

- name: install our packages  dnf:    name: "{{ copied | json_query('results[].dest') }}"    disable_gpg_check: true

in 02_packages.yml

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Fri February 03, 2023 11:10 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I had to play around with yum vs. dnf (I only have dnf loaded) and then hit the below issues:

TASK [copy our required packages] ***********************************************************************************************************************************************************changed: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)changed: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)changed: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)changed: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)changed: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)TASK [install our packages] *****************************************************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for linux_compat-0.1.2-1.ppc"}PLAY RECAP **********************************************************************************************************************************************************************************localhost                  : ok=10   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

28. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Andrey Klyachkin

Posted Thu February 09, 2023 09:33 AM

Hi Stan,

it looks like you had a problem during DNF upgrade or something similar. DNF and YUM can't be installed at the same time. Do you still have YUM installed?

rpm -q yum

Otherwise I can't understand why DNF would try to install YUM.

------------------------------
Andrey Klyachkin

https://www.power-devops.com
------------------------------

Original Message

Original Message:
Sent: Thu February 09, 2023 08:57 AM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I received an error when using the updated install package.

TASK [update all packages] *******************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "failures": [], "msg": "Depsolve Error occured: \n Problem 1: cannot install both dnf-4.2.17-32_50.ppc and dnf-4.2.17-1.noarch\n - package yum-4.2.17-1.noarch requires dnf = 4.2.17-1, but none of the providers can be installed\n - cannot install the best update candidate for package dnf-4.2.17-1.noarch\n - problem with installed package yum-4.2.17-1.noarch\n Problem 2: package yum-4.2.17-1.noarch requires dnf-data = 4.2.17-1, but none of the providers can be installed\n - cannot install both dnf-data-4.2.17-32_50.ppc and dnf-data-4.2.17-1.noarch\n - cannot install the best update candidate for package yum-4.2.17-1.noarch\n - cannot install the best update candidate for package dnf-data-4.2.17-1.noarch", "rc": 1, "results": []}

Tried dfn update but got a similar error.

root@awx:/home/awx-install #dnf update
Last metadata expiration check: 0:07:58 ago on Thu Feb 9 07:44:29 CST 2023.
Error:
Problem 1: cannot install both dnf-4.2.17-32_50.ppc and dnf-4.2.17-1.noarch
- package yum-4.2.17-1.noarch requires dnf = 4.2.17-1, but none of the providers can be installed
- cannot install the best update candidate for package dnf-4.2.17-1.noarch
- problem with installed package yum-4.2.17-1.noarch
Problem 2: package yum-4.2.17-1.noarch requires dnf-data = 4.2.17-1, but none of the providers can be installed
- cannot install both dnf-data-4.2.17-32_50.ppc and dnf-data-4.2.17-1.noarch
- cannot install the best update candidate for package yum-4.2.17-1.noarch
- cannot install the best update candidate for package dnf-data-4.2.17-1.noarch
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Do I need to use the dnf update --allowerasing then attempt the AWX reinstall?

Thank you for your help!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu February 09, 2023 04:13 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Stanley,

the new files are available from https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz

If you do a new install, just unpack them and run ansible-playbook -i localhost, -c local awx_install.yml.

If you already have AWX 17.1.0 and want to update it, unpack the archive, stop nginx and supervisord, and update the 3 RPMs, I wrote above in the answer to Steve. Then start supervisord and nginx again. It should work.

Regarding your question about ssh-askpass, I have no idea why is it asking for it.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed February 08, 2023 02:15 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Where are the new AWX install files available?

The error I am getting is that the host is unreachable but ssh & ansible work from the command line with on errors.

mh-p9-nim | UNREACHABLE! => {

10

"changed": false,

11

"msg": "Failed to connect to the host via ssh: Control socket connect(/tmp/bwrap_244_vgwmryb1/awx_244_7s37x9b0/cp): Socket operation on non-socket\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nroot@mh-p9-nim: Permission denied (publickey,password,keyboard-interactive).",

12

"unreachable": true

I have searched and can't find where the file ssh-askpass should be installed from?

Can I reinstall the new AWX version over the existing AWX to refresh the install?

Thank you for your help!

Stan

------------------------------
Stanley

Original Message:
Sent: Wed February 08, 2023 10:56 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

I re-uploaded the file with new xmlssec RPMs and new playbook.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon February 06, 2023 12:33 PM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I tried to upload the /var/log/supervisor/awx-uwsgi.log however I get "Access denied" so have pasted it below.

# tail /var/log/supervisor/awx-uwsgi.log[pid: 13631776|app: 0|req: 4/15] 10.207.114.45 () {58 vars in 1037 bytes} [Mon Feb  6 11:57:54 2023] POST /api/login/ => generated 41 bytes in 83 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)##########################################################################################################<<< "Refreshed" the browser to re-create the initial "Error!  Failed to fetch custom login ..." message >>##########################################################################################################[pid: 2752938|app: 0|req: 5/16] 10.207.114.45 () {52 vars in 860 bytes} [Mon Feb  6 17:18:53 2023] GET / => generated 1082 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 168 bytes (1 switches on core 0)[pid: 13631776|app: 0|req: 5/17] 10.207.114.45 () {52 vars in 876 bytes} [Mon Feb  6 17:18:53 2023] GET /api/ => generated 186 bytes in 79 msecs (HTTP/1.1 200) 11 headers in 402 bytes (1 switches on core 0)2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.[pid: 11272526|app: 0|req: 2/18] 10.207.114.45 () {52 vars in 892 bytes} [Mon Feb  6 17:18:53 2023] GET /api/v2/auth/ => generated 41 bytes in 175 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)#

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 07:08 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for that, the install was able to run through without any further issues.

Once I completed the install and started the browser session to AWX I then saw the below (so the same as "before" the updated 17.1) it seems.

I re-checked openldap and libxml2.

# dnf list openldap libxml2Last metadata expiration check: 1:37:58 ago on Mon Feb  6 05:18:01 EST 2023.Installed Packageslibxml2.ppc                                                                               2.10.3-1                                                                               @AIX_Toolboxopenldap.ppc                                                                              2.5.12-1                                                                               @AIX_Toolbox#

Checking the /var/log/nginx/access.log I see the following (2nd line == screenshot above, 3rd line == once I clicked the "X" in screenshot above)

10.1.1.1 - - [06/Feb/2023:06:52:11 -0500] "GET / HTTP/1.1" 200 1082 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:12 -0500] "GET /api/v2/auth/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:13 -0500] "GET /api/ HTTP/1.1" 200 186 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

If I then enter "admin" and the password I get the following on the screen.

And this in the /var/log/nginx/access.log

10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "GET /api/login/ HTTP/1.1" 200 5720 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "POST /api/login/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

Any thoughts?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 03:19 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

can you please add:

disable_gpg_check: true

into dnf call in the playbook? The packages are not signed.

It should something like:

- name: install our packages  dnf:    name: "{{ copied | json_query('results[].dest') }}"    disable_gpg_check: true

in 02_packages.yml

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Fri February 03, 2023 11:10 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I had to play around with yum vs. dnf (I only have dnf loaded) and then hit the below issues:

TASK [copy our required packages] ***********************************************************************************************************************************************************changed: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)changed: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)changed: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)changed: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)changed: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)TASK [install our packages] *****************************************************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for linux_compat-0.1.2-1.ppc"}PLAY RECAP **********************************************************************************************************************************************************************************localhost                  : ok=10   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

29. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Andrey Klyachkin

Posted Thu February 09, 2023 09:39 AM

Stan,

do you install from IBM repos or from your local mirror?

The latest version of YUM in IBM repos is 3.4.3:

# /opt/freeware/bin/dnf info --all yum
Last metadata expiration check: 0:03:28 ago on Thu Feb  9 14:33:17 2023.
Available Packages
Name         : yum
Version      : 3.4.3
Release      : 8
Architecture : noarch
Size         : 1.3 M
Source       : yum-3.4.3-8.src.rpm
Repository   : AIX_Toolbox_noarch
Summary      : RPM installer/updater
URL          : http://yum.baseurl.org/
License      : GPLv2+
Description  : Yum is a utility that can check for and automatically download and
             : install updated RPM packages. Dependencies are obtained and downloaded
             : automatically, prompting the user for permission as necessary.

Your output shows another version:

Problem 2: package yum-4.2.17-1.noarch requires dnf-data = 4.2.17-1, but none of the providers can be installed

------------------------------
Andrey Klyachkin

https://www.power-devops.com
------------------------------

Original Message

Original Message:
Sent: Thu February 09, 2023 08:57 AM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I received an error when using the updated install package.

TASK [update all packages] *******************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "failures": [], "msg": "Depsolve Error occured: \n Problem 1: cannot install both dnf-4.2.17-32_50.ppc and dnf-4.2.17-1.noarch\n - package yum-4.2.17-1.noarch requires dnf = 4.2.17-1, but none of the providers can be installed\n - cannot install the best update candidate for package dnf-4.2.17-1.noarch\n - problem with installed package yum-4.2.17-1.noarch\n Problem 2: package yum-4.2.17-1.noarch requires dnf-data = 4.2.17-1, but none of the providers can be installed\n - cannot install both dnf-data-4.2.17-32_50.ppc and dnf-data-4.2.17-1.noarch\n - cannot install the best update candidate for package yum-4.2.17-1.noarch\n - cannot install the best update candidate for package dnf-data-4.2.17-1.noarch", "rc": 1, "results": []}

Tried dfn update but got a similar error.

root@awx:/home/awx-install #dnf update
Last metadata expiration check: 0:07:58 ago on Thu Feb 9 07:44:29 CST 2023.
Error:
Problem 1: cannot install both dnf-4.2.17-32_50.ppc and dnf-4.2.17-1.noarch
- package yum-4.2.17-1.noarch requires dnf = 4.2.17-1, but none of the providers can be installed
- cannot install the best update candidate for package dnf-4.2.17-1.noarch
- problem with installed package yum-4.2.17-1.noarch
Problem 2: package yum-4.2.17-1.noarch requires dnf-data = 4.2.17-1, but none of the providers can be installed
- cannot install both dnf-data-4.2.17-32_50.ppc and dnf-data-4.2.17-1.noarch
- cannot install the best update candidate for package yum-4.2.17-1.noarch
- cannot install the best update candidate for package dnf-data-4.2.17-1.noarch
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Do I need to use the dnf update --allowerasing then attempt the AWX reinstall?

Thank you for your help!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu February 09, 2023 04:13 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Stanley,

the new files are available from https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz

If you do a new install, just unpack them and run ansible-playbook -i localhost, -c local awx_install.yml.

If you already have AWX 17.1.0 and want to update it, unpack the archive, stop nginx and supervisord, and update the 3 RPMs, I wrote above in the answer to Steve. Then start supervisord and nginx again. It should work.

Regarding your question about ssh-askpass, I have no idea why is it asking for it.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed February 08, 2023 02:15 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Where are the new AWX install files available?

The error I am getting is that the host is unreachable but ssh & ansible work from the command line with on errors.

mh-p9-nim | UNREACHABLE! => {

10

"changed": false,

11

"msg": "Failed to connect to the host via ssh: Control socket connect(/tmp/bwrap_244_vgwmryb1/awx_244_7s37x9b0/cp): Socket operation on non-socket\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nroot@mh-p9-nim: Permission denied (publickey,password,keyboard-interactive).",

12

"unreachable": true

I have searched and can't find where the file ssh-askpass should be installed from?

Can I reinstall the new AWX version over the existing AWX to refresh the install?

Thank you for your help!

Stan

------------------------------
Stanley

Original Message:
Sent: Wed February 08, 2023 10:56 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

I re-uploaded the file with new xmlssec RPMs and new playbook.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon February 06, 2023 12:33 PM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I tried to upload the /var/log/supervisor/awx-uwsgi.log however I get "Access denied" so have pasted it below.

# tail /var/log/supervisor/awx-uwsgi.log[pid: 13631776|app: 0|req: 4/15] 10.207.114.45 () {58 vars in 1037 bytes} [Mon Feb  6 11:57:54 2023] POST /api/login/ => generated 41 bytes in 83 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)##########################################################################################################<<< "Refreshed" the browser to re-create the initial "Error!  Failed to fetch custom login ..." message >>##########################################################################################################[pid: 2752938|app: 0|req: 5/16] 10.207.114.45 () {52 vars in 860 bytes} [Mon Feb  6 17:18:53 2023] GET / => generated 1082 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 168 bytes (1 switches on core 0)[pid: 13631776|app: 0|req: 5/17] 10.207.114.45 () {52 vars in 876 bytes} [Mon Feb  6 17:18:53 2023] GET /api/ => generated 186 bytes in 79 msecs (HTTP/1.1 200) 11 headers in 402 bytes (1 switches on core 0)2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.[pid: 11272526|app: 0|req: 2/18] 10.207.114.45 () {52 vars in 892 bytes} [Mon Feb  6 17:18:53 2023] GET /api/v2/auth/ => generated 41 bytes in 175 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)#

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 07:08 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for that, the install was able to run through without any further issues.

Once I completed the install and started the browser session to AWX I then saw the below (so the same as "before" the updated 17.1) it seems.

I re-checked openldap and libxml2.

# dnf list openldap libxml2Last metadata expiration check: 1:37:58 ago on Mon Feb  6 05:18:01 EST 2023.Installed Packageslibxml2.ppc                                                                               2.10.3-1                                                                               @AIX_Toolboxopenldap.ppc                                                                              2.5.12-1                                                                               @AIX_Toolbox#

Checking the /var/log/nginx/access.log I see the following (2nd line == screenshot above, 3rd line == once I clicked the "X" in screenshot above)

10.1.1.1 - - [06/Feb/2023:06:52:11 -0500] "GET / HTTP/1.1" 200 1082 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:12 -0500] "GET /api/v2/auth/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:13 -0500] "GET /api/ HTTP/1.1" 200 186 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

If I then enter "admin" and the password I get the following on the screen.

And this in the /var/log/nginx/access.log

10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "GET /api/login/ HTTP/1.1" 200 5720 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "POST /api/login/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

Any thoughts?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 03:19 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

can you please add:

disable_gpg_check: true

into dnf call in the playbook? The packages are not signed.

It should something like:

- name: install our packages  dnf:    name: "{{ copied | json_query('results[].dest') }}"    disable_gpg_check: true

in 02_packages.yml

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Fri February 03, 2023 11:10 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I had to play around with yum vs. dnf (I only have dnf loaded) and then hit the below issues:

TASK [copy our required packages] ***********************************************************************************************************************************************************changed: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)changed: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)changed: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)changed: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)changed: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)TASK [install our packages] *****************************************************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for linux_compat-0.1.2-1.ppc"}PLAY RECAP **********************************************************************************************************************************************************************************localhost                  : ok=10   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

30. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Steve Munday

Posted Thu February 09, 2023 11:38 AM

Andrey,

Hmm, I'm finding at at 7.3.0.2 I have an issue which disappears when running 7.3.1.1, details below. On the 7.3.1.1 I had to do a reinstall as the packages were already loaded.

##########################
## On the 7.3.0.2 system #
##########################
# oslevel -s
7300-00-02-2220
# lslpp -l | grep -i openss
  CentrifyDC.openssl       5.8.1.371  COMMITTED  Centrify DirectControl OpenSSL
  openssh.base.client   8.1.112.1201  COMMITTED  Open Secure Shell Commands
  openssh.base.server   8.1.112.1201  COMMITTED  Open Secure Shell Server
  openssh.license       8.1.112.1201  COMMITTED  Open Secure Shell License
  openssh.man.en_US     8.1.112.1201  COMMITTED  Open Secure Shell
  openssh.msg.EN_US     8.1.112.1201  COMMITTED  Open Secure Shell Messages -
  openssh.msg.en_US     8.1.112.1201  COMMITTED  Open Secure Shell Messages -
  openssl.base            3.0.7.1000  COMMITTED  Open Secure Socket Layer
  openssl.license         3.0.7.1000  COMMITTED  Open Secure Socket License
  openssl.man.en_US       3.0.7.1000  COMMITTED  Open Secure Socket Layer
  openssh.base.client   8.1.112.1201  COMMITTED  Open Secure Shell Commands
  openssh.base.server   8.1.112.1201  COMMITTED  Open Secure Shell Server
  openssl.base            3.0.7.1000  COMMITTED  Open Secure Socket Layer
# rpm -qa | grep openldap
openldap-2.5.12-1.ppc
amrsomanim3:/# rpm -qa | grep libxml2
libxml2-devel-2.10.3-1.ppc
libxml2-2.10.3-1.ppc
libxml2-python3-2.10.3-1.ppc
# dnf install xmlsec1-1.2.28-2.aix7.2.ppc.rpm
Last metadata expiration check: 21:38:34 ago on Wed Feb  8 12:19:57 EST 2023.
Error:
 Problem: conflicting requests
  - nothing provides libxmlsec1-openssl.a(libxmlsec1-openssl.so.1) needed by xmlsec1-1.2.28-2.ppc
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
# dnf install xmlsec1-openssl-1.2.28-2.aix7.2.ppc.rpm
Last metadata expiration check: 21:39:26 ago on Wed Feb  8 12:19:57 EST 2023.
Error:
 Problem: conflicting requests
  - nothing provides libxmlsec1.a(libxmlsec1.so.1) needed by xmlsec1-openssl-1.2.28-2.ppc
  - nothing provides xmlsec1 = 1.2.28 needed by xmlsec1-openssl-1.2.28-2.ppc
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
#

I copied in the below (dated 7th Feb) from a 7.3.1.1 system where the install worked fine and get the issue with the originals and the "replacement" ones.

/opt/freeware/lib# ls -la libxml*
-rwxr-xr-x    1 root     system      1620376 Feb 07 10:08 libxmlsec1-openssl.a
-rwxr-xr-x    1 root     system      4371499 Jul 14 2020  libxmlsec1-openssl.a.20230209
-rwxr-xr-x    1 root     system       742200 Feb 07 10:08 libxmlsec1-openssl.so.1
-rwxr-xr-x    1 root     system      2101467 Jul 14 2020  libxmlsec1-openssl.so.1.20230209
-rwxr-xr-x    1 root     system      2095470 Feb 07 10:08 libxmlsec1.a
-rwxr-xr-x    1 root     system      3733085 Jul 14 2020  libxmlsec1.a.20230209
-rwxr-xr-x    1 root     system       931620 Feb 07 10:08 libxmlsec1.so.1
-rwxr-xr-x    1 root     system      1746787 Jul 14 2020  libxmlsec1.so.1.20230209
/opt/freeware/lib# 

===
##########################
## On the 7.3.1.1 system #
##########################
# oslevel -s
7300-01-01-2246
# lslpp -l | grep -i openss
  CentrifyDC.openssl       5.8.1.371  COMMITTED  Centrify DirectControl OpenSSL
  openssh.base.client   8.1.112.1201  COMMITTED  Open Secure Shell Commands
  openssh.base.server   8.1.112.1201  COMMITTED  Open Secure Shell Server
  openssh.license       8.1.112.1201  COMMITTED  Open Secure Shell License
  openssh.man.en_US     8.1.112.1201  COMMITTED  Open Secure Shell
  openssh.msg.EN_US     8.1.112.1201  COMMITTED  Open Secure Shell Messages -
  openssh.msg.en_US     8.1.112.1201  COMMITTED  Open Secure Shell Messages -
  openssl.base            3.0.7.1000  COMMITTED  Open Secure Socket Layer
  openssl.license         3.0.7.1000  COMMITTED  Open Secure Socket License
  openssl.man.en_US       3.0.7.1000  COMMITTED  Open Secure Socket Layer
  openssh.base.client   8.1.112.1201  COMMITTED  Open Secure Shell Commands
  openssh.base.server   8.1.112.1201  COMMITTED  Open Secure Shell Server
  openssl.base            3.0.7.1000  COMMITTED  Open Secure Socket Layer
# rpm -qa | grep openldap
openldap-2.5.12-1.ppc
# rpm -qa | grep libxml2
libxml2-2.10.3-1.ppc
libxml2-devel-2.10.3-1.ppc
libxml2-python3-2.10.3-1.ppc
# dnf reinstall xmlsec1-1.2.28-2.aix7.2.ppc.rpm
Last metadata expiration check: 23:12:26 ago on Wed Feb  8 12:13:38 EST 2023.
Dependencies resolved.
=============================================================================================================================================================================================
 Package                                      Architecture                             Version                                       Repository                                         Size
=============================================================================================================================================================================================
Reinstalling:
 xmlsec1                                      ppc                                      1.2.28-2                                      @commandline                                      1.2 M

Transaction Summary
=============================================================================================================================================================================================

Total size: 1.2 M
Installed size: 4.2 M
Is this ok [y/N]: n
Operation aborted.
# dnf reinstall xmlsec1-openssl-1.2.28-2.aix7.2.ppc.rpm
Last metadata expiration check: 23:12:56 ago on Wed Feb  8 12:13:38 EST 2023.
Dependencies resolved.
=============================================================================================================================================================================================
 Package                                            Architecture                           Version                                     Repository                                       Size
=============================================================================================================================================================================================
Reinstalling:
 xmlsec1-openssl                                    ppc                                    1.2.28-2                                    @commandline                                    777 k

Transaction Summary
=============================================================================================================================================================================================

Total size: 777 k
Installed size: 3.1 M
Is this ok [y/N]: n
Operation aborted.

Is something slightly mis-aligned perhaps?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM
------------------------------

Original Message

Original Message:
Sent: Thu February 09, 2023 09:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Stan,

do you install from IBM repos or from your local mirror?

The latest version of YUM in IBM repos is 3.4.3:

# /opt/freeware/bin/dnf info --all yumLast metadata expiration check: 0:03:28 ago on Thu Feb  9 14:33:17 2023.Available PackagesName         : yumVersion      : 3.4.3Release      : 8Architecture : noarchSize         : 1.3 MSource       : yum-3.4.3-8.src.rpmRepository   : AIX_Toolbox_noarchSummary      : RPM installer/updaterURL          : http://yum.baseurl.org/License      : GPLv2+Description  : Yum is a utility that can check for and automatically download and             : install updated RPM packages. Dependencies are obtained and downloaded             : automatically, prompting the user for permission as necessary.

Your output shows another version:

Problem 2: package yum-4.2.17-1.noarch requires dnf-data = 4.2.17-1, but none of the providers can be installed

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Thu February 09, 2023 08:57 AM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I received an error when using the updated install package.

TASK [update all packages] *******************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "failures": [], "msg": "Depsolve Error occured: \n Problem 1: cannot install both dnf-4.2.17-32_50.ppc and dnf-4.2.17-1.noarch\n - package yum-4.2.17-1.noarch requires dnf = 4.2.17-1, but none of the providers can be installed\n - cannot install the best update candidate for package dnf-4.2.17-1.noarch\n - problem with installed package yum-4.2.17-1.noarch\n Problem 2: package yum-4.2.17-1.noarch requires dnf-data = 4.2.17-1, but none of the providers can be installed\n - cannot install both dnf-data-4.2.17-32_50.ppc and dnf-data-4.2.17-1.noarch\n - cannot install the best update candidate for package yum-4.2.17-1.noarch\n - cannot install the best update candidate for package dnf-data-4.2.17-1.noarch", "rc": 1, "results": []}

Tried dfn update but got a similar error.

root@awx:/home/awx-install #dnf update
Last metadata expiration check: 0:07:58 ago on Thu Feb 9 07:44:29 CST 2023.
Error:
Problem 1: cannot install both dnf-4.2.17-32_50.ppc and dnf-4.2.17-1.noarch
- package yum-4.2.17-1.noarch requires dnf = 4.2.17-1, but none of the providers can be installed
- cannot install the best update candidate for package dnf-4.2.17-1.noarch
- problem with installed package yum-4.2.17-1.noarch
Problem 2: package yum-4.2.17-1.noarch requires dnf-data = 4.2.17-1, but none of the providers can be installed
- cannot install both dnf-data-4.2.17-32_50.ppc and dnf-data-4.2.17-1.noarch
- cannot install the best update candidate for package yum-4.2.17-1.noarch
- cannot install the best update candidate for package dnf-data-4.2.17-1.noarch
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Do I need to use the dnf update --allowerasing then attempt the AWX reinstall?

Thank you for your help!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu February 09, 2023 04:13 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Stanley,

the new files are available from https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz

If you do a new install, just unpack them and run ansible-playbook -i localhost, -c local awx_install.yml.

If you already have AWX 17.1.0 and want to update it, unpack the archive, stop nginx and supervisord, and update the 3 RPMs, I wrote above in the answer to Steve. Then start supervisord and nginx again. It should work.

Regarding your question about ssh-askpass, I have no idea why is it asking for it.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed February 08, 2023 02:15 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Where are the new AWX install files available?

The error I am getting is that the host is unreachable but ssh & ansible work from the command line with on errors.

mh-p9-nim | UNREACHABLE! => {

10

"changed": false,

11

"msg": "Failed to connect to the host via ssh: Control socket connect(/tmp/bwrap_244_vgwmryb1/awx_244_7s37x9b0/cp): Socket operation on non-socket\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nroot@mh-p9-nim: Permission denied (publickey,password,keyboard-interactive).",

12

"unreachable": true

I have searched and can't find where the file ssh-askpass should be installed from?

Can I reinstall the new AWX version over the existing AWX to refresh the install?

Thank you for your help!

Stan

------------------------------
Stanley

Original Message:
Sent: Wed February 08, 2023 10:56 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

I re-uploaded the file with new xmlssec RPMs and new playbook.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon February 06, 2023 12:33 PM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I tried to upload the /var/log/supervisor/awx-uwsgi.log however I get "Access denied" so have pasted it below.

# tail /var/log/supervisor/awx-uwsgi.log[pid: 13631776|app: 0|req: 4/15] 10.207.114.45 () {58 vars in 1037 bytes} [Mon Feb  6 11:57:54 2023] POST /api/login/ => generated 41 bytes in 83 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)##########################################################################################################<<< "Refreshed" the browser to re-create the initial "Error!  Failed to fetch custom login ..." message >>##########################################################################################################[pid: 2752938|app: 0|req: 5/16] 10.207.114.45 () {52 vars in 860 bytes} [Mon Feb  6 17:18:53 2023] GET / => generated 1082 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 168 bytes (1 switches on core 0)[pid: 13631776|app: 0|req: 5/17] 10.207.114.45 () {52 vars in 876 bytes} [Mon Feb  6 17:18:53 2023] GET /api/ => generated 186 bytes in 79 msecs (HTTP/1.1 200) 11 headers in 402 bytes (1 switches on core 0)2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.[pid: 11272526|app: 0|req: 2/18] 10.207.114.45 () {52 vars in 892 bytes} [Mon Feb  6 17:18:53 2023] GET /api/v2/auth/ => generated 41 bytes in 175 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)#

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 07:08 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for that, the install was able to run through without any further issues.

Once I completed the install and started the browser session to AWX I then saw the below (so the same as "before" the updated 17.1) it seems.

I re-checked openldap and libxml2.

# dnf list openldap libxml2Last metadata expiration check: 1:37:58 ago on Mon Feb  6 05:18:01 EST 2023.Installed Packageslibxml2.ppc                                                                               2.10.3-1                                                                               @AIX_Toolboxopenldap.ppc                                                                              2.5.12-1                                                                               @AIX_Toolbox#

Checking the /var/log/nginx/access.log I see the following (2nd line == screenshot above, 3rd line == once I clicked the "X" in screenshot above)

10.1.1.1 - - [06/Feb/2023:06:52:11 -0500] "GET / HTTP/1.1" 200 1082 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:12 -0500] "GET /api/v2/auth/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:13 -0500] "GET /api/ HTTP/1.1" 200 186 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

If I then enter "admin" and the password I get the following on the screen.

And this in the /var/log/nginx/access.log

10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "GET /api/login/ HTTP/1.1" 200 5720 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "POST /api/login/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

Any thoughts?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 03:19 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

can you please add:

disable_gpg_check: true

into dnf call in the playbook? The packages are not signed.

It should something like:

- name: install our packages  dnf:    name: "{{ copied | json_query('results[].dest') }}"    disable_gpg_check: true

in 02_packages.yml

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Fri February 03, 2023 11:10 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I had to play around with yum vs. dnf (I only have dnf loaded) and then hit the below issues:

TASK [copy our required packages] ***********************************************************************************************************************************************************changed: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)changed: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)changed: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)changed: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)changed: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)TASK [install our packages] *****************************************************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for linux_compat-0.1.2-1.ppc"}PLAY RECAP **********************************************************************************************************************************************************************************localhost                  : ok=10   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

31. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Andrey Klyachkin

Posted Thu February 09, 2023 12:03 PM

Steve,

I don't understand why it functions on 7.3.1.1 ;-) Please install both xmlsec and xmlsec-openssl RPMs in the same command.

------------------------------
Andrey Klyachkin

https://www.power-devops.com
------------------------------

Original Message

Original Message:
Sent: Thu February 09, 2023 11:37 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hmm, I'm finding at at 7.3.0.2 I have an issue which disappears when running 7.3.1.1, details below. On the 7.3.1.1 I had to do a reinstall as the packages were already loaded.

############################ On the 7.3.0.2 system ############################ oslevel -s7300-00-02-2220# lslpp -l | grep -i openss  CentrifyDC.openssl       5.8.1.371  COMMITTED  Centrify DirectControl OpenSSL  openssh.base.client   8.1.112.1201  COMMITTED  Open Secure Shell Commands  openssh.base.server   8.1.112.1201  COMMITTED  Open Secure Shell Server  openssh.license       8.1.112.1201  COMMITTED  Open Secure Shell License  openssh.man.en_US     8.1.112.1201  COMMITTED  Open Secure Shell  openssh.msg.EN_US     8.1.112.1201  COMMITTED  Open Secure Shell Messages -  openssh.msg.en_US     8.1.112.1201  COMMITTED  Open Secure Shell Messages -  openssl.base            3.0.7.1000  COMMITTED  Open Secure Socket Layer  openssl.license         3.0.7.1000  COMMITTED  Open Secure Socket License  openssl.man.en_US       3.0.7.1000  COMMITTED  Open Secure Socket Layer  openssh.base.client   8.1.112.1201  COMMITTED  Open Secure Shell Commands  openssh.base.server   8.1.112.1201  COMMITTED  Open Secure Shell Server  openssl.base            3.0.7.1000  COMMITTED  Open Secure Socket Layer# rpm -qa | grep openldapopenldap-2.5.12-1.ppcamrsomanim3:/# rpm -qa | grep libxml2libxml2-devel-2.10.3-1.ppclibxml2-2.10.3-1.ppclibxml2-python3-2.10.3-1.ppc# dnf install xmlsec1-1.2.28-2.aix7.2.ppc.rpmLast metadata expiration check: 21:38:34 ago on Wed Feb  8 12:19:57 EST 2023.Error: Problem: conflicting requests  - nothing provides libxmlsec1-openssl.a(libxmlsec1-openssl.so.1) needed by xmlsec1-1.2.28-2.ppc(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)# dnf install xmlsec1-openssl-1.2.28-2.aix7.2.ppc.rpmLast metadata expiration check: 21:39:26 ago on Wed Feb  8 12:19:57 EST 2023.Error: Problem: conflicting requests  - nothing provides libxmlsec1.a(libxmlsec1.so.1) needed by xmlsec1-openssl-1.2.28-2.ppc  - nothing provides xmlsec1 = 1.2.28 needed by xmlsec1-openssl-1.2.28-2.ppc(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)#I copied in the below (dated 7th Feb) from a 7.3.1.1 system where the install worked fine and get the issue with the originals and the "replacement" ones./opt/freeware/lib# ls -la libxml*-rwxr-xr-x    1 root     system      1620376 Feb 07 10:08 libxmlsec1-openssl.a-rwxr-xr-x    1 root     system      4371499 Jul 14 2020  libxmlsec1-openssl.a.20230209-rwxr-xr-x    1 root     system       742200 Feb 07 10:08 libxmlsec1-openssl.so.1-rwxr-xr-x    1 root     system      2101467 Jul 14 2020  libxmlsec1-openssl.so.1.20230209-rwxr-xr-x    1 root     system      2095470 Feb 07 10:08 libxmlsec1.a-rwxr-xr-x    1 root     system      3733085 Jul 14 2020  libxmlsec1.a.20230209-rwxr-xr-x    1 root     system       931620 Feb 07 10:08 libxmlsec1.so.1-rwxr-xr-x    1 root     system      1746787 Jul 14 2020  libxmlsec1.so.1.20230209/opt/freeware/lib# ===############################ On the 7.3.1.1 system ############################ oslevel -s7300-01-01-2246# lslpp -l | grep -i openss  CentrifyDC.openssl       5.8.1.371  COMMITTED  Centrify DirectControl OpenSSL  openssh.base.client   8.1.112.1201  COMMITTED  Open Secure Shell Commands  openssh.base.server   8.1.112.1201  COMMITTED  Open Secure Shell Server  openssh.license       8.1.112.1201  COMMITTED  Open Secure Shell License  openssh.man.en_US     8.1.112.1201  COMMITTED  Open Secure Shell  openssh.msg.EN_US     8.1.112.1201  COMMITTED  Open Secure Shell Messages -  openssh.msg.en_US     8.1.112.1201  COMMITTED  Open Secure Shell Messages -  openssl.base            3.0.7.1000  COMMITTED  Open Secure Socket Layer  openssl.license         3.0.7.1000  COMMITTED  Open Secure Socket License  openssl.man.en_US       3.0.7.1000  COMMITTED  Open Secure Socket Layer  openssh.base.client   8.1.112.1201  COMMITTED  Open Secure Shell Commands  openssh.base.server   8.1.112.1201  COMMITTED  Open Secure Shell Server  openssl.base            3.0.7.1000  COMMITTED  Open Secure Socket Layer# rpm -qa | grep openldapopenldap-2.5.12-1.ppc# rpm -qa | grep libxml2libxml2-2.10.3-1.ppclibxml2-devel-2.10.3-1.ppclibxml2-python3-2.10.3-1.ppc# dnf reinstall xmlsec1-1.2.28-2.aix7.2.ppc.rpmLast metadata expiration check: 23:12:26 ago on Wed Feb  8 12:13:38 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                      Architecture                             Version                                       Repository                                         Size=============================================================================================================================================================================================Reinstalling: xmlsec1                                      ppc                                      1.2.28-2                                      @commandline                                      1.2 MTransaction Summary=============================================================================================================================================================================================Total size: 1.2 MInstalled size: 4.2 MIs this ok [y/N]: nOperation aborted.# dnf reinstall xmlsec1-openssl-1.2.28-2.aix7.2.ppc.rpmLast metadata expiration check: 23:12:56 ago on Wed Feb  8 12:13:38 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                     Repository                                       Size=============================================================================================================================================================================================Reinstalling: xmlsec1-openssl                                    ppc                                    1.2.28-2                                    @commandline                                    777 kTransaction Summary=============================================================================================================================================================================================Total size: 777 kInstalled size: 3.1 MIs this ok [y/N]: nOperation aborted.

Is something slightly mis-aligned perhaps?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Thu February 09, 2023 09:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Stan,

do you install from IBM repos or from your local mirror?

The latest version of YUM in IBM repos is 3.4.3:

# /opt/freeware/bin/dnf info --all yumLast metadata expiration check: 0:03:28 ago on Thu Feb  9 14:33:17 2023.Available PackagesName         : yumVersion      : 3.4.3Release      : 8Architecture : noarchSize         : 1.3 MSource       : yum-3.4.3-8.src.rpmRepository   : AIX_Toolbox_noarchSummary      : RPM installer/updaterURL          : http://yum.baseurl.org/License      : GPLv2+Description  : Yum is a utility that can check for and automatically download and             : install updated RPM packages. Dependencies are obtained and downloaded             : automatically, prompting the user for permission as necessary.

Your output shows another version:

Problem 2: package yum-4.2.17-1.noarch requires dnf-data = 4.2.17-1, but none of the providers can be installed

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Thu February 09, 2023 08:57 AM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I received an error when using the updated install package.

TASK [update all packages] *******************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "failures": [], "msg": "Depsolve Error occured: \n Problem 1: cannot install both dnf-4.2.17-32_50.ppc and dnf-4.2.17-1.noarch\n - package yum-4.2.17-1.noarch requires dnf = 4.2.17-1, but none of the providers can be installed\n - cannot install the best update candidate for package dnf-4.2.17-1.noarch\n - problem with installed package yum-4.2.17-1.noarch\n Problem 2: package yum-4.2.17-1.noarch requires dnf-data = 4.2.17-1, but none of the providers can be installed\n - cannot install both dnf-data-4.2.17-32_50.ppc and dnf-data-4.2.17-1.noarch\n - cannot install the best update candidate for package yum-4.2.17-1.noarch\n - cannot install the best update candidate for package dnf-data-4.2.17-1.noarch", "rc": 1, "results": []}

Tried dfn update but got a similar error.

root@awx:/home/awx-install #dnf update
Last metadata expiration check: 0:07:58 ago on Thu Feb 9 07:44:29 CST 2023.
Error:
Problem 1: cannot install both dnf-4.2.17-32_50.ppc and dnf-4.2.17-1.noarch
- package yum-4.2.17-1.noarch requires dnf = 4.2.17-1, but none of the providers can be installed
- cannot install the best update candidate for package dnf-4.2.17-1.noarch
- problem with installed package yum-4.2.17-1.noarch
Problem 2: package yum-4.2.17-1.noarch requires dnf-data = 4.2.17-1, but none of the providers can be installed
- cannot install both dnf-data-4.2.17-32_50.ppc and dnf-data-4.2.17-1.noarch
- cannot install the best update candidate for package yum-4.2.17-1.noarch
- cannot install the best update candidate for package dnf-data-4.2.17-1.noarch
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Do I need to use the dnf update --allowerasing then attempt the AWX reinstall?

Thank you for your help!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu February 09, 2023 04:13 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Stanley,

the new files are available from https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz

If you do a new install, just unpack them and run ansible-playbook -i localhost, -c local awx_install.yml.

If you already have AWX 17.1.0 and want to update it, unpack the archive, stop nginx and supervisord, and update the 3 RPMs, I wrote above in the answer to Steve. Then start supervisord and nginx again. It should work.

Regarding your question about ssh-askpass, I have no idea why is it asking for it.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed February 08, 2023 02:15 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Where are the new AWX install files available?

The error I am getting is that the host is unreachable but ssh & ansible work from the command line with on errors.

mh-p9-nim | UNREACHABLE! => {

10

"changed": false,

11

"msg": "Failed to connect to the host via ssh: Control socket connect(/tmp/bwrap_244_vgwmryb1/awx_244_7s37x9b0/cp): Socket operation on non-socket\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nroot@mh-p9-nim: Permission denied (publickey,password,keyboard-interactive).",

12

"unreachable": true

I have searched and can't find where the file ssh-askpass should be installed from?

Can I reinstall the new AWX version over the existing AWX to refresh the install?

Thank you for your help!

Stan

------------------------------
Stanley

Original Message:
Sent: Wed February 08, 2023 10:56 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

I re-uploaded the file with new xmlssec RPMs and new playbook.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon February 06, 2023 12:33 PM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I tried to upload the /var/log/supervisor/awx-uwsgi.log however I get "Access denied" so have pasted it below.

# tail /var/log/supervisor/awx-uwsgi.log[pid: 13631776|app: 0|req: 4/15] 10.207.114.45 () {58 vars in 1037 bytes} [Mon Feb  6 11:57:54 2023] POST /api/login/ => generated 41 bytes in 83 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)##########################################################################################################<<< "Refreshed" the browser to re-create the initial "Error!  Failed to fetch custom login ..." message >>##########################################################################################################[pid: 2752938|app: 0|req: 5/16] 10.207.114.45 () {52 vars in 860 bytes} [Mon Feb  6 17:18:53 2023] GET / => generated 1082 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 168 bytes (1 switches on core 0)[pid: 13631776|app: 0|req: 5/17] 10.207.114.45 () {52 vars in 876 bytes} [Mon Feb  6 17:18:53 2023] GET /api/ => generated 186 bytes in 79 msecs (HTTP/1.1 200) 11 headers in 402 bytes (1 switches on core 0)2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.[pid: 11272526|app: 0|req: 2/18] 10.207.114.45 () {52 vars in 892 bytes} [Mon Feb  6 17:18:53 2023] GET /api/v2/auth/ => generated 41 bytes in 175 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)#

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 07:08 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for that, the install was able to run through without any further issues.

Once I completed the install and started the browser session to AWX I then saw the below (so the same as "before" the updated 17.1) it seems.

I re-checked openldap and libxml2.

# dnf list openldap libxml2Last metadata expiration check: 1:37:58 ago on Mon Feb  6 05:18:01 EST 2023.Installed Packageslibxml2.ppc                                                                               2.10.3-1                                                                               @AIX_Toolboxopenldap.ppc                                                                              2.5.12-1                                                                               @AIX_Toolbox#

Checking the /var/log/nginx/access.log I see the following (2nd line == screenshot above, 3rd line == once I clicked the "X" in screenshot above)

10.1.1.1 - - [06/Feb/2023:06:52:11 -0500] "GET / HTTP/1.1" 200 1082 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:12 -0500] "GET /api/v2/auth/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:13 -0500] "GET /api/ HTTP/1.1" 200 186 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

If I then enter "admin" and the password I get the following on the screen.

And this in the /var/log/nginx/access.log

10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "GET /api/login/ HTTP/1.1" 200 5720 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "POST /api/login/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

Any thoughts?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 03:19 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

can you please add:

disable_gpg_check: true

into dnf call in the playbook? The packages are not signed.

It should something like:

- name: install our packages  dnf:    name: "{{ copied | json_query('results[].dest') }}"    disable_gpg_check: true

in 02_packages.yml

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Fri February 03, 2023 11:10 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I had to play around with yum vs. dnf (I only have dnf loaded) and then hit the below issues:

TASK [copy our required packages] ***********************************************************************************************************************************************************changed: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)changed: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)changed: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)changed: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)changed: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)TASK [install our packages] *****************************************************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for linux_compat-0.1.2-1.ppc"}PLAY RECAP **********************************************************************************************************************************************************************************localhost                  : ok=10   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

32. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

IBM Champion

Steve Munday

Posted Thu February 09, 2023 01:27 PM

Andrey,

Hi, installing both at the same time overcame the issue. Odd really as I did one-at-a-time on my "Test" system.

Anyway, once that was done I found I had another oddity in that according to DNF I didn't have the linux-compatabilty, redis, supervisor, or ansible-awx installed when clearly they are as I'd been using them for many months! I can only think that "maybe" the YUM to DNF move de-listed them within the rpm DB.

No issue, I installed them however hit upon an "old" issue (the nofiles and -1) so used the workaround (set ulimit within ../init.d/supervisord) you suggested and that sorted things out. I'm now able to connect using AD accounts (again) :-)

Posted Thu September 17, 2020 09:09 AM
I just tested another solution, if you for some reasons won't want to change nofiles attribute.

You can the function start_app in /etc/rc.d/init.d/supervisord and add ulimit -n 65536 (or lower) before 
starting supervisord in it:

start_app() {
if [[ -f $PIDFILE ]] ; then
print -u2 -- "Supervisord is already started"
exit 1
fi
# workaround if root have too many allowed file descriptors
ulimit -n 65536
/opt/freeware/bin/supervisord -c /opt/freeware/etc/supervisord.conf
}

------------------------------
Andrey Klyachkin
------------------------------

So, all now sorted. THANK-YOU again for all your help with this.

Regards, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM
------------------------------

Original Message

Original Message:
Sent: Thu February 09, 2023 12:03 PM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

I don't understand why it functions on 7.3.1.1 ;-) Please install both xmlsec and xmlsec-openssl RPMs in the same command.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Thu February 09, 2023 11:37 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hmm, I'm finding at at 7.3.0.2 I have an issue which disappears when running 7.3.1.1, details below. On the 7.3.1.1 I had to do a reinstall as the packages were already loaded.

############################ On the 7.3.0.2 system ############################ oslevel -s7300-00-02-2220# lslpp -l | grep -i openss  CentrifyDC.openssl       5.8.1.371  COMMITTED  Centrify DirectControl OpenSSL  openssh.base.client   8.1.112.1201  COMMITTED  Open Secure Shell Commands  openssh.base.server   8.1.112.1201  COMMITTED  Open Secure Shell Server  openssh.license       8.1.112.1201  COMMITTED  Open Secure Shell License  openssh.man.en_US     8.1.112.1201  COMMITTED  Open Secure Shell  openssh.msg.EN_US     8.1.112.1201  COMMITTED  Open Secure Shell Messages -  openssh.msg.en_US     8.1.112.1201  COMMITTED  Open Secure Shell Messages -  openssl.base            3.0.7.1000  COMMITTED  Open Secure Socket Layer  openssl.license         3.0.7.1000  COMMITTED  Open Secure Socket License  openssl.man.en_US       3.0.7.1000  COMMITTED  Open Secure Socket Layer  openssh.base.client   8.1.112.1201  COMMITTED  Open Secure Shell Commands  openssh.base.server   8.1.112.1201  COMMITTED  Open Secure Shell Server  openssl.base            3.0.7.1000  COMMITTED  Open Secure Socket Layer# rpm -qa | grep openldapopenldap-2.5.12-1.ppcamrsomanim3:/# rpm -qa | grep libxml2libxml2-devel-2.10.3-1.ppclibxml2-2.10.3-1.ppclibxml2-python3-2.10.3-1.ppc# dnf install xmlsec1-1.2.28-2.aix7.2.ppc.rpmLast metadata expiration check: 21:38:34 ago on Wed Feb  8 12:19:57 EST 2023.Error: Problem: conflicting requests  - nothing provides libxmlsec1-openssl.a(libxmlsec1-openssl.so.1) needed by xmlsec1-1.2.28-2.ppc(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)# dnf install xmlsec1-openssl-1.2.28-2.aix7.2.ppc.rpmLast metadata expiration check: 21:39:26 ago on Wed Feb  8 12:19:57 EST 2023.Error: Problem: conflicting requests  - nothing provides libxmlsec1.a(libxmlsec1.so.1) needed by xmlsec1-openssl-1.2.28-2.ppc  - nothing provides xmlsec1 = 1.2.28 needed by xmlsec1-openssl-1.2.28-2.ppc(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)#I copied in the below (dated 7th Feb) from a 7.3.1.1 system where the install worked fine and get the issue with the originals and the "replacement" ones./opt/freeware/lib# ls -la libxml*-rwxr-xr-x    1 root     system      1620376 Feb 07 10:08 libxmlsec1-openssl.a-rwxr-xr-x    1 root     system      4371499 Jul 14 2020  libxmlsec1-openssl.a.20230209-rwxr-xr-x    1 root     system       742200 Feb 07 10:08 libxmlsec1-openssl.so.1-rwxr-xr-x    1 root     system      2101467 Jul 14 2020  libxmlsec1-openssl.so.1.20230209-rwxr-xr-x    1 root     system      2095470 Feb 07 10:08 libxmlsec1.a-rwxr-xr-x    1 root     system      3733085 Jul 14 2020  libxmlsec1.a.20230209-rwxr-xr-x    1 root     system       931620 Feb 07 10:08 libxmlsec1.so.1-rwxr-xr-x    1 root     system      1746787 Jul 14 2020  libxmlsec1.so.1.20230209/opt/freeware/lib# ===############################ On the 7.3.1.1 system ############################ oslevel -s7300-01-01-2246# lslpp -l | grep -i openss  CentrifyDC.openssl       5.8.1.371  COMMITTED  Centrify DirectControl OpenSSL  openssh.base.client   8.1.112.1201  COMMITTED  Open Secure Shell Commands  openssh.base.server   8.1.112.1201  COMMITTED  Open Secure Shell Server  openssh.license       8.1.112.1201  COMMITTED  Open Secure Shell License  openssh.man.en_US     8.1.112.1201  COMMITTED  Open Secure Shell  openssh.msg.EN_US     8.1.112.1201  COMMITTED  Open Secure Shell Messages -  openssh.msg.en_US     8.1.112.1201  COMMITTED  Open Secure Shell Messages -  openssl.base            3.0.7.1000  COMMITTED  Open Secure Socket Layer  openssl.license         3.0.7.1000  COMMITTED  Open Secure Socket License  openssl.man.en_US       3.0.7.1000  COMMITTED  Open Secure Socket Layer  openssh.base.client   8.1.112.1201  COMMITTED  Open Secure Shell Commands  openssh.base.server   8.1.112.1201  COMMITTED  Open Secure Shell Server  openssl.base            3.0.7.1000  COMMITTED  Open Secure Socket Layer# rpm -qa | grep openldapopenldap-2.5.12-1.ppc# rpm -qa | grep libxml2libxml2-2.10.3-1.ppclibxml2-devel-2.10.3-1.ppclibxml2-python3-2.10.3-1.ppc# dnf reinstall xmlsec1-1.2.28-2.aix7.2.ppc.rpmLast metadata expiration check: 23:12:26 ago on Wed Feb  8 12:13:38 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                      Architecture                             Version                                       Repository                                         Size=============================================================================================================================================================================================Reinstalling: xmlsec1                                      ppc                                      1.2.28-2                                      @commandline                                      1.2 MTransaction Summary=============================================================================================================================================================================================Total size: 1.2 MInstalled size: 4.2 MIs this ok [y/N]: nOperation aborted.# dnf reinstall xmlsec1-openssl-1.2.28-2.aix7.2.ppc.rpmLast metadata expiration check: 23:12:56 ago on Wed Feb  8 12:13:38 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                     Repository                                       Size=============================================================================================================================================================================================Reinstalling: xmlsec1-openssl                                    ppc                                    1.2.28-2                                    @commandline                                    777 kTransaction Summary=============================================================================================================================================================================================Total size: 777 kInstalled size: 3.1 MIs this ok [y/N]: nOperation aborted.

Is something slightly mis-aligned perhaps?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Thu February 09, 2023 09:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Stan,

do you install from IBM repos or from your local mirror?

The latest version of YUM in IBM repos is 3.4.3:

# /opt/freeware/bin/dnf info --all yumLast metadata expiration check: 0:03:28 ago on Thu Feb  9 14:33:17 2023.Available PackagesName         : yumVersion      : 3.4.3Release      : 8Architecture : noarchSize         : 1.3 MSource       : yum-3.4.3-8.src.rpmRepository   : AIX_Toolbox_noarchSummary      : RPM installer/updaterURL          : http://yum.baseurl.org/License      : GPLv2+Description  : Yum is a utility that can check for and automatically download and             : install updated RPM packages. Dependencies are obtained and downloaded             : automatically, prompting the user for permission as necessary.

Your output shows another version:

Problem 2: package yum-4.2.17-1.noarch requires dnf-data = 4.2.17-1, but none of the providers can be installed

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Thu February 09, 2023 08:57 AM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I received an error when using the updated install package.

TASK [update all packages] *******************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "failures": [], "msg": "Depsolve Error occured: \n Problem 1: cannot install both dnf-4.2.17-32_50.ppc and dnf-4.2.17-1.noarch\n - package yum-4.2.17-1.noarch requires dnf = 4.2.17-1, but none of the providers can be installed\n - cannot install the best update candidate for package dnf-4.2.17-1.noarch\n - problem with installed package yum-4.2.17-1.noarch\n Problem 2: package yum-4.2.17-1.noarch requires dnf-data = 4.2.17-1, but none of the providers can be installed\n - cannot install both dnf-data-4.2.17-32_50.ppc and dnf-data-4.2.17-1.noarch\n - cannot install the best update candidate for package yum-4.2.17-1.noarch\n - cannot install the best update candidate for package dnf-data-4.2.17-1.noarch", "rc": 1, "results": []}

Tried dfn update but got a similar error.

root@awx:/home/awx-install #dnf update
Last metadata expiration check: 0:07:58 ago on Thu Feb 9 07:44:29 CST 2023.
Error:
Problem 1: cannot install both dnf-4.2.17-32_50.ppc and dnf-4.2.17-1.noarch
- package yum-4.2.17-1.noarch requires dnf = 4.2.17-1, but none of the providers can be installed
- cannot install the best update candidate for package dnf-4.2.17-1.noarch
- problem with installed package yum-4.2.17-1.noarch
Problem 2: package yum-4.2.17-1.noarch requires dnf-data = 4.2.17-1, but none of the providers can be installed
- cannot install both dnf-data-4.2.17-32_50.ppc and dnf-data-4.2.17-1.noarch
- cannot install the best update candidate for package yum-4.2.17-1.noarch
- cannot install the best update candidate for package dnf-data-4.2.17-1.noarch
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Do I need to use the dnf update --allowerasing then attempt the AWX reinstall?

Thank you for your help!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu February 09, 2023 04:13 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Stanley,

the new files are available from https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz

If you do a new install, just unpack them and run ansible-playbook -i localhost, -c local awx_install.yml.

If you already have AWX 17.1.0 and want to update it, unpack the archive, stop nginx and supervisord, and update the 3 RPMs, I wrote above in the answer to Steve. Then start supervisord and nginx again. It should work.

Regarding your question about ssh-askpass, I have no idea why is it asking for it.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed February 08, 2023 02:15 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Where are the new AWX install files available?

The error I am getting is that the host is unreachable but ssh & ansible work from the command line with on errors.

mh-p9-nim | UNREACHABLE! => {

10

"changed": false,

11

"msg": "Failed to connect to the host via ssh: Control socket connect(/tmp/bwrap_244_vgwmryb1/awx_244_7s37x9b0/cp): Socket operation on non-socket\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nroot@mh-p9-nim: Permission denied (publickey,password,keyboard-interactive).",

12

"unreachable": true

I have searched and can't find where the file ssh-askpass should be installed from?

Can I reinstall the new AWX version over the existing AWX to refresh the install?

Thank you for your help!

Stan

------------------------------
Stanley

Original Message:
Sent: Wed February 08, 2023 10:56 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

I re-uploaded the file with new xmlssec RPMs and new playbook.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon February 06, 2023 12:33 PM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I tried to upload the /var/log/supervisor/awx-uwsgi.log however I get "Access denied" so have pasted it below.

# tail /var/log/supervisor/awx-uwsgi.log[pid: 13631776|app: 0|req: 4/15] 10.207.114.45 () {58 vars in 1037 bytes} [Mon Feb  6 11:57:54 2023] POST /api/login/ => generated 41 bytes in 83 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)##########################################################################################################<<< "Refreshed" the browser to re-create the initial "Error!  Failed to fetch custom login ..." message >>##########################################################################################################[pid: 2752938|app: 0|req: 5/16] 10.207.114.45 () {52 vars in 860 bytes} [Mon Feb  6 17:18:53 2023] GET / => generated 1082 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 168 bytes (1 switches on core 0)[pid: 13631776|app: 0|req: 5/17] 10.207.114.45 () {52 vars in 876 bytes} [Mon Feb  6 17:18:53 2023] GET /api/ => generated 186 bytes in 79 msecs (HTTP/1.1 200) 11 headers in 402 bytes (1 switches on core 0)2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.[pid: 11272526|app: 0|req: 2/18] 10.207.114.45 () {52 vars in 892 bytes} [Mon Feb  6 17:18:53 2023] GET /api/v2/auth/ => generated 41 bytes in 175 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)#

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 07:08 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for that, the install was able to run through without any further issues.

Once I completed the install and started the browser session to AWX I then saw the below (so the same as "before" the updated 17.1) it seems.

I re-checked openldap and libxml2.

# dnf list openldap libxml2Last metadata expiration check: 1:37:58 ago on Mon Feb  6 05:18:01 EST 2023.Installed Packageslibxml2.ppc                                                                               2.10.3-1                                                                               @AIX_Toolboxopenldap.ppc                                                                              2.5.12-1                                                                               @AIX_Toolbox#

Checking the /var/log/nginx/access.log I see the following (2nd line == screenshot above, 3rd line == once I clicked the "X" in screenshot above)

10.1.1.1 - - [06/Feb/2023:06:52:11 -0500] "GET / HTTP/1.1" 200 1082 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:12 -0500] "GET /api/v2/auth/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:13 -0500] "GET /api/ HTTP/1.1" 200 186 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

If I then enter "admin" and the password I get the following on the screen.

And this in the /var/log/nginx/access.log

10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "GET /api/login/ HTTP/1.1" 200 5720 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "POST /api/login/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

Any thoughts?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 03:19 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

can you please add:

disable_gpg_check: true

into dnf call in the playbook? The packages are not signed.

It should something like:

- name: install our packages  dnf:    name: "{{ copied | json_query('results[].dest') }}"    disable_gpg_check: true

in 02_packages.yml

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Fri February 03, 2023 11:10 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I had to play around with yum vs. dnf (I only have dnf loaded) and then hit the below issues:

TASK [copy our required packages] ***********************************************************************************************************************************************************changed: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)changed: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)changed: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)changed: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)changed: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)TASK [install our packages] *****************************************************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for linux_compat-0.1.2-1.ppc"}PLAY RECAP **********************************************************************************************************************************************************************************localhost                  : ok=10   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

33. RE: AWX fails on AIX 7200-05-05-2246 after yum update

Like

Stanley Speegle

Posted Thu February 09, 2023 01:35 PM

Andrey,

Thank you for the the updated Ansible-playbook and files.

These are the steps performed to get AWX working.

Manually removed the old yum version with "dnf erase old yum-4.2.17-1"
Run dnf update
/etc/rc.d/init.d/supervisord stop
/etc/rc.d/init.d/redis stop
/etc/rc.d/init.d/nginx stop
/etc/rc.d/init.d/postgresql stop
run ansible-playbook -i localhost, -c local awx_install.yml
Create user postgres with ID 64200 - required because the new postgres files are owned by 64200
/etc/rc.d/init.d/postgresql start
/etc/rc.d/init.d/redis start
/etc/rc.d/init.d/supervisord start
/etc/rc.d/init.d/nginx start
Verify AWX login and run ad-hoc shell commands.
Reboot LPAR to verify all is still working after reboot.

AWX looks to be working correctly now!

I see an issue with gcc10 missing files but AWX is now working.

root@awx:/ #rpm -V gcc10 | grep missing
missing /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/10/include-fixed/README
missing /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/10/include-fixed/ctype.h
missing /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/10/include-fixed/langinfo.h
missing /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/10/include-fixed/ldfcn.h
missing /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/10/include-fixed/rpc
missing /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/10/include-fixed/rpc/svc.h
missing /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/10/include-fixed/sys/inttypes.h
missing /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/10/include-fixed/sys/localedef.h
missing /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/10/include-fixed/sys/socket.h
missing /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/10/include-fixed/sys/times.h

Thanks again for all your help!

Have a great day!

Stan

------------------------------
Stanley
------------------------------

Original Message

Original Message:
Sent: Thu February 09, 2023 04:13 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Stanley,

the new files are available from https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz

If you do a new install, just unpack them and run ansible-playbook -i localhost, -c local awx_install.yml.

If you already have AWX 17.1.0 and want to update it, unpack the archive, stop nginx and supervisord, and update the 3 RPMs, I wrote above in the answer to Steve. Then start supervisord and nginx again. It should work.

Regarding your question about ssh-askpass, I have no idea why is it asking for it.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed February 08, 2023 02:15 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Where are the new AWX install files available?

The error I am getting is that the host is unreachable but ssh & ansible work from the command line with on errors.

mh-p9-nim | UNREACHABLE! => {

10

"changed": false,

11

"msg": "Failed to connect to the host via ssh: Control socket connect(/tmp/bwrap_244_vgwmryb1/awx_244_7s37x9b0/cp): Socket operation on non-socket\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nPermission denied, please try again.\r\nssh_askpass: exec(/usr/sbin/ssh-askpass): No such file or directory\r\nroot@mh-p9-nim: Permission denied (publickey,password,keyboard-interactive).",

12

"unreachable": true

I have searched and can't find where the file ssh-askpass should be installed from?

Can I reinstall the new AWX version over the existing AWX to refresh the install?

Thank you for your help!

Stan

------------------------------
Stanley

Original Message:
Sent: Wed February 08, 2023 10:56 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

I re-uploaded the file with new xmlssec RPMs and new playbook.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon February 06, 2023 12:33 PM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I tried to upload the /var/log/supervisor/awx-uwsgi.log however I get "Access denied" so have pasted it below.

# tail /var/log/supervisor/awx-uwsgi.log[pid: 13631776|app: 0|req: 4/15] 10.207.114.45 () {58 vars in 1037 bytes} [Mon Feb  6 11:57:54 2023] POST /api/login/ => generated 41 bytes in 83 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)##########################################################################################################<<< "Refreshed" the browser to re-create the initial "Error!  Failed to fetch custom login ..." message >>##########################################################################################################[pid: 2752938|app: 0|req: 5/16] 10.207.114.45 () {52 vars in 860 bytes} [Mon Feb  6 17:18:53 2023] GET / => generated 1082 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 168 bytes (1 switches on core 0)[pid: 13631776|app: 0|req: 5/17] 10.207.114.45 () {52 vars in 876 bytes} [Mon Feb  6 17:18:53 2023] GET /api/ => generated 186 bytes in 79 msecs (HTTP/1.1 200) 11 headers in 402 bytes (1 switches on core 0)2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.2023-02-06 17:18:54,097 ERROR    django.request Internal Server Error: /api/v2/auth/Traceback (most recent call last):  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner    response = get_response(request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response    response = self.process_exception_by_middleware(e, request)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response    response = wrapped_callback(request, *callback_args, **callback_kwargs)  File "/opt/freeware/lib64/python3.7/contextlib.py", line 74, in inner    return func(*args, **kwds)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view    return view_func(*args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view    return self.dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/generics.py", line 328, in dispatch    return super(APIView, self).dispatch(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 509, in dispatch    response = self.handle_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 469, in handle_exception    self.raise_uncaught_exception(exc)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception    raise exc  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/rest_framework/views.py", line 506, in dispatch    response = handler(request, *args, **kwargs)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/api/views/__init__.py", line 546, in get    auth_backends = list(load_backends(settings.AUTHENTICATION_BACKENDS, force_load=True).items())  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/utils.py", line 35, in load_backends    backend = module_member(auth_backend)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 62, in module_member    module = import_module(mod)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/utils.py", line 56, in import_module    __import__(name)  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/sso/backends.py", line 31, in <module>    from social_core.backends.saml import OID_USERID  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/social_core/backends/saml.py", line 10, in <module>    from onelogin.saml2.auth import OneLogin_Saml2_Auth  File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/onelogin/saml2/auth.py", line 14, in <module>    import xmlsecImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-021 Additional errors occurred but are not reported.        0509-192 Examine .loader section symbols with the                 'dump -Tv' command.[pid: 11272526|app: 0|req: 2/18] 10.207.114.45 () {52 vars in 892 bytes} [Mon Feb  6 17:18:53 2023] GET /api/v2/auth/ => generated 41 bytes in 175 msecs (HTTP/1.1 500) 5 headers in 185 bytes (1 switches on core 0)#

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 07:08 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for that, the install was able to run through without any further issues.

Once I completed the install and started the browser session to AWX I then saw the below (so the same as "before" the updated 17.1) it seems.

I re-checked openldap and libxml2.

# dnf list openldap libxml2Last metadata expiration check: 1:37:58 ago on Mon Feb  6 05:18:01 EST 2023.Installed Packageslibxml2.ppc                                                                               2.10.3-1                                                                               @AIX_Toolboxopenldap.ppc                                                                              2.5.12-1                                                                               @AIX_Toolbox#

Checking the /var/log/nginx/access.log I see the following (2nd line == screenshot above, 3rd line == once I clicked the "X" in screenshot above)

10.1.1.1 - - [06/Feb/2023:06:52:11 -0500] "GET / HTTP/1.1" 200 1082 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:12 -0500] "GET /api/v2/auth/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:52:13 -0500] "GET /api/ HTTP/1.1" 200 186 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

If I then enter "admin" and the password I get the following on the screen.

And this in the /var/log/nginx/access.log

10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "GET /api/login/ HTTP/1.1" 200 5720 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"10.1.1.1 - - [06/Feb/2023:06:57:54 -0500] "POST /api/login/ HTTP/1.1" 500 41 "https://x.y.z/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0" "-"#

Any thoughts?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon February 06, 2023 03:19 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Steve,

can you please add:

disable_gpg_check: true

into dnf call in the playbook? The packages are not signed.

It should something like:

- name: install our packages  dnf:    name: "{{ copied | json_query('results[].dest') }}"    disable_gpg_check: true

in 02_packages.yml

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Fri February 03, 2023 11:10 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

I had to play around with yum vs. dnf (I only have dnf loaded) and then hit the below issues:

TASK [copy our required packages] ***********************************************************************************************************************************************************changed: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)changed: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)changed: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)changed: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)changed: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)TASK [install our packages] *****************************************************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to validate GPG signature for linux_compat-0.1.2-1.ppc"}PLAY RECAP **********************************************************************************************************************************************************************************localhost                  : ok=10   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 09:28 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Thanks for the link (FYI, I had to use https://downloads.power-devops.com rather than https://dl.power-devops.com), I'll stand up a separate AWX with the latest rebuild and see how I get on.

Out of interest, are there any plans to roll out any new(er) builds of AWX on AIX?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri February 03, 2023 08:49 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

I uploaded a new AWX rebuild, but didn't test it. If you want, you can download it here - https://dl.power-devops.com/ansible-awx-17.1.0-2.aix7.2.install.tar.gz. It is the same AWX version, but built with newer IBM RPMs (incl. libxml2 and openldap).

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:52 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Andrey,

Hi there, thanks for your reply. So, do you know when that might be possible?

Also, would a downgrade to an older version of openldap act as a workaround? I moved back to openldap-2.4.58-4.ppc (the various "missing" files are now available again) however I still get the Error 500 message having stopped/started AWX :-(

# rpm -qa | grep -i openlopenldap-2.4.58-4.ppc# ls -la /opt/freeware/lib/libldap_r.alrwxrwxrwx    1 root     system           15 Jan 30 11:13 /opt/freeware/lib/libldap_r.a -> libldap_r-2.4.a# ls -la /opt/freeware/lib | grep -i libldap-rwxr-xr-x    1 root     system      1122886 Oct 13 07:12 libldap-2.4.a-rwxr-xr-x    1 root     system       506822 Oct 13 07:12 libldap-2.4.so.2lrwxrwxrwx    1 root     system           13 Jan 30 11:13 libldap.a -> libldap-2.4.a-rwxr-xr-x    1 root     system      1256736 Oct 13 07:12 libldap_r-2.4.a-rwxr-xr-x    1 root     system       564875 Oct 13 07:12 libldap_r-2.4.so.2lrwxrwxrwx    1 root     system           15 Jan 30 11:13 libldap_r.a -> libldap_r-2.4.a#

** Further update **
I've managed to workaround the issue by reverting to a slightly older version of libxml2 (as well as reverting openldap back to 2.4.58-4).

# tail -n 50 pgstartup.logImportError:    0509-130 Symbol resolution failed for /opt/freeware/lib/libxmlsec1.a[libxmlsec1.so.1] because:        0509-136   Symbol xmlIOFTPClose (number 17) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPRead (number 18) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPOpen (number 19) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlIOFTPMatch (number 20) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPInit (number 117) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].        0509-136   Symbol xmlNanoFTPCleanup (number 118) is not exported from                   dependent module /opt/freeware/lib/libxml2.a[libxml2.so.2].# dnf downgrade libxml2-2.9.14-1Last metadata expiration check: 7:32:12 ago on Mon Jan 30 04:38:40 EST 2023.Dependencies resolved.============================================================================================================================================================================================= Package                                            Architecture                           Version                                      Repository                                      Size=============================================================================================================================================================================================Downgrading: libxml2                                            ppc                                    2.9.14-1                                     AIX_Toolbox                                    5.3 M libxml2-devel                                      ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.5 M libxml2-python3                                    ppc                                    2.9.14-1                                     AIX_Toolbox                                    1.4 MTransaction Summary=============================================================================================================================================================================================Downgrade  3 Packages. . . .# rpm -qa | grep -i libxml2libxml2-2.9.14-1.ppclibxml2-python3-2.9.14-1.ppclibxml2-devel-2.9.14-1.ppc

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Mon January 30, 2023 11:39 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Steve,

_ldap.so is a part of AWX package (/var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so). It means I have to create a new version of the package built with new IBM RPMs.

Regards
Andrey

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Mon January 30, 2023 11:25 AM
From: Steve Munday
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Ayappan,

Re: "I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.".

This is new to me, how should I go about the rebuild?

Also, would a dnf downgrade openldap-x.y.z resolve the issue?

Many thanks, Steve

------------------------------
Steve Munday
AIX, IBM i, HMC, PowerVM

Original Message:
Sent: Fri January 27, 2023 02:31 AM
From: Ayappan P
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

In openldap 2.5.X, there is no libldap_r.a library. The community has merged it with libldap.a. So now there is only libldap.a library.
I think you need to rebuild that "_ldap.so" python3 module with the new openldap & openldap-devel 2.5.X installed.

------------------------------
Ayappan P

Original Message:
Sent: Thu January 26, 2023 01:11 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

Found this in the awx-uwsgi.log

ImportError: Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module libldap_r.a(libldap_r-2.4.so.2) could not be loaded.
Could not load module libldap_r.a(libldap_r-2.4.so.2).
System error: No such file or directory
Could not load module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so.
Dependent module /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so could not be loaded.
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
goodbye to uWSGI.

The file _ldap.so exists.
ls -l /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so
-rwxr-xr-x 1 root system 213145 Mar 10 2021 /var/lib/awx/venv/awx/lib/python3.7/site-packages/_ldap.so

But these are not found on the system.
find / -name libldap_r.a
find / -name libldap_r-2.4.so.2

Maybe the files were removed by the yum upgrade

Found the files on a system that was not upgraded and scp them back to the awx system
scp server1:/opt/freeware/lib/libldap_r.a /opt/freeware/lib/libldap_r.a
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2
scp server1:/opt/freeware/lib/libldap_r-2.4.so.2 /opt/freeware/lib/libldap_r-2.4.so.2

I can now get to the login screen but the login fails with this message:

Failed to fetch custom login configuration settings. System defaults will be shown instead.

Details
GET /api/v2/auth/ 500
A server error has occurred.

Then the login fails with this:
There was a problem signing in. Please try again.

Thank you for any assistance you can provide!

Stan

------------------------------
Stanley

Original Message:
Sent: Thu January 26, 2023 04:34 AM
From: Andrey Klyachkin
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Stanley,

was postgresql removed during your action? If it is, you may need to create your database structures again.

------------------------------
Andrey Klyachkin

https://www.power-devops.com

Original Message:
Sent: Wed January 25, 2023 01:31 PM
From: Stanley Speegle
Subject: AWX fails on AIX 7200-05-05-2246 after yum update

Hi Team,

AWX 17.1 was working until yum update --allowerasing was ran to update the installed packages. The --allowerasing was needed because of conflicting packages being updated. The yum update worked and yum check is clean.

The first item was the postgres DB failed to start with an error "postgres user does not exist" Created a system user account postgres and group postgres with ID 64200 as that is the number showing as owner of the postgres database directories. Looks like the new version of postgresql.ppc 11.18-1 requires a postgres user account to start up. Now the database is up and running.

/etc/rc.d/init.d/postgresql status
PostgreSQL DB is running with PID 12452296.

The Nginx web page is up but the AWX login page shows "internal server error"

The supervisord.log shows awx-dispatcher, awx-wsbroadcast & awx-daphne spawning and exiting over and over.

supervisord.log
2023-01-25 10:58:26,834 INFO Included extra file "/opt/freeware/etc/supervisord.d/awx.ini" during parsing
2023-01-25 10:58:26,834 INFO Set uid to user 0 succeeded
2023-01-25 10:58:26,834 INFO Increased RLIMIT_NPROC limit to 200
2023-01-25 10:58:26,849 INFO RPC interface 'supervisor' initialized
2023-01-25 10:58:26,849 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-01-25 10:58:26,858 INFO daemonizing the supervisord process
2023-01-25 10:58:26,862 INFO supervisord started with pid 11665866
2023-01-25 10:58:27,886 INFO spawned: 'awx-config-watcher' with pid 7143892
2023-01-25 10:58:27,908 INFO spawned: 'awx-uwsgi' with pid 6226256
2023-01-25 10:58:27,946 INFO spawned: 'awx-daphne' with pid 12190058
2023-01-25 10:58:27,985 INFO spawned: 'awx-wsbroadcast' with pid 11927938
2023-01-25 10:58:28,011 INFO spawned: 'awx-rsyslogd' with pid 11600364
2023-01-25 10:58:28,087 INFO spawned: 'awx-dispatcher' with pid 14549330
2023-01-25 10:58:28,147 INFO spawned: 'awx-callback-receiver' with pid 12648936
2023-01-25 10:58:29,499 INFO success: awx-config-watcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-wsbroadcast entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,499 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:29,825 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:29,915 INFO spawned: 'awx-wsbroadcast' with pid 11927940
2023-01-25 10:58:29,943 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:30,013 INFO spawned: 'awx-callback-receiver' with pid 12648938
2023-01-25 10:58:30,140 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:30,168 INFO spawned: 'awx-dispatcher' with pid 14549332
2023-01-25 10:58:30,715 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:30,761 INFO spawned: 'awx-daphne' with pid 9437502
2023-01-25 10:58:30,762 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:31,013 INFO success: awx-callback-receiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,013 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:31,058 INFO spawned: 'awx-callback-receiver' with pid 12648940
2023-01-25 10:58:31,060 INFO exited: awx-dispatcher (exit status 1; not expected)
2023-01-25 10:58:31,820 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:31,826 INFO spawned: 'awx-wsbroadcast' with pid 11927942
2023-01-25 10:58:32,047 INFO exited: awx-callback-receiver (exit status 1; not expected)
2023-01-25 10:58:32,157 INFO spawned: 'awx-dispatcher' with pid 12648942
2023-01-25 10:58:32,609 INFO exited: awx-daphne (exit status 1; not expected)
2023-01-25 10:58:32,653 INFO spawned: 'awx-daphne' with pid 9437504
2023-01-25 10:58:32,655 INFO exited: awx-wsbroadcast (exit status 1; not expected)
2023-01-25 10:58:33,929 INFO success: awx-daphne entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,929 INFO success: awx-dispatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-01-25 10:58:33,937 INFO spawned: 'awx-callback-receiver' with pid 11927944
2023-01-25 10:58:33,938 INFO exited: awx-dispatcher (exit status 1; not expected)

Here is the list of the installed packages. They are all from the AIX Linux-toolbox.

/opt/freeware/bin/yum list installed
Installed Packages
AIX-rpm.ppc 7.2.5.103-13 @System
GeoIP.ppc 1.6.12-1 @AIX_Toolbox
ansible.noarch 2.9.25-1 @AIX_Toolbox_noarch
ansible-awx.ppc 17.1.0-1aix @/ansible-awx-17.1.0-1aix.aix7.2.ppc
apr.ppc 1.7.0-1 @AIX_Toolbox
apr-util.ppc 1.6.1-1 @AIX_Toolbox
autoconf.noarch 2.71-1 @AIX_Toolbox_noarch
automake.noarch 1.16.5-1 @AIX_Toolbox_noarch
bash.ppc 5.1.16-1 @AIX_Toolbox
bzip2.ppc 1.0.8-2 @AIX_Toolbox
ca-certificates.ppc 2021.2.52-3 @AIX_Toolbox
check.ppc 0.13.0-1 @AIX_Toolbox
clamav.ppc 0.104.2-3 @AIX_Toolbox
cloud-init.ppc 0.7.5-4.5 @AIX_Toolbox
curl.ppc 7.86.0-1 @AIX_Toolbox
cyrus-sasl.ppc 2.1.28-1 @AIX_Toolbox
db.ppc 1:5.3.28-1 @AIX_Toolbox
diffutils.ppc 3.8-1 @AIX_Toolbox
dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-automatic.ppc 4.2.17-32_50 @AIX_Toolbox_72
dnf-data.ppc 4.2.17-32_50 @AIX_Toolbox_72
expat.ppc 2.5.0-1 @AIX_Toolbox
expect.ppc 5.45.4-2 @AIX_Toolbox
file.ppc 5.41-1 @AIX_Toolbox
file-libs.ppc 5.41-1 @AIX_Toolbox
fontconfig.ppc 2.11.95-4 @AIX_Toolbox
freetype2.ppc 2.12.1-1 @AIX_Toolbox
gcc.ppc 1:10-2 @AIX_Toolbox_72
gcc-c++.ppc 1:10-2 @AIX_Toolbox_72
gcc-cpp.ppc 1:10-2 @AIX_Toolbox_72
gcc10.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-c++.ppc 10.3.0-6 @AIX_Toolbox_72
gcc10-cpp.ppc 10.3.0-6 @AIX_Toolbox_72
gcc8.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-c++.ppc 8.3.0-6 @AIX_Toolbox_72
gcc8-cpp.ppc 8.3.0-6 @AIX_Toolbox_72
gdbm.ppc 1.23-1 @AIX_Toolbox
gettext.ppc 0.21-2 @AIX_Toolbox
git.ppc 2.38.3-1 @AIX_Toolbox
git-core.ppc 2.38.3-1 @AIX_Toolbox
git-core-doc.noarch 2.38.3-1 @AIX_Toolbox_noarch
glib2.ppc 2.56.1-3 @AIX_Toolbox
gmp.ppc 6.2.1-1 @AIX_Toolbox
gnupg2.ppc 2.2.35-1 @AIX_Toolbox
gnutls.ppc 3.7.2-1 @AIX_Toolbox
golang.ppc 1.18.7-1 @AIX_Toolbox_72
golang-bin.ppc 1.18.7-1 @AIX_Toolbox_72
golang-src.noarch 1.18.7-1 @AIX_Toolbox_noarch
gpgme.ppc 1.13.1-100 @AIX_Toolbox
grep.ppc 3.7-2 @AIX_Toolbox
info.ppc 6.7-1 @AIX_Toolbox
jbigkit-libs.ppc 2.1-1 @AIX_Toolbox
json-c.ppc 0.15-2 @AIX_Toolbox
krb5-libs.ppc 1.18.5-2 @AIX_Toolbox
less.ppc 598-1 @AIX_Toolbox
libXft.ppc 2.3.2-4waixX11 @AIX_Toolbox
libXrender.ppc 0.9.8-3waixX11 @AIX_Toolbox
libassuan.ppc 2.5.5-1 @AIX_Toolbox
libcomps.ppc 0.1.11-101 @AIX_Toolbox
libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
libffi.ppc 3.4.2-1 @AIX_Toolbox
libffi-devel.ppc 3.4.2-1 @AIX_Toolbox
libgcc.ppc 1:10-2 @AIX_Toolbox_72
libgcc10.ppc 10.3.0-6 @AIX_Toolbox_72
libgcc8.ppc 8.3.0-6 @AIX_Toolbox_72
libgcrypt.ppc 1.9.4-1 @AIX_Toolbox
libgd.ppc 2.3.3-1 @AIX_Toolbox
libgomp.ppc 1:10-2 @AIX_Toolbox_72
libgomp10.ppc 10.3.0-6 @AIX_Toolbox_72
libgomp8.ppc 8.3.0-6 @AIX_Toolbox_72
libgpg-error.ppc 1.45-1 @AIX_Toolbox
libiconv.ppc 1.17-1 @AIX_Toolbox
libjpeg.ppc 9e-1 @AIX_Toolbox
libksba.ppc 1.6.2-1 @AIX_Toolbox
libmodulemd.ppc 1.5.2-100 @AIX_Toolbox
libmpc.ppc 1.2.1-1 @AIX_Toolbox
libnghttp2.ppc 1.46.0-1 @AIX_Toolbox
libpng.ppc 1.6.37-1 @AIX_Toolbox
librepo.ppc 1.11.0-101 @AIX_Toolbox
libsmartcols.ppc 2.34-100 @AIX_Toolbox
libsodium.ppc 1.0.18-1 @AIX_Toolbox
libsolv.ppc 0.7.9-32_51 @AIX_Toolbox_72
libssh2.ppc 1.10.0-2 @AIX_Toolbox
libstdc++.ppc 1:10-2 @AIX_Toolbox_72
libstdc++10.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++10-devel.ppc 10.3.0-6 @AIX_Toolbox_72
libstdc++8.ppc 8.3.0-6 @AIX_Toolbox_72
libstdc++8-devel.ppc 8.3.0-6 @AIX_Toolbox_72
libtasn1.ppc 4.16.0-1 @AIX_Toolbox
libtextstyle.ppc 0.21-2 @AIX_Toolbox
libtiff.ppc 4.4.0-1 @AIX_Toolbox
libtool.ppc 2.4.7-1 @AIX_Toolbox
libtool-ltdl.ppc 2.4.7-1 @AIX_Toolbox
libunistring.ppc 0.9.10-1 @AIX_Toolbox
libwebp.ppc 1.0.2-1 @AIX_Toolbox
libxml2.ppc 2.10.3-1 @AIX_Toolbox
libxslt.ppc 1.1.37-1 @AIX_Toolbox
libyaml.ppc 0.2.5-1 @AIX_Toolbox
libzstd.ppc 1.5.2-1 @AIX_Toolbox
linux_compat.ppc 0.1.2-1 @/linux_compat-0.1.2-1.aix7.2.ppc
lz4.ppc 1.9.3-1 @AIX_Toolbox
m4.ppc 1.4.18-1 @AIX_Toolbox
mpfr.ppc 4.1.0-1 @AIX_Toolbox
ncurses.ppc 6.3-1 @AIX_Toolbox
nettle.ppc 3.7.3-1 @AIX_Toolbox
nginx.ppc 1.21.4-2 @AIX_Toolbox
npth.ppc 1.5-1 @AIX_Toolbox
openldap.ppc 2.5.12-1 @AIX_Toolbox
osinstall.noarch 1.6-1 @System
p11-kit.ppc 0.23.22-1 @AIX_Toolbox
p11-kit-tools.ppc 0.23.22-1 @AIX_Toolbox
pcre.ppc 8.44-2 @AIX_Toolbox
pcre2.ppc 10.40-1 @AIX_Toolbox
perl.ppc 1:5.34.1-1 @AIX_Toolbox
pinentry.ppc 1.0.0-1 @AIX_Toolbox
pkg-config.ppc 0.29.2-2 @AIX_Toolbox
postgresql.ppc 11.18-1 @AIX_Toolbox
postgresql-libs.ppc 11.18-1 @AIX_Toolbox
postgresql-server.ppc 11.18-1 @AIX_Toolbox
pysqlite.ppc 2.8.3-2 @AIX_Toolbox
python.ppc 2.7.18-4 @AIX_Toolbox
python-PyYAML.ppc 3.11-1 @System
python-argparse.noarch 1.2.1-1 @System
python-boto.noarch 2.34.0-1 @System
python-cheetah.ppc 2.4.4-2 @System
python-configobj.noarch 5.0.5-1 @System
python-devel.ppc 2.7.18-4 @AIX_Toolbox
python-iniparse.noarch 0.4-1 @System
python-jsonpatch.noarch 1.8-1 @System
python-jsonpointer.noarch 1.0-1 @System
python-oauth.noarch 1.0.1-1 @System
python-prettytable.noarch 0.7.2-1 @System
python-pycurl.ppc 7.43.0.5-1 @AIX_Toolbox
python-pyserial.ppc 2.7-1 @System
python-requests.noarch 2.4.3-1 @System
python-setuptools.noarch 0.9.8-2 @System
python-six.noarch 1.10.0-1 @System
python-tools.ppc 2.7.18-4 @AIX_Toolbox
python-urlgrabber.noarch 3.10.1-1 @System
python-xml.ppc 0.8.4-1 @System
python3.ppc 3.7.15-2 @AIX_Toolbox
python3-dateutil.noarch 2.8.0-1 @System
python3-devel.ppc 3.7.15-2 @AIX_Toolbox
python3-dnf.ppc 4.2.17-32_50 @AIX_Toolbox_72
python3-dnf-plugin-migrate.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-dnf-plugins-core.ppc 4.0.16-32_50 @AIX_Toolbox_72
python3-gpg.ppc 1.13.1-100 @AIX_Toolbox
python3-hawkey.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-jinja2.noarch 2.10.3-1 @AIX_Toolbox_noarch
python3-jmespath.noarch 0.9.4-1 @AIX_Toolbox_noarch
python3-libcomps.ppc 0.1.11-101 @AIX_Toolbox
python3-libdnf.ppc 0.39.1-32_51 @AIX_Toolbox_72
python3-librepo.ppc 1.11.0-101 @AIX_Toolbox
python3-markupsafe.ppc 1.1.1-1 @AIX_Toolbox
python3-pyyaml.ppc 5.4.1.1-1 @AIX_Toolbox
python3-six.noarch 1.13.0-1 @AIX_Toolbox_noarch
readline.ppc 8.1-1 @AIX_Toolbox
redis.ppc 5.0.9-1aix @/redis-5.0.9-1aix.aix7.2.ppc
rpm-python3.ppc 4.15.1-1_py3 @System
rsync.ppc 3.2.7-1 @AIX_Toolbox
sed.ppc 4.8-1 @AIX_Toolbox
serf.ppc 1.3.9-2 @AIX_Toolbox
sqlite.ppc 3.39.3-1 @AIX_Toolbox
sshpass.ppc 1.09-1 @AIX_Toolbox
subversion.ppc 1.10.8-1 @AIX_Toolbox
supervisor.noarch 4.2.0-2aix @/supervisor-4.2.0-2aix.aix7.2.noarch
tcl.ppc 8.6.11-1 @AIX_Toolbox
tk.ppc 8.6.11-1 @AIX_Toolbox
unzip.ppc 6.0-3 @AIX_Toolbox
wget.ppc 1.21.2-1 @AIX_Toolbox
xmlsec1.ppc 1.2.28-1 @/xmlsec1-1.2.28-1.aix7.2.ppc
xmlsec1-openssl.ppc 1.2.28-1 @/xmlsec1-openssl-1.2.28-1.aix7.2.ppc
xz-libs.ppc 5.2.5-1 @AIX_Toolbox
yum-metadata-parser.ppc 1.1.4-2 @System
zchunk-libs.ppc 1.1.4-102 @AIX_Toolbox
zeromq.ppc 4.3.4-1 @AIX_Toolbox
zeromq-devel.ppc 4.3.4-1 @AIX_Toolbox
zip.ppc 3.0-4 @AIX_Toolbox
zlib.ppc 1.2.13-1 @AIX_Toolbox

Any assistance on the next steps to correct this issue would be a great help.

Thank you,

Stan

------------------------------
Stanley
------------------------------

AIX Open Source