AIX Open Source

 View Only
Expand all | Collapse all

Installing Ansible AWX on AIX 7.2

  • 1.  Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Thu September 03, 2020 01:23 AM
    How to install Ansible AWX on AIX 7.2 - Installing Ansible AWX on IBM AIX

    Video with the installation - Installing Ansible AWX on IBM AIX
    YouTube remove preview
    Installing Ansible AWX on IBM AIX
    Installing Ansible AWX (open source version of Ansible Tower) on IBM AIX. Yes, it is possible! Yes, it works! Yes, you can do it too! Link to the installatio...
    View this on YouTube >

    Have fun automating AIX! :-)

    ------------------------------
    Andrey Klyachkin
    ------------------------------


  • 2.  RE: Installing Ansible AWX on AIX 7.2

    Posted Fri September 04, 2020 06:56 AM
    Edited by C- -T Fri September 04, 2020 06:56 AM
    i'm merely interessted in the linux ppc64 package, so i looked inside "https://dl.power-devops.com/ansible-awx-14.1.0-1.el7.ppc64le.rpm", but this all looks i little bit clumsy. any chance i can get hold of the source rpm, spec file, build script whatever to redo it locally?

    ------------------------------
    I regret starting this entire conversation
    ------------------------------



  • 3.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Sat September 05, 2020 02:10 PM
    Official source code is available on github - https://github.com/ansible/awx/ I didn't do any changes to AWX code. In the code you can also find installer ansible-playbook, which describes the whole build process. The dependencies are also open and available on github. My code you can find in https://github.com/power-devops/linux_compat

    ------------------------------
    Andrey Klyachkin
    ------------------------------



  • 4.  RE: Installing Ansible AWX on AIX 7.2

    Posted Mon September 07, 2020 09:48 AM
    those awx zealots will never support rpm packing and they get really bitchy if you even dare to ask about it, because you know docker is REALLY cool. thats why i asked about YOUR rpm building script spec file or source rpm package.

    ------------------------------
    I regret starting this entire conversation
    ------------------------------



  • 5.  RE: Installing Ansible AWX on AIX 7.2

    Posted Wed September 16, 2020 04:46 AM
    very interesting , good job and thanks for sharing

    ------------------------------
    Bis Matrimony
    Marriage Consultant
    Bis Matrimony
    Ranni
    ------------------------------



  • 6.  RE: Installing Ansible AWX on AIX 7.2

    Posted Tue April 19, 2022 09:42 AM
    Edited by C- -T Tue April 19, 2022 09:44 AM
    this is about your awx package 17.1.0 for rhel ppc64le:

    i installed the mentioned package, but i am unable to run any jobs.

    example job stdout from awx gui:

      File "/usr/lib/python2.7/site-packages/ansible/constants.py", line 19, in <module>
        from ansible.config.manager import ConfigManager, ensure_type, get_ini_config_value
      File "/usr/lib/python2.7/site-packages/ansible/config/manager.py", line 17, in <module>
        from yaml import load as yaml_load
      File "/var/lib/awx/venv/ansible/lib/python3.6/site-packages/yaml/__init__.py", line 399
        class YAMLObject(metaclass=YAMLObjectMetaclass):
                                  ^
    SyntaxError: invalid syntax​


    seems like the standard awx venv wants to run python2 code with python3 interpreter or something like that. i also tried to create a new  venv but this fail with some cryptograhy atrocities.

    (aix-custom-venv) root@lpgaixmgmtlx01:/var/lib/awx/venv/aix-custom-venv>pip3  install -U "ansible == 2.9.27"
    Collecting ansible==2.9.27
      Using cached ansible-2.9.27.tar.gz (14.8 MB)
      Preparing metadata (setup.py) ... done
    Collecting jinja2
      Using cached Jinja2-3.0.3-py3-none-any.whl (133 kB)
    Collecting PyYAML
      Using cached PyYAML-6.0.tar.gz (124 kB)
      Installing build dependencies ... done
      Getting requirements to build wheel ... done
      Preparing metadata (pyproject.toml) ... done
    Collecting cryptography
      Using cached cryptography-36.0.2.tar.gz (572 kB)
      Installing build dependencies ... done
      Getting requirements to build wheel ... error
      ERROR: Command errored out with exit status 1:
       command: /var/lib/awx/venv/aix-custom-venv/bin/python /var/lib/awx/venv/aix-custom-venv/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpozbpdbks
           cwd: /tmp/pip-install-ulvd5ypb/cryptography_88110136f1d74748b2e5c3ada1d52dbb
      Complete output (62 lines):
    
          =============================DEBUG ASSISTANCE=============================
          If you are seeing a compilation error please try the following steps to
          successfully install cryptography:
          1) Upgrade to the latest pip and try again. This will fix errors for most
             users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
          2) Read https://cryptography.io/en/latest/installation/ for specific
             instructions for your platform.
          3) Check our frequently asked questions for more information:
             https://cryptography.io/en/latest/faq/
          4) Ensure you have a recent Rust toolchain installed:
             https://cryptography.io/en/latest/installation/#rust
    
          Python: 3.6.8
          platform: Linux-3.10.0-1160.59.1.el7.ppc64le-ppc64le-with-redhat-7.9-Maipo
          pip: n/a
          setuptools: 59.6.0
          setuptools_rust: 1.1.2
          =============================DEBUG ASSISTANCE=============================
    
      Traceback (most recent call last):
        File "/var/lib/awx/venv/aix-custom-venv/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/var/lib/awx/venv/aix-custom-venv/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/var/lib/awx/venv/aix-custom-venv/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-mp7b9_bq/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 163, in get_requires_for_build_wheel
          config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-mp7b9_bq/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 143, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-mp7b9_bq/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 158, in run_setup
          exec(compile(code, __file__, 'exec'), locals())
        File "setup.py", line 54, in <module>
          rust_version=">=1.41.0",
        File "/tmp/pip-build-env-mp7b9_bq/overlay/lib/python3.6/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/usr/lib64/python3.6/distutils/core.py", line 108, in setup
          _setup_distribution = dist = klass(attrs)
        File "/tmp/pip-build-env-mp7b9_bq/overlay/lib/python3.6/site-packages/setuptools/dist.py", line 463, in __init__
          for k, v in attrs.items()
        File "/usr/lib64/python3.6/distutils/dist.py", line 281, in __init__
          self.finalize_options()
        File "/tmp/pip-build-env-mp7b9_bq/overlay/lib/python3.6/site-packages/setuptools/dist.py", line 837, in finalize_options
          ep(self)
        File "/tmp/pip-build-env-mp7b9_bq/overlay/lib/python3.6/site-packages/setuptools/dist.py", line 858, in _finalize_setup_keywords
          ep.load()(self, ep.name, value)
        File "/tmp/pip-build-env-mp7b9_bq/overlay/lib64/python3.6/site-packages/cffi/setuptools_ext.py", line 219, in cffi_modules
          add_cffi_module(dist, cffi_module)
        File "/tmp/pip-build-env-mp7b9_bq/overlay/lib64/python3.6/site-packages/cffi/setuptools_ext.py", line 49, in add_cffi_module
          execfile(build_file_name, mod_vars)
        File "/tmp/pip-build-env-mp7b9_bq/overlay/lib64/python3.6/site-packages/cffi/setuptools_ext.py", line 25, in execfile
          exec(code, glob, glob)
        File "src/_cffi_src/build_openssl.py", line 118, in <module>
          extra_link_args=extra_link_args(compiler_type()),
        File "src/_cffi_src/utils.py", line 59, in build_ffi_for_binding
          extra_link_args=extra_link_args,
        File "src/_cffi_src/utils.py", line 73, in build_ffi
          ffi = FFI()
        File "/tmp/pip-build-env-mp7b9_bq/overlay/lib64/python3.6/site-packages/cffi/api.py", line 48, in __init__
          import _cffi_backend as backend
      ModuleNotFoundError: No module named '_cffi_backend'
      ----------------------------------------
    WARNING: Discarding https://files.pythonhosted.org/packages/10/a7/51953e73828deef2b58ba1604de9167843ee9cd4185d8aaffcb45dd1932d/cryptography-36.0.2.tar.gz#sha256=70f8f4f7bb2ac9f340655cbac89d68c527af5bb4387522a8413e841e3e6628c9 (from https://pypi.org/simple/cryptography/) (requires-python:>=3.6). Command errored out with exit status 1: /var/lib/awx/venv/aix-custom-venv/bin/python /var/lib/awx/venv/aix-custom-venv/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpozbpdbks Check the logs for full command output.


    so...how to get this going?



    ------------------------------
    I regret starting this entire conversation
    ------------------------------



  • 7.  RE: Installing Ansible AWX on AIX 7.2

    Posted Tue September 15, 2020 09:54 PM
    Vry interesting , good job and thanks for sharing

    ------------------------------
    Bis Matrimony
    Marriage Consultant
    Bis Matrimony
    Ranni
    ------------------------------



  • 8.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Fri September 11, 2020 02:52 PM
    Audrey, hi, thank-you for sharing the steps to deploy AWX on AIX 7.2.

    I followed them all and although I had to re-run the install a second time to cleardown some issues I had I now have a successfully installed AWX.  Currently I am hitting a "502 Bad Gateway" error with the following being displayed in the error.log

    2020/09/11 11:52:54 [crit] 12190084#0: *1 connect() to unix:/var/run/tower/uwsgi.sock failed (13: Permission denied) while connecting to upstream, client: 1.2.3.4, server: _, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:/var/run/tower/uwsgi.sock:", host: "host-name"​
    The permissions are as follows:

    For /var/run drwxr-xr-x 5 root system 256 Sep 11 11:52 run For /var/run/tower drwxr-xr-x 2 root awx 256 Sep 11 11:31 tower For /var/run/tower/uwsgi.sock -rwxr--r-- 1 root system 0 Sep 10 11:38 uwsgi.sock ​
    I'm running AIX 7200-04-02-2028, it's a PoC LPAR at the moment.

    Do you have any pointers for me as to resolve this "Permission denied" issue?  I've already set uwsgi.sock to 777 but then get the following so clearly that didn't work!

    connect() to unix:/var/run/tower/uwsgi.sock failed (57: Socket operation on non-socket) while connecting to upstream​
    Many thanks, Steve

    ------------------------------
    Pfizer Support
    ------------------------------



  • 9.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Fri September 11, 2020 03:34 PM
    Hi Steve,

    i think the problem is that the group awx can't write into /var/run/tower. Try chmod 0775 /var/run/tower, remove /var/run/tower/uwsgi.sock and restart supervisord.

    ------------------------------
    Andrey Klyachkin
    ------------------------------



  • 10.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Mon September 14, 2020 04:29 AM
    Andrey,

    Hi, good morning.  I ran the chmod and removed the .sock before starting everything however I now get the following:

    2020/09/14 05:02:31 [crit] 6226384#0: *3 connect() to unix:/var/run/tower/uwsgi.sock failed (2: No such file or directory) while connecting to upstream, client: 1.2.3.4, server: _, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:/var/run/tower/uwsgi.sock:", host: "<LPAR-name>"​​


    Many thanks, Steve



    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 11.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Mon September 14, 2020 06:21 AM
    Steve,

    when you start supervisord, it starts uwsgi and uwsgi makes the socket file /var/run/tower/uwsgi.sock:

    # cd /var/run
    # ls -ld tower
    drwxrwxr-x 2 awx awx 256 Sep 14 10:17 tower
    # cd tower
    # ls -l
    total 0
    srw-rw-rw- 1 awx awx 0 Sep 14 10:17 daphne.sock
    lrwxrwxrwx 1 awx awx 8 Sep 14 10:17 daphne.sock.lock -> 17564110
    srw-rw---- 1 awx awx 0 Sep 14 10:17 uwsgi.sock
    # /etc/rc.d/init.d/supervisord stop
    # ps -ef | grep super
    root 17236262 16449826 0 10:18:53 pts/2 0:00 grep super
    # set -o vi
    # ps -ef | grep uwsgi
    # ls -l uwsgi.sock
    uwsgi.sock not found
    # pwd
    /var/run/tower
    # /etc/rc.d/init.d/supervisord start
    # ls -l
    total 0
    srw-rw---- 1 awx awx 0 Sep 14 10:19 uwsgi.sock
    #

    Do you have some strange umask?

    ------------------------------
    Andrey Klyachkin
    ------------------------------



  • 12.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Mon September 14, 2020 08:10 AM
    Andrey,

    Thanks for the guidance.  As you can see from the below "supervisord stop" failed to find the process yet when I "started" it it said it was already running!

    <LPAR-name>:/var/run/tower# /etc/rc.d/init.d/supervisord stop kill: 16384330: 0403-003 The specified process does not exist. <LPAR-name>:/var/run/tower# ps -ef | grep uwsgi root 11010514 10486044 0 07:10:22 pts/0 0:00 grep uwsgi <LPAR-name>:/var/run/tower# ls -l uwsgi.sock ls: 0653-341 The file uwsgi.sock does not exist. <LPAR-name>:/var/run/tower# /etc/rc.d/init.d/supervisord start Supervisord is already started <LPAR-name>:/var/run/tower# ls -l total 0 ​


    I'm wondering whether it's worth me removing Postrgesql which will also remove awx and start over having ensured /var/run/tower and awx no-llonger exists.

    Thanks, Steve



    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 13.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Mon September 14, 2020 10:57 AM
    ** Update **
    1) Checked umask, it defaults to 022 and the awx stanza only has admin = false
    2) Removed Postgresql & ansible-awx and re-installed ainsible-awx.
    3) Now getting

    2020/09/14 09:23:12 [alert] 7667992#0: unlink() "/var/run/nginx.pid" failed (2: No such file or directory)​


    4) Should I erase and re-install nginx?

    Thanks, Steve



    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 14.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Mon September 14, 2020 02:49 PM
    It's interesting. nginx is a "standard" AIX package from AIX Toolbox for Linux Applications. If you get such a message, it looks for me very suspicious. I suppose, you have some other problem on the server. Can you post output of ls -ld /var/run? Anyway you can reinstall nginx if you want. Just make backup of nginx config files installed with AWX before reinstalling nginx.

    ------------------------------
    Andrey Klyachkin
    ------------------------------



  • 15.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Tue September 15, 2020 05:36 AM
    Andrey,

    Hi, here's the output from the command.

    # ls -ld /var/run drwxr-xr-x 5 root system 256 Sep 14 09:57 /var/run​


    As for the LPAR, it's a freshly installed 7200-04-02-2028 currently without further patches though.  wget, yum, ainsible were all installed ahead of ansible-awx.



    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 16.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Tue September 15, 2020 05:59 AM
    Andrey,

    Checking the supervisor log I see the following (from a few days ago I admit) so pass it on "just in case".

    2020-09-09 06:39:12,980 INFO success: awx-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-09-09 06:39:15,836 INFO exited: awx-uwsgi (exit status 1; not expected) 2020-09-09 06:39:17,806 INFO spawnerr: unknown error during fork for 'awx-uwsgi': ENOMEM 2020-09-09 06:39:19,443 INFO spawnerr: unknown error during fork for 'awx-uwsgi': ENOMEM 2020-09-09 06:39:19,624 INFO exited: awx-rsyslogd (terminated by SIGKILL; not expected) 2020-09-09 06:39:20,688 INFO spawnerr: unknown error during fork for 'awx-rsyslogd': ENOMEM 2020-09-09 06:39:21,924 INFO spawnerr: unknown error during fork for 'awx-uwsgi': ENOMEM 2020-09-09 06:39:22,343 INFO spawnerr: unknown error during fork for 'awx-rsyslogd': ENOMEM 2020-09-09 06:39:24,899 INFO spawnerr: unknown error during fork for 'awx-rsyslogd': ENOMEM 2020-09-09 06:39:25,835 INFO spawnerr: unknown error during fork for 'awx-uwsgi': ENOMEM 2020-09-09 06:39:25,918 INFO gave up: awx-uwsgi entered FATAL state, too many start retries too quickly 2020-09-09 06:39:28,309 INFO spawnerr: unknown error during fork for 'awx-rsyslogd': ENOMEM 2020-09-09 06:39:28,310 INFO gave up: awx-rsyslogd entered FATAL state, too many start retries too quickly 2020-09-09 06:39:32,229 INFO exited: awx-config-watcher (terminated by SIGKILL; not expected) 2020-09-09 06:39:32,809 INFO spawnerr: unknown error during fork for 'awx-config-watcher': ENOMEM # lparstat System configuration: type=Shared mode=Uncapped smt=8 lcpu=16 mem=6144MB psize=4 ent=0.20 %user %sys %wait %idle physc %entc lbusy vcsw phint %nsp ----- ----- ------ ------ ----- ----- ------ ----- ----- ----- 0.1 0.0 0.0 99.9 0.00 0.2 1.4 22434257 0 74​


    The nginx.pid has "appeared" (didn't re-install nginx) however I'm back to the old error.

    2020/09/15 06:17:44 [crit] 6750522#0: *1 connect() to unix:/var/run/tower/uwsgi.sock failed (2: No such file or directory) while connecting to upstream, client: 1.2.3.4, server: _, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:/var/run/tower/uwsgi.sock:", host: "<LPAR-name>" 2020/09/15 06:17:44 [error] 6750522#0: *1 open() "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/ui/static/favicon.ico" failed (2: No such file or directory), client: 1.2.3.4, server: _, request: "GET /favicon.ico HTTP/1.1", host: "<LPAR-name>" # # ls -la /var/run/tower/ total 0 drwxrwxr-x 2 root awx 256 Sep 02 16:07 . drwxr-xr-x 5 root system 256 Sep 14 09:57 ..


    Also, when I start supervisord and it says "already started" I see nothing in ps -ef so where is it hiding?

    Thanks, Steve



    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 17.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Tue September 15, 2020 08:58 AM
    ENOMEM means not enough memory. Either you don't have enough memory in your LPAR or the awx user has some limitations.
    In the first case you could see something like PGSP_KILL in errpt output.
    In the second case check the attributes data,stack,rss of the awx users (lsuser -a data stack rss awx). Try to set them higher, even may be -1. In my environment they are quite low though - data=262144 stack=65536 rss=65536.

    If supervisord starts it must be visible with ps -ef. I would suggest you first make sure, it is not running, then remove its logs (rm -f /var/log/supervisor/*) and its pid file (rm -f /opt/freeware/var/run/supervisor/supervisord.pid), then start it again (/etc/rc.d/init.d/supervisord start). After that you should be able to see supervisord running (ps -ef | grep supervisor) and to control it with supervisorctl (/opt/freeware/bin/supervisorctl status).

    If it is not running after that, you can check the logs (/var/log/supervisor/supervisord.log). The file should be relative small.

    ------------------------------
    Andrey Klyachkin
    ------------------------------



  • 18.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Tue September 15, 2020 10:43 AM
    Andrey,

    Hi, that certainly got supervisord started however I then ran of of Paging Space (set at 16GB) so had to re-boot.  After that supervisord wasn't running (by default) so I manually started it and whilst the tatus is RUNNING I see "STARTING" (only) for awx-uwsgi.

    /# ps -ef | grep -i superv /# lsuser -a data stack rss awx awx data=262144 stack=65536 rss=65536 /# rm -f /opt/freeware/var/run/supervisor/supervisord.pid /# rm -f /var/log/supervisor/* /# /etc/rc.d/init.d/supervisord start Unlinking stale socket /opt/freeware/var/run/supervisor/supervisor.sock /# ps -ef | grep supervisor root 12583322 1 9 11:06:58 - 0:00 /opt/freeware/bin/python3_64 /opt/freeware/bin/supervisord -c /opt/freeware/etc/supervisord.conf root 13566370 8192274 0 11:07:09 vty0 0:00 grep supervisor /# /opt/freeware/bin/supervisorctl status awx-config-watcher RUNNING pid 6226380, uptime 0:00:26 tower-processes:awx-callback-receiver RUNNING pid 10617218, uptime 0:00:26 tower-processes:awx-daphne RUNNING pid 12321164, uptime 0:00:26 tower-processes:awx-dispatcher RUNNING pid 11272580, uptime 0:00:26 tower-processes:awx-rsyslogd RUNNING pid 12386704, uptime 0:00:26 tower-processes:awx-uwsgi STARTING tower-processes:awx-wsbroadcast RUNNING pid 12255650, uptime 0:00:26 /# lsps -a Page Space Physical Volume Volume Group Size %Used Active Auto Type Chksum hd6 hdisk0 rootvg 16384MB 1 yes yes lv 0 /# cd /var/run /# /# cd /var/run /var/run# ls -ld tower drwxrwxr-x 2 root awx 256 Sep 15 11:08 tower /var/run# cd tower /var/run/tower# ls -l total 0 /var/run/tower# /etc/rc.d/init.d/supervisord stop /var/run/tower# ps -ef | grep super root 12583322 1 1 11:06:58 - 0:00 /opt/freeware/bin/python3_64 /opt/freeware/bin/supervisord -c /opt/freeware/etc/supervisord.conf root 28115306 8192274 0 11:09:01 vty0 0:00 grep super /var/run/tower# ps -ef | grep uwsgi awx 8126812 1 0 11:07:00 - 0:01 /var/lib/awx/venv/awx/bin/uwsgi -s /var/run/tower/uwsgi.sock -w awx.wsgi --vacuum -p 5 -t 120 -R 1000 -M --no-orphans --master-fifo=/var/lib/awx/awxfifo --lazy-apps -b 32768 --stats /var/lib/awx/uwsgi.stats --chmod-socket=660 --uid awx --gid awx awx 9699684 1 0 11:07:01 - 0:01 /var/lib/awx/venv/awx/bin/uwsgi -s /var/run/tower/uwsgi.sock -w awx.wsgi --vacuum -p 5 -t 120 -R 1000 -M --no-orphans --master-fifo=/var/lib/awx/awxfifo --lazy-apps -b 32768 --stats /var/lib/awx/uwsgi.stats --chmod-socket=660 --uid awx --gid awx ​
    I then see Paging Space being eaten up!!

    awx 27918680 1 0 11:08:41 - 0:01 /var/lib/awx/venv/awx/bin/uwsgi -s /var/run/tower/uwsgi.sock -w awx.wsgi --vacuum -p 5 -t 120 -R 1000 -M --no-orphans --master-fifo=/var/lib/awx/awxfifo --lazy-apps -b 32768 --stats /var/lib/awx/uwsgi.stats --chmod-socket=660 --uid awx --gid awx awx 27984218 1 0 11:08:41 - 0:01 /var/lib/awx/venv/awx/bin/uwsgi -s /var/run/tower/uwsgi.sock -w awx.wsgi --vacuum -p 5 -t 120 -R 1000 -M --no-orphans --master-fifo=/var/lib/awx/awxfifo --lazy-apps -b 32768 --stats /var/lib/awx/uwsgi.stats --chmod-socket=660 --uid awx --gid awx awx 28049756 1 0 11:08:41 - 0:01 /var/lib/awx/venv/awx/bin/uwsgi -s /var/run/tower/uwsgi.sock -w awx.wsgi --vacuum -p 5 -t 120 -R 1000 -M --no-orphans --master-fifo=/var/lib/awx/awxfifo --lazy-apps -b 32768 --stats /var/lib/awx/uwsgi.stats --chmod-socket=660 --uid awx --gid awx root 28246384 8192274 1 11:10:05 vty0 0:00 grep uwsgi /var/run/tower# lsps -a Page Space Physical Volume Volume Group Size %Used Active Auto Type Chksum hd6 hdisk0 rootvg 16384MB 74 yes yes lv 0 /var/run/tower# lsps -a Page Space Physical Volume Volume Group Size %Used Active Auto Type Chksum hd6 hdisk0 rootvg 16384MB 77 yes yes lv 0 ​/var/run/tower# lsps -a Page Space Physical Volume Volume Group Size %Used Active Auto Type Chksum hd6 hdisk0 rootvg 16384MB 86 yes yes lv 0


    Any ideas why Paging Space is being eaten up so much?

    Thanks, Steve



    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 19.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Tue September 15, 2020 01:18 PM
    Andrey,

    Hi, here's the latest which occured after the LPAR re-started due to Paging Space filling up.  Upon restart supervisord wasn't running, see excerpt below.

    /# lsuser -a data stack rss awx awx data=262144 stack=65536 rss=65536 /# /etc/rc.d/init.d/supervisord start Supervisord is already started /# rm -f /var/log/supervisor/* /# rm -f /opt/freeware/var/run/supervisor/supervisord.pid /# /etc/rc.d/init.d/supervisord start Unlinking stale socket /opt/freeware/var/run/supervisor/supervisor.sock /# ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 11:04:26 - 0:00 /etc/init root 2425272 1 0 11:04:45 - 0:01 /usr/sbin/syncd 60 root 3146112 1 0 11:04:33 - 0:00 /usr/ccs/bin/shlap64 root 3604954 1 0 11:04:45 - 0:00 /usr/lib/errdemon root 4915536 1 0 11:04:48 - 0:00 /usr/sbin/srcmstr root 5570990 4915536 0 11:04:53 - 0:00 /usr/sbin/portmap root 5767604 4915536 0 11:04:53 - 0:00 /usr/sbin/inetd root 5833146 4915536 0 11:04:53 - 0:00 /usr/sbin/aixmibd root 5898700 4915536 0 11:04:57 - 0:00 /usr/sbin/clcomd -d root 6095294 4915536 0 11:04:53 - 0:00 /usr/sbin/snmpd root 6160832 4915536 0 11:04:53 - 0:00 /usr/sbin/netcd root 7668066 4915536 0 11:05:10 - 0:00 /opt/rsct/bin/IBM.HostRMd root 9896256 4915536 0 11:05:08 - 0:00 /opt/rsct/bin/rmcd -a IBM root 9961838 1 0 11:58:34 - 0:00 /usr/sbin/uprintfd root 10420596 4915536 0 11:05:11 - 0:00 /opt/rsct/bin/IBM.MgmtDom root 10551634 1 1 11:58:34 vty0 0:00 -ksh awx 10617232 28967290 3 13:49:32 - 0:00 /var/lib/awx/venv/awx/bin root 10682758 1 0 11:58:34 - 0:00 /usr/sbin/cron root 10944854 4915536 0 11:05:11 - 0:00 /opt/rsct/bin/IBM.Service root 11207008 4915536 0 11:05:11 - 0:00 /opt/rsct/bin/IBM.ConfigR root 11993474 4915536 0 11:05:12 - 0:00 /opt/rsct/bin/IBM.DRMd root 28443028 10551634 3 13:49:32 vty0 0:00 ps -ef - 28705154 - - - <idle> awx 28967290 33095938 44 13:49:31 - 0:00 /var/lib/awx/venv/awx/bin awx 29098366 1 120 13:49:30 - 0:00 /var/lib/awx/venv/awx/bin awx 29426056 1 119 13:49:30 - 0:00 /var/lib/awx/venv/awx/bin awx 29688208 1 120 13:49:30 - 0:00 /var/lib/awx/venv/awx/bin awx 30409126 1 120 13:49:30 - 0:00 /var/lib/awx/venv/awx/bin awx 30540202 1 120 13:49:30 - 0:00 /var/lib/awx/venv/awx/bin awx 30605744 28967290 14 13:49:32 - 0:00 /var/lib/awx/venv/awx/bin awx 30736816 1 102 13:49:29 - 0:00 /var/lib/awx/venv/awx/bin awx 30867892 1 120 13:49:29 - 0:00 /var/lib/awx/venv/awx/bin awx 30998968 1 120 13:49:29 - 0:00 /var/lib/awx/venv/awx/bin awx 31130044 1 93 13:49:29 - 0:00 /var/lib/awx/venv/awx/bin awx 31326658 1 120 13:49:29 - 0:00 /var/lib/awx/venv/awx/bin awx 31523272 1 97 13:49:27 - 0:00 /var/lib/awx/venv/awx/bin awx 31719886 1 112 13:49:27 - 0:00 /var/lib/awx/venv/awx/bin awx 31916500 1 102 13:49:27 - 0:00 /var/lib/awx/venv/awx/bin root 31981844 9896256 0 12:01:29 - 0:00 [trspoolm] awx 32113114 1 90 13:49:27 - 0:00 /var/lib/awx/venv/awx/bin awx 32178652 1 103 13:49:27 - 0:00 /var/lib/awx/venv/awx/bin awx 32244190 33095938 103 13:49:27 - 0:00 /var/lib/awx/venv/awx/bin awx 32309730 33095938 92 13:49:27 - 0:00 /var/lib/awx/venv/awx/bin awx 32637420 33095938 0 13:49:27 - 0:00 rsyslogd -n -i /var/run/a awx 32702958 33095938 120 13:49:27 - 0:00 /var/lib/awx/venv/awx/bin awx 32899572 33095938 104 13:49:26 - 0:00 /var/lib/awx/venv/awx/bin root 33095938 1 7 13:49:25 - 0:00 /opt/freeware/bin/python3 root 33358108 33095938 3 13:49:26 - 0:00 python3 /usr/bin/config-w /# lsps -a Page Space Physical Volume Volume Group Size %Used Active Auto Type Chksum hd6 hdisk0 rootvg 16384MB 41 yes yes lv 0​


    As you can seet (almost) immediately the %age Paging Space is at 41% so something is very wrong.  Should there be all these /var/lib/awx/venv/awx/bin processes running I wonder?

    Thanks, Steve



    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 20.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Wed September 16, 2020 06:10 AM
      |   view attached
    Andrey,

    Hi, good morning.

    With the LPAR re-booted I then started supervisord (which was successful) I then checked ps -ef and saw loads of uwsgi processes running.  At this point the Paging Space started to rise and rise.  I then stopped supervisord (which was successful) however the uwsgi processes still existed and the Paging Space continued to rise until it maxed out so I had to re-boot the LPAR where I still saw loads of /var/lib/awx/venc/awx/bin processes running but no supervisord "and" Paging Space rose again.

    I re-booted yet again and captured some more logs and .ini file contents for your review in case something "jumps out" (see attached).

    Many thanks, Steve




    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------

    Attachment(s)



  • 21.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Wed September 16, 2020 06:28 AM
    Checkingawx-uwsgi.log​
    I see (at the top) the following

    spawned uWSGI master process (pid: 10617190) spawned uWSGI worker 1 (pid: 10092860, cores: 1) spawned uWSGI worker 2 (pid: 10223948, cores: 1) spawned uWSGI worker 3 (pid: 10682712, cores: 1) spawned uWSGI worker 4 (pid: 10748240, cores: 1) spawned uWSGI worker 5 (pid: 10944852, cores: 1) malloc(): Not enough space [core/utils.c line 1799] !!! tried memory allocation of 18446744073709551608 bytes !!! VACUUM: unix socket /var/run/tower/uwsgi.sock removed. unlink(): No such file or directory [core/uwsgi.c line 1673] ​


    Could it be this malloc() which is throwing things off?

    Thanks, Steve



    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 22.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Wed September 16, 2020 08:02 AM
    I'll create an environment with AIX 7.2 TL4 SP2 and try to install AWX into it. I didn't see such problem and couldn't repeat it. How much memory do you in the LPAR?

    ------------------------------
    Andrey Klyachkin
    ------------------------------



  • 23.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Wed September 16, 2020 08:18 AM
    Andrey,

    I allocated 6GB to the LPAR, along with 0.2 PU and 2 VP.

    Thanks, Steve

    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 24.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Wed September 16, 2020 11:52 AM
    Steve,

    I've got the same error on AIX 7.2 TL4 SP2. Let's try to find out why.


    ------------------------------
    Andrey Klyachkin
    ------------------------------



  • 25.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Wed September 16, 2020 12:32 PM
    Andrey,

    Hi, glad you've been able to reproduce the issue.  Fingers crossed you uncover the issue and the resolution is straight forward.  I guess I could always move back to 7.2 TL4 SP1 however that doesn't correct the root cause.

    Many thanks, Steve

    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 26.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Wed September 16, 2020 12:48 PM
    Steve,

    I've caught the problem. It has nothing to do with AIX version, but with number of max open file descriptors. Could you check lsuser -a nofiles root ? If the value is -1, I'm getting the problem with uwsgi. If I set the value to 2000, everything works fine.

    ------------------------------
    Andrey Klyachkin
    ------------------------------



  • 27.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Thu September 17, 2020 05:15 AM
    Andrey,

    Hi, good morning.  My nofiles (for root) was also -1 so set it to 2000 and then tried again.  I still got Paging Space issues so re-booted and checked a few logs.

    I started supervisord however got errors (see below) as postgresql doesn't auto start so I then started Postgresql which resolved that issue.

    Is the server running locally and accepting connections on Unix domain socket "/tmp/.s.PGSQL.5432"

    Next up was an issue with the the following:

    Invalid HTTP_HOST header: '<LPAR-name>'. The domain name provided is not valid according to RFC 1034/1035.

    So I used the IP address instead which got me here so SUCCESS :-)



    First of all Thank-you so much for working to resolve the nofiles issue.  Secondly, how did you know that it was nofiles causing the problem as I'm keen to understand and learn?

    Many thanks, Steve

    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 28.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Thu September 17, 2020 06:06 AM
    Steve,

    I'm glad to hear, that I could help you!

    If you look in the log, you've sent, you'll find the following entry in it:

    detected max file descriptor number: 9223372036854775807

    As for me the number is TOO big. Almost as big as the number of bytes it tries to allocate:

    !!! tried memory allocation of 18446744073709551608 bytes !!!

    I just made an assumption, that uwsgi tries to allocate space for all of these descriptors, then tested it in my environment and yes, it worked for me. If I set nofiles=-1, I become the same big number and then paging space issues. If I set e.g. nofiles=65536 I don't have any paging space issues. You don't need to reboot after the change, just relogin.


    ------------------------------
    Andrey Klyachkin
    ------------------------------



  • 29.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Thu September 17, 2020 06:48 AM
    Andrey,

    Hi, thanks very much for explaining how to track down the issue and the resolution.  I only needed to add a symlink for Postgresql as the others were already defined.  I just re-booted and everything is now auto-starting :-)

    All the best, Steve

    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 30.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Thu September 17, 2020 09:09 AM
    I just tested another solution, if you for some reasons won't want to change nofiles attribute. You can the function start_app in /etc/rc.d/init.d/supervisord and add ulimit -n 65536 (or lower) before starting supervisord in it:

    start_app() {
    if [[ -f $PIDFILE ]] ; then
    print -u2 -- "Supervisord is already started"
    exit 1
    fi
    # workaround if root have too many allowed file descriptors
    ulimit -n 65536
    /opt/freeware/bin/supervisord -c /opt/freeware/etc/supervisord.conf
    }

    ------------------------------
    Andrey Klyachkin
    ------------------------------



  • 31.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Thu September 17, 2020 10:43 AM
    Andrey,

    Hi, this is a nice alternative "just in case" changing nofiles for root isn't allowed so as to comply for company policy(s).

    Thanks again.

    Steve

    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 32.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Thu September 17, 2020 06:07 AM
    btw if you need autostart, just make symlinks for postgresql, redis, nginx and supervisord in /etc/rc.d/rc2.d (if you use runlevel 2).

    ------------------------------
    Andrey Klyachkin
    ------------------------------



  • 33.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Wed September 16, 2020 12:05 PM
    btw I have a running AWX instance on 7.2 TL4 SP1 without any problems...

    ------------------------------
    Andrey Klyachkin
    ------------------------------



  • 34.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Fri September 18, 2020 10:56 AM
    Andrey,

    Hi, making progress now within AWX however when I run a very simple Playbook I get the following error:

    Playbook
    --- - hosts: all gather_facts: true tasks: - debug: msg: "{{ ansible_os_family }}" - debug: msg: "{{ ansible_processor }}" ​
    Error
    in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error 79 connecting to unix socket: /var/run/redis/redis.sock. Connection refused.​
    The permissions are as follows:
    /var/run/redis# ls -la total 8 drwxr-xr-x 2 redis redis 256 Sep 09 06:35 . drwxr-xr-x 5 root system 256 Sep 17 07:19 .. srwxrwxr-x 1 redis redis 0 Sep 09 06:35 redis.sock -rw-r--r-- 1 redis redis 9 Sep 09 06:35 redis_6379.pid ​
    I'm wondering where I should be looking for the cause?

    Thanks, Steve

    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 35.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Fri September 18, 2020 11:38 AM
    Andrey,

    Hi, my fault.  Seems that redis hadn't remained running (!!) however the .pid was still there.  Anyway, I deleted it and actually got redis to re-start so making more progress.

    Thanks, Steve

    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 36.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Fri September 18, 2020 12:27 PM
    Steve,

    good that you've found it!

    Andrey

    ------------------------------
    Andrey Klyachkin
    ------------------------------



  • 37.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Wed September 23, 2020 01:53 PM
    Andrey,

    Hi, just wanted to provide an update as I just re-downloaded/re-installed AWX 14.1.0 on a fresh AIX 7.2 TL4 SP2 and everything went through nice and clean.  Many thanks for making the changes within the process.

    All the best, Steve

    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 38.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Thu September 24, 2020 05:28 AM
    Steve,

    thank you for the update and the feedback!

    Cheers
    Andrey

    ------------------------------
    Andrey Klyachkin
    ------------------------------



  • 39.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Thu September 24, 2020 05:54 AM
    Andrey,

    Hi, now that AWX is working nicely I'm moving on to trying things out with the power_aix collection however whilst I've had things working nicely for some time via the CLI I am struggling to see how to link things together when it comes to AWX (Tower) and a collection.  I'm thinking it's all to do with how I define where AWX should look for collections.  Checking a few things on line I've gone with creating a collections (sub)folder per project.

    Do you have any hints / tips to share?

    Many thanks, Steve

    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 40.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Thu September 24, 2020 07:02 AM
    Steve, 

    my way of doing things is quite complex ;-)

    First of all I recommend to install and use some local git repository to save all the cookbooks there. My choice is Gitlab. You can find official version here, or unofficial port to Power Linux here. Because you work for a large company, you might already have some internal source code management system, you can connect to. Check that it provides Git interface.

    There are 2 strategies how to keep data in your source code management:
    • everything together in one repo
    • separate repos for separate roles/projects

    It is your choice. I think, you should try the both ways and you'll find your best way.

    I don't recommend to go production and make some real PoCs without an SCM, but if you want, you can always save your cookbooks into /var/lib/awx/projects. Just create the path and make it accessible for user awx:

    # mkdir -p /var/lib/awx/projects
    # chown awx:awx /var/lib/awx/projects

    After you uploaded all your playbooks to repos, you define your projects and jobs in AWX and schedule their execution.

    Please note, that AWX starts all the playbooks under the user awx and not root or your own user.

    I packaged ibm.power_aix and ibm.power_i collections together with AWX. You can find them under /var/lib/awx/venv/vendor/awx_ansible_collections/ansible_collections/ibm. But to use them in AWX or in Ansible CLI you must change /etc/ansible/ansible.cfg. Find the section [defaults] there and add the line:

    collections_paths = /var/lib/awx/venv/vendor/awx_ansible_collections

    If you did it correctly, you can verify it with ansible-config:

    # ansible-config dump --only-changed
    COLLECTIONS_PATHS(/etc/ansible/ansible.cfg) = ['/var/lib/awx/venv/vendor/awx_ansible_collections']

    ansible-galaxy will not install ibm.power_aix in this case:

    # ansible-galaxy collection install ibm.power_aix
    Process install dependency map
    Starting collection install process
    Skipping 'ibm.power_aix' as it is already installed

    Cheers
    Andrey



    ​​

    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 41.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Thu September 24, 2020 08:28 AM
    Andrey,

    Ah-ha!  I'll remove my (IBM) collections then :-)

    I did as you recommended and here's the result as I read somewhere that I'd need the Galaxy Server defined too.

    # ansible-config dump --only-changed COLLECTIONS_PATHS(/etc/ansible/ansible.cfg) = ['/var/lib/awx/venv/vendor/awx_ansible_collections'] GALAXY_SERVER_LIST(/etc/ansible/ansible.cfg) = ['release_galaxy'] ​


    I'll start playing around with flrtvc now.

    Many thanks, Steve



    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 42.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Mon September 28, 2020 11:33 AM
    Andrey,

    Hi, I seem to be going around in circles with credentials!!

    Here's what I'm wanting to do.

    1. Starting as "awx", run a playbook
    2. Switch to "aix-admin-service-account" on AWX LPAR ["aix-admin-service-account" is on all LPARs and has public/private keys deployed]
    3. Connect to "remote" LPAR as "aix-admin-service-account"
    4. Execute task/command
    5. End

    I'm just not quite seeing "how to correctly configure the necessary credentials", do you have any pointers please?

    Many thanks, Steve

    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 43.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Mon September 28, 2020 04:29 PM
    Hi Steve,

    as far as I undertand, you want to patch your AIX estate with flrtvc. Let's try to go through the jungle together.

    For testing purposes I created the directory /var/lib/awx/projects/patchaix on AWX server and the playbook flrtvc-sec.yml there:

    # mkdir -p /var/lib/awx/projects/patchaix
    # cat >/var/lib/awx/projects/patchaix/flrtvc-sec.yml
    - hosts: all
      collections:
        - ibm.power_aix
      vars:
        ansible_python_interpreter: /opt/freeware/bin/python
      tasks:
        - name: query and install latest security fixes
          flrtvc:
            path: /usr/sys/inst.images
            verbose: yes
            apar: sec
            save_report: yes
            extend_fs: yes

    ​​​​Now let's go into AWX and first create a project pointing to the directory. AWX -> Projects -> click green plus symbol. Then enter the values. It should look like:

    AWX Project

    Name: patchaix
    Description: install the latest fixes on AIX
    Organization: default
    SCM type: manual
    Playbook directory: patchaix

    Save, ready for the next step - Inventories. The same green plus, just new name. In my case - nodes_to_patch. I use static inventory (not smart inventory). Save. Hosts tab, green plus. Enter server names. I test with just one AIX server and its name is just "node1". For a small test it is enough to have one node.

    Nodes are defined, let's define credentials. In my environment the user "aix-admin-service-account" is called aixsvc. It exists on all AIX nodes (I have just one in the environment :-) and has an authentication with SSH key. SSH key is secured by passphrase. Let's go to credentials, click again green plus.

    Name is aixsvc, organization is default. Credential type is machine. User name is aixsvc. We don't need a password, because we authenticate with an SSH key. The private SSH key we paste into the field "SSH private key". Into the field "Private key passphrase" I enter the passphrase to the SSH key. Privilege escalation method is sudo in my case. You can work with RBAC, if you want, but it requires a little bit tuning. "Privilege escalation user" is "root". Without RBAC only root can install security fixes. Because I don't have password requirement in sudo, I leave the field "Privilege escalation password" empty. Save.

    Now let's create the last piece in the puzzle - template. Templates -> green plus -> Job template.
    Name: patch_aix
    Job type: Run
    Inventory: nodes_to_patch
    Project: patchaix
    Playbook: flrtvc-sec.yml
    Credentials: aixsvc
    Verbosity: normal
    Enable privilege escalation: yes

    Something like:
    Job definition

    After save, we can launch. If you have wget installed on the target node, it should work:

    AWX Job Log



    On the node1 it installed one fix:

    # emgr -l

    ID  STATE LABEL      INSTALL TIME      UPDATED BY ABSTRACT
    === ===== ========== ================= ========== ======================================
    1   S     IJ25927s2a 09/28/20 21:59:15            IJ25927 for AIX 7.2 TL04 SP00 to SP02



    Hope, it helps!

    Andrey




    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 44.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Tue September 29, 2020 06:28 AM
    Andrey,

    Hi, I followed all the steps you set out and ran the "new" job however I still get the following error.

    Identity added: /tmp/awx_27_0y_qy6w_/artifacts/27/ssh_key_data (/tmp/awx_27_0y_qy6w_/artifacts/27/ssh_key_data) PLAY [FLRTVC on AIX playbook] ************************************************** TASK [Gathering Facts] ********************************************************* [WARNING]: Platform aix on host localhost is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See https://docs.ansible.com/ansible/2.9/referen ce_appendices/interpreter_discovery.html for more information. ok: [localhost] TASK [FLRTVC] ****************************************************************** fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "Sorry, user awx is not allowed to execute '/bin/sh -c echo BECOME-SUCCESS-nkijvhlifoqhskjokhxjfqchfajjeuxv ; /usr/bin/python' as root on <LPAR-name>.\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} PLAY RECAP ********************************************************************* localhost : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 ​


    1) I've done nothing with or to "awx" account on the AWX LPAR, does it need to be added to /etc/sudoers (which currently doesn't exist) for example?
    2) The SSH key (of my AIX admin service account) needing to be pasted into the credentials is id_rsa and NOT the id_rsa.pub, correct?

    Many thanks, Steve



    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 45.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Tue September 29, 2020 07:19 AM
    Steve,

    the error message you pasted shows that you didn't follow the steps or use another playbook. Please paste the playbook to use, if you want to debug it.

    Another error is that you use localhost in your inventory. localhost is a special case in Ansible, don't use it, if you don't need problems, or learn how to use it correctly.

    1) no, just don't use localhost, but the "official" name of the LPAR and let ansible connect to it by ssh, using your aix-admin-service-account.

    2) correct


    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 46.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Tue September 29, 2020 09:50 AM
    Andrey,

    Hi, thanks for the guidance around "localhost".  I updated the inventory and used an alternate "node" LPAR rather than looping back to the AWX LPAR itself.  Once done, I have a successful execution of the flrtvc playbook (my variant) which Emails me a report of the exposures.

    I'm still waiting for some DNS changes to be completed so tried to include the IP address of the AWX LPAR in the inventory but got an error related to permissions so will check into that.

    I'll keep you posted as to progress.

    Many thanks, Steve

    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 47.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Mon January 18, 2021 07:42 AM
    Andrey,

    Hi, Happy New Year!

    I decided to install a fresh AWX 15.0.1 on a fresh AIX 7.2 (7200-04-02-2028) so downloaded the image and installed the pre-reqs as usual.  I then expanded the installation media into /tmp/awx/awx-install before changing to that directory.  Here's what happened:

    abc123-awx:/tmp/awx# aix7.2.install.tar.gz | tar xf -                       <
    abc123-awx:/tmp/awx# cd awx-install
    abc123-awx:/tmp/awx/awx-install# ls -la
    total 32
    drwxr-x--x    4 501      staff           256 Oct 24 17:46 .
    drwxr-xr-x    3 root     system         4096 Jan 18 07:58 ..
    -rw-r-----    1 501      staff           308 Sep 29 09:59 awx_install.yml
    -rw-r-----    1 501      staff           149 Sep 29 09:59 awx_update.yml
    drwxr-x--x    2 501      staff          4096 Oct 27 07:10 files
    drwxr-x--x    2 501      staff           256 Oct 27 07:10 tasks
    abc123-awx:/tmp/awx/awx-install# calhost, -c local awx_install.yml          <
    
    PLAY [all] *********************************************************************
    
    TASK [Gathering Facts] *********************************************************
    ok: [localhost]
    
    TASK [copy prereq check script] ************************************************
    changed: [localhost]
    
    TASK [execute prereq check script] *********************************************
    changed: [localhost]
    
    TASK [remove prereq check script] **********************************************
    changed: [localhost]
    
    TASK [check if X11.base.lib is installed] **************************************
    changed: [localhost]
    
    TASK [check if rsyslog.base is installed] **************************************
    changed: [localhost]
    
    TASK [check if krb5.client.rte is installed] ***********************************
    changed: [localhost]
    
    TASK [update all packages] *****************************************************
    ok: [localhost]
    
    TASK [install required packages] ***********************************************
    ok: [localhost]
    
    TASK [make a temp install directory] *******************************************
    ok: [localhost]
    
    TASK [copy our required packages] **********************************************
    ok: [localhost] => (item=linux_compat-0.1.2-1.aix7.2.ppc.rpm)
    ok: [localhost] => (item=redis-5.0.9-1aix.aix7.2.ppc.rpm)
    ok: [localhost] => (item=supervisor-4.2.0-2aix.aix7.2.noarch.rpm)
    ok: [localhost] => (item=xmlsec1-1.2.28-1.aix7.2.ppc.rpm)
    ok: [localhost] => (item=xmlsec1-openssl-1.2.28-1.aix7.2.ppc.rpm)
    
    TASK [install our packages] ****************************************************
    ok: [localhost]
    
    TASK [get shells] **************************************************************
    changed: [localhost]
    
    TASK [set shells] **************************************************************
    changed: [localhost]
    
    TASK [copy awx rpm] ************************************************************
    changed: [localhost]
    
    TASK [install awx.rpm] *********************************************************
    ok: [localhost]
    
    TASK [remove temporary directory] **********************************************
    fatal: [localhost]: FAILED! => {"changed": false, "msg": "rmtree failed: [Errno 16] The requested resource is busy.: '/tmp/awx/awx-install'"}
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=16   changed=9    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
    
    abc123-awx:/tmp/awx/awx-install# ls -la
    total 8
    drwxr-x--x    2 501      staff           256 Jan 18 07:59 .
    drwxr-xr-x    3 root     system         4096 Jan 18 07:59 ..
    abc123-awx:/tmp/awx/awx-install#​

    Seems that as I'm "in" /tmp/awx/awx-install to run the playbook the resource (directory) is going to be busy and so the install fails.

    Thoughts?

    Many thanks, Steve

    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 48.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Mon January 18, 2021 08:12 AM
    Hi Steve,

    yes, the playbook uses /tmp/awx as a place to store some temporary files and tries to clean up it afterwards:

    - name: remove temporary directory
      file:
      path: /tmp/awx
      state: absent

    The easiest way to avoid the problem, is to use another directory, e.g. /tmp/awx-install or /tmp/awx15...

    Regards
    Andrey Klyachkin

    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 49.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Mon January 18, 2021 10:22 AM
    Andrey,

    Hi, I used /tmp/awx15 instead and all worked just fine :-)

    Regards, Steve

    ------------------------------
    Steve Munday
    AIX, IBM i, PowerVM
    ------------------------------



  • 50.  RE: Installing Ansible AWX on AIX 7.2

    Posted Wed November 11, 2020 12:22 AM
    Hello, 

    I need some help with this regards,
    Looks like my Ansible AWX tower broke and is throwing error as "502 Bad Gateway".
    I tried restarting host and below services too.

    nginx
    postgresql
    redis
    supervisord

    There were some stale PID's for supervisor which i delete but superisor log is showing below part.
    A help will be much appreciated.




    2020-11-10 15:00:22,271 INFO supervisord started with pid 13959612
    2020-11-10 15:00:23,282 INFO spawned: 'awx-config-watcher' with pid 7995814
    2020-11-10 15:00:23,287 INFO spawnerr: can't find command '/var/lib/awx/venv/awx/bin/uwsgi'
    2020-11-10 15:00:23,289 INFO spawnerr: can't find command '/var/lib/awx/venv/awx/bin/daphne'
    2020-11-10 15:00:23,289 INFO spawnerr: can't find command '/usr/bin/awx-manage'
    2020-11-10 15:00:23,299 INFO spawned: 'awx-rsyslogd' with pid 8388898
    2020-11-10 15:00:23,303 INFO spawnerr: can't find command '/usr/bin/awx-manage'
    2020-11-10 15:00:23,304 INFO spawnerr: can't find command '/usr/bin/awx-manage'
    2020-11-10 15:00:23,307 INFO exited: awx-config-watcher (exit status 127; not expected)
    2020-11-10 15:00:24,387 INFO spawned: 'awx-config-watcher' with pid 7995816
    2020-11-10 15:00:24,391 INFO spawnerr: can't find command '/var/lib/awx/venv/awx/bin/uwsgi'
    2020-11-10 15:00:24,393 INFO spawnerr: can't find command '/var/lib/awx/venv/awx/bin/daphne'
    2020-11-10 15:00:24,393 INFO spawnerr: can't find command '/usr/bin/awx-manage'
    2020-11-10 15:00:24,393 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
    2020-11-10 15:00:24,394 INFO spawnerr: can't find command '/usr/bin/awx-manage'
    2020-11-10 15:00:24,394 INFO spawnerr: can't find command '/usr/bin/awx-manage'
    2020-11-10 15:00:24,395 INFO exited: awx-rsyslogd (exit status 1; not expected)
    2020-11-10 15:00:24,424 INFO spawned: 'awx-rsyslogd' with pid 8388900
    2020-11-10 15:00:24,427 INFO exited: awx-config-watcher (exit status 127; not expected)
    2020-11-10 15:00:24,505 INFO exited: awx-rsyslogd (exit status 1; not expected)
    2020-11-10 15:00:25,515 INFO spawned: 'awx-rsyslogd' with pid 8388902
    2020-11-10 15:00:25,592 INFO exited: awx-rsyslogd (exit status 1; not expected)
    2020-11-10 15:00:26,602 INFO spawned: 'awx-config-watcher' with pid 8388904
    2020-11-10 15:00:26,607 INFO spawnerr: can't find command '/var/lib/awx/venv/awx/bin/uwsgi'
    2020-11-10 15:00:26,608 INFO spawnerr: can't find command '/var/lib/awx/venv/awx/bin/daphne'
    2020-11-10 15:00:26,608 INFO spawnerr: can't find command '/usr/bin/awx-manage'
    2020-11-10 15:00:26,609 INFO spawnerr: can't find command '/usr/bin/awx-manage'

    ------------------------------
    Gaurav Mishra
    ------------------------------



  • 51.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Wed November 11, 2020 03:35 AM
    Hi Guarav,


    it looks like you have removed AWX:

    2020-11-10 15:00:26,608 INFO spawnerr: can't find command '/var/lib/awx/venv/awx/bin/daphne'

    or installed/switched on something, that prevents execution of the commands. May be Trusted Execution?

    Regards
    Andrey Klyachkin

    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 52.  RE: Installing Ansible AWX on AIX 7.2

    Posted Wed November 11, 2020 08:54 AM
    Hi  Andrey,  

    I performed an Live kernel upgrade but took an alt disk prior to that.  since it was not working on new OS level i.e 7.2 TL4 SP1, i rolled it back to previous disk which was having oslevel as "7200-03-02-1845" but seeing above error.
    Not sure but should i execute below playbook and see if it reconfigures?

    ansible-playbook -i localhost, -c local awx_install.yml

    Thanks again
    Regards,
    Gaurav Mishra

    ------------------------------
    Gaurav Mishra
    ------------------------------



  • 53.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Wed November 11, 2020 09:49 AM
    Hi Gaurav,

    yes, it looks like you messed up your installation and the only way is to reinstall AWX. It doesn't matter with TL4 or with TL3.

    Regards
    Andrey

    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 54.  RE: Installing Ansible AWX on AIX 7.2

    Posted Wed November 11, 2020 02:09 PM
    Hey Andrey,

    Yeah looks like.
    Can you suggest how to cleanup old AWX.
    I tried to execute "ansible-playbook -i localhost, -c local awx_install.yml"  again  but it got failed on creating database structures.

    TASK [create database structures] ***************************************************************************************************************************
    FAILED - RETRYING: create database structures (3 retries left).
    FAILED - RETRYING: create database structures (2 retries left).
    FAILED - RETRYING: create database structures (1 retries left).
    fatal: [localhost]: FAILED! => {"attempts": 3, "changed": true, "cmd": ["su", "-", "awx", "-c", "awx-manage", "migrate"], "delta": "0:00:00.045606", "end": "2020-11-11 13:36:27.799373", "msg": "non-zero return code", "rc": 127, "start": "2020-11-11 13:36:27.753767", "stderr": "-bash: awx-manage: command not found", "stderr_lines": ["-bash: awx-manage: command not found"], "stdout": "", "stdout_lines": []}

    PLAY RECAP **************************************************************************************************************************************************
    localhost : ok=25 changed=15 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0

    ------------------------------
    Gaurav Mishra
    ------------------------------



  • 55.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Wed November 11, 2020 03:28 PM
    If it went to this point, it means, that YUM still thinks, that AWX is installed. You need first to uninstall RPM packages manually:

    rpm -e ansible-awx linux_compat redis supervisor xmlsec1 xmlsec1-openssl

    If it doesn't work, post the output here before proceeding further. If it works for you, then ran the AWX install playbook again.

    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 56.  RE: Installing Ansible AWX on AIX 7.2

    Posted Thu November 12, 2020 12:29 AM
    Hi Andrew, 

    It did worked, 
    I tried installing again and its stuck now on below.

    one thing i have observed is,  current version of ansible installed is "ansible-2.9.10-1.noarch".   which does not give me ansible-playbook command to execute.
    bash-5.0# ansible-playbook -i localhost, -c local awx_install.yml
    bash: /usr/bin/ansible-playbook: A file or directory in the path name does not exist.

    once i installed ansible.pcc (2.6.2-1)   command is executable and it gives below error.. something which i am noticing for the first time.



    bash-5.0# ansible-playbook -i localhost, -c local awx_install.yml

    PLAY [all] **************************************************************************************************************************************************

    TASK [Gathering Facts] **************************************************************************************************************************************
    ok: [localhost]

    TASK [copy prereq check script] *****************************************************************************************************************************
    changed: [localhost]

    TASK [execute prereq check script] **************************************************************************************************************************
    changed: [localhost]

    TASK [remove prereq check script] ***************************************************************************************************************************
    changed: [localhost]

    TASK [check if X11.base.lib is installed] *******************************************************************************************************************
    changed: [localhost]

    TASK [check if rsyslog.base is installed] *******************************************************************************************************************
    changed: [localhost]

    TASK [check if krb5.client.rte is installed] ****************************************************************************************************************
    changed: [localhost]

    TASK [update all packages] **********************************************************************************************************************************
    changed: [localhost]

    TASK [install required packages] ****************************************************************************************************************************
    fatal: [localhost]: FAILED! => {"msg": "the connection plugin 'local' was not found"}
    to retry, use: --limit @/home/gmishra/awx-install/awx_install.retry

    PLAY RECAP **************************************************************************************************************************************************
    localhost : ok=8 changed=7 unreachable=0 failed=1


    Thank you
    Reards,
    Gaurav Mishra

    ------------------------------
    Gaurav Mishra
    ------------------------------



  • 57.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Thu November 12, 2020 02:57 AM
    Hi Guarav,

    AWX requires ansible 2.8 and newer. Ansible 2.9.10 from IBM has ansible-playbook, but as you can read on this forum it is not anymore installed in /usr/bin, but in /opt/freeware/bin. So you have to update your PATH variable to include /opt/freeware/bin or write the whole path to the command: /opt/freeware/bin/ansible-playbook.

    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 58.  RE: Installing Ansible AWX on AIX 7.2

    Posted Thu November 12, 2020 10:07 PM
    Hi,

    It went a step ahead post cleanup and installation of ansible 2.9. now stuck on below.
    Looks like still some cleanup is required.


    TASK [create superuser] *************************************************************************************************************************************
    fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["su", "-", "awx", "-c", "awx-manage", "createsuperuser", "--username", "admin", "--email", "root@localhost", "--noinput"], "delta": "0:00:05.185482", "end": "2020-11-12 21:36:47.141282", "msg": "non-zero return code", "rc": 1, "start": "2020-11-12 21:36:41.955800", "stderr": "CommandError: Error: That username is already taken.", "stderr_lines": ["CommandError: Error: That username is already taken."], "stdout": "", "stdout_lines": []}

    PLAY RECAP **************************************************************************************************************************************************
    localhost : ok=26 changed=20 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0


    Thank you
    Regards,
    Gaurav Mishra

    ------------------------------
    Gaurav Mishra
    ------------------------------



  • 59.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Mon November 16, 2020 06:24 AM
    Hi Gaurav,

    if you don't need the database, you can always drop it (su - postgres, dropdb awx) and it will be recreated by the playbook. If you need it, you can change the administrator user's name in the playbook (tasks/04_awxconfig.yml, su - awx -c awx-manage createsuperuser --username admin --email root@localhost --noinput) to something else and then delete the unneeded user.

    Regards
    Andrey

    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 60.  RE: Installing Ansible AWX on AIX 7.2

    Posted Mon November 16, 2020 04:31 PM
    Thanks Andrey, it went through and i can access the ansible web portal now.
    But looks like there is some post manual changes in permission or Database required since i am getting below error while i am syncing any inventory or running any command on the host from tower.

    Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/ansible_runner/runner.py", line 179, in run use_poll=self.config.pexpect_use_poll, File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/pexpect/pty_spawn.py", line 204, in __init__ self._spawn(command, args, preexec_fn, dimensions) File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/pexpect/pty_spawn.py", line 276, in _spawn 'executable: %s.' % self.command) pexpect.exceptions.ExceptionPexpect: The command was not found or was not executable: ansible. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/redis/connection.py", line 552, in connect sock = self._connect() File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/redis/connection.py", line 905, in _connect sock.connect(self.path) PermissionError: [Errno 13] Permission denied During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/main/tasks.py", line 1479, in run res = ansible_runner.interface.run(**params) File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/ansible_runner/interface.py", line 178, in run r.run() File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/ansible_runner/runner.py", line 200, in run stdout_handle.write(_decode('\n')) File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/ansible_runner/utils.py", line 312, in write self._emit_event(line) File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/ansible_runner/utils.py", line 349, in _emit_event self._event_callback(event_data) File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/ansible_runner/runner.py", line 69, in event_callback should_write = self.event_handler(event_data) File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/main/tasks.py", line 1242, in event_handler self.dispatcher.dispatch(event_data) File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/awx/main/queue.py", line 35, in dispatch self.connection.rpush(self.queue, json.dumps(obj, cls=AnsibleJSONEncoder)) File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/redis/client.py", line 1981, in rpush return self.execute_command('RPUSH', name, *values) File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/redis/client.py", line 875, in execute_command conn = self.connection or pool.get_connection(command_name, **options) File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/redis/connection.py", line 1185, in get_connection connection.connect() File "/var/lib/awx/venv/awx/lib/python3.7/site-packages/redis/connection.py", line 557, in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error 13 connecting to unix socket: /var/run/redis/redis.sock. Permission denied.SHOW LESS
    Run


    ------------------------------
    Gaurav Mishra
    ------------------------------



  • 61.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Tue November 17, 2020 03:48 AM
    I suppose, you have some problem with redis permission.

    Check, that any member of the redis group can write to /var/run/redis/redis.sock:

    # ls -ld /var/run
    # ls -ld /var/run/redis
    # ls -l /var/run/redis/redis.sock

    Check, that the unix socket is enabled in /opt/freeware/etc/redis.conf and has correct permissions:

    unixsocket /var/run/redis/redis.sock
    unixsocketperm 0775

    Check, that redis runs.

    ps -ef | grep redis

    Check, that the user awx is a member of redis group.

    # lsgroup redis

    ​​​​

    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 62.  RE: Installing Ansible AWX on AIX 7.2

    Posted Tue November 17, 2020 01:43 PM
    Thanks Andrey, it went through.
    Now any way we can explicitely specify /opt/freeware/bin during runtime.  Looks like its not picking up ansible path by default.
    I added it in root's and awx users .profile still below error.

    Thanks




    ------------------------------
    Gaurav Mishra
    ------------------------------



  • 63.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Tue November 17, 2020 03:30 PM
    If you added it to the root profile after starting AWX, you must restart supervisord. In the latest AWX RPM 15.0.1 I added to /etc/rc.d/init.d/supervisord:

    # workaround if root have too many allowed descriptors
    ulimit -n 65536
    # /opt/freeware/bin must be in PATH
    export PATH=/opt/freeware/bin:$PATH

    It solves the problem with PATH. If you don't see the lines in /etc/rc.d/init.d/supervisord you should think about update. Otherwise you can add the lines into the script in function start_app() before starting supervisord and restart supervisord.

    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 64.  RE: Installing Ansible AWX on AIX 7.2

    Posted Mon November 30, 2020 01:14 PM
    Any comment on below error from Tower logs...

    <codendcpapp347.agrium.com> (255, b'', b'Control socket connect(/tmp/awx_48__59ptqfc/cp): Socket operation on non-socket\r\nPermission denied, please try again.\r\nPermission denied, please try again.\r\nroot@codendcpapp347.agrium.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).\r\n')
    47
    codendcpapp347.agrium.com | FAILED | rc=-1 >>
    48
    Failed to connect to the host via ssh: Control socket connect(/tmp/awx_48__59ptqfc/cp): Socket operation on non-socket
    49
    Permission denied, please try again.
    50
    Permission denied, please try again.
    51
    root@codendcpapp347.agrium.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).


    ------------------------------
    Gaurav Mishra
    ------------------------------



  • 65.  RE: Installing Ansible AWX on AIX 7.2

    Posted Thu November 19, 2020 07:41 PM
    Thanks Andrey for the help, I am able to proceed.
    Will update for any further help required.

    Much appreciated.

    Regards,
    Gaurav Mishra

    ------------------------------
    Gaurav Mishra
    ------------------------------



  • 66.  RE: Installing Ansible AWX on AIX 7.2

    Posted Wed November 25, 2020 03:41 AM
    Hi,
    I try to install awx on ppc64le following your nice procedure. I'm able to run the ansible playbook.
    .
    "1. Login to your AIX server",
    "2. Create a password for the AWX superuser using the following command:",
    "su - awx -c awx-manage changepassword admin",
    "3. Start the browser and go to:",
    "https://awxmistral/#login"
    .
    The process are running.
    # /opt/freeware/bin/supervisorctl
    awx-config-watcher RUNNING pid 5702074, uptime 0:28:03
    tower-processes:awx-callback-receiver RUNNING pid 7536964, uptime 0:28:02
    tower-processes:awx-daphne RUNNING pid 4850128, uptime 0:28:03
    tower-processes:awx-dispatcher RUNNING pid 14877096, uptime 0:00:05
    tower-processes:awx-rsyslogd RUNNING pid 6160838, uptime 0:28:02
    tower-processes:awx-uwsgi RUNNING pid 6095326, uptime 0:28:03
    tower-processes:awx-wsbroadcast RUNNING pid 12976528, uptime 0:00:08
    .
    But I'm still unable to connect to my AWX GUI, message "This site can't be reached"
    In fact the nginx process is not running. When I try to start it, I have this error.
    .
    # /etc/rc.d/init.d/nginx start
    Starting nginx... exec(): 0509-036 Cannot load program /opt/freeware/sbin/nginx because of the following errors:
    0509-022 Cannot load module /opt/freeware/lib/libXpm.a(shr_64.o).
    0509-150 Dependent module /opt/freeware/lib/libX11.a(shr_64.o) could not be loaded.
    0509-152 Member shr_64.o is not found in archive
    0509-022 Cannot load module nginx.
    0509-150 Dependent module /opt/freeware/lib/libXpm.a(shr_64.o) could not be loaded.
    .
    I installed the rpm
    # rpm -qa |grep Xp
    libXpm-3.5.13-1.ppc
    And run command /usr/sbin/updtvpkg but I still have the same error about libXpm.a
    .
    Do you have any idea how I can recover from this nginx error ? 
    Thanks for your support.





    ------------------------------
    Thierry Huche
    IBM France
    33499580675
    ------------------------------



  • 67.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Wed November 25, 2020 05:25 AM
    Hi Thierry,

    AIX Toolbox for Linux applications has no libXpm RPM package (https://www.ibm.com/support/pages/aix-toolbox-linux-applications-downloads-alpha#L). It looks like either you have RPM packages from Michael Perzl or you have a mix of different packages. It will not work. I compiled and tested AWX only with RPM packages from AIX Toolbox for Linux applications, so you have first to "upgrade" your RPMs.

    Regards
    Andrey

    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 68.  RE: Installing Ansible AWX on AIX 7.2

    Posted Tue December 22, 2020 01:39 PM

    Hi Andrey,

    Getting below error while running any command against servers from AWX. credentials are verified and are working fine.

    is there any config change/setting i need to do.

    I am just running command/shell from AWX for a test server.

    [WARNING]: sftp transfer mechanism failed on [dendcpnim02]. Use ANSIBLE_DEBUG=1
    to see detailed information
    [WARNING]: scp transfer mechanism failed on [dendcpnim02]. Use ANSIBLE_DEBUG=1
    to see detailed information
    dendcpnim02 | UNREACHABLE! => {
    "changed": false,
    "msg": "Data could not be sent to remote host \"dendcpnim02\". Make sure this host can be reached over ssh: Control socket connect(/tmp/awx_63_buvqii2u/cp): Socket operation on non-socket\r\nPermission denied, please try again.\r\nPermission denied, please try again.\r\nroot@dendcpnim02: Permission denied (publickey,password,keyboard-interactive).\r\n",
    "unreachable": true
    }



    ------------------------------
    Gaurav Mishra
    ------------------------------



  • 69.  RE: Installing Ansible AWX on AIX 7.2

    Posted Fri March 26, 2021 03:11 PM
    Hello,
    I have problem with uwsgi:

    Could not load program uwsgi:
    rtld: 0712-001 Symbol _GLOBAL__AIXI_libintl_so was referenced
    from module uwsgi(), but a runtime definition
    of the symbol was not found.
    rtld: 0712-001 Symbol _GLOBAL__AIXD_libintl_so was referenced
    from module uwsgi(), but a runtime definition
    of the symbol was not found.
    rtld: 0712-002 fatal error: exiting.

    Any idea whats wrong?
    All my rpms are from IBM toolbox, downloaded ansible-awx-17.1.0.aix7.2.install.tar.gz, AIX is 7200-04-02-2028

    ------------------------------
    maesho po
    ------------------------------



  • 70.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Mon March 29, 2021 07:32 AM
    Hi Maesho,

    I checked on AIX 7.2 TL4 SP2 and as for me it works.


    Can you check the output of ldd /var/lib/awx/venv/awx/bin/uwsgi?



    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 71.  RE: Installing Ansible AWX on AIX 7.2

    Posted Tue April 06, 2021 03:06 AM
    Hello,
    sorry I forgot to provide this output, here it is:

    root@utaxanst:/root # ldd /var/lib/awx/venv/awx/bin/uwsgi
    /var/lib/awx/venv/awx/bin/uwsgi needs:
             /usr/lib/libpthread.a(shr_xpg5_64.o)
             /usr/lib/liblinux_compat.a(liblinux_compat.so.1)
             /opt/freeware/lib/libpcre.a(libpcre.so.1)
             /usr/lib/libssl.a(libssl.so.1.0.2)
             /usr/lib/libcrypto.a(libcrypto.so.1.0.2)
             /opt/freeware/lib/libxml2.a(libxml2.so.2)
             /opt/freeware/lib64/liblzma.a(liblzma.so.5)
             /opt/freeware/lib/libiconv.a(libiconv.so.2)
             /opt/freeware/lib64/libintl.a(libintl.so.8)
             /usr/lib/libdl.a(shr_64.o)
             /opt/freeware/lib64/libexpat.a(libexpat.so.1)
             /opt/freeware/lib/libffi.a(libffi.so.6)
             /opt/freeware/lib64/libncurses.so
             /opt/freeware/lib64/libsqlite3.so
             /opt/freeware/lib64/libpython3.7m.so
             /usr/lib/libcrypt.a(shr_64.o)
             /usr/lib/libc.a(shr_64.o)
             /unix
             /opt/freeware/lib64/libgcc_s.a(shr.o)
             /opt/freeware/lib/libz.a(libz.so.1)
             /usr/lib/libpthreads.a(shr_xpg5_64.o)


    Caould be that we dont have most actual repository...in our company we dont have direct acces to IBM repo but we syncing it to local repo.....


    root@utaxanst:/root # yum list installed
    Installed Packages
    AIX-rpm.ppc                                                               7.2.4.1-57                                                         installed
    GeoIP.ppc                                                                 1.6.12-1                                                           @AIX_Toolbox
    ansible.noarch                                                            2.9.7-1                                                            @AIX_Toolbox_noarch
    ansible-awx.ppc                                                           17.1.0-1aix                                                        installed
    atk.ppc                                                                   2.20.0-3                                                           installed
    bash.ppc                                                                  5.0.18-1                                                           installed
    bzip2.ppc                                                                 1.0.8-2                                                            installed
    ca-certificates.ppc                                                       2020.06.01-1                                                       installed
    cairo.ppc                                                                 1.14.6-2waixX11                                                    installed
    coreutils.ppc                                                             8.32-1                                                             installed
    cups.ppc                                                                  2.2.3-3                                                            installed
    cups-ipptool.ppc                                                          2.2.3-3                                                            installed
    cups-libs.ppc                                                             2.2.3-3                                                            installed
    cups-lpd.ppc                                                              2.2.3-3                                                            installed
    curl.ppc                                                                  7.71.1-1                                                           installed
    cyrus-sasl.ppc                                                            2.1.26-3                                                           installed
    db.ppc                                                                    6.2.38-2                                                           installed
    dbus.ppc                                                                  1.13.18-1                                                          installed
    expat.ppc                                                                 2.2.9-2                                                            installed
    file.ppc                                                                  5.38-1                                                             installed
    file-libs.ppc                                                             5.38-1                                                             installed
    fontconfig.ppc                                                            2.11.95-4                                                          installed
    freetype2.ppc                                                             2.8-1                                                              installed
    gcc.ppc                                                                   8.3.0-2                                                            installed
    gcc-cpp.ppc                                                               8.3.0-2                                                            installed
    gdbm.ppc                                                                  1.18.1-1                                                           installed
    gdk-pixbuf.ppc                                                            2.35.1-2waixX11                                                    installed
    gettext.ppc                                                               0.19.8.1-5                                                         installed
    git.ppc                                                                   2.20.4-1                                                           @AIX_Toolbox
    glib2.ppc                                                                 2.56.1-2                                                           installed
    gmp.ppc                                                                   6.1.2-1                                                            installed
    gnutls.ppc                                                                3.6.14-1                                                           installed
    grep.ppc                                                                  3.4-1                                                              installed
    gtk2.ppc                                                                  2.24.30-2waixX11                                                   installed
    gzip.ppc                                                                  1.10-1                                                             installed
    harfbuzz.ppc                                                              1.2.7-2                                                            installed
    host-power-tools.ppc                                                      4.2.0-1                                                            installed
    infinidat-odm.ppc                                                         2.2.3.2-1                                                          installed
    info.ppc                                                                  6.6-2                                                              installed
    jbigkit-libs.ppc                                                          2.1-1                                                              installed
    krb5-libs.ppc                                                             1.16.1-4                                                           installed
    less.ppc                                                                  557-1                                                              installed
    libXcursor.ppc                                                            1.1.14-1                                                           installed
    libXcursor-devel.ppc                                                      1.1.14-1                                                           installed
    libXft.ppc                                                                2.3.2-4waixX11                                                     installed
    libXrender.ppc                                                            0.9.8-2waixX11                                                     installed
    libffi.ppc                                                                3.2.1-3                                                            installed
    libgcc.ppc                                                                8.3.0-2                                                            installed
    libgcrypt.ppc                                                             1.8.5-1                                                            installed
    libgd.ppc                                                                 2.2.5-3                                                            @AIX_Toolbox
    libgpg-error.ppc                                                          1.36-1                                                             installed
    libiconv.ppc                                                              1.16-1                                                             installed
    libjpeg.ppc                                                               9d-1                                                               installed
    libmpc.ppc                                                                1.1.0-1                                                            installed
    libnghttp2.ppc                                                            1.41.0-1                                                           installed
    libpng.ppc                                                                1.6.27-3                                                           installed
    libsodium.ppc                                                             1.0.18-1                                                           @AIX_Toolbox
    libssh2.ppc                                                               1.9.0-1                                                            installed
    libstdc++.ppc                                                             8.3.0-2                                                            installed
    libtasn1.ppc                                                              4.16.0-1                                                           installed
    libtiff.ppc                                                               4.1.0-1                                                            installed
    libunistring.ppc                                                          0.9.9-2                                                            installed
    libwebp.ppc                                                               1.0.2-1                                                            installed
    libxml2.ppc                                                               2.9.9-1                                                            installed
    libxslt.ppc                                                               1.1.32-1                                                           @AIX_Toolbox
    libyaml.ppc                                                               0.2.2-1                                                            @AIX_Toolbox
    linux_compat.ppc                                                          0.1.2-1                                                            installed
    lynx.ppc                                                                  2.8.8-3ssl                                                         installed
    lz4.ppc                                                                   1.9.2-1                                                            installed
    lzo.ppc                                                                   2.10-2                                                             installed
    make.ppc                                                                  1:4.2.1-5                                                          installed
    mpfr.ppc                                                                  4.0.2-2                                                            installed
    mtr.ppc                                                                   0.84-1                                                             installed
    ncurses.ppc                                                               6.2-2                                                              installed
    nettle.ppc                                                                3.6-1                                                              installed
    nginx.ppc                                                                 1.17.8-1                                                           @AIX_Toolbox
    openldap.ppc                                                              2.4.50-1                                                           installed
    p11-kit.ppc                                                               0.23.16-1                                                          installed
    p11-kit-tools.ppc                                                         0.23.16-1                                                          installed
    pango.ppc                                                                 1.40.1-2waixX11                                                    installed
    parallel.ppc                                                              20190622-1                                                         installed
    pcre.ppc                                                                  8.44-1                                                             installed
    perl.ppc                                                                  1:5.30.3-1                                                         installed
    pixman.ppc                                                                0.34.0-1                                                           installed
    postgresql.ppc                                                            11.7-2                                                             @AIX_Toolbox
    postgresql-libs.ppc                                                       11.7-2                                                             @AIX_Toolbox
    postgresql-server.ppc                                                     11.7-2                                                             @AIX_Toolbox
    pysqlite.ppc                                                              2.8.3-2                                                            installed
    python.ppc                                                                2.7.18-1                                                           installed
    python-dateutil.noarch                                                    2.6.0-1                                                            installed
    python-devel.ppc                                                          2.7.18-1                                                           installed
    python-iniparse.noarch                                                    0.4-1                                                              installed
    python-pycurl.ppc                                                         7.43.0-1                                                           installed
    python-six.noarch                                                         1.10.0-1                                                           installed
    python-tools.ppc                                                          2.7.18-1                                                           installed
    python-urlgrabber.noarch                                                  3.10.1-1                                                           installed
    python3.ppc                                                               3.7.9-1                                                            @AIX_Toolbox
    python3-jinja2.noarch                                                     2.10.3-1                                                           @AIX_Toolbox_noarch
    python3-jmespath.noarch                                                   0.9.4-1                                                            @AIX_Toolbox_noarch
    python3-markupsafe.ppc                                                    1.1.1-1                                                            @AIX_Toolbox
    python3-pyyaml.ppc                                                        5.3.1-1                                                            @AIX_Toolbox
    python3-six.noarch                                                        1.13.0-1                                                           @AIX_Toolbox_noarch
    readline.ppc                                                              8.0-2                                                              installed
    redis.ppc                                                                 5.0.9-1aix                                                         installed
    rsync.ppc                                                                 3.2.3-1                                                            installed
    screen.ppc                                                                4.7.0-1                                                            installed
    shared-mime-info.ppc                                                      1.6-2                                                              installed
    socat.ppc                                                                 1.7.3.4-1                                                          installed
    sqlite.ppc                                                                3.32.3-1                                                           installed
    sshpass.ppc                                                               1.06-2                                                             @AIX_Toolbox
    sudo_ids.ppc                                                              1.8.31p1-2                                                         installed
    supervisor.noarch                                                         4.2.0-2aix                                                         installed
    tar.ppc                                                                   1.32-1                                                             installed
    tcl.ppc                                                                   8.6.8-3                                                            @AIX_Toolbox
    tcsh.ppc                                                                  6.11-3                                                             installed
    tightvnc-server.ppc                                                       1.3.10-1                                                           installed
    unrar.ppc                                                                 5.9.2-1                                                            installed
    unzip.ppc                                                                 6.0-3                                                              installed
    vim-common.ppc                                                            8.1.2424-1                                                         installed
    vim-enhanced.ppc                                                          8.1.2424-1                                                         installed
    wget.ppc                                                                  1.20.3-1                                                           installed
    xmlsec1.ppc                                                               1.2.28-1                                                           installed
    xmlsec1-openssl.ppc                                                       1.2.28-1                                                           installed
    xz-libs.ppc                                                               5.2.5-1                                                            installed
    yum.noarch                                                                3.4.3-7                                                            installed
    yum-metadata-parser.ppc                                                   1.1.4-2                                                            installed
    zip.ppc                                                                   3.0-4                                                              installed
    zlib.ppc                                                                  1.2.11-1                                                           installed






  • 72.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Thu April 08, 2021 11:02 AM
    Hi Maesho,

    yes, the problem is because you have an old version of gettext. I was able to reproduce the error with your version of gettext. Please update the package to the newest version:

    # yum update gettext
    Setting up Update Process
    Resolving Dependencies
    --> Running transaction check
    ---> Package gettext.ppc 0:0.19.8.1-5 will be updated
    ---> Package gettext.ppc 0:0.20.2-1 will be an update
    --> Finished Dependency Resolution

    Dependencies Resolved

    ===========================================================================================================================================================================
    Package Arch Version Repository Size
    ===========================================================================================================================================================================
    Updating:
    gettext ppc 0.20.2-1 AIX_Toolbox 11 M

    Transaction Summary
    ===========================================================================================================================================================================
    Upgrade 1 Package

    Then restart supervisord:

    # /etc/rc.d/init.d/supervisord stop
    # /etc/rc.d/init.d/supervisord start
    # /opt/freeware/bin/supervisorctl status
    awx-config-watcher RUNNING pid 9699766, uptime 0:00:26
    tower-processes:awx-callback-receiver RUNNING pid 9044308, uptime 0:00:26
    tower-processes:awx-daphne RUNNING pid 5439754, uptime 0:00:26
    tower-processes:awx-dispatcher RUNNING pid 12124422, uptime 0:00:26
    tower-processes:awx-rsyslogd RUNNING pid 17826070, uptime 0:00:26
    tower-processes:awx-uwsgi RUNNING pid 9896332, uptime 0:00:26
    tower-processes:awx-wsbroadcast RUNNING pid 9830766, uptime 0:00:26
    ​​​​

    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 73.  RE: Installing Ansible AWX on AIX 7.2

    Posted Fri June 17, 2022 06:44 AM
    Hi Andrey, thankyou for sharing your work, really appreciated !
    after AWX successfull installation I find first issue with very simple playbook, I run a job on a single AIX client but I receive this error:

    Identity added: /tmp/bwrap_13_vkj2xzml/awx_13_5ovhho7d/artifacts/13/ssh_key_data (root@host1)

    PLAY [host1.example.com] ***************************************************

    TASK [Gathering Facts] *********************************************************
    fatal: [host1.example.com]: FAILED! => {"ansible_facts": {}, "changed": false, "failed_modules": {"setup": {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "failed": true, "module_stderr": "Shared connection to svlsrv80.iside.bcc.it closed.\\r\\n", "module_stdout": "/bin/sh: /usr/bin/python: not found.\\r\\n", "msg": "MODULE FAILURE\\nSee stdout/stderr for the exact error", "rc": 127, "warnings": ["No python interpreters found for host host1.example.com (tried ['/usr/bin/python', 'python3.7', 'python3.6', 'python3.5', 'python2.7', 'python2.6', '/usr/libexec/platform-python', '/usr/bin/python3', 'python'])"]}}, "msg": "The following modules failed to execute: setup\\n"}

    PLAY RECAP *********************************************************************
    host1.example.com : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

    the error is clear, there is not any python installation on client and it is true, but do I really need to install python on all my aix client ? Should not be "agentless" and free by any additional software ?
    I tried the same playbook on "localhost" host where python is installed and it works.
    this is the playbook (very very simple ... )

    ---
    - hosts: host1.example.com
    tasks:
    - name: Print message
    debug:
    msg: Hello Ansible World


    ------------------------------
    michele dainotti
    ------------------------------



  • 74.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Fri June 17, 2022 07:27 AM
    Hi Michele,

    yes, Python is a prerequisite for Ansible. It is very easy with AIX 7.3 because it has Python in it. On AIX 7.2 and AIX 7.1 you have to install Python first to manage them with Ansible.

    You can do it manually (in this I case I recommend to install yum/dnf first).

    You can try using power_aix_bootstrap role from ibm.power_aix collection.

    E.g. see the article on using Ansible on AIX - https://www.ansible.com/blog/aix-patch-management-with-ansible

    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 75.  RE: Installing Ansible AWX on AIX 7.2

    Posted Fri June 17, 2022 08:00 AM
    thankyou Andrey, ok now it is clear to me.
    Another question, is there a way to configure proxy while setting up a new project in order to reach external resource (e.g. github.com)
    At the moment I cannot sync because my server is behind a proxy.
    thankyou


    ------------------------------
    michele dainotti
    ------------------------------



  • 76.  RE: Installing Ansible AWX on AIX 7.2

    IBM Champion
    Posted Fri June 17, 2022 08:33 AM
    Michele,

    I never did it, but Tower documentation says, that you can add extra environment variables in "Settings > Jobs":

    {
    "https_proxy": "http://192.168.1.1:3128"
    }


    ------------------------------
    Andrey Klyachkin

    https://www.power-devops.com
    ------------------------------



  • 77.  RE: Installing Ansible AWX on AIX 7.2

    Posted Fri June 17, 2022 08:53 AM
    Yes it is the same solution I found on google but seems like "edit" function is not yet available:


    I also tried to export as a varible (export http_proxy=.....) without success.. however it is not so important.

    thankyou again, bye
    Michele


    ------------------------------
    michele dainotti
    ------------------------------