Hello @Andrey Klyachkin and thanks for your suggestions!
Regarding your bullet point #2 --> it seems to be related to the v2 to v3 API change, all other collections are working (except https://galaxy.ansible.com/ui/repo/published/fedora/linux_system_roles/ , but this is not urgently needed). But I can´t point to the error which causes the problem, maybe the great number of revisions that are tried while downloading?
Regarding your bullet point #4 --> I´ve tried it on another AIX server of mine and get the same results, even though the ansible environment there was in a "virgin" state:
[USER@SERVER2:/home/USER/TEST]# ansible-galaxy collection install community.general --force -vvv
ansible-galaxy [core 2.14.2]
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/USER/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /opt/freeware/lib/python3.9/site-packages/ansible
ansible collection location = /home/USER/.ansible/collections:/usr/share/ansible/collections
executable location = /opt/freeware/bin/ansible-galaxy.orig
python version = 3.9.18 (main, Sep 19 2023, 04:57:07) [GCC 10.3.0] (/opt/freeware/bin/python3.9)
jinja version = 3.1.2
libyaml = True
Using /etc/ansible/ansible.cfg as config file
Starting galaxy collection install process
Process install dependency map
Opened /home/USER/.ansible/galaxy_token
ERROR! Unexpected Exception, this is probably a bug:
the full traceback was:
Traceback (most recent call last):
File "/opt/freeware/lib/python3.9/site-packages/resolvelib/resolvers.py", line 171, in _merge_into_criterion
crit = self.state.criteria[name]
KeyError: 'community.general'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/freeware/lib/python3.9/site-packages/ansible/cli/__init__.py", line 647, in cli_executor
exit_code = cli.run()
File "/opt/freeware/lib/python3.9/site-packages/ansible/cli/galaxy.py", line 714, in run
return context.CLIARGS['func']()
File "/opt/freeware/lib/python3.9/site-packages/ansible/cli/galaxy.py", line 115, in method_wrapper
return wrapped_method(*args, **kwargs)
File "/opt/freeware/lib/python3.9/site-packages/ansible/cli/galaxy.py", line 1363, in execute_install
self._execute_install_collection(
File "/opt/freeware/lib/python3.9/site-packages/ansible/cli/galaxy.py", line 1400, in _execute_install_collection
install_collections(
File "/opt/freeware/lib/python3.9/site-packages/ansible/galaxy/collection/__init__.py", line 730, in install_collections
dependency_map = _resolve_depenency_map(
File "/opt/freeware/lib/python3.9/site-packages/ansible/galaxy/collection/__init__.py", line 1777, in _resolve_depenency_map
return collection_dep_resolver.resolve(
File "/opt/freeware/lib/python3.9/site-packages/resolvelib/resolvers.py", line 453, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/opt/freeware/lib/python3.9/site-packages/resolvelib/resolvers.py", line 318, in resolve
name, crit = self._merge_into_criterion(r, parent=None)
File "/opt/freeware/lib/python3.9/site-packages/resolvelib/resolvers.py", line 173, in _merge_into_criterion
crit = Criterion.from_requirement(self._p, requirement, parent)
File "/opt/freeware/lib/python3.9/site-packages/resolvelib/resolvers.py", line 79, in from_requirement
cands = build_iter_view(provider.find_matches([requirement]))
File "/opt/freeware/lib/python3.9/site-packages/ansible/galaxy/dependency_resolution/providers.py", line 506, in find_matches
return self._find_matches(requirements)
File "/opt/freeware/lib/python3.9/site-packages/ansible/galaxy/dependency_resolution/providers.py", line 395, in _find_matches
signatures = src_server.get_collection_signatures(first_req.namespace, first_req.name, version)
File "/opt/freeware/lib/python3.9/site-packages/ansible/galaxy/api.py", line 121, in wrapped
return method(self, *args, **kwargs)
File "/opt/freeware/lib/python3.9/site-packages/ansible/galaxy/api.py", line 904, in get_collection_signatures
self._set_cache()
File "/opt/freeware/lib/python3.9/site-packages/ansible/galaxy/api.py", line 46, in wrapped
return func(*args, **kwargs)
File "/opt/freeware/lib/python3.9/site-packages/ansible/galaxy/api.py", line 433, in _set_cache
fd.write(to_bytes(json.dumps(self._cache), errors='surrogate_or_strict'))
File "/opt/freeware/lib64/python3.9/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/opt/freeware/lib64/python3.9/json/encoder.py", line 202, in encode
return ''.join(chunks)
MemoryError
[USER@SERVER2:/home/USER/TEST]# oslevel -s
7200-05-06-2320
I haven´t tried so far to upgrade the package (for my central AIX Ansible project) via a tar.gz since I need some of the modules and don´t want to ruin it, so I´m waiting on version 7.4 of community.general.
Thanks for your input,
With kind regards,
Stephan Dietl
------------------------------
Stephan Dietl
------------------------------
Original Message:
Sent: Tue October 24, 2023 02:24 AM
From: Andrey Klyachkin
Subject: Ansible Galaxy Collection community.general can´t be upgraded AIX 7.2
Hi Stephan,
I don't have the problem you describe and can still download community.general from Ansible Galaxy. But some thoughts:
- from time to time I also have occasionally KeyErrors, but they are not persistent. Next time I do the same operation, everything works.
- a month or two ago Red Hat updated Galaxy. Even if I didn't encounter any problems after the update and I don't think Red Hat broke some APIs, it could happen.
- if you work in an enterprise environment and through proxy, your proxy server could check traffic for viruses and malware and return broken answers to you in some specific cases.
- I'd also try to issue the same command on another AIX server just to check that the problem has nothing to do with the server itself.
I hope it helps! If you find the reason of the problem let us all know ;-)
------------------------------
Andrey Klyachkin
https://www.power-devops.com
Original Message:
Sent: Mon October 23, 2023 02:43 PM
From: Stephan Dietl
Subject: Ansible Galaxy Collection community.general can´t be upgraded AIX 7.2
Hello!
Anybody also experiencing this? Any ideas :) ?
Thanks,
With kind regards,
Stephan Dietl
------------------------------
Stephan Dietl
Original Message:
Sent: Fri October 20, 2023 08:54 AM
From: Stephan Dietl
Subject: Ansible Galaxy Collection community.general can´t be upgraded AIX 7.2
Hello!
According to my logs between 14th. of September and 2.nd of October my Ansible Galaxy Upgrade Script stopped working only for "community.general" (others work fine), here the very verbose output of a manual trigger:
[USER@SERVER:/SCRIPTS/ANSIBLE/]# ansible-galaxy collection install community.general --force -vvvv
ansible-galaxy [core 2.14.2]
config file = /SCRIPTS/ANSIBLE/ansible.cfg
configured module search path = ['/SCRIPTS/ANSIBLE/collections']
ansible python module location = /opt/freeware/lib/python3.9/site-packages/ansible
ansible collection location = /SCRIPTS/ANSIBLE/collections
executable location = /opt/freeware/bin/ansible-galaxy.orig
python version = 3.9.18 (main, Sep 19 2023, 04:57:07) [GCC 10.3.0] (/opt/freeware/bin/python3.9)
jinja version = 3.1.2
libyaml = True
Using /SCRIPTS/ANSIBLE/ansible.cfg as config file
Galaxy cache file at '/home/die/.ansible/galaxy_cache/api.json' has an invalid version, clearing
Starting galaxy collection install process
Found installed collection community.general:7.5.0 at '/SCRIPTS/ANSIBLE/collections/ansible_collections/community/general'
Found installed collection ibm.cloudcollection:1.51.0 at '/SCRIPTS/ANSIBLE/collections/ansible_collections/ibm/cloudcollection'
Process install dependency map
Initial connection to galaxy_server: https://galaxy.ansible.com
Opened /home/die/.ansible/galaxy_token
Calling Galaxy at https://galaxy.ansible.com/api/
Found API version 'v3, pulp-v3, v1' with Galaxy server default (https://galaxy.ansible.com/api/)
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/?limit=100
Calling Galaxy at https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/index/community/general/versions/?limit=100&offset=100
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/4.1.0/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/1.3.7/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/2.5.9/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/1.3.13/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/5.8.7/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/4.8.4/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/1.0.0/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/5.5.0/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/3.8.5/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/6.6.3/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/3.8.4/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/7.2.0/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/4.0.1/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/6.0.0/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/5.3.0/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/4.8.0/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/4.8.5/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/1.3.9/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/6.6.2/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/4.7.0/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/1.3.3/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/5.0.1/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/3.4.0/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/6.6.4/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/0.2.0/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/5.2.0/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/6.3.0/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/2.5.8/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/1.1.0/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/4.0.2/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/2.5.3/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/5.8.3/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/4.8.10/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/7.3.0/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/2.1.1/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/3.3.1/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/2.0.1/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/7.0.1/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/5.1.1/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/5.8.8/
Calling Galaxy at https://galaxy.ansible.com/api/v3/collections/community/general/versions/4.8.8/
ERROR! Unexpected Exception, this is probably a bug:
the full traceback was:
Traceback (most recent call last):
File "/opt/freeware/lib/python3.9/site-packages/resolvelib/resolvers.py", line 171, in _merge_into_criterion
crit = self.state.criteria[name]
KeyError: 'community.general'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/freeware/lib/python3.9/site-packages/ansible/cli/__init__.py", line 647, in cli_executor
exit_code = cli.run()
File "/opt/freeware/lib/python3.9/site-packages/ansible/cli/galaxy.py", line 714, in run
return context.CLIARGS['func']()
File "/opt/freeware/lib/python3.9/site-packages/ansible/cli/galaxy.py", line 115, in method_wrapper
return wrapped_method(*args, **kwargs)
File "/opt/freeware/lib/python3.9/site-packages/ansible/cli/galaxy.py", line 1363, in execute_install
self._execute_install_collection(
File "/opt/freeware/lib/python3.9/site-packages/ansible/cli/galaxy.py", line 1400, in _execute_install_collection
install_collections(
File "/opt/freeware/lib/python3.9/site-packages/ansible/galaxy/collection/__init__.py", line 730, in install_collections
dependency_map = _resolve_depenency_map(
File "/opt/freeware/lib/python3.9/site-packages/ansible/galaxy/collection/__init__.py", line 1777, in _resolve_depenency_map
return collection_dep_resolver.resolve(
File "/opt/freeware/lib/python3.9/site-packages/resolvelib/resolvers.py", line 453, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/opt/freeware/lib/python3.9/site-packages/resolvelib/resolvers.py", line 318, in resolve
name, crit = self._merge_into_criterion(r, parent=None)
File "/opt/freeware/lib/python3.9/site-packages/resolvelib/resolvers.py", line 173, in _merge_into_criterion
crit = Criterion.from_requirement(self._p, requirement, parent)
File "/opt/freeware/lib/python3.9/site-packages/resolvelib/resolvers.py", line 79, in from_requirement
cands = build_iter_view(provider.find_matches([requirement]))
File "/opt/freeware/lib/python3.9/site-packages/ansible/galaxy/dependency_resolution/providers.py", line 506, in find_matches
return self._find_matches(requirements)
File "/opt/freeware/lib/python3.9/site-packages/ansible/galaxy/dependency_resolution/providers.py", line 395, in _find_matches
signatures = src_server.get_collection_signatures(first_req.namespace, first_req.name, version)
File "/opt/freeware/lib/python3.9/site-packages/ansible/galaxy/api.py", line 121, in wrapped
return method(self, *args, **kwargs)
File "/opt/freeware/lib/python3.9/site-packages/ansible/galaxy/api.py", line 904, in get_collection_signatures
self._set_cache()
File "/opt/freeware/lib/python3.9/site-packages/ansible/galaxy/api.py", line 46, in wrapped
return func(*args, **kwargs)
File "/opt/freeware/lib/python3.9/site-packages/ansible/galaxy/api.py", line 433, in _set_cache
fd.write(to_bytes(json.dumps(self._cache), errors='surrogate_or_strict'))
File "/opt/freeware/lib/python3.9/site-packages/ansible/module_utils/common/text/converters.py", line 114, in to_bytes
return obj.encode(encoding, errors)
MemoryError
Any hints what I can do in this case, not sure if installing via .tar.gz does all the things the install via online call does..also tried changing the API URL and v2/v3 entries in the GALAXY.yml, but still not working.
OSLEVEL is 7200-05-06-2320
RPM packages and their corresponding entries in PIP:
# rpm -qa|grep -iE "python|ansible"
python3.9-dateutil-2.8.0-2.noarch
python3.9-libcomps-0.1.15-101.ppc
python3.9-certifi-2019.9.11-2.noarch
python3.9-idna-2.8-2.noarch
python3.9-pycparser-2.19-2.noarch
rpm-python3.9-4.15.1-32_2.ppc
python3.9-hawkey-0.39.1-32_52.ppc
python3-dnf-4.2.17-32_51.ppc
python3-pycparser-2.19-2.noarch
python3-charset-normalizer-2.0.12-2.noarch
python3.9-pycurl-7.44.1-2.ppc
python3-chardet-3.0.4-2.noarch
python3-libdnf-0.39.1-32_52.ppc
python3-jmespath-0.9.4-2.noarch
python3-libcomps-0.1.15-101.ppc
python3-dateutil-2.8.0-2.noarch
python3-pip-22.2.2-1.noarch
python3-distro-1.5.0-2.noarch
python3-dnf-plugins-core-4.0.16-32_52.ppc
python3.9-dnf-plugin-versionlock-4.0.16-32_52.ppc
python3-dnf-plugin-versionlock-4.0.16-32_52.ppc
python3-cryptography-3.4.7-4.ppc
python3.9-setuptools-65.5.1-1.noarch
python3.9-resolvelib-0.5.4-1.noarch
python3.9-requests-2.31.0-1.noarch
python3.9-3.9.18-1.ppc
python3.9-jinja2-3.1.2-1.noarch
python3.9-packaging-23.1-1.noarch
python3-devel-3.9.18-1.ppc
python3-librepo-1.11.0-103.ppc
python-requests-2.4.3-1.noarch
python3.9-six-1.13.0-2.noarch
python3.9-gpg-1.13.1-101.ppc
python3.9-pyyaml-5.4.1.1-2.ppc
python3.9-charset-normalizer-2.0.12-2.noarch
python3.9-jmespath-0.9.4-2.noarch
python3.9-cffi-1.13.2-2.ppc
python3.9-urllib3-1.26.7-2.noarch
rpm-python3-4.15.1-32_2.ppc
python3.9-libdnf-0.39.1-32_52.ppc
python3.9-dnf-4.2.17-32_51.ppc
python3-urllib3-1.26.7-2.noarch
python3-idna-2.8-2.noarch
python3-certifi-2019.9.11-2.noarch
python3.9-chardet-3.0.4-2.noarch
python3-pycurl-7.44.1-2.ppc
python3-cffi-1.13.2-2.ppc
python3-hawkey-0.39.1-32_52.ppc
python3-pyyaml-5.4.1.1-2.ppc
python3-gpg-1.13.1-101.ppc
python3-six-1.13.0-2.noarch
python3.9-pip-22.2.2-1.noarch
python3.9-distro-1.5.0-2.noarch
python3.9-dnf-plugins-core-4.0.16-32_52.ppc
python3.9-dnf-plugin-migrate-4.0.16-32_52.ppc
python3-dnf-plugin-migrate-4.0.16-32_52.ppc
python3.9-cryptography-3.4.7-4.ppc
python3-setuptools-65.5.1-1.noarch
python3.9-markupsafe-2.1.2-1.ppc
python3-markupsafe-2.1.2-1.ppc
python3.9-pyparsing-2.4.4-2.noarch
ansible-7.2.0-1.noarch
python3-requests-2.31.0-1.noarch
python3-3.9.18-1.ppc
python3.9-devel-3.9.18-1.ppc
ansible-core-2.14.2-2.noarch
python3-jinja2-3.1.2-1.noarch
python3.9-librepo-1.11.0-103.ppc
[USER@SERVER:/SCRIPTS/ANSIBLE]# python3.9 -m pip list
Package Version
------------------- ---------
ansible 7.2.0
ansible-builder 3.0.0
ansible-cmdb 1.31
ansible-core 2.14.2
ansible-generator 3.1.1
ansible-runner 2.3.3
attrs 23.1.0
bindep 2.11.0
bitmath 1.3.3.1
cachez 0.1.2
certifi 2019.9.11
cffi 1.13.2
chardet 3.0.4
charset-normalizer 2.0.12
cryptography 3.4.7
dataclasses 0.6
distro 1.8.0
docutils 0.20.1
gpg 1.13.1
idna 2.8
importlib-metadata 6.2.1
Jinja2 3.1.2
jmespath 0.9.4
jsonschema 4.17.3
jsonxs 0.6
lockfile 0.12.2
lxml 4.9.2
Mako 1.2.4
MarkupSafe 2.1.2
packaging 23.1
Parsley 1.3
pbr 5.11.1
persist-queue 0.8.1
pexpect 4.8.0
pip 22.2.2
psutil 5.9.5
ptyprocess 0.7.0
pycparser 2.19
pycurl 7.44.1
pyparsing 2.4.4
pyrsistent 0.19.3
python-daemon 3.0.1
python-dateutil 2.8.0
python-tss-sdk 1.2.1
PyYAML 5.4.1
requests 2.31.0
requirements-parser 0.5.0
resolvelib 0.5.4
retryz 0.1.9
sentry-sdk 1.25.1
setuptools 65.5.1
six 1.13.0
storops 1.2.11
toml 0.10.2
types-setuptools 67.8.0.0
urllib3 1.26.16
ushlex 0.99.1
zipp 3.15.0
Thanks in advance,
With kind regards,
Stephan Dietl
------------------------------
Stephan Dietl
------------------------------