Thanks Ranjit for taking care of this,
There are some python messages in Minion log, not explaining much, at least to me :) see below.
Minion process still visible in process list but no jobs going through. And it is happening after some time, Minion restart helps.
Further investigation shows possible incompatibility between Salt Minion and Master being lower version.
Anyway, it would be great to have the latest version of 3006 LTS available in Toolbox with all the bugfixes so we can start using it and check if the problem still persists.
Thank you in advance.
2024-08-05 12:01:12,439 [tornado.application :640 ][ERROR ][13894108] Exception in callback functools.partial(<
function wrap.<locals>.null_wrapper at 0xa000000019129d0>, <salt.ext.tornado.concurrent.Future object at 0xa000000018b5fd0>)
Traceback (most recent call last):
File "/opt/freeware/lib/python3.9/site-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
ret = callback()
File "/opt/freeware/lib/python3.9/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
return fn(*args, **kwargs)
File "/opt/freeware/lib/python3.9/site-packages/salt/ext/tornado/ioloop.py", line 628, in _discard_future_result
future.result()
File "/opt/freeware/lib/python3.9/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/opt/freeware/lib/python3.9/site-packages/salt/ext/tornado/gen.py", line 1070, in run
yielded = self.gen.send(value)
File "/opt/freeware/lib/python3.9/site-packages/salt/transport/tcp.py", line 797, in _do_send
yield self._stream.write(item)
File "/opt/freeware/lib/python3.9/site-packages/salt/ext/tornado/iostream.py", line 389, in write
self._check_closed()
File "/opt/freeware/lib/python3.9/site-packages/salt/ext/tornado/iostream.py", line 927, in _check_closed
raise StreamClosedError(real_error=self.error)
salt.ext.tornado.iostream.StreamClosedError: Stream is closed
2024-08-05 12:01:20,440 [tornado.application :353 ][ERROR ][13894108] Future <salt.ext.tornado.concurrent.Futur
e object at 0xa00000001aa0c10> exception was never retrieved: Traceback (most recent call last):
File "/opt/freeware/lib/python3.9/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/opt/freeware/lib/python3.9/site-packages/salt/minion.py", line 2699, in handle_event
yield _minion.req_channel.send(
File "/opt/freeware/lib/python3.9/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/opt/freeware/lib/python3.9/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/opt/freeware/lib/python3.9/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/opt/freeware/lib/python3.9/site-packages/salt/channel/client.py", line 295, in send
ret = yield self._crypted_transfer(load, timeout=timeout, raw=raw)
File "/opt/freeware/lib/python3.9/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/opt/freeware/lib/python3.9/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/opt/freeware/lib/python3.9/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/opt/freeware/lib/python3.9/site-packages/salt/channel/client.py", line 252, in _crypted_transfer
ret = yield _do_transfer()
File "/opt/freeware/lib/python3.9/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/opt/freeware/lib/python3.9/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/opt/freeware/lib/python3.9/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/opt/freeware/lib/python3.9/site-packages/salt/channel/client.py", line 233, in _do_transfer
data = yield self.transport.send(
------------------------------
Zbigniew Branny
------------------------------
Original Message:
Sent: Tue August 06, 2024 05:22 AM
From: Ranjit Ranjan
Subject: Salt 3006.9
Hi,
We will look into it and share the time line later.
Is it possible to share the minimal program/steps to reproduce the crash ?
Thanks
Ranjit
------------------------------
Ranjit Ranjan
Original Message:
Sent: Mon August 05, 2024 09:28 AM
From: Zbigniew Branny
Subject: Salt 3006.9
Hi,
would it be possible to compile the latest SaltStack components, LTS version 3006.9 and provide as RPM packages via AIX Toolbox service?
https://repo.saltproject.io/salt/py3/src/3006.9/
There is older version 3006.2 available in AIX Toolbox.
AIX Toolbox for Open Source Software : Downloads alpha
I tried to check it by myself but the whole compilation process on AIX machine doesn't look easy...
Our company use internal SaltStack based tool as main automation service. Currently I'm dealing with as issue (crashing Salt Minion 3006.2) therefore it would be great to run the latest LTS Minion with all the fixes.
Thanks in advance.
------------------------------
Zbigniew Branny
Senior Service Architect, Power Cloud Capacity
Tietoevry Tech Services
------------------------------