Hello,
on a busy SSH/SFTP server we encounter constantly login delays and ran now openssh in debug mode to find the reason. As the README says, openssh 9.9 comes compiled with auditing and EFS capabilities, but there is no possibility (shown) to turn these functions off, like the OS counterparts. So the first impression of the debug messages are lots of auditing events, even when auditing is shutdown. But the most shocking discovery, and probably the reason for the login delays, was that on every connection, before the kex exchange init, in the very beginging of a ssh handshake, the openssh daemon runs the command oslevel! We first thought, this is ran by some auditing script/config, but no, oslevel is hard coded in the sshd binary:
strings /usr/sbin/sshd* | grep oslevel
getoslevel
oslevel : %s
/usr/bin/oslevel
getoslevel
oslevel : %s
/usr/bin/oslevel
The second last line is also what appears in the log:
Oct 30 17:45:53 sftp local6:debug sshd-session[57214588]: debug1: oslevel : 7.2.0.0\n\n
And this is an example showing a delay of more than 10 seconds (3. and 4. line) for the oslevel command, before the ssh authentication goes further:
Oct 30 14:09:11 sftp local6:debug sshd-session[34865828]: debug1: Remote protocol version 2.0, remote software version OpenSSH_9.9
Oct 30 14:09:11 sftp local6:debug sshd-session[34865828]: debug1: compat_banner: match: OpenSSH_9.9 pat OpenSSH* compat 0x04000000
Oct 30 14:09:25 sftp local6:debug sshd-session[34865828]: debug1: oslevel : 7.2.0.0\n\n
Oct 30 14:09:25 sftp local6:debug sshd-session[34865828]: debug1: using libzNX from /opt/freeware/ssh/libzNX_s.a(libz.so.1) \n
Oct 30 14:09:25 sftp local6:debug sshd-session[34865828]: debug1: init_libz_ptrs success
Oct 30 14:09:25 sftp local6:debug sshd-session[34865828]: debug1: init_func_ptrs passed
Oct 30 14:09:25 sftp local6:debug sshd-session[34865828]: debug1: value of krb5 is : 1
Oct 30 14:09:25 sftp local6:debug sshd-session[34865828]: debug1: value of gssapi is : 1
Oct 30 14:09:25 sftp local6:debug sshd-session[34865828]: debug1: permanently_set_uid: 202/201 [preauth]
Oct 30 14:09:25 sftp local6:debug sshd-session[34865828]: debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Oct 30 14:09:25 sftp local6:debug sshd-session[34865828]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Oct 30 14:09:25 sftp local6:debug sshd-session[34865828]: debug1: SSH2_MSG_KEXINIT received [preauth]
We have about 400 to 600 connections per minute! So 400 to 600 times oslevel gets executed even before any user authentication takes place!
Since we have old debug logs with openssh 8, we see no oslevel there, but downgrading seems imposible for AIX, simply because there are no more old openssh packages officially available.
My first attempt for a workaround will be to replace the oslevel with something like:
#!/bin/ksh -f
if [[ -n "$1" ]] ; then
exec /usr/bin/oslevel.orig "$@"
else
print 7.2.0.0
fi
Any ideas, besides compiling own sshd or switch to another OS?
Best regards,
P. Tanovski
------------------------------
P T
------------------------------