AIX Open Source

AIX Open Source

Share your experiences and connect with fellow developers to discover how to build and manage open source software for the AIX operating system

 View Only
Expand all | Collapse all

Is Sudo 1.9.16.0 in plan for Toolbox update

  • 1.  Is Sudo 1.9.16.0 in plan for Toolbox update

    Posted Thu October 31, 2024 02:15 PM

    Hello, Toolbox team!

    We have seen a couple of cases reporting some sudo hangs, and have some suspicion this might be related to some fixes in 1.9.16

    For the meantime, I have advised them to test the latest sudo bff to see if this resolves the hang. If it does resolve the hang, I will point to this thread for them to follow.

    A summary of related changes for sudo 1.9.16.0 :

    https://github.com/sudo-project/sudo/commit/41978a56e042977923c1a55191b887218c536145

    2024-04-27 Todd C. Miller

    • src/exec_monitor.c, src/exec_pty.c, src/sudo.h:
    • If user's tty goes away, tell monitor to revoke the tty in its session.
    • Previously, we would simply close the pty leader in the main sudo process. This had the effect of revoking the pty, but the foreground process would not necessarily receive SIGHUP. By using TIOCNOTTY in the monitor, the running command has a better chance of getting SIGHUP. Once the monitor has revoked the pty, the main sudo process will close the pty leader, invalidating the pty. GitHub issue #367. [3d5708b425df]


    ------------------------------
    Jan Harris
    AIX Development Support (Liaison to the AIX Toolbox for Open Source)
    IBM (Contract)
    Austin
    ------------------------------


  • 2.  RE: Is Sudo 1.9.16.0 in plan for Toolbox update

    Posted Thu October 31, 2024 02:31 PM

    Jan,

    Is there an ETA on when IBM will put the new version of sudo into the toolbox.  We are also being impacted by the sudo hangs, but we don't like to use any rpms except ones that are part of the IBM toolbox.

    Thanks,

    Bryan Lenherr

    Fiserv



    ------------------------------
    Bryan Lenherr
    ------------------------------



  • 3.  RE: Is Sudo 1.9.16.0 in plan for Toolbox update

    Posted Thu October 31, 2024 03:31 PM

    I don't know if it is even in plan yet, Bryan. The purpose of my initiating the thread was to request that the Toolbox team update sudo. My suggestion to a couple of customers was to temporarily install the installp format bff file set (to avoid getting in to the "rpm" database,) to at least test to see if this resolves the issue. If it does resolve the issue, that will help put a priority on the update.



    ------------------------------
    Jan Harris
    AIX Development Support (Liaison to the AIX Toolbox for Open Source)
    IBM (Contract)
    Austin
    ------------------------------



  • 4.  RE: Is Sudo 1.9.16.0 in plan for Toolbox update

    Posted Sun November 03, 2024 08:37 AM

    Thanks Jan,

    I think we need to debug this issue and find out if it is because of new sudo or something else.

    We would like to get a good recreation scenario.

    If anyone who is facing sudo hang issue can share the recreation steps that will help.

     

    Thanks,

     

    Sanket Rathi

     






  • 5.  RE: Is Sudo 1.9.16.0 in plan for Toolbox update

    Posted Mon November 04, 2024 07:45 PM

    Hi, Sanket

    Yes, I agree. I have tried many methods to reproduce a hang, to no avail so far. I have asked another customer for any reproduction steps if possible.

    I have tried timeouts, kill -STOP, then kill -CONT of parent processes... so far I cannot reproduce the any hangs A colleague did examime pdump and could see that  sudo is trying to write to stream. As the stream is full, it is waiting to write. 



    ------------------------------
    Jan Harris
    AIX Development Support (Liaison to the AIX Toolbox for Open Source)
    IBM (Contract)
    Austin
    ------------------------------



  • 6.  RE: Is Sudo 1.9.16.0 in plan for Toolbox update

    Posted Mon November 04, 2024 07:47 PM

    Hello Sankit

    I want to mention that sometimes this issue is intermittent, it doesn't happen frequently on a regular basis



    ------------------------------
    Ahmed Adel Fahmy Hassan
    ------------------------------



  • 7.  RE: Is Sudo 1.9.16.0 in plan for Toolbox update

    Posted Mon November 04, 2024 07:48 PM

    It seems to be consistent for at least 2 customers, so I would hesitate to say it is not frequent.



    ------------------------------
    Jan Harris
    AIX Development Support (Liaison to the AIX Toolbox for Open Source)
    IBM (Contract)
    Austin
    ------------------------------



  • 8.  RE: Is Sudo 1.9.16.0 in plan for Toolbox update

    Posted Thu October 31, 2024 04:28 PM

    Bryan, do you have any good re-creation scenarios? I have been trying to reproduce this in my 73 LPAR with sudo-1.9.15p5-1.ppc, I have killed the parent ssh process, the parent shell, I have tried kill -STOP, then kill -CONT - I have tried everything but water boarding my shell :->  I  not been able to reproduce any sudo hangs.



    ------------------------------
    Jan Harris
    AIX Development Support (Liaison to the AIX Toolbox for Open Source)
    IBM (Contract)
    Austin
    ------------------------------



  • 9.  RE: Is Sudo 1.9.16.0 in plan for Toolbox update

    Posted Mon November 04, 2024 08:30 PM

    I have tried multiple methods to cause a hang

    As userA:

    sudo su -
    Then in root shell:
    TMOUT=1
    sleep 300 &
    tail -f /tmp/file &
    
    The root shell exits, the sudo processes are both killed as expected.
    
    # proctree $$
    5046728    /usr/sbin/srcmstr
       19071300    /usr/sbin/sshd
          14156076    sshd: userA [priv]
             14549354    sshd: userA@pts/1
                19267970    -bash
                   15466956    sudo su -
                      11927892    sudo su -
                         14483884    -ksh
                            13107568    tail -f /tmp/file
                            17301986    proctree 14483884
    

    In another shell:

    kill -STOP  14549354 
    kill -CONT  14549354 
    
    

    I tried -STOP for other processes as well. In all cases, I could not produce a hang.



    ------------------------------
    Jan Harris
    AIX Development Support (Liaison to the AIX Toolbox for Open Source)
    IBM (Contract)
    Austin
    ------------------------------



  • 10.  RE: Is Sudo 1.9.16.0 in plan for Toolbox update

    Posted Tue November 05, 2024 05:40 AM

    Hi Jan,

    Maybe it's driven/triggered by a couple of old bash hanging problems? (I think around the bash 4.2-3 level)

    If you're using version 5 of bash, you're probably not subject to the bash issues.



    ------------------------------
    José Pina Coelho
    IT Specialist at Kyndryl
    ------------------------------



  • 11.  RE: Is Sudo 1.9.16.0 in plan for Toolbox update

    Posted Tue November 05, 2024 08:44 AM

    Thanks for your input, José

    Good point - but they are not using bash in at least one I'm working on, bash is not installed (72)

     In that case, user's shell is ksh. It is difficult to know so far which is the "original" process that causes the issue, since what I have is proctrees of presumably subsequent hangs after the first issue.



    ------------------------------
    Jan Harris
    AIX Development Support (Liaison to the AIX Toolbox for Open Source)
    IBM (Contract)
    Austin
    ------------------------------



  • 12.  RE: Is Sudo 1.9.16.0 in plan for Toolbox update

    Posted Tue November 05, 2024 10:03 AM

    I did notice they have a sudo process in inittab

    inittab:cdstart:2:once:sudo su - cduser -c /cdndm/ONDDBP1/ndm/bin/cdstart > /dev/console 2>&1

    I am not wondering if that could be related. I'm testing some options there too...



    ------------------------------
    Jan Harris
    AIX Development Support (Liaison to the AIX Toolbox for Open Source)
    IBM (Contract)
    Austin
    ------------------------------



  • 13.  RE: Is Sudo 1.9.16.0 in plan for Toolbox update

    Posted Tue November 05, 2024 11:51 AM

    Jan,

     I think you have been working with my colleague (Emmanuel) on the above issue.

    We are seeing this sporadically in more than one country with different configurations, but always with just one id (ansible) causing the hangs, that uses ssh keys to get into a server, then sudos to root.  As a workaround to the issue, I wrote a BigFix fixlet that just kills all ansible sudo sessions.  Killing the ansible sudo sessions allows sudo access once again.  I don't have high confidence it is bash related, but it is interesting that it started when we migrated from Centrify to IDM ldap authentication for users.  Every LDAP user in IDM has their shell using bash as their shell, so it still could be related somehow with bash, even though the ansible ID is local using KSH.  In Centrify previously the default shell was ksh.

    Since this is so sporadic, I hesitate to rollout any sudo updates that aren't from the IBM toolbox.

    Bryan Lenherr

    Fiserv



    ------------------------------
    Bryan Lenherr
    ------------------------------



  • 14.  RE: Is Sudo 1.9.16.0 in plan for Toolbox update

    Posted Tue November 05, 2024 11:56 AM

    Well, that's one superfluous sudo...  anything on the console alog ?



    ------------------------------
    José Pina Coelho
    IT Specialist at Kyndryl
    ------------------------------



  • 15.  RE: Is Sudo 1.9.16.0 in plan for Toolbox update

    Posted Tue November 05, 2024 12:36 PM

    Pretty full of database errors, so not much.

    An interesting turn of events on the other "sudo hang"

    In that case, we found, in audit, some mkdir errors for /var/run.

    Turns out the they had an invalid link in /var/run, so sudo could not write to this directory.

    So I found that the files in /var/run/sudo  are timestamp files that sudo creates to track when a user last authenticated. These files determine how long a user can run sudo commands without re-entering their password, based on the timestamp_timeout setting in /etc/sudoers.

    I created a hang by linking to a file, then deleting it, then doing sudo su - as a non-passwordless sudo user.

    So at this point, I think we just have a "fluke" that multiple sudo hangs came in from multiple customers in 24 hours :-)

    @Bryan Lenherr it might be good for us to check auditing for some other clues. I see they don't have anything  useful being tracked at this point, but I will update that case with some recommendations.  It seems the ansible is a local user, so I'm not sure what the LDAP impact could/should be on sudo. We could try to rule that out by just changing SYSTEM=compat for ansible. I will also get a truss from them.



    ------------------------------
    Jan Harris
    AIX Development Support (Liaison to the AIX Toolbox for Open Source)
    IBM (Contract)
    Austin
    ------------------------------