AIX

 View Only
  • 1.  Diagnosing Connection Timeouts

    Posted Wed December 06, 2023 01:22 PM

    We have a VIO client running AIX 7.2 TL 06. It has an app that is listening on the main adapter and the loopback. When a connection comes in on the main adapter it tries to reach out to the loopback to get a response and return it to the client on the main adapter. Unfortunately, the connections on the loopback never get to an established state, are always listening in a TIME_WAIT state, and the ephemeral ports keep changing. So that appears like the app is timing out, resetting and trying again on the new ports over and over.  How may we diagnose what the TIME_WAIT connections are waiting on?

    $ netstat -aAn | grep 32775
    f10010000b796bc0 tcp4       0      0  127.0.0.1.32775       127.0.0.1.60262       TIME_WAIT
    f10010000697fbc0 tcp4       0      0  127.0.0.1.32775       127.0.0.1.60263       TIME_WAIT
    f1001000084193c0 tcp4       0      0  127.0.0.1.32775       127.0.0.1.60264       TIME_WAIT
    f1001000059b6bc0 tcp        0      0  *.32775               *.*                   LISTEN
    $ netstat -aAn | grep 32775
    f10010000b84fbc0 tcp4       0      0  127.0.0.1.32775       127.0.0.1.60298       TIME_WAIT
    f10010000bcdebc0 tcp4       0      0  127.0.0.1.32775       127.0.0.1.60299       TIME_WAIT
    f10010000bd77bc0 tcp4       0      0  127.0.0.1.32775       127.0.0.1.60300       TIME_WAIT
    f1001000059b6bc0 tcp        0      0  *.32775               *.*                   LISTEN
    $ netstat -aAn | grep 32775
    f100100005a1b3c0 tcp4       0      0  127.0.0.1.32775       127.0.0.1.60838       TIME_WAIT
    f100100005a7ebc0 tcp4       0      0  127.0.0.1.32775       127.0.0.1.60839       TIME_WAIT
    f100100005abebc0 tcp4       0      0  127.0.0.1.32775       127.0.0.1.60840       TIME_WAIT
    f1001000059b6bc0 tcp        0      0  *.32775               *.*                   LISTEN
    $ netstat -aAn | grep 32775
    f10010000b96cbc0 tcp4       0      0  127.0.0.1.32775       127.0.0.1.60983       TIME_WAIT
    f10010000837ebc0 tcp4       0      0  127.0.0.1.32775       127.0.0.1.60984       TIME_WAIT
    f10010000a37abc0 tcp4       0      0  127.0.0.1.32775       127.0.0.1.60985       TIME_WAIT
    f10010000beb03c0 tcp4       0      0  127.0.0.1.32775       127.0.0.1.60987       TIME_WAIT
    f10010000997a3c0 tcp4       0      0  127.0.0.1.32775       127.0.0.1.60988       TIME_WAIT
    f10010000b780bc0 tcp4       0      0  127.0.0.1.32775       127.0.0.1.60989       TIME_WAIT
    f1001000059b6bc0 tcp        0      0  *.32775               *.*                   LISTEN



    ------------------------------
    Ed Stuart
    ------------------------------


  • 2.  RE: Diagnosing Connection Timeouts

    Posted Thu December 07, 2023 04:30 AM

    TIME_WAIT is only reached after the connection is established and closed.

    After delivering the last FIN-ACK, the connection moves to TIME_WAIT and stays there for tcp_timewait*15seconds.

    Have you tried to capture the loopback traffic to see what's happening?



    ------------------------------
    José Pina Coelho
    IT Specialist at Kyndryl
    ------------------------------



  • 3.  RE: Diagnosing Connection Timeouts

    Posted Thu December 07, 2023 05:58 AM

    This does not look like a general question but is very specific to your AIX 7.2 virtual machine.

    I suggest you raise a PMR with AIX Support.



    ------------------------------
    Nigel Griffiths
    ------------------------------



  • 4.  RE: Diagnosing Connection Timeouts

    Posted Thu December 07, 2023 10:03 AM

    TIME_WAIT is part of the socket shutdown. The simplest way to work around this is the enable clean_partial_conns:

           no -p -o clean_partial_conns=1

    This will allow new connections to open even if the hold down time has not completed. This will only affect NEWLY opened connections - ie. the old connections in TIMED_WAIT state will stay there. 

    This is a very low risk change to make.

    Ideally you will reboot the system to clean everything up. If you wait long enough, they will go away. 



    ------------------------------
    Grover Davidson
    ------------------------------