Informix

 View Only

  • 1.  too many open connections, consider increasing the number of threads

    Posted Thu March 12, 2020 01:15 PM
    The following message is written repeatedly in the /var/log/messages file in the RHEL 7.5 (64 bit) system :-
    "too many open connections, consider increasing the number of threads".
    The above message is written to the /var/log/messages file after approx. every 2-3 seconds, so much so that the system becomes totally unresponsive. At this stage if you check the load of the system (using the "w -u" or "top" command), it displays that the system load to be more than 120. At this time, there are no error messages written in the online.log file of the databaseserver (only normal messages are written like "checkpoint completed: <number> seconds", "logical log <number> completed" etc ...). Even dbaccess command goes into a hang state.

    Then, after approx. 10-15 minutes, the system load starts reducing and it comes back to less than 1. Then, the user are able to connect.

    Just wanted to know as to what could be the problem and how do we set it right.

    Thanks & Regards,
    Neville Monteiro.

    Sent with ProtonMail Secure Email.


    #Informix


  • 2.  RE: too many open connections, consider increasing the number of threads

    IBM Champion
    Posted Thu March 12, 2020 01:23 PM
    Running nfs ? Just tweak the nfsd

    Cheers
    Paul

    > The following message is written repeatedly in the /var/log/messages file
    > in the RHEL 7.5 (64 bit) system :-
    > "too many open connections, consider increasing the number of threads".
    > The above message is written to the /var/log/messages file after approx.
    > every 2-3 seconds, so much so that the system becomes totally
    > unresponsive. At this stage if you check the load of the system (using the
    > "w -u" or "top" command), it displays that the system load to be more than
    > 120. At this time, there are no error messages written in the online.log
    > file of the databaseserver (only normal messages are written like
    > "checkpoint completed:
    #Informix


  • 3.  RE: too many open connections, consider increasing the number of threads

    IBM Champion
    Posted Thu March 12, 2020 01:34 PM
    Neville:

    I have a client that has seen a similar issue. They normally run around 4,000 active connections, but occasionally the number of connections starts to ramp up to over 30,000 bringing the system to its knees.

    The cause was an IO slow down that caused users get impatient waiting for a response from their applications and so to close their Windows based front-end applications and open a new copy. The windows apps were not shutting down the network connections to the database server. It was taking a long time for the OS to recognize that the clients were no longer attached to the other end of the connection and since the Informix server was not trying to write to those pipes it didn't know that the clients were gone.

    The solution was to adjust the keepalive setting in the OS to shorten the timeout and to add the keepalive option to the sqlhosts file lines.

    Art

    Art S. Kagel, President and Principal Consultant
    ASK Database Management


    Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference.  Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves.






    Original Message


  • 4.  RE: too many open connections, consider increasing the number of threads

    Posted Thu March 12, 2020 01:44 PM

    If you are doing a lot of (connect) disconnects on the system it's possible that you can run into issue related to TIME_WAIT. To confirm - look at the output of 

        netstat -nta

    or, you could run something like

        netstat -nta | awk 'substr($1,1,3)=="tcp" {print $NF}' | sort | uniq -c

    There is nothing that can be done about it other than reduce frequency of disconnects, there are few workarounds including allowing reuse of the socket and reducing time spent in this state - but they are not without consequences.

    If you find that it's your problem - here are couple of references:

    TIME_WAIT and its design implications for protocols and scalable client server systems

    kernel: Possible SYN flooding on port #. Sending cookies. - Red Hat Customer Portal

    more can be found by searching for the TIME_WAIT


     



    ------------------------------
    Vladimir Kolobrodov
    ------------------------------

    Original Message