IBM Security Verify

 View Only
  • 1.  worker threads

    Posted Wed October 27, 2021 08:06 AM
    Edited by Andreas Rühl Wed October 27, 2021 08:21 AM
    Hello,
    after an unexpected behavior of WebSEAL in an overload situation I have some questions.

    If I have a WebSEAL with three junctions (A, B and C)
    Junction A should be limited to 300 connections
    Junction B should be limited to 200 connections
    Junction C should be limited to 100 connections

    How should be the following parameters should be sized?
    [server] worker-threads
    [junction] worker-thread-hard-limit

    In the past we was are running with:
    worker-threads = 300
    A: worker-thread-hard-limit = 100
    B: worker-thread-hard-limit = 66
    C: worker-thread-hard-limit = 33

    Now we have changed to:
    worker-threads = 600
    A: worker-thread-hard-limit = 50
    B: worker-thread-hard-limit = 33
    C: worker-thread-hard-limit = 16

    Gets WebSEAL in trouble or do we get a resource problem (ports for example), if we change the settings like the following and running lots of instances on one host:
    worker-threads = 1000
    A: worker-thread-hard-limit = 100
    B: worker-thread-hard-limit = 100
    C: worker-thread-hard-limit = 100

    What happens, if the [server] worker-thread limit is reached?

    I have seen lots of requests which are rejected on TCP layer (WebSEAL answers with a RST for SYN requests). Which parameter limits the incomming requests? There are no messages in the webseald.log. Only DPWWM1493E Junction '/A' has reached it's worker thread hard limit.

    # The maximum number of idle client persistent connections. This value
    # should be less than the maximum number of connections supported by the
    # WebSEAL server to ensure that the idle connections do not consume all of
    # the available connections.
    max-idle-persistent-connections = 512
    Which parameter is referenced here?

    I have also seen, there are hundrets of HPDCO0192W LDAP server ldap:636 has failed. and HPDCO0193W LDAP server ldap:636 has recovered. , but the LDAP server was always available. Are these connections limited by worker-threads too?
    I can see these messages in a less count (2-5/hour) every day. Why is WebSEAL loosing this connection if there is no network reason?

    ------------------------------
    Andreas Rühl
    ------------------------------


  • 2.  RE: worker threads

    Posted Wed October 27, 2021 04:55 PM

    Andreas,

    First let me try to answer all of your questions:

    > How should be the following parameters should be sized?

      In order to size the parameters you need to understand what the parameters do.  The worker-threads parameter controls the number of worker threads which will be used by WebSEAL to handle all web requests.  When a connection request comes into WebSEAL it will add the connection request to a list of requests which are to be processed.  When a worker thread become free it will grab a connection request from the list and start processing the request.  The worker-thread-hard-limit configuration parameter controls the maximum number of worker threads which can be handling a request for a particular junction at the one time.  So, when a connection request is first detected by WebSEAL it has no idea as to which junction the request is for until it starts processing the request.  So, the worker thread will start processing the request, even if the junction limit has been reached.  It is not until the worker thread detects the junction which will be used for the request that it can apply the hard limit and potentially return an error for the request without passing it to the junction.
    > Which parameter is referenced here?
      The max-idle-persistent-connections parameter is referring to the number of socket/network connections which are available to the server itself.
    > Are these connections limited by worker-threads too?
      The LDAP connections are not limited by the worker-threads.
    Your actual problem appears to be a socket starvation problem on the system itself.  It sounds like the server is running out of sockets which can be allocated to network connections.  You can check this by running the tools -> connections command from the CLI.  

    I hope that this helps.



    Scott A. Exton
    Senior Software Engineer
    Chief Programmer - IBM Security Access Manager

    IBM Master Inventor



    "Andreas Rühl via IBM Community" ---27/10/2021 10:06:08 pm---Hello, after an unexpected behavior of WebSEAL in an overload situation I have some questions.





  • 3.  RE: worker threads

    Posted Tue December 14, 2021 08:56 AM
    To add further,

    1) You may want to go through the following short course on "Understanding and tuning Reverse Proxy worker threads"
    https://www.securitylearningacademy.com/course/view.php?id=4981

    2) To get the output, using RestAPI, similar to "isam > tools > connections" (CLI command for netstat) refer to the following (/core/cli RAPI):
    https://github.com/IBM-Security/isam-support/tree/master/appliance/restapi/All

    Jigar



    ------------------------------
    Jigar Amin
    Accelerated Value Specialist
    IBM
    ------------------------------



  • 4.  RE: worker threads

    Posted Thu January 27, 2022 02:22 PM
    I am going to provide my thoughts on the LDAP connection since we have the same thing showing up in our logs.  Our ldap environment sits behind a firewall and my understanding is the firewall is closing those connections once it reaches the set connection timeout.  Would this apply to your environment?

    Troy

    ------------------------------
    Troy Burkle
    ------------------------------