I have 18 WebSEAL instances with an /isam junction receiving uniform traffic, based on the request logs. The /isam junction has four federation servers (in a cluster) as backends. What are the ways I can ensure requests are spread uniformly across all four backends? I feel my primary master is receiving more requests compared to the other three. Is there a way to verify this and ensure traffic is uniform across all four nodes?

Maybe others can chime in, but the two big things would be configuring the junction with multiple backends as well as configuring the webseal config sections (i.e. tfim-cluster and rtss-cluster sections) with multiple runtime servers.  Don't have multiple runtimes behind any sort of external load balancer if possible (ideally you want webseal direct to the runtimes).  Set priority on the backends on the junction (i.e. /isam) going to the runtime all the same as well as the priority in the webseal config the same for those servers.  This should make them load balance evenly in theory.

Granted, if you have servers at a different regions, then different priorities for the different regions may make sense to try to keep things local unless there is a failure.  Meaning, all servers at the same region as the webseal get the same higher priority number than all servers at a region that is further away from that webseal instance.  Make sure the junction to the runtime is setup as stateful as I think that is a requirement, depending on what you are using the runtime for.

If your 4 federation servers are already at the same priority levels as backends to the junction and in the webseal sections, then it sounds like maybe the webseal is not randomly selecting a server.  In that case perhaps altering the order within the junction config and/or webseal config?  But that seems kind of a pain if that is neccessary.

I have 18 WebSEAL instances with an /isam junction receiving uniform traffic, based on the request logs. The /isam junction has four federation servers (in a cluster) as backends. What are the ways I can ensure requests are spread uniformly across all four backends? I feel my primary master is receiving more requests compared to the other three. Is there a way to verify this and ensure traffic is uniform across all four nodes?

Thanks Matt for the suggestions. I've gone through the configuration across all the WebSEALs to ensure everything is aligned with your advice, but the imbalance persists. All four backend nodes are explicitly set to Priority 9 both in the junction definition (pdadmin show confirms this) and in the [tfim-cluster] and [rtss-cluster] stanzas of the webseald.conf files. The traffic distribution on the 4 nodes of the the junction /isam looks like this: 

primary master -Node 1: ~35%

secondary master -Node 2: ~32%

tertiary master -Node 3: ~16%

quaternary master -Node 4: ~16%

The consistency of this '35/32/16/16' split suggests it isn't a random network glitch, but rather that WebSEAL is intentionally treating Nodes 1 & 2 as a 'Primary' and Nodes 3 & 4 as a 'Secondary,' despite the identical priorities. I will continue to investigate. thank you!

Maybe others can chime in, but the two big things would be configuring the junction with multiple backends as well as configuring the webseal config sections (i.e. tfim-cluster and rtss-cluster sections) with multiple runtime servers.  Don't have multiple runtimes behind any sort of external load balancer if possible (ideally you want webseal direct to the runtimes).  Set priority on the backends on the junction (i.e. /isam) going to the runtime all the same as well as the priority in the webseal config the same for those servers.  This should make them load balance evenly in theory.

Granted, if you have servers at a different regions, then different priorities for the different regions may make sense to try to keep things local unless there is a failure.  Meaning, all servers at the same region as the webseal get the same higher priority number than all servers at a region that is further away from that webseal instance.  Make sure the junction to the runtime is setup as stateful as I think that is a requirement, depending on what you are using the runtime for.

If your 4 federation servers are already at the same priority levels as backends to the junction and in the webseal sections, then it sounds like maybe the webseal is not randomly selecting a server.  In that case perhaps altering the order within the junction config and/or webseal config?  But that seems kind of a pain if that is neccessary.

I have 18 WebSEAL instances with an /isam junction receiving uniform traffic, based on the request logs. The /isam junction has four federation servers (in a cluster) as backends. What are the ways I can ensure requests are spread uniformly across all four backends? I feel my primary master is receiving more requests compared to the other three. Is there a way to verify this and ensure traffic is uniform across all four nodes?

It would be good if someone from IBM could chime in and explain how the load balancing for junctions and cluster configurations within the webseal conf work when the server priorities are the same.  I've always wondered this, and it seems by your findings it may not work as expected.

How are you identifying which traffic is going to which node?  Are you just looking at the junction logs?  The reason I ask is I am curious if the issue is the junction traffic or the traffic for say tfim-cluster or rtss-cluster activities (oauth-eas, mfa, etc.).  The only other thing I wonder is if their load balancing is actually using the ordering, but only sending requests to the other servers if the others have active requests.  Hence, maybe the webseal instance(s) is not getting busy enough to send traffic to the other nodes if webseal is not doing a round-robin selection of which server to send to.

Thanks for sharing your findings.

Thanks Matt for the suggestions. I've gone through the configuration across all the WebSEALs to ensure everything is aligned with your advice, but the imbalance persists. All four backend nodes are explicitly set to Priority 9 both in the junction definition (pdadmin show confirms this) and in the [tfim-cluster] and [rtss-cluster] stanzas of the webseald.conf files. The traffic distribution on the 4 nodes of the the junction /isam looks like this: 

primary master -Node 1: ~35%

secondary master -Node 2: ~32%

tertiary master -Node 3: ~16%

quaternary master -Node 4: ~16%

The consistency of this '35/32/16/16' split suggests it isn't a random network glitch, but rather that WebSEAL is intentionally treating Nodes 1 & 2 as a 'Primary' and Nodes 3 & 4 as a 'Secondary,' despite the identical priorities. I will continue to investigate. thank you!

Maybe others can chime in, but the two big things would be configuring the junction with multiple backends as well as configuring the webseal config sections (i.e. tfim-cluster and rtss-cluster sections) with multiple runtime servers.  Don't have multiple runtimes behind any sort of external load balancer if possible (ideally you want webseal direct to the runtimes).  Set priority on the backends on the junction (i.e. /isam) going to the runtime all the same as well as the priority in the webseal config the same for those servers.  This should make them load balance evenly in theory.

Granted, if you have servers at a different regions, then different priorities for the different regions may make sense to try to keep things local unless there is a failure.  Meaning, all servers at the same region as the webseal get the same higher priority number than all servers at a region that is further away from that webseal instance.  Make sure the junction to the runtime is setup as stateful as I think that is a requirement, depending on what you are using the runtime for.

If your 4 federation servers are already at the same priority levels as backends to the junction and in the webseal sections, then it sounds like maybe the webseal is not randomly selecting a server.  In that case perhaps altering the order within the junction config and/or webseal config?  But that seems kind of a pain if that is neccessary.

I have 18 WebSEAL instances with an /isam junction receiving uniform traffic, based on the request logs. The /isam junction has four federation servers (in a cluster) as backends. What are the ways I can ensure requests are spread uniformly across all four backends? I feel my primary master is receiving more requests compared to the other three. Is there a way to verify this and ensure traffic is uniform across all four nodes?

It would be good if someone from IBM could chime in and explain how the load balancing for junctions and cluster configurations within the webseal conf work when the server priorities are the same.  I've always wondered this, and it seems by your findings it may not work as expected.

How are you identifying which traffic is going to which node?  Are you just looking at the junction logs?  The reason I ask is I am curious if the issue is the junction traffic or the traffic for say tfim-cluster or rtss-cluster activities (oauth-eas, mfa, etc.).  The only other thing I wonder is if their load balancing is actually using the ordering, but only sending requests to the other servers if the others have active requests.  Hence, maybe the webseal instance(s) is not getting busy enough to send traffic to the other nodes if webseal is not doing a round-robin selection of which server to send to.

Thanks for sharing your findings.

Thanks Matt for the suggestions. I've gone through the configuration across all the WebSEALs to ensure everything is aligned with your advice, but the imbalance persists. All four backend nodes are explicitly set to Priority 9 both in the junction definition (pdadmin show confirms this) and in the [tfim-cluster] and [rtss-cluster] stanzas of the webseald.conf files. The traffic distribution on the 4 nodes of the the junction /isam looks like this: 

primary master -Node 1: ~35%

secondary master -Node 2: ~32%

tertiary master -Node 3: ~16%

quaternary master -Node 4: ~16%

The consistency of this '35/32/16/16' split suggests it isn't a random network glitch, but rather that WebSEAL is intentionally treating Nodes 1 & 2 as a 'Primary' and Nodes 3 & 4 as a 'Secondary,' despite the identical priorities. I will continue to investigate. thank you!

Maybe others can chime in, but the two big things would be configuring the junction with multiple backends as well as configuring the webseal config sections (i.e. tfim-cluster and rtss-cluster sections) with multiple runtime servers.  Don't have multiple runtimes behind any sort of external load balancer if possible (ideally you want webseal direct to the runtimes).  Set priority on the backends on the junction (i.e. /isam) going to the runtime all the same as well as the priority in the webseal config the same for those servers.  This should make them load balance evenly in theory.

Granted, if you have servers at a different regions, then different priorities for the different regions may make sense to try to keep things local unless there is a failure.  Meaning, all servers at the same region as the webseal get the same higher priority number than all servers at a region that is further away from that webseal instance.  Make sure the junction to the runtime is setup as stateful as I think that is a requirement, depending on what you are using the runtime for.

If your 4 federation servers are already at the same priority levels as backends to the junction and in the webseal sections, then it sounds like maybe the webseal is not randomly selecting a server.  In that case perhaps altering the order within the junction config and/or webseal config?  But that seems kind of a pain if that is neccessary.

I have 18 WebSEAL instances with an /isam junction receiving uniform traffic, based on the request logs. The /isam junction has four federation servers (in a cluster) as backends. What are the ways I can ensure requests are spread uniformly across all four backends? I feel my primary master is receiving more requests compared to the other three. Is there a way to verify this and ensure traffic is uniform across all four nodes?