I will describe our development environment (where the issue was found) to begin. We have a single WebSphere 7 installation, with a single DMGR and node agent. This installation has 5 servers, with a single cluster of 2 JVMs, which I will call Server1 and Server2 for the purposes of this explanation. To conserve resources during performance testing on another server (unrelated to this issue), I shutdown Server2, but did not modify our WebSphere plugin file.

One of our applications interfaces with another, non-WebSphere application (Siebel). This application depends on another server that must timeout in 5 minutes. Accordingly, our various IO timeouts are set to 5 minutes in the plugin and WebSphere servers. However, the calls through our application to Siebel may take longer than 5 minutes to complete in Siebel, which has been accounted for in the response to the client via the application and the plugin. Since the application call in Siebel will be successful, even if lasting longer than 5 minutes, we have set PostBufferSize to 0 to ensure duplicate requests are not sent. (In our testing, if PostBufferSize was set to the default, requests would timeout in the plugin after 5 minutes and be sent to the next JVM, creating duplicate records in Siebel.)

The issue was discovered due to our extended downtime of Server1. After the RetryInterval of 60 seconds expires, the next incoming client's request is routed to Server2. Since Server2 is down, the request is not completed, but is also not redirected to Server1 due to PostBufferSize=0, returning a 404 to the user. While I can fix this in development by manually editing the plugin file, we are concerned that a production server may be inadvertently shutdown or otherwise be unreachable without our immediate knowledge of it. In this case, end-users would have a 25% chance (4 servers in production) to receive a 404 every 5 minutes (RetryInterval=600) until we fix the issue with the JVM.

It there any setting we can use in WebSphere or the plugin file to retry POST requests only when the server is unreachable, but not when the IO timeout is reached? Or am I misunderstanding the cause of our issue?

Thank you in advance,

Adam

Hi Adam,

  So doubts about your question:
 

1. "I shutdown Server2, but did not modify our WebSphere plugin file". What do you need to modify in WebSphere plugin file?

     2.- "While I can fix this in development by manually editing the plugin file". what do you edit manually PostBufferSize?
 
     3.- If you have both servers down, how do you want to serve requests?
   
  thank in advance.
 
kind regards

[quote author=91692909 post=537288437]What do you need to modify in WebSphere plugin file?[/quote]
I can modify the plugin file so it does not know of the existence of Server2. Then it won't try to route requests to it.

[quote author=91692909 post=537288437]what do you edit manually PostBufferSize?[/quote]
I'm not sure I understand your question. If you mean where, we changed it in the plugin file. If you mean why, we had to prevent POST requests from being retried on another server due to how one of our applications work.
[quote author=208322613 post=537286353]One of our applications interfaces with another, non-WebSphere application (Siebel). This application depends on another server that must timeout in 5 minutes. Accordingly, our various IO timeouts are set to 5 minutes in the plugin and WebSphere servers. However, the calls through our application to Siebel may take longer than 5 minutes to complete in Siebel, which has been accounted for in the response to the client via the application and the plugin. Since the application call in Siebel will be successful, even if lasting longer than 5 minutes, we have set PostBufferSize to 0 to ensure duplicate requests are not sent.[/quote]


[quote author=91692909 post=537288437] If you have both servers down, how do you want to serve requests?[/quote]
We don't. Sorry for the confusion. Server1 is up and serving requests without issue. Server2 is down.

When a tester first begins working in the environment in the morning, the first request is sent to Server1; no issue. The second request is sent to Server2; this returns a 404 because PostBufferSize=0 means the request will not be retried on another server. The plugin marks Server2 down for the next 60 seconds (RetryInterval=60) and sends every request to Server1. After 60 seconds have passed of every request going to Server1, the next request is sent to Server2; this returns a 404 because PostBufferSize=0 means the request will not be retried on another server.

I need to leave PostBufferSize=0, but figure out a way so that either (1) valid requests are not sent to Server2 until it is back up or (2) valid requests sent to Server2 when it is down are then resent to Server1.

As a sidenote, I also posted this in IBM developerWorks.

Answered in developerWorks by IBM employee Brian Paskins.

 [quote]There is no way with the version of plugin you are using to route the traffic to another server if you have the PostBufferSize set to 0 and the server is truly down for some reason.  What you can do is have a small program running on the server that hits each web server every 5 seconds or so to test, which is what most load balances do.[/quote]

Hi Adam,

  Thank you very much for share the solution.

regards,