We are currently upgrading wm from 8.2 to 9.6 in clustered environment in RedHatEnterprise6. initially we kept all configurations from 8.2, until the OutOfMemoryError appears. we have increased java heap space to 3G RAM for all integration servers(4 IS servers), except one back end server which is used as batch server(java based solution), the heap space has been increased to 3G, but we noticed that this sever still behaves unexpected inconsistently.

Please advise us on how we should proceed further to resolve this issue, any stress test can be recommended to help verify if the heap space is sufficient. Thanks!

You can take a heap dump and analyse that to find out if any service or threads that could be causing any leak.