Hi, I hope someone can shed some light of what is happening with the Integration managing its memory.

We have an intermittant issue where the used memory increases sharply for a reason as yet to be identfied, but at the same time the total memory available reduces.

We are running IS 9.5 on a RedHat Linux 64bit VM machine with the following memory settings
JAVA_MIN_MEM=3072M
JAVA_MAX_MEM=3072M
JAVA_MAX_PERM_SIZE=256M

Normally the IS runs happliy consuming no more than 50% of memory most of the time.
Then periodically, which can be once every few days or once a month or twice a day even an as yet unknown event occurs which causes the memory used to increase significantly, by nearly a gig sometimes, but to make matters worse the Total Memory available drops at the same time, making the used percentage quite high.
Once this happens, the used and total memory never recover to their original state and if a 2nd unknown event occurs, the memory consumptions becomes so high it will causes the server to restart. Normally though we restart it before it gets to the stage.

Any help to understand what sort of issue would cause the total memory reduce like that would be much appreciated.

See the figures taken from the stats.log below.

Time Total MB Used MB(Converted from Free memory)
12:44 3047.0 977.1
12:45 3047.0 1215.3
12:46 3047.0 1413.7
12:47 3047.0 1596.7
12:48 3050.5 986.7
12:49 3050.5 1194.1
12:50 3050.5 1333.0
12:51 3050.5 1546.0
12:52 3050.0 628.5
12:53 3050.0 755.6
12:54 3050.0 1122.4

12:55 2516.5 1552.6
12:56 2589.0 2029.7
12:57 2589.0 1993.7
12:58 2589.0 2247.9
12:59 2589.0 2079.6
13:00 2589.0 2348.7
13:01 2589.0 2047.7
13:02 2589.0 2153.3
13:03 2589.0 2151.7
13:04 2589.0 2231.2
13:05 2589.0 2158.8
13:06 2589.0 2316.3

The attached screen shot demonstrates the issue better.

Many thanks in advance,

Mark

Hi,
You say its an intermittent issue, did you talk to SAG support on this issue?

And we are unsure about the unknown event? During the max memory utilization do you see some message/errors on server.log, I see you have monitored the stats.log

Hi Manesh,

I have not involved SAG yet, although that is the next step if I can’t find a resolution through this forum.

I have checked the server.log(s) at the times when the issue happens and I cannot find a single error anywhere in there or the our internal error logs. No unusual or large doucments are showing in MWS / Monitoring /Integration / B2B / Transactions at those times either. What confused me most is why the total memory drops, when it is needed the most.

Hi Mark,

It’s really a strange issue. Did you get a chance to increase log levels. If not, can you kindly please do it and let us know what you will observe for next a couple of days.

Even if you approach SAG, they request for logs but in logs there is no such data exist to analyze, so no use of approaching SAG at this time.

Thanks,

Hi Mark,

Could you post the output of TOP or HTOP commands? Or any other command which ideally gives the memory usage of every process at the OS level.

There might be a chance that some OS level process is being priortized than the IS JVM, this might be reducing your total memory.

/Naidu

Hi Naidu,

Please see the two attached screen shots.

The first is the Admin Console showing the statistics pages, with a Test and Production linux session, both showing the output of the Top command. At this point the production server had just and Integration Server restarted and was running nicely.

The second is the Admin Console showing the statistics pages, with a Production linux session, showing the output of the Top command. At this point the production Integration Server had a “memory consumption” event earlier in the day and had not yet been restarted.

Thanks

https://higherlogicdownload.s3.amazonaws.com/IMWUC/webMethods/Files3/260ac0d5f06217395002b4a96766e1956498262c_2_1035x496.png 1.5x, https://higherlogicdownload.s3.amazonaws.com/IMWUC/webMethods/Files3/260ac0d5f06217395002b4a96766e1956498262c_2_1380x662.png 2x" data-dominant-color="AEB3AD">

MemoryUsage2014-12-15-AfterEvent.PNG1637×787 97.4 KB

https://higherlogicdownload.s3.amazonaws.com/IMWUC/webMethods/Files3/f76cdd9f3e81cb8b3eab450d5cb63a3dca3e06b6_2_1035x507.png 1.5x, https://higherlogicdownload.s3.amazonaws.com/IMWUC/webMethods/Files3/f76cdd9f3e81cb8b3eab450d5cb63a3dca3e06b6_2_1380x676.png 2x" data-dominant-color="797D7A">

MemoryUsage2014-12-08-AfterRestart.PNG1583×777 120 KB