Originally posted by: filippo gadotti

Hi,

Got this behaviour in one of my servers which I can't quite explain. I'd like some tips.

I've got an alarm that my swap usage was up to 80%, which turned to be a fact when I checked `topas`.

The problems I've found were:

1) Looking into a `vmstat 5 10`, I've found lots of PI's with 0 SR's. How's that even possible?

kthr memory page faults cpu

---

-----------

---

------------

---

r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
12 1 12245252 92621 0 229 0 0 0 0 1884 56260 24568 84 15 0 0 2.50 99.9
12 1 12245413 90313 0 190 0 0 0 0 1764 51759 23827 86 14 0 0 2.50 99.9
12 1 12245193 88040 0 218 0 0 0 0 1734 69307 25347 85 15 0 0 2.50 99.9
14 1 12246377 83810 0 157 0 0 0 0 1960 80471 24057 84 16 0 0 2.50 100.0
13 1 12246050 79785 0 183 0 0 0 0 2280 103138 21990 81 19 0 0 2.50 100.0
12 1 12245988 77393 0 173 0 0 0 0 1881 51984 22331 84 16 0 0 2.50 100.0
14 1 12246180 74721 0 179 0 0 0 0 1792 52624 20610 79 21 0 0 2.50 99.9
15 1 12246131 72304 0 176 0 0 0 0 2109 58504 23344 82 18 0 0 2.50 99.9
15 1 12246673 68231 0 187 0 0 0 0 2272 73068 25319 85 15 0 0 2.50 99.9
13 1 12246305 66342 0 172 0 0 0 0 1966 104313 21884 83 17 0 0 2.50 100.0

2) I went off to find who were consuming my precious memory and it I found that I actually know very little in how to figure that in a AIX.

Found this command somewhere which seemed reasonable reading the manual:

`ps -ealf | head -1 ; ps -ealf | sort -rn +9 | head` which seemed reasonable by looking in the manual. and gave an output of the sort:

F S UID PID PPID C PRI NI ADDR SZ STIME TTY TIME CMD
242001 A util 1581080 1 76 60 20 fb34510 150044 10:55:40 pts/0 103:43 /usr...
242001 A util 569540 1 0 60 20 d235510 142580 11:01:09 pts/0 68:55 /usr/...
242001 A util 1425464 1 4 60 20 43c6510 129916 23:17:58 - 168:02 /usr...
202001 A util 245864 1 83 60 24 da9e510 113008 13:37:22 pts/2 43:26 /usr/...
242001 A util 1163370 1 0 68 24 d69d510 103572 09:55:52 pts/13 17:24 /usr/...
242001 A util 466984 1 0 60 20 5d0c510 83064 11:00:34 pts/0 22:57 /usr/...
242001 A raid 1048782 1 7 60 20 e5b8510 78724 16:41:18 pts/6 0:36 /usr/...
242001 A util 659612 1 13 60 20 edc3510 76400 11:13:17 pts/0 10:57 /usr/...
242001 A util 1134736 1 0 60 20 eb91510 75188 06:21:23 - 27:23 /usr/...

Where SZ is supposed to be the size in 1k units, according to the man page; which also didn't make too much sense for these are Java processes with Xms=1G (or so) and as shown the biggest process has ~150mb. Again.. ?

And for the last, my server is not currently at maximum load but still shows ~20% of swap space usage. How do I explain that?

Well.. I'm pretty lost. These things were easier to figure with Solaris :(

Would someone share some thoughts?

thanks,

f.
#AIX-Forum

Originally posted by: Kosala

Hi Filipo,

To have a high page space usage, you do not need to have a high load on the server. My first suspect is one of the processes running on the server leaking memory. The easiest way to findout would be to look for a process where memory foot print growing rapidly.

I would recommend using ps command with -o for a formated output.

Following could be a starting point:
while true; do ps -eko vsz,pid | sort -rn | head -n 5; sleep 5; done

HTH,
Kosala
#AIX-Forum

Originally posted by: filippo gadotti

Hey, thanks!

Didn't really understand the output from your command. I'll be looking the man page in detail in a min. (the output was this)

1736736
1683622
1609830
1536042
1515536

Anyway, I wanted to comment that memory leakage is unlikely; as I mention it is a JVM, so I guess it would be IBM's fault if there was :P in the application level we have also the Xmx set, so our leakages shouldn't go too far (and the sum of our Xmx's ins less than the total RAM).

Thanks again,

f.
#AIX-Forum

Originally posted by: Kosala

Hi,

The comand is simple, ps -ek will list all the processes including the kernel processes (-k). The -o can specify multiple fields for each process... in this case I was more interested in the virtual size (vsz) which is the memory foot print including the page space.

Looking at the output, I can see there are several processes having a mem foot print of nearly 2G. I suppose these are the Java JVMs... how many JVMs do you have in your server.

If you have Xmx set for the JVMs, I would suggest you look at the processes other than java. For example:

ps -eko vsz,pid,command | grep -v java

This server has about 48G mem rt? What does

ps -eko vsz | awk '{vsz+=$1} END { print vsz}'

command say?

Kosala
#AIX-Forum