Originally posted by: TradeStone

Hi, We have a customer running a Web Application in WebSphere 7.0.0.17, on AIX , GPFS 3,4,0,22. At some point the customer runs out of resources( (inodes). They have increased the ulimit, which moderates the issue, but just delays the need to restart (WebSphere). The fishy thing about this, is that the inodes being used are all clustered around the same inode range. In other words, one "file" appears to be opened multiple times and then not released. We have done a number of things, including assuring that the application does not refference the file in question, and also setting access permissions on the file so that the application cannot actually open the file (logs an error instead). Can anybody please review this output of lsof and give me some idea of what might be going on?

Here is a fragment from the output:

java 26673266 pkmd999 342r VREG 0,0 18446744073709551615 65550 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 357r VREG 0,0 0 65537 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 376r VREG 0,0 0 65539 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 379r VREG 0,0 0 65538 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 384r VREG 0,0 0 65540 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 387r VREG 0,0 12297829382473034410 65537 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 388r VREG 0,0 0 65538 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 393r VREG 0,0 0 65537 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 400r VREG 0,0 0 65537 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 434r VREG 0,0 12297829382473034410 65537 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 435r VREG 0,0 12297829382473034410 65540 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 436r VREG 0,0 450971566191 65537 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 439r VREG 0,0 18446744073709551615 65538 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 440r VREG 0,0 18446744073709551615 65541 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 441r VREG 0,0 0 65542 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 442r VREG 0,0 0 65537 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 443r VREG 0,0 18446744073709551615 65537 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 444r VREG 0,0 18446744073709551615 65537 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 445r VREG 0,0 0 65537 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 446r VREG 0,0 0 65537 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 447r VREG 0,0 0 65546 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 450r VREG 0,0 18446744073709551615 65541 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 451r VREG 0,0 0 65542 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 452r VREG 0,0 18446744073709551615 65537 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 453r VREG 0,0 18446744073709551615 65537 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 454r VREG 0,0 0 65537 /prod/file/te/techspecs (/dev/lv2223GPFS)
java 26673266 pkmd999 455r VREG 0,0 0 65537 /prod/file/te/techspecs (/dev/lv2223GPFS)

Note that the inode range of in-use inodes isalways around 65537-65550

The rest of the output is similar, with about 1100 of these same inodes in use. file attached

Thanks!

Carl

Originally posted by: j.gann

observation: the java process opens _a lot_ of files, some of them multiple times.

suggested action: feed this info to the developer of the java program.

you are not running out of inodes (this is a filesystem object) but out of file descriptors (this is a process object and therefore can be set via ulimit on a per-user basis)

j.gann

Originally posted by: TradeStone

Hi,

I am the developer.  These point to image files served through a servlet, and we are closing the InputSteam in a finally block in every case.

What is strange to me, is that it is only the files occupying these particular inodes that behave this way.  If it were a code defect, I would expect to see the open files spread over a random set of inodes, not clustered around inode 65537, which in and of itself is suspicious, as it is the border of 2^16.

Thanks,

Carl

Originally posted by: j.gann

Hi Carl,

I see, but from the resulting table of open files I would suspect that the close() in the finally block does not succeed every time. Can the return value of the close() and its errno be verified? (close can fail due to timeout e.g.)

A speciality on the side is the use of gpfs. If it's possible to run the app on a different filesystem (jfs2) for comparison that might shed some more light and if it does not happen on jfs2 give (the usually very competent) gpfs support something to chew on.

regards

Joachim