Originally posted by: SystemAdmin

Hi,
Stats:
AIX 5.3, P590 series. 8 CPUs, 27GB RAM. DB SGA 4GB
MAXCLIENT=MAXPERM% = 20% (MIN IS 10%)
lru_file_repage =1
strict value for maxperm and for maxclient both set to 1.
File system=jfs2.
paging = 1% all the time (means no paging)

Consider the following at point in time:
From vmstat -v, it is clear the file cache is at 19.8% meaning almost all of 20% of 27gb = 5.4gb is in used by cache.

From svmon and vmstat , free memory is 6gb.
From nmon also it says total mem=27gb, free=6gb and used=21gb and file cache used is 5.4 gb.

I would like to know where i am doing wrong below:
Objective:
Try to add the elements/processes and sum it upto 27gb.

Tot_memory = File_cache + unixprocess + shared segment memory +freemem
27gb = 5.4 gb + 9gb + 4gb +6g (as you see i am missing 2.4gb uncounted for)

Running the above at different times more or less am short of 2-2.4 gb always. Where is this 2.4 gb used by???

Filecache, freemem and shared segment calcualtion is pretty much straight forward, and i think you all know how it is to be calcualted. (I even checked ipcs -b and i do not have a memory segment orphan etc).

How I comeup with 9gb is simple.
Running ps vg command and adding (rss-trs)/1024 (mb) to sum for those process ids where PID is also available in (select pid from v$process).
To this i add sum of rss for rest of the remaining process(oracle lile listener, dbconsole) and root processes and all ther processs).

Oracle processs adds upto major like 8gb, rest 1gb is for all remaining processes.
So am sure am doing wrong in calcualting memory from ps vg output. Or is there any other type of mem i misses?

What about jave heap, is this a seperate memory that wont come in ps vg and also in ipcs -b?? Am at a loss.

My only obj for now is to find ways to add all process memory + shared segments + free + file cache and this should sum to 27gb or atleast the difference should not be so high as 2.4 gb missing.
I checked i dont have a leaking unless u tell me how to check??
Cheers
#AIX-Forum

Originally posted by: hdkutz

Take a look at this thread.
http://www-128.ibm.com/developerworks/forums/thread.jspa?threadID=187721
There is a Link to developerworks which contains the answer to your Questions.

Cheers,
ku
#AIX-Forum

Originally posted by: SystemAdmin

Hi,
I have looked at the problem and I do not find it useful at all to relate to my issues.

Cheers
#AIX-Forum

Originally posted by: SystemAdmin

I have some news, maybe you can verify it.

One of my aix friend told me that 5-10% of the memory is reserved by the IBM Machine for logical patition
operation.

For example if you have a 16 cpu, 32gb p590 machine with two parititions each having allocated
16gb of real memory. then 5-10% 32gb can be consumed by the partition for its various functions.

My question is for sure this 5-10$ memory is not assigned to any process when we do ps vg, so
how would a unix admin find it from running commands at the whole p590 , of how much this is actually
allocated and is this 5-10 of the whole or per parition??

cheers
#AIX-Forum

Originally posted by: SystemAdmin

I suspect what your friend was referring to is actually the memory that is not actually assigned to LPARs and so you would not see it from within AIX, but only by looking at the CEC properties from the HMC. The types of things that take this extra memory are the hypervisor which takes on the order of 256 MB of memory give or take, then there are some small chunks of memory set aside for memory device management called the TCE, and finally for each LPAR that is created there is a page table space. The calculate the amount of page table space needed for each LPAR, you divide the MAX memory value from the LPAR definition and divide that value by 64 and then round up to the next power of 2. For example if you have a 32 GB LPAR but you set the MAXimum memory for the LPAR to anywhere from 32.25 - 64 GB then this will set aside 1 GB of memory for this LPAR to be used by the hypervisor for the page table. If you set the max to just 32 GB then you would only use 0.5 GB for page table space.

This really has nothing to do with the memory issue that started this thread.
#AIX-Forum

Originally posted by: SystemAdmin

Yes I concur with you as I just confirmed this from our unix admin and on our 32gb machine, around 2gb is reserved.

So the question remains, on a single partition of 27gb memory, we always see memory not counted for between 2-3gb always. I wonder where we are doing the wrong in our calculation which we verified with oracle support as the correct way to calculate oracle processes memory.

If you know any other means to calculate memory please let me know,

cheers
#AIX-Forum

Originally posted by: hdkutz

http://www.ibm.com/developerworks/aix/library/au-aixoptimization-memtun1
It was in the thread i posted.

Cheers,
ku
#AIX-Forum

Originally posted by: SystemAdmin

thanks for all replied but am sorry none has actually answered my questions.
I will wait for any one who can post some thing new.

For example I know on linux you have some memory reserved for page tables and slab area, but am not able to find
if similar exists on aix.

Cheers
#AIX-Forum

Originally posted by: DavidMcKee

If you are looking to see how much memory is being used by the kernel heap/page table area/etc. use the command "svmon -swS".

You can not sum the RSS for all the processes and get anything useful, RSS=working segments+code segments. If you have multiple ksh/oracle processes then you would be adding the same memory being used by the code segment multiple times. In "svmon -P pid", code segment is Esid 1, you will notice that if you run this command against multiple ksh processes that they all have the same Vsid for Esid 1.

Javaheap can be allocated in the native heap or it can be allocated in memory-mapped segments. If you track all of the mmap/shared memory segments and all of the working segments you will have it covered.

If you want to know the amount of memory used by shared memory segments, -b does not work.
Instead, try "ipcs -mS1", the hex numbers in the SID column can be used with the svmon command to not only tell you how much memory is being used, but also how much is in real memory and how much is paged out.

IE:I added -b to show how the SEGSZ comapered to what svmon reports.


 $ ipcs -mbS1


 IPC status from /dev/mem as of Fri Jan  4 10:08:57 EST 2008


 T        ID     KEY        MODE       OWNER    GROUP     SEGSZ   SID


 Shared Memory:


 m         1 0x78000034 --rw-rw-rw-     root   system  16777216   0xcd79



 $ svmon -S 0xcd79



 Vsid      Esid Type Description              PSize  Inuse   Pin Pgsp Virtual


 cd79         - work                              s     35     0 1459  1474



 Message was edited by: DavidMcKee

#AIX-Forum

Originally posted by: SystemAdmin

Hi,
If you read my question at the top of this thread, you will notice that i have provided a formula for my calculation results which do not include summing rss for oracle processes but rss-trs .

Cheers
#AIX-Forum

Originally posted by: SystemAdmin

David, if you can let me know your email address I will send you the results of output
for all of the commands , perhaps you can then find something which i over looked.

Just for your information I have already calculated the shared segment which includes 2 sgas from oracle one for
main db instance and another for asm to manage raw disks.

Cheers
shahid
#AIX-Forum

Originally posted by: SystemAdmin

Update:
I ran the command svmon -swS and it produces a long list with the following format:
Vsid Esid Type Description PSize Inuse Pin Pgsp Virtua
0 - work kernel segment (lgpg_vsid=0) L 16 16 0 16
7003 - work kernel heap s 65536 0 0 65536
5002 - work kernel heap s 65536 0 0 65536
9004 - work kernel heap s 54327 26033 0 54327
3001 - work kernel heap s 46344 634 0 46344
2000 - work mbuf pool s 26400 26400 0 26400
b005 - work kernel heap s 16817 15328 0 16817
d005 - work vmm software hat s 16384 16384 0 16384
3000 - work page table area s 9777 17 9760 9777
2c4162 - work other kernel segments s 4599 4578 0 4599
2001 - work other kernel segments s 1526 0 0 1526
13809c - work other kernel segments s 1313 1313 0 1313
88044 - work other kernel segments s 1313 1313 0 1313
228114 - work other kernel segments s 1313 1313 0 1313
5802c - work other kernel segments s 1313 1313 0 1313
21810c - work other kernel segments s 1313 1313 0 1313
d806c - work other kernel segments s 1313 1313 0 1313
1b80dc - work other kernel segments s 1313 1313 0 1313
1480a4 - work other kernel segments s 1313 1313 0 1313
68034 - work other kernel segments s 1313 1313 0 1313
c8064 - work other kernel segments s 1313 1313 0 1313
28014 - work other kernel segments s 1313 1313 0 1313
e8074 - work other kernel segments s 1313 1313 0 1313
1a80d4 - work other kernel segments s 1313 1313 0 1313
b805c - work other kernel segments s 1313 1313 0 1313
48024 - work other kernel segments s 1313 1313 0 1313
108084 - work other kernel segments s 1313 1313 0 1313
23811c - work other kernel segments s 1313 1313 0 1313
1e80f4 - work other kernel segments s 1313 1313 0 1313
128094 - work other kernel segments s 1313 1313 0 1313
1980cc - work other kernel segments s 1313 1313 0 1313
1580ac - work other kernel segments s 1313 1313 0 1313
208104 - work other kernel segments s 1313 1313 0 1313
7803c - work other kernel segments s 1313 1313 0 1313
27813c - work other kernel segments s 1313 1313 0 1313
a8054 - work other kernel segments s 1313 1313 0 1313
8004 - work other kernel segments s 1313 1313 0 1313
1880c4 - work other kernel segments s 1313 1313 0 1313
248124 - work other kernel segments s 1313 1313 0 1313
3801c - work other kernel segments s 1313 1313 0 1313
1c80e4 - work other kernel segments s 1313 1313 0 1313
268134 - work other kernel segments s 1313 1313 0 1313
11808c - work other kernel segments s 1313 1313 0 1313
1800c - work other kernel segments s 1313 1313 0 1313
1f80fc - work other kernel segments s 1313 1313 0 1313
1d80ec - work other kernel segments s 1313 1313 0 1313
9804c - work other kernel segments s 1313 1313 0 1313
f807c - work other kernel segments s 1313 1313 0 1313
1680b4 - work other kernel segments s 1313 1313 0 1313
1780bc - work other kernel segments s 1313 1313 0 1313
25812c - work other kernel segments s 1313 1313 0 1313
1000 - work kernel heap s 1271 9 0 1271
7000 - work vmm data segment s 636 636 0 636
5001 - work page space disk map s 540 540 0 540
13f09e - work loader segment s 344 0 0 344
2cd165 - work vmm scb segment s 338 338 0 338
2e9b7c - work s 204 65 0 204
2c2161 - work other kernel segments s 180 176 0 180
23c716 - work s 179 50 0 179
2f8974 - work s 163 43 0 163
22411a - work s 162 161 0 162
1e70f0 - work process and thread tables s 157 157 0 157
28d145 - work lfs segment s 136 0 0 136
115089 - work vmm vmintervals s 129 129 0 129
28a145 - work other kernel segments s 107 0 0 107
2a2151 - work other kernel segments s 90 82 0 90
1e50f1 - work process and thread tables s 79 79 0 79
300788 - work s 78 20 0 78
ee87f - work s 71 21 0 71
1d00e8 - work other kernel segments s 69 69 0 69
270138 - work other kernel segments s 69 69 0 69
c0060 - work other kernel segments s 69 69 0 69
30018 - work other kernel segments s 69 69 0 69
50028 - work other kernel segments s 69 69 0 69
1a00d0 - work other kernel segments s 69 69 0 69
200100 - work other kernel segments s 69 69 0 69
80040 - work other kernel segments s 69 69 0 69
110088 - work other kernel segments s 69 69 0 69
230118 - work other kernel segments s 69 69 0 69
1f00f8 - work other kernel segments s 69 69 0 69
260130 - work other kernel segments s 69 69 0 69
f0078 - work other kernel segments s 69 69 0 69
120090 - work other kernel segments s 69 69 0 69
d0068 - work other kernel segments s 69 69 0 69
1800c0 - work other kernel segments s 69 69 0 69
1b00d8 - work other kernel segments s 69 69 0 69
a0050 - work other kernel segments s 69 69 0 69
20010 - work other kernel segments s 69 69 0 69
280140 - work other kernel segments s 69 69 0 69
40020 - work other kernel segments s 69 69 0 69
220110 - work other kernel segments s 69 69 0 69
100080 - work other kernel segments s 69 69 0 69
e0070 - work other kernel segments s 69 69 0 69
10008 - work other kernel segments s 69 69 0 69
130098 - work other kernel segments s 69 69 0 69
1600b0 - work other kernel segments s 69 69 0 69
1e00f0 - work other kernel segments s 69 69 0 69
210108 - work other kernel segments s 69 69 0 69
b0058 - work other kernel segments s 69 69 0 69
1c00e0 - work other kernel segments s 69 69 0 69
60030 - work other kernel segments s 69 69 0 69
1400a0 - work other kernel segments s 69 69 0 69
240120 - work other kernel segments s 69 69 0 69
1700b8 - work other kernel segments s 69 69 0 69
1500a8 - work other kernel segments s 69 69 0 69
1900c8 - work other kernel segments s 69 69 0 69
70038 - work other kernel segments s 69 69 0 69
90048 - work other kernel segments s 69 69 0 69
250128 - work other kernel segments s 69 69 0 69
15e6a7 - work s 63 17 0 63
1b26d1 - work s 61 16 0 61
2d2761 - work s 61 18 0 61
139c94 - work s 46 12 0 46
14c2ae - work s 46 14 0 46
19e6c7 - work s 44 13 0 44
11308a - work vmm vmintervals s 43 43 0 43
280148 - work s 41 40 0 41
1986c4 - work s 32 9 0 32
322799 - work s 31 9 0 31
2d2961 - work s 31 9 0 31
6003 - work other kernel segments s 31 31 0 31
22a71d - work s 26 7 0 26
e2679 - work s 26 7 0 26
16003 - work s 25 25 0 25
c846c - work s 21 20 0 21
8c84e - work s 20 6 0 20
4c02e - work s 19 18 0 19
28c146 - work other kernel segments s 19 0 0 19
26d135 - work jfs segment s 17 17 0 17
24b126 - work mpdata debug s 16 16 0 16
24d125 - work mpdata debug s 16 16 0 16
249127 - work mpdata debug s 16 16 0 16
24f124 - work mpdata debug s 16 16 0 16
255129 - work mpdata debug s 16 16 0 16
257128 - work mpdata debug s 16 16 0 16
28b146 - work lfs segment s 15 0 0 15
b8454 - work s 12 3 0 12
ac45e - work s 12 11 0 12
30a98d - work s 12 3 0 12
13e497 - work s 10 3 0 10
290148 - work other kernel segments s 10 0 0 10
28e147 - work other kernel segments s 10 0 0 10
292149 - work other kernel segments s 10 0 0 10
2c976c - work s 9 2 0 9
1c30e0 - work vmm rmalloc heap s 9 9 0 9
2401a - work s 8 7 0 8
29a14d - work other kernel segments s 5 0 0 5
29814c - work other kernel segments s 5 0 0 5
29414a - work other kernel segments s 5 0 0 5
16c0bf - work s 5 0 0 5
29614b - work other kernel segments s 5 0 0 5
29e14f - work other kernel segments s 5 0 0 5
100088 - work s 5 4 0 5
2a0150 - work other kernel segments s 5 0 0 5
10d085 - work vmm ame segment s 4 0 0 4
29c14e - work other kernel segments s 4 0 0 4
242329 - work s 2 1 0 2
1862cb - work s 2 1 0 2
2801c - work s 2 1 0 2
112081 - work s 2 1 0 2
27013 - work kernel heap s 0 0 0 0
1ee0f7 - work other kernel segments s 0 0 0 0
1ef0f4 - work process and thread tables s 0 0 0 0
10b085 - work kernel heap s 0 0 0 0
10b086 - work vmm ame segment s 0 0 0 0
f1078 - work kernel heap s 0 0 0 0
1ef0f7 - work kernel heap s 0 0 0 0

I then added the Inuse column and multiply it by 4k , then divide the sum twice by 1024 and got 1.4 gb of memory.

Thats good news!!!!. As I told you i can not count for memory like 2.4 gb to 3gb. so atleast i now know where 1.4 gb is going to? Can you please confirm that the above list in full is a separate memory and not overlapping with
any of the process ids so not to calculate twice??

However I still am missing like 1 to 1.5 gb of memory.????

I have ran your command ipcs -mS1 but it gave me the same result as of ipcs -b to be honest, the two major shared segments are actually two oracl sgas which i already know from oracle init.ora , while there are other shared segments but hardly worth adding, i mean they add up to 50-100 mb only.

Perhaps the rest of memory is in java heap so I like to understand further what did you mean by the following:
"Javaheap can be allocated in the native heap or it can be allocated in memory-mapped segments. If you track all of the mmap/shared memory segments and all of the working segments you will have it covered."

How will i find out mmap/shared memory for the java heap?? if it is output of ipcs then i already added this memory.....

Cheers
#AIX-Forum

Originally posted by: DavidMcKee

The -s option in svmon should ensure that only system segments will be included in the report.

There is a column PSize in the svmon output, I noticed that the PSize column for segment 0 was L(large page:16M), adjust for that and you will have another 255.9MB accounted for.

Here is a link that goes over java memory usage on AIX

http://www-128.ibm.com/developerworks/aix/library/au-JavaPart2.html

ipcs will list the mmap as well as the shared memory segments, so if you have java applications that are allocating the javaheap in memory-mapped segments then the ipcs command will account for those.

Message was edited by: DavidMcKee

#AIX-Forum

Originally posted by: SystemAdmin

Hi,

I do not understand what you mean by:
"There is a column PSize in the svmon output, I noticed that the PSize column for segment 0 was L(large page:16M), adjust for that and you will have another 255.9MB accounted for."

PSize stands for paging size? while we are interested in RAM ?And I thought we are not using large page on this machine but default 4k size as we hardly have paging??
How can 16M becomes 255.9M?There is only one L line in the output?

Please find below output for ipcs command plus svmon and lsps. Please tell me where is the java heap mmap?

oracle @ tadrac1: /u1/app/oracle/product/admin >ipcs -mbS1
IPC status from /dev/mem as of Fri Jan 4 22:21:06 saust 2008
T ID KEY MODE OWNER GROUP SEGSZ SID
Shared Memory:
m 1048576 0x0000ecf8 --rw-rw---- root system 6048 0x36c3be
m 1048577 0x4154520c --rw-rw-rw- twsuser tws 1048576 0x31a385
m 1048578 0x41545200 --rw-rw-rw- twsuser tws 1048576 0x2e0378
m 3 0x4154520b --rw-rw-rw- twsuser tws 524288 0xba455
m 4 0x4154520d --rw-rw-rw- twsuser tws 1048576 0x26c53e
m 5 0x4154520e --rw-rw-rw- root tws 1048576 0x28854c
m 6 0x0d009321 --rw-rw---- root system 1440 0x32a39d
m 7 0x7800021a --rw-rw-rw- root system 16777216 0x3c616
m 8 0xffffffff --rw-rw---- root system 4096 0x23a715
m 9 0xffffffff --rw-rw---- root system 4096 0x18a6cd
m 10 0xffffffff --rw-rw---- root system 4096 0x27c736
m 11 0x020040f5 --rw-rw-rw- patrol adm 2048 0x1d28e1
m 12 0x010040f4 --rw-rw-rw- patrol adm 2296 0x1ae8df
m 1048589 0x410040f7 --rw-rw-rw- patrol adm 768 0x258924
m 14 0x410040f9 --rw-rw-rw- patrol adm 256 0x21e907
m 15 0x410040fa --rw-rw-rw- patrol adm 256 0x1b68d3
m 16 0x410040fb --rw-rw-rw- patrol adm 256 0x20c90e
m 7340049 0x410040fc --rw-rw-rw- patrol adm 3840 0x1e88fc
m 18 0x410040fd --rw-rw-rw- patrol adm 256 0x1a68db
m 1048595 0x410040fe --rw-rw-rw- patrol adm 768 0x22691b
m 3145748 0x410040f6 --rw-rw-rw- patrol adm 244600 0x1ee8ff
m 1048597 0xb3fff550 --rw-r----- oracle dba 218107904 0xbc856
m 2097174 0x410040f8 --rw-rw-rw- patrol adm 1280 0x130890
m 23 0xd29e674c --rw-rw---- oracle dba 3959435264 0x14caae 0x11a
a85 0x12ca9e 0x62a39 0x13ca96 0x144aaa 0xe8a7c 0x140aa8 0xe4a7a 0x15caa6 0xe0a78
0x158aa4 0xfca76 0x154aa2 0x10ca8e
m 5242904 0x410040ff --rw-rw-rw- patrol adm 2816 0x10ea8f
m 1048601 0x41004101 --rw-rw-rw- patrol adm 768 0x136a93
m 26 0x4154520a --rw-rw-rw- twsuser tws 1048576 0x37f9b7
oracle @ tadrac1: /u1/app/oracle/product/admin >lsps -s

Total Paging Space Percent Used
30720MB 1%

oracle @ tadrac1: /u1/app/oracle/product/admin >svmon -G
size inuse free pin virtual
memory 7077887 1902231 5175656 473659 1820713
pg space 7864320 10119

work pers clnt
pin 473653 0 6
in use 1820713 0 81518
oracle @ tadrac1: /u1/app/oracle/product/admin >

Cheers
#AIX-Forum

Originally posted by: DavidMcKee

From: http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds5/svmon.htm

PSize

Indicates the size of the pages inside the segment. The page size is represented by a letter (for example, s for 4KB pages and L for 16MB pages).

The reason for 255.9.

16*16MB=256MB

65536*4KB=256MB

If it was originally assumed that the 16 for Vsid 0 was in 4KB units, then the difference would be (65536-16)*4/1024=255.9375MB

If you want to check a java process to see what it is using for memory

svmon -nmP pid|egrep -v 'clnt|pers'

If it has any memory-mapped segments you will see them.

If you have the output from "ipcs -mS1", you can see who is using that segment with "svmon -lS sid".

#AIX-Forum

Originally posted by: SystemAdmin

Thanks for your valuable replies.

After applying all of the concept , am now short of 900-1gb of memory uncounted for?

Is there any other structure we missed? like memory for managing raw devices because this db server has
database on raw devices.
#AIX-Forum

Originally posted by: SystemAdmin

Hi David,

In order to find out the missing memory, I embarked on a different exercise.
We have created a new AIX 5.3 LPAR with the following settings and I calculated the mmeory parameters exactly as per your formul, and in this 1gb box, am left with around 150-200mb uncounted for.

I am copying here all the details, perhaps if you can throw some light on the calculations.

root @ unixdev: /home/root >vmstat -v
262144 memory pages
244915 lruable pages
123422 free pages
4 memory pools
76482 pinned pages
80.0 maxpin percentage
15.0 minperm percentage
20.0 maxperm percentage
6.3 numperm percentage
15539 file pages
0.0 compressed percentage
0 compressed pages
5.9 numclient percentage
20.0 maxclient percentage
14469 client pages
0 remote pageouts scheduled
20 pending disk I/Os blocked with no pbuf
15169 paging space I/Os blocked with no psbuf
5584 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
0 external pager filesystem I/Os blocked with no fsbuf
0 Virtualized Partition Memory Page Faults
0.00 Time resolving virtualized partition memory page faults
root @ unixdev: /home/root >

System configuration: lcpu=4 mem=1024MB ent=0.20

kthr memory page faults cpu
<hr />
-----------<hr />
------------<hr />
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
0 0 127333 123416 0 0 0 0 0 0 16 123 213 2 3 95 0 0.02 7.7
0 0 127437 123246 0 0 0 0 0 0 17 162 273 2 4 93 1 0.02 9.4
0 0 127437 123241 0 0 0 0 0 0 17 60 218 1 3 96 0 0.01 6.7

^Croot @ unixdev: /home/root >lsps -s
Total Paging Space Percent Used
768MB 1%

ps vg output:
<hr />
PID TTY STAT TIME PGIN SIZE RSS LIM TSIZ TRS %CPU %MEM COMMAND
0 - A 33:23 9 52 52 xx 0 0 0.0 0.0 swapper
1 - A 0:34 1275 668 692 xx 25 28 0.0 0.0 /etc/ini
8196 - A 139:48 0 40 40 xx 0 0 0.0 0.0 wait
12294 - A 0:03 63 52 52 xx 0 0 0.0 0.0 sched
16392 - A 1:57 9 104 104 xx 0 0 0.0 0.0 lrud
20490 - A 0:00 0 48 48 xx 0 0 0.0 0.0 vmptacrt
24588 - A 0:00 0 60 60 xx 0 0 0.0 0.0 xmfreed
28686 - A 0:00 0 48 48 xx 0 0 0.0 0.0 memp_rbd
32784 - A 0:00 0 48 48 xx 0 0 0.0 0.0 memgrdd
36882 - A 0:00 0 48 48 xx 0 0 0.0 0.0 psgc
40980 - A 0:12 595 124 124 xx 0 0 0.0 0.0 pilegc
45078 - A 4:40 1226 52 52 xx 0 0 0.0 0.0 xmgc
49176 - A 245:29 0 40 40 xx 0 0 0.1 0.0 wait
53274 - A 62:37 0 40 40 xx 0 0 0.0 0.0 wait
57372 - A 140:15 0 40 40 xx 0 0 0.0 0.0 wait
61470 - A 0:41 445 48 48 xx 0 0 0.0 0.0 netm
65568 - A 22:17 67 116 116 xx 0 0 0.0 0.0 gil
69666 - A 0:00 0 60 60 xx 0 0 0.0 0.0 wlmsched
77888 - A 0:00 0 48 48 xx 0 0 0.0 0.0 lvmbb
90180 - A 0:00 0 48 48 xx 0 0 0.0 0.0 jfsz
94414 - A 0:00 60 780 792 xx 58 60 0.0 0.0 /usr/lib
98458 - A 0:00 0 208 208 xx 0 0 0.0 0.0 dog
106710 - A 1:36 553 4520 4060 xx 823 0 0.0 1.0 /usr/sbi
114808 - A 0:04 69 68 68 xx 0 0 0.0 0.0 rtcmd
118882 - A 41:09 61004 492 484 xx 3 4 0.0 0.0 /usr/sbi
123044 - A 1:56 1020 2724 2076 32768 217 44 0.0 0.0 /opt/pat
135234 - A 0:00 822 128 164 xx 33 36 0.0 0.0 /usr/ccs
139336 - A 0:11 69 52 52 xx 0 0 0.0 0.0 rgsr
155744 - A 0:12 238 48 48 xx 0 0 0.0 0.0 n4bg
159830 - A 1:21 614 128 128 xx 0 0 0.0 0.0 kbiod
163920 - A 0:01 83 60 60 xx 0 0 0.0 0.0 nfsSM
168018 - A 0:05 228 52 52 xx 0 0 0.0 0.0 rdpgc
176262 - A 0:00 292 820 728 xx 36 12 0.0 0.0 /usr/sbi
188522 - A 0:00 173 416 404 xx 25 20 0.0 0.0 /usr/sbi
192612 - A 0:08 178 324 324 xx 0 0 0.0 0.0 j2pg
196740 - A 7:29 1668 184 184 xx 0 0 0.0 0.0 rpc.lockd
209072 - A 0:00 0 48 48 xx 0 0 0.0 0.0 random
217236 - A 0:00 227 976 984 xx 33 28 0.0 0.0 /usr/sbi
221422 - A 0:00 2 292 248 xx 4 0 0.0 0.0 /usr/sbi
233684 - A 0:00 4 784 784 xx 57 72 0.0 0.0 telnetd
237732 pts/1 A 0:00 437 556 744 xx 191 224 0.0 0.0 -ksh
241810 - A 40:20 2 400 396 xx 48 72 0.0 0.0 /usr/sbi
258184 - A 8:44 4981 740 748 xx 150 44 0.0 0.0 /usr/sbi
262290 - A 0:28 2647 524 548 xx 40 36 0.0 0.0 /usr/sbi
266378 - A 0:00 202 2996 3000 xx 34 44 0.0 1.0 /usr/sbi
282806 - A 0:00 0 784 748 xx 57 72 0.0 0.0 telnetd
286896 - A 0:01 181 468 300 xx 360 0 0.0 0.0 /usr/sbi
290968 - A 8:34 1775 200 200 xx 0 0 0.0 0.0 nfsd
311532 - A 0:00 1 148 124 xx 3 0 0.0 0.0 /usr/sbi
315626 - A 198:01 12202 3340 3240 32768 188 24 0.1 1.0 ./snmpma
331972 - A 0:00 29 2832 2760 xx 48 0 0.0 0.0 /usr/sbi
340140 - A 0:00 443 1848 1740 xx 85 80 0.0 0.0 /usr/sbi
344242 - A 0:00 4 180 144 xx 9 0 0.0 0.0 /usr/sbi
368840 - A 4:33 9656 3076 3020 32768 2928 104 0.0 1.0 bgscolle
376882 - A 280:02 47083 26516 27992 32768 3629 1728 0.1 5.0 PatrolAg
381012 - A 1:57 1177 200 192 xx 25 16 0.0 0.0 /var/adm
385140 pts/0 A 0:00 2 620 700 xx 68 80 0.0 0.0 ps vg
393460 - A 0:04 307 376 360 32768 12 12 0.0 0.0 /opt/pat
397524 - A 66:32 779 20924 20756 xx 39 0 0.0 4.0 /usr/jav
418002 - A 1:56 413 240 196 xx 73 8 0.0 0.0 /usr/lpp
438524 - A 62:03 3175 2692 2836 xx 187 196 0.0 1.0 /usr/bin
487678 - A 0:00 47 1416 1064 xx 58 0 0.0 0.0 /usr/sbi
508158 - A 0:00 57 1512 1248 xx 116 4 0.0 0.0 /usr/sbi
512088 - A 0:00 0 476 656 32768 191 224 0.0 0.0 /usr/bin
516100 - A 0:05 396 1316 1116 xx 230 52 0.0 0.0 /usr/sbi
520196 - A 3:10 1942 1768 1564 xx 150 20 0.0 0.0 /usr/sbi
524332 - A 7:29 12713 2456 2396 xx 511 240 0.0 0.0 /usr/sbi
528436 - A 0:00 69 1820 1504 xx 198 0 0.0 0.0 /usr/sbi
561402 pts/3 A 0:00 23 564 784 xx 191 224 0.0 0.0 -ksh
647396 pts/3 A 0:00 33 556 780 xx 191 224 0.0 0.0 -ksh
680090 pts/0 A 0:00 25 556 780 xx 191 224 0.0 0.0 -ksh
688308 pts/4 A 0:00 4 560 768 xx 191 224 0.0 0.0 -ksh
725176 - A 0:37 27 476 416 32768 135 0 0.0 0.0 /opt/pat
761884 - A 0:00 26 476 656 32768 191 224 0.0 0.0 /usr/bin
782360 - A 0:00 47 784 856 xx 57 72 0.0 0.0 telnetd
790654 - A 0:00 114 784 744 xx 57 72 0.0 0.0 telnetd

root @ unixdev: /home/root >ipcs -b
IPC status from /dev/mem as of Sun Jan 6 17:28:54 saust 2008
T ID KEY MODE OWNER GROUP QBYTES
Message Queues:
T ID KEY MODE OWNER GROUP SEGSZ
Shared Memory:
m 0 0xffffffff --rw-rw---- root system 4096
m 1 0xffffffff --rw-rw---- root system 4096
m 2 0xffffffff --rw-rw---- root system 4096
m 3 0x78000053 --rw-rw-rw- root system 16777216
m 10485764 0x02007c41 --rw-rw-rw- patrol adm 2048
m 4194309 0x01007c3f --rw-rw-rw- patrol adm 2296
m 8388614 0x41007c4e --rw-rw-rw- patrol adm 32600
m 6291463 0x41007c46 --rw-rw-rw- patrol adm 256
m 7340040 0x41007c43 --rw-rw-rw- patrol adm 768
m 5242889 0x41007c47 --rw-rw-rw- patrol adm 256
m 4194314 0x41007c48 --rw-rw-rw- patrol adm 256
m 14680075 0x41007c49 --rw-rw-rw- patrol adm 256
m 19922956 0x41007c4a --rw-rw-rw- patrol adm 2816
m 4194317 0x41007c4b --rw-rw-rw- patrol adm 256
m 5242894 0x41007c4c --rw-rw-rw- patrol adm 256
m 7340047 0x41007c4d --rw-rw-rw- patrol adm 768
m 16 0x780010c1 --rw-rw-rw- root system 6000
m 7340049 0x41007c4f --rw-rw-rw- patrol adm 256
m 7340050 0x41007c50 --rw-rw-rw- patrol adm 768
m 19 0x0d001b72 --rw-rw---- root system 1440
T ID KEY MODE OWNER GROUP NSEMS
Semaphores:
s 3145728 0x0102b086 --ra-ra-r-- root system 1
s 1 0x62001c97 --ra-r--r-- root system 1
s 16777218 0xc1007c4d --ra-ra-ra- patrol adm 2
s 7340035 0xc1007c47 --ra-ra-ra- patrol adm 2
s 20971524 0x81007c3f --ra-ra-ra- patrol adm 2
s 5242885 0xc1007c4c --ra-ra-ra- patrol adm 2
s 4194310 0xc1007c43 --ra-ra-ra- patrol adm 2
s 18874375 0x82007c41 --ra-ra-ra- patrol adm 2
s 4194312 0xc1007c48 --ra-ra-ra- patrol adm 2
s 4194313 0xc1007c49 --ra-ra-ra- patrol adm 2
s 4194314 0xc1007c4a --ra-ra-ra- patrol adm 2
s 4194315 0xc1007c4b --ra-ra-ra- patrol adm 2
s 7340044 0xc1007c4e --ra-ra-ra- patrol adm 2
s 17825805 0xc1007c4f --ra-ra-ra- patrol adm 2
s 4194318 0xc1007c50 --ra-ra-ra- patrol adm 2
s 4194319 0xc1007c46 --ra-ra-ra- patrol adm 2
s 16 0x010019a1 --ra------- root system 1
root @ unixdev: /home/root >
root @ unixdev: /home/root >svmon -swS|more

Vsid Esid Type Description PSize Inuse Pin Pgsp Virtual
1 - work kernel heap s 49211 39919 0 49211
0 - work kernel segment s 11394 7556 0 11394
2042 - work other kernel segments s 7344 7312 0 7344
2000 - work mbuf pool s 4416 4416 0 4416
2165 - work vmm scb segment s 1845 1845 0 1845
1001 - work other kernel segments s 1299 0 0 1299
3000 - work page table area s 1096 17 1079 1096
2145 - work lfs segment s 530 0 0 530
7000 - work vmm data segment s 524 524 0 524
1c01c - work other kernel segments s 486 486 0 486
4004 - work other kernel segments s 486 486 0 486
c00c - work other kernel segments s 486 486 0 486
14014 - work other kernel segments s 486 486 0 486
2005 - work vmm software hat s 256 256 0 256
1d09e - work loader segment s 256 0 0 256
1000 - work kernel heap s 226 225 0 226
2002 - work other kernel segments s 224 224 0 224
18508 - work s 70 20 0 70
1041 - work other kernel segments s 67 67 0 67
13523 - work s 62 18 0 62
1d56d - work s 60 16 0 60
170f0 - work process and thread tables s 53 53 0 53
111a1 - work s 43 13 0 43
6576 - work s 41 13 0 41
5025 - work other kernel segments s 34 0 0 34
16546 - work s 31 9 0 31
5675 - work s 29 9 0 29
e51e - work s 28 9 0 28
160f1 - work process and thread tables s 26 26 0 26
10010 - work other kernel segments s 25 25 0 25
8008 - work other kernel segments s 25 25 0 25
18018 - work other kernel segments s 25 25 0 25
20 - work other kernel segments s 25 25 0 25
84b8 - work s 24 7 0 24
19389 - work s 24 6 0 24
11031 - work other kernel segments s 24 22 0 24
17547 - work s 23 7 0 23
12135 - work jfs segment s 18 17 0 18
5255 - work s 18 17 0 18
2125 - work mpdata debug s 16 16 0 16
127 - work mpdata debug s 16 16 0 16
3124 - work mpdata debug s 16 16 0 16
1126 - work mpdata debug s 16 16 0 16
3003 - work other kernel segments s 15 15 0 15
6001 - work page space disk map s 14 14 0 14
6026 - work other kernel segments s 14 0 0 14
1d34d - work s 13 12 0 13
9359 - work s 13 3 0 13
5355 - work s 12 3 0 12
8028 - work other kernel segments s 10 0 0 10
14344 - work s 10 9 0 10
30e0 - work vmm rmalloc heap s 9 9 0 9
9029 - work other kernel segments s 9 0 0 9
1146 - work lfs segment s 9 0 0 9
e089 - work vmm vmintervals s 9 9 0 9
Standard inputroot @ unixdev: /home/root >svmon -swS|more

Vsid Esid Type Description PSize Inuse Pin Pgsp Virtual
1 - work kernel heap s 49211 39919 0 49211
0 - work kernel segment s 11394 7556 0 11394
2042 - work other kernel segments s 7344 7312 0 7344
2000 - work mbuf pool s 4416 4416 0 4416
2165 - work vmm scb segment s 1845 1845 0 1845
1001 - work other kernel segments s 1299 0 0 1299
3000 - work page table area s 1096 17 1079 1096
2145 - work lfs segment s 530 0 0 530
7000 - work vmm data segment s 524 524 0 524
1c01c - work other kernel segments s 486 486 0 486
4004 - work other kernel segments s 486 486 0 486
c00c - work other kernel segments s 486 486 0 486
14014 - work other kernel segments s 486 486 0 486
2005 - work vmm software hat s 256 256 0 256
1d09e - work loader segment s 256 0 0 256
1000 - work kernel heap s 226 225 0 226
2002 - work other kernel segments s 224 224 0 224
18508 - work s 70 20 0 70
1041 - work other kernel segments s 67 67 0 67
13523 - work s 62 18 0 62
1d56d - work s 60 16 0 60
170f0 - work process and thread tables s 53 53 0 53
111a1 - work s 43 13 0 43
6576 - work s 41 13 0 41
5025 - work other kernel segments s 34 0 0 34
16546 - work s 31 9 0 31
5675 - work s 29 9 0 29
e51e - work s 28 9 0 28
160f1 - work process and thread tables s 26 26 0 26
10010 - work other kernel segments s 25 25 0 25
8008 - work other kernel segments s 25 25 0 25
18018 - work other kernel segments s 25 25 0 25
20 - work other kernel segments s 25 25 0 25
84b8 - work s 24 7 0 24
19389 - work s 24 6 0 24
11031 - work other kernel segments s 24 22 0 24
17547 - work s 23 7 0 23
12135 - work jfs segment s 18 17 0 18
5255 - work s 18 17 0 18
2125 - work mpdata debug s 16 16 0 16
127 - work mpdata debug s 16 16 0 16
3124 - work mpdata debug s 16 16 0 16
1126 - work mpdata debug s 16 16 0 16
3003 - work other kernel segments s 15 15 0 15
6001 - work page space disk map s 14 14 0 14
6026 - work other kernel segments s 14 0 0 14
1d34d - work s 13 12 0 13
9359 - work s 13 3 0 13
5355 - work s 12 3 0 12
8028 - work other kernel segments s 10 0 0 10
14344 - work s 10 9 0 10
30e0 - work vmm rmalloc heap s 9 9 0 9
9029 - work other kernel segments s 9 0 0 9
1146 - work lfs segment s 9 0 0 9
e089 - work vmm vmintervals s 9 9 0 9
7027 - work other kernel segments s 8 0 0 8
13003 - work s 8 8 0 8
c02c - work other kernel segments s 6 0 0 6
11f1 - work s 6 5 0 6
1e02e - work s 6 5 0 6
b03b - work s 5 4 0 5
a02a - work other kernel segments s 5 0 0 5
b02b - work other kernel segments s 5 0 0 5
a01a - work s 5 4 0 5
f15f - work s 5 4 0 5
d08a - work vmm vmintervals s 4 4 0 4
e02e - work other kernel segments s 3 0 0 3
d02d - work other kernel segments s 3 0 0 3
10030 - work other kernel segments s 3 0 0 3
f02f - work other kernel segments s 3 0 0 3
a03a - work s 2 1 0 2
61d6 - work s 2 1 0 2
c01c - work s 2 1 0 2
2085 - work vmm ame segment s 2 0 0 2
f85f - work s 2 1 0 2
11011 - work other kernel segments s 0 0 0 0
b02a - work kernel heap s 0 0 0 0
8009 - work kernel heap s 0 0 0 0
5044 - work kernel heap s 0 0 0 0
12012 - work other kernel segments s 0 0 0 0
12013 - work kernel heap s 0 0 0 0
1021 - work other kernel segments s 0 0 0 0
c02d - work kernel heap s 0 0 0 0
13012 - work kernel heap s 0 0 0 0
6047 - work kernel heap s 0 0 0 0

...rest are 0 for Inuse.
Cheers
#AIX-Forum

Originally posted by: DavidMcKee

Check "shared library text", in 32-bit apps this is Esid d and can be see with svmon. All 32-bit apps would be using the same segment, so it should only be counted once. But it could grow to be over 200MB, or it can be small, like 5MB.

I have no idea how much memory is used by kernel extensions and device drivers (genkex, kdb??), but that might cover the rest.

#AIX-Forum

Originally posted by: SystemAdmin

Hi David,

shared text when counted once amounts for 15mb.
1e09d d work shared library text s 3985 0 0 3987
Still am missing some 100mb
Now the memory for kernel extensions and device drivers (genkex, kdb?? How do I calculate this?

One other question:

Output from svmon -P should cover all real memory allocations (is this assumption correct?).
If yes then perhaps like 'work shared library text' there might be other segments which are not linked to
aix processes and I should count them as well?

Cheers
#AIX-Forum

Originally posted by: DavidMcKee

I missed that you were subtracting the TRS from RSS, that should cover the redundancy, though you will be short one TRS per set. Shared library data is also included in the RSS, so after subtracting the TRS from RSS you should have the working segments and the shared library data left that is unique to that process.
#AIX-Forum