Originally posted by: sajid1

Hi,

I want to know about all the parameters of vmstat,iostat and sar command in such way that i can compare and reach the conlusion.I am not AIX admin I am DBA.

Quadri
#AIX-Forum

Originally posted by: orphy

Well, I don't mean to be rude but this is a broad topic. A question like this could get you thousands of answers. What exactly is this "performance issue" that you are having?
Orphy
#AIX-Forum

Originally posted by: sajid1

Thanks Orphy,

Actually I am facing database performence issue on production.Sometime it seems hang while shows hdisk0 and hdisk1 100% utilization.For your information database and datafiles are not located on hdisk0 and hdisk1 it is on hdisk3.In this context i want to run vmstat,iostat to check out the issue and try to know what actual happened.

example:- vmstat 2 5
it will display five times output in two second so i want to know how to compare in between output and what are the all parameters example r , b , avm, fre , re etc.

Quadri
#AIX-Forum

Originally posted by: orphy

In that case, simply look at the man pages (e.g. man vmstat) of the commands to find out what those columns mean.

It sounded like you could possibly have a lot of paging going on to the paging device(s) on those disks which I assume is mirrored. Use topas to check that about paging. And if you have the topas_nmon command on your box, you could use that too.

Is this is a new performance issue? Have there been an upgrade (OS, DB, etc) of some sort recently?
Orphy
#AIX-Forum

Originally posted by: sajid1

Hi Orphy,
Is this is a new performance issue? Have there been an upgrade (OS, DB, etc) of some sort recently?
---No not new issue it is occuring since eight month.No upgrade niether OS or DB.

Thanks
#AIX-Forum

Originally posted by: alethad

What database are you running? Ho much is your paging size and your RAM?
There may some parameters you can change either in the database and/or the OS to help with your performance issue to use less paging.

It may be that you just don't have quite enough paging configured.

If you are running Oracle there is a very good white paper on tuning AIX with Oracle. I have given it out to other forum questions just last month.

Good luck.
#AIX-Forum

Originally posted by: sajid1

Hi Alethad,

What database are you running? Ho much is your paging size and your RAM?
--10.2.0.4.0,
$ lsps -a
Page Space Physical Volume Volume Group Size %Used Active Auto Type
hd6 hdisk0 rootvg 19712MB 29 yes yes lv

There may some parameters you can change either in the database and/or the OS to help with your performance issue to use less paging.
--How can i get this?

It may be that you just don't have quite enough paging configured.
--How can i understand this?

If you are running Oracle there is a very good white paper on tuning AIX with Oracle. I have given it out to other forum questions just last month.
--kindly provide the link?

Thanks,
Quadri
#AIX-Forum

Originally posted by: SystemAdmin

I am not a DBA but an AIX SA and deal with this all the time. I have been tuning the OS for Oracle 9i, 10g and 11g for the last 5 years.

There have been several papers from the ORacle IBM Compentency Center on tuning AIX for Oracle.

Tuning AIX for ORacle all depends on several things

The SGA, PGA , types of Drives, ASM in use, GPOFS, is it RAC.

First step should be contact your SA, if you have one and explain the problem

If this is not oracle, the same approach can still be used from the OS point of view but there may be specific database paramters that will need to be changed
Can you send me results of commands below :

cat /etc/tunables/nextboot

lsattr -El sys0

lspv

lsdev -Cc disk

errpt

vmstat 5 5

lsps -a

a print screen of topas output and I may be able to suggest a few things

(-l is lower case L )
iostat -d -l -V -D 5 2
#AIX-Forum

Originally posted by: SystemAdmin

We face the same issue especially when rman backups are running. Mainly it is the paging space in use as AIX is busy caching file pages. So you have sufficient RAM and an appropriate paging space, then think about tuning file caching, committing dirty buffers, frequency of rman, etc

r/
R
#AIX-Forum

Originally posted by: sajid1

Hi Rnoel,

We face the same issue especially when rman backups are running. Mainly it is the paging space in use as AIX is busy caching file pages. So you have sufficient RAM and an appropriate paging space, then think about tuning file caching, committing dirty buffers, frequency of rman, etc
----How can i get and comapre with culprit instance either it has sufficient RAM and paging space or not??

Thanks,
#AIX-Forum

Originally posted by: SystemAdmin

If you can provide the information from the commands below , I can maybe make suggestions.

I have several databases where I worked and they range in size from 2GB to 150+TB. I have had to do a lot of tuning over the years but I start by gathering info.

The commands below are a starting point for you to try to analyse the problem

Is this an LPAR ?

Are you using APV capabilities like CPU Pooling ?

Is it oracle ?

If so are you using ASM or Filesystems ?

cat /etc/tunables/nextboot ( shows me what the current modified tunables are )

lsattr -El sys0 ( Lets me see the System Settings )

lspv ( just need an overview of the disks )

lsdev -Cc disk ( What are the disks types SAN, Local, MPIO ... )

errpt ( Are you get any errors )

vmstat -I 5 5 ( shows #CPU and Memory )

lsps -a ( How is paging )

a print screen of topas output and I may be able to suggest a few things ( Let me get a quick overview of the system )

(-l is lower case L )
iostat -d -l -V -D 5 2 ( Disk activity with extended stats)

you may have to enable this
The lsattr -El sys0 will show you iostat value , if false
Use the command below to enable it ( No reboot is needed )

chdev -l sys0 -a iostat=true

Will help if you can send the information
#AIX-Forum

Originally posted by: sajid1

Many Thanks Deank,

Find below output of your action plan almost for all.For your information The output is not taken while issue occured(hdisk0 and hdisk1 shows 100% using).
Is this an LPAR ?
--yes
Are you using APV capabilities like CPU Pooling ?
--How can i get this
Is it oracle ?
--yes

If so are you using ASM or Filesystems ?
--no idea,in oracle we are not using ASM,HOW CAN I FIND IT.

cat /etc/tunables/nextboot ( shows me what the current modified tunables are )
--$ cat /etc/tunables/nextboot
cat: 0652-050 Cannot open /etc/tunables/nextboot.

lsattr -El sys0 ( Lets me see the System Settings )
--$ lsattr -El sys0
SW_dist_intr false Enable SW distribution of interrupts True
autorestart true Automatically REBOOT system after a crash True
boottype disk N/A False
capacity_inc 0.01 Processor capacity increment False
capped true Partition is capped False
conslogin enable System Console Login False
cpuguard enable CPU Guard True
dedicated false Partition is dedicated False
ent_capacity 5.00 Entitled processor capacity False
frequency 2660000000 System Bus Frequency False
fullcore false Enable full CORE dump True
fwversion IBM,EM310_063 Firmware version and revision levels False
id_to_partition 0X800004F39C800009 Partition ID False
id_to_system 0X800004F39C800000 System ID False
iostat true Continuously maintain DISK I/O history True
keylock normal State of system keylock at boot time False
log_pg_dealloc true Log predictive memory page deallocation events True
max_capacity 6.00 Maximum potential processor capacity False
max_logname 9 Maximum login name length at boot time True
maxbuf 20 Maximum number of pages in block I/O BUFFER CACHE True
maxmbuf 0 Maximum Kbytes of real memory allowed for MBUFS True
maxpout 0 HIGH water mark for pending write I/Os per file True
maxuproc 2048 Maximum number of PROCESSES allowed per user True
min_capacity 3.00 Minimum potential processor capacity False
minpout 0 LOW water mark for pending write I/Os per file True
modelname IBM,9117-MMA Machine name False
ncargs 128 ARG/ENV list size in 4K byte blocks True
nfs4_acl_compat secure NFS4 ACL Compatibility Mode True
pre430core false Use pre-430 style CORE dump True
pre520tune disable Pre-520 tuning compatibility mode True
realmem 16777216 Amount of usable physical memory in Kbytes False
rtasversion 1 Open Firmware RTAS version False
sed_config select Stack Execution Disable (SED) Mode True
systemid IBM,026533AA0 Hardware system identifier False
variable_weight 0 Variable processor capacity weight False

lspv ( just need an overview of the disks )
--
$ lspv
hdisk0 00c33aa0621dd148 rootvg active
hdisk1 00c33aa0a5823689 rootvg active
hdisk2 none None
hdisk3 00c33aa02b95a0ab datavg active
hdisk4 00c33aa0c8ec5953 datavg active

lsdev -Cc disk ( What are the disks types SAN, Local, MPIO ... )
--
hdisk0 Available 04-08-00 SAS Disk Drive
hdisk1 Available 04-08-00 SAS Disk Drive
hdisk2 Available 04-08-00 SAS Disk Drive
hdisk3 Available 03-00-02 1815 DS4800 Disk Array Device
hdisk4 Available 01-00-02 1815 DS4800 Disk Array Device
$

errpt ( Are you get any errors )
--$ errpt
IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
40E9A4E1 0112141010 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0111154910 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0111154810 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0111154410 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0111154410 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0111154210 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0111154210 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0111154110 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0111154110 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
A6DF45AA 0111140810 I O RMCdaemon The daemon is started.
38A8D4BB 0111140710 I H ent5 HEA PORT DOWN
D6A51BF7 0111140710 I H ent4 HEA PORT UP
EC0BCCD4 0111140710 T H ent1 ETHERNET DOWN
2BFA76F6 0111140510 T S SYSPROC SYSTEM SHUTDOWN BY USER
9DBCFDEE 0111140710 T O errdemon ERROR LOGGING TURNED ON
192AC071 0111135310 T O errdemon ERROR LOGGING TURNED OFF
12337A8D 0111114910 T S DR_KER_MEM Affected memory not available for DR rem
40E9A4E1 0111114010 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0111101510 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0106092510 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0106092010 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0106091810 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0106091710 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0106091610 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0106091510 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0104091010 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0102185310 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0102184010 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0102183210 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0102182410 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0102135410 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0102135110 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0102133110 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0102132810 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 0102112110 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1231020709 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1230020209 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
DE9A52D1 1229040709 I S rmt0 AAA1
40E9A4E1 1228191709 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1228174209 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1228103809 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1228103009 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1228102509 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1228101509 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1228093409 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226191209 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226181209 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226181109 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226180409 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226180009 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226175909 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226174009 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226173709 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226173509 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226173309 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226172809 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226172609 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226172509 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226165509 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226165409 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226165309 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226165209 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226165109 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226165009 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226164909 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226162009 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226161809 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1226132909 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
DE9A52D1 1226040609 I S rmt1 AAA1
40E9A4E1 1223165509 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1223164509 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1223092909 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1223092509 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1223092109 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1223092009 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1223092009 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1223091609 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1223091309 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1223091209 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
DE9A52D1 1223040609 I S rmt1 AAA1
40E9A4E1 1221095209 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1221095109 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
40E9A4E1 1221095009 P S SYSPROC SOFTWARE PROGRAM ABNORMALLY TERMINATED
38A8D4BB 1125184609 I H ent5 HEA PORT DOWN
D6A51BF7 1125184609 I H ent4 HEA PORT UP
EC0BCCD4 1125184609 T H ent1 ETHERNET DOWN

vmstat -I 5 5 ( shows #CPU and Memory )
--
System configuration: lcpu=10 mem=16384MB ent=5.00

kthr memory page faults cpu

---

-----------

---

------------

---

r b p avm fre fi fo pi po fr sr in sy cs us sy id wa pc ec
2 1 0 3288545 394840 197 111 23 0 0 0 618 21204 2485 23 1 73 3 1.23 24.7
2 1 0 3288110 394689 28 110 74 0 0 0 692 17157 2070 15 1 82 3 0.79 15.9
2 1 0 3288985 393228 107 105 6 0 0 0 407 20298 2443 24 5 70 2 1.45 28.9
2 1 0 3288105 392182 344 143 13 0 0 0 544 23713 2848 21 2 74 4 1.12 22.4
2 0 0 3288506 387194 126 966 5 0 0 0 1995 42868 6240 26 3 70 1 1.48 29.6
lsps -a ( How is paging )
--
Page Space Physical Volume Volume Group Size %Used Active Auto Type
hd6 hdisk0 rootvg 19712MB 32 yes yes lv

a print screen of topas output and I may be able to suggest a few things ( Let me get a quick overview of the system )
--
Topas Monitor for host: rmsdb EVENTS/QUEUES FILE/TTY
Sat Jan 16 17:21:33 2010 Interval: 2 Cswitch 5434 Readch 1052.3K
Syscall 37611 Writech 1367.5K
Kernel 7.4 |### | Reads 2015 Rawin 0
User 30.5 |######### | Writes 1968 Ttyout 580
Wait 2.7 |# | Forks 0 Igets 0
Idle 59.4 |################# | Execs 0 Namei 338
Physc = 1.93 %Entc= 38.7 Runqueue 2.0 Dirblk 0
Waitqueue 0.0
Network KBPS I-Pack O-Pack KB-In KB-Out
en3 705.5 2237.8 1667.0 486.2 219.2 PAGING MEMORY
lo0 113.2 429.7 429.7 56.6 56.6 Faults 1172 Real,MB 16384
en2 0.0 0.0 0.0 0.0 0.0 Steals 0 % Comp 61.7
PgspIn 13 % Noncomp 30.3
Disk Busy% KBPS TPS KB-Read KB-Writ PgspOut 0 % Client 30.3
dac1 0.0 1902.3 90.2 490.5 1411.8 PageIn 137
hdisk3 25.9 1902.3 90.2 490.5 1411.8 PageOut 347 PAGING SPACE
hdisk0 8.9 69.8 17.4 53.8 16.0 Sios 485 Size,MB 19712
hdisk1 0.4 16.0 4.0 0.0 16.0 % Used 32.0
hdisk2 0.0 0.0 0.0 0.0 0.0 NFS (calls/sec) % Free 67.9
hdisk4 0.0 0.0 0.0 0.0 0.0 ServerV2 0
dac0utm 0.0 0.0 0.0 0.0 0.0 ClientV2 0 Press:
dac1utm 0.0 0.0 0.0 0.0 0.0 ServerV3 0 "h" for help
dac0 0.0 0.0 0.0 0.0 0.0 ClientV3 0 "q" to quit

Name PID CPU% PgSp Owner
oracle 3616978 14.3 1043.6 oracle
oracle 2064612 8.9 9.4 oracle
oracle 3903670 6.2 10.9 oracle
java 4132952 3.0 55.3 oretail
oracle 938236 0.7 10.9 oracle
oracle 1064968 0.3 7.9 oracle
oracle 397510 0.1 4.1 oracle
topas 1929240 0.1 2.9 sajid
java 757886 0.1 116.4 oretail
aioserve 1167442 0.1 0.4 root
aioserve 1319010 0.1 0.4 root

(-l is lower case L )
iostat -d -l -V -D 5 2 ( Disk activity with extended stats)
--occuring error,not running

you may have to enable this
The lsattr -El sys0 will show you iostat value , if false
Use the command below to enable it ( No reboot is needed )

--$ lsattr -El sys0
SW_dist_intr false Enable SW distribution of interrupts True
autorestart true Automatically REBOOT system after a crash True
boottype disk N/A False
capacity_inc 0.01 Processor capacity increment False
capped true Partition is capped False
conslogin enable System Console Login False
cpuguard enable CPU Guard True
dedicated false Partition is dedicated False
ent_capacity 5.00 Entitled processor capacity False
frequency 2660000000 System Bus Frequency False
fullcore false Enable full CORE dump True
fwversion IBM,EM310_063 Firmware version and revision levels False
id_to_partition 0X800004F39C800009 Partition ID False
id_to_system 0X800004F39C800000 System ID False
iostat true Continuously maintain DISK I/O history True
keylock normal State of system keylock at boot time False
log_pg_dealloc true Log predictive memory page deallocation events True
max_capacity 6.00 Maximum potential processor capacity False
max_logname 9 Maximum login name length at boot time True
maxbuf 20 Maximum number of pages in block I/O BUFFER CACHE True
maxmbuf 0 Maximum Kbytes of real memory allowed for MBUFS True
maxpout 0 HIGH water mark for pending write I/Os per file True
maxuproc 2048 Maximum number of PROCESSES allowed per user True
min_capacity 3.00 Minimum potential processor capacity False
minpout 0 LOW water mark for pending write I/Os per file True
modelname IBM,9117-MMA Machine name False
ncargs 128 ARG/ENV list size in 4K byte blocks True
nfs4_acl_compat secure NFS4 ACL Compatibility Mode True
pre430core false Use pre-430 style CORE dump True
pre520tune disable Pre-520 tuning compatibility mode True
realmem 16777216 Amount of usable physical memory in Kbytes False
rtasversion 1 Open Firmware RTAS version False
sed_config select Stack Execution Disable (SED) Mode True
systemid IBM,026533AA0 Hardware system identifier False
variable_weight 0 Variable processor capacity weight False

Thanks,

Quadri
#AIX-Forum

Originally posted by: SystemAdmin

There is a real good write up at this location http://www.ibm.com/developerworks/aix/library/au-aixoracle/index.html
Thanks for the quick reply of info

I'll list what I see and then give some recommendations.

Guess the other question IS THIS PRODUCTION or do you have a test system to play with.

1. You are using CPU pooling
vnstat command shows this :
System configuration: lcpu=10 mem=16384MB ent=5.00
kthr memory page faults cpu
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa pc ec
2 1 0 3288545 394840 197 111 23 0 0 0 618 21204 2485 23 1 73 3 1.23 24.7
2 1 0 3288110 394689 28 110 74 0 0 0 692 17157 2070 15 1 82 3 0.79 15.9
2 1 0 3288985 393228 107 105 6 0 0 0 407 20298 2443 24 5 70 2 1.45 28.9

The ent=5.00 indicates CPU pooling alomg with the pc and ent values.

2. The vmstat shows you are currently not CPU bound or in an IO wait state.

3, The vmstat shows you still have some memory left on the freelist

4. The vmstat does show you are using paging space and actualy always reading from paging space. (pi column)

5. The lsps command shows you are using 32% of paging space
For a database system this usually means there is not enough memory or configed incorrectly.
you are trying to extend your 16GB of real memory by 6GB more by using paging space
This will work but at a cost of IO to paging space hdisk0

Page Space Physical VG Size %Used Active Auto Type
hd6 hdisk0 rootvg 19712MB 32 yes yes lv

6. What I have started to do with version 5.3 and 6.1 is modify the paging system to favour paging files all the time over program space.
When the page stealer wakes up and decides it wants a page I actually force out files (persistent storage) over programs space.
Most the time my lsps command shows 0-1% used.
Since you are not root you can not show me the /etc/tunables/nextboot

First I would suggest the following settings ( I use them ALL the time )
The SysAdmin will have to do this
If possible get the Sysadmin to send you a copy of the /etc/tunables/nextboot before the changes

vmo -P -o lru_file_repage=0
vmo -P -o lru_poll_interval=10
vmo -P -o minperm%=5
vmo -P -o maxperm%=90
vmo -P -o maxclient%=90

This should cut back on the amount of page space being used and therefore less activity on hdisk0 when a demand for pages occurs.
When pages are requested persistemnt files will be favoured for paging activites until persistenmt storage occupies only 5% of memory
Then Process space will be pagd to paging disk

I am guessing you are not using ASM since all the hdiusks are assign to volume groups.

The other thing I would like if you can provide it is:

lsfs -q
mount
df -g
lsvg -l rootvg
lsvg -l datavg

ps -ef | grep ASM ( this will confirm if you are using ASM or not )

I can make suggestions based on real life experience on tuning IO to disks
Hope this helps ....

Originally posted by: sajid1

Hi,

Guess the other question IS THIS PRODUCTION or do you have a test system to play with.
-----This is production and the issue is occuring since almost 1 year and all the works are effecting while issue occur.

Many Thanks,

Quadri
#AIX-Forum

Originally posted by: sajid1

Hi,

Find below output as required by you.
Kindly send action plan in detail if possible for you i am struggling to sort out this since one year
It is very important for me to resolve this because in coming days some more brands are going to implement to the Oracle Retail sever.
$cat/etc/tunables/nextboot

1. IBM_PROLOG_BEGIN_TAG
2. This is an automatically generated prolog.

#

1. bos530 src/bos/usr/sbin/perf/tune/nextboot 1.1

#

1. Licensed Materials - Property of IBM

#

1. (C) COPYRIGHT International Business Machines Corp. 2002
2. All Rights Reserved

#

1. US Government Users Restricted Rights - Use, duplication or
2. disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

#

1. IBM_PROLOG_END_TAG

$ lsfs -q
Name Nodename Mount Pt VFS Size Options Auto
Accounting
/dev/hd4 -- / jfs2 2097152 -- yes
no
/dev/hd1 -- /home jfs2 2097152 -- yes
no
/dev/hd2 -- /usr jfs2 10485760 -- yes
no
/dev/hd9var -- /var jfs2 4194304 -- yes
no
/dev/hd3 -- /tmp jfs2 12582912 -- yes
no
/proc -- /proc procfs -- -- yes
no
/dev/hd10opt -- /opt jfs2 524288 -- yes
no
/dev/fslv00 -- /vol01 jfs2 1572864000 rw ye
s no
/dev/fslv01 -- /vol02 jfs2 964689920 rw yes
no
/dev/fslv02 -- /vol03 jfs2 387973120 rw yes
no
/dev/lv02 -- /mkcd/cd_images jfs -- rw no
no
Filesystem Helper: 0506-519 Device open failed
$
$ mount
node mounted mounted over vfs date options

---

---------------

---

---

------------

---

/dev/hd4 / jfs2 Jan 11 14:06 rw,log=/dev/hd8
/dev/hd2 /usr jfs2 Jan 11 14:06 rw,log=/dev/hd8
/dev/hd9var /var jfs2 Jan 11 14:07 rw,log=/dev/hd8
/dev/hd3 /tmp jfs2 Jan 11 14:07 rw,log=/dev/hd8
/dev/hd1 /home jfs2 Jan 11 14:07 rw,log=/dev/hd8
/proc /proc procfs Jan 11 14:07 rw
/dev/hd10opt /opt jfs2 Jan 11 14:07 rw,log=/dev/hd8
/dev/fslv00 /vol01 jfs2 Jan 11 14:07 rw,log=/dev/loglv
00
/dev/fslv01 /vol02 jfs2 Jan 11 14:07 rw,log=/dev/loglv
00
/dev/fslv02 /vol03 jfs2 Jan 11 14:07 rw,log=/dev/loglv
00
$
$ df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 1.00 0.95 6% 1975 1% /
/dev/hd2 5.00 3.02 40% 39807 6% /usr
/dev/hd9var 2.00 0.91 55% 592 1% /var
/dev/hd3 6.00 3.13 48% 4427 1% /tmp
/dev/hd1 1.00 0.90 10% 244 1% /home
/proc - - - - - /proc
/dev/hd10opt 0.25 0.18 29% 1284 3% /opt
/dev/fslv00 750.00 187.66 75% 2397949 6% /vol01
/dev/fslv01 460.00 284.27 39% 571592 1% /vol02
/dev/fslv02 185.00 49.83 74% 186173 2% /vol03
$ lsvg -l rootvg
rootvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
hd5 boot 1 2 2 closed/syncd N/A
hd6 paging 77 154 2 open/syncd N/A
hd8 jfs2log 1 2 2 open/syncd N/A
hd4 jfs2 4 8 2 open/syncd /
hd2 jfs2 20 40 2 open/syncd /usr
hd9var jfs2 8 16 2 open/syncd /var
hd3 jfs2 24 48 2 open/syncd /tmp
hd1 jfs2 4 8 2 open/syncd /home
hd10opt jfs2 1 2 2 open/syncd /opt
lg_dumplv sysdump 8 8 1 open/syncd N/A
loglv01 jfslog 1 1 1 closed/syncd N/A
lv02 jfs 7 7 1 closed/syncd /mkcd/cd_images
$ lsvg -l datavg
datavg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
loglv00 jfs2log 1 1 1 open/syncd N/A
fslv00 jfs2 750 750 2 open/syncd /vol01
fslv01 jfs2 460 460 2 open/syncd /vol02
fslv02 jfs2 185 185 2 open/syncd /vol03
$
$ ps -ef|grep ASM
--No output

Many Thanks,

Quadri
#AIX-Forum

Originally posted by: SystemAdmin

Are vol01, 02 and 03 strictly oracle data and index files (.dbf) or do the contain archive logs, temp space , ....

What I would do is implement the following first.
Adjust the tunables so they favour files being paged out when a free page is required, thus protect program space as much as possible. If propgrams are not being paged out then there should be little to no paging activity on the rootvg disk

edit /etc/tunables/nextboot and add the following
this can also be done by using the vmo -P and ioo -P command
vmo:
lru_poll_interval = "10"
lru_file_repage = "0"
minperm% = "5"
maxperm% = "90"
maxclient% = "90"

ioo:
maxpgahead = "16"
j2_dynamicBufferPreallocation = "16"
j2_maxPageReadAhead = "128"
pv_min_pbuf = "1024"
j2_minPageReadAhead = "8"
I would start with this and see how your application behaves.
Use vmstat and topas
Once this is done I would start to look at the filesystems and use a feature called CIO on the mount. Read up on it - Only your datafiles should be mounted using cio ..........
#AIX-Forum

Originally posted by: sajid1

Many Thanks Deank,

I will do as per the action plan soon.

Regards,

Quadri
#AIX-Forum

Originally posted by: sajid1

Hi,

Are vol01, 02 and 03 strictly oracle data and index files (.dbf) or do the contain archive logs, temp space , ....
--In vol01 there are located ORACLE_HOME and all datafiles except control02.ctl,redo5a.log,redo4b.log and redo6b.log these files are located on vol02.Archive files are also located on vol02.In vol03 there is nothing related to oracle database.

Thanks,

Quadri
#AIX-Forum

Originally posted by: MarkTaylor

>> Once this is done I would start to look at the filesystems and use a feature called CIO on the mount.
>> Read up on it - Only your datafiles should be mounted using cio

You are better off enabling this in Oracle itself .. Ref:

http://www.dba-oracle.com/t_aix_cio.htm

HTH
Mark Taylor
#AIX-Forum

Originally posted by: sajid1

Hi Mark,

I am not able to open provided link please copy and paste it here.It is very critical issue.

Regards,

Quadri
#AIX-Forum

Originally posted by: sajid1

Hi Mark,

Kindly answer me I am waiting your update.

Regards,

Quadri
#AIX-Forum

Originally posted by: sajid1

Hi,

Guess the other question IS THIS PRODUCTION or do you have a test system to play with.
---Yes i have test what do you think better to perform first on test instance?
---Anything need to start or down to perform your task?

Many Thanks,

Quadri
#AIX-Forum

Originally posted by: SystemAdmin

Sorry for the delay working on other issues

1. Yes, I would recommend doing in test and then reboot

2. Since you mix data , index and other files this will be a little more tricky.

I will followup later tonight
#AIX-Forum

Originally posted by: sajid1

Hi Deank,

I have performed as per your action plan on test instance.For your information where i performed there is no issue for performence.Let me know what is next for me and what i have to monitor on test??

Thanks,

Quadri
#AIX-Forum

Originally posted by: MarkTaylor

Hi, The link did not work for you ? it works fine for me ..

Try googling for "oracle cio filesystemio_options setall"

..
Oracle init.ora: filesystemio_options = SETALL

HTH
Mark Taylor
#AIX-Forum

Originally posted by: sajid1

Hi MarK,

can you send me all the steps which i have to perform?
Do i have to define async to setall using "alter system set filesystemio_options=setall" only or some other task also?

Many Thanks,

Quadri
#AIX-Forum

Originally posted by: MarkTaylor

Hi, did you try googling ???

You need to set "filesystemio_options = SETALL" in the Oracle init.ora file ..

HTH
Mark Taylor
#AIX-Forum

Originally posted by: sajid1

Many thanks Mark,

did you try googling ???
---Yes,but provided site is not opening here.

ok I will perform as per your action plan soon on PRODUCTION.Just confirm me if anything go wrong can i revert back?
Thanks for patience.

Regards,

Quadri
#AIX-Forum

Originally posted by: sajid1

Hi,

I have changed async to setall in init.ora file, i used same as you required,For your information setall what i changed in capital letter not in small is it ok?

Many Thanks,

Quadri
#AIX-Forum

Originally posted by: sajid1

FYI The database version is 10.2.0.4
#AIX-Forum

Originally posted by: sajid1

Hi Mark,

It seems that issue is resolved after performing your action but now hdisk3 and hdisk2 is showing 100% most of the time and system went little bit slow.I was facing same issue on one of another production server also since some days back that is oracle 9i there i did set same as you said and issue seems resolved but same thing occured there also that is hdisk2 and hdisk3 100%.

Manyyyy Thanks,

Quadri
#AIX-Forum

Originally posted by: sajid1

Hi Mark,

The issue is resolved Really its great job,I have raised SRs and Tkts to oracle and aix support site but did not get any solution but you resolved it.

Many Thanks Mark.

I will be thankful if you provide me your personal account id so that i can send you sometime while critical issue occur.

Regards,

Sajid
#AIX-Forum