Originally posted by: sajid1

Hi,

DB 10.2.0.4 and 9.2.0.4
AIX 5.3

There are three Production database and All database server have same issue that is Disk I/O utilization 100% and while it goes to 100% (frequently happen) the system will damn slow.

What I noticed, there are logical four disks that are
Hdisk1 , hdisk0,hdiak3,hdisk4 ( Now added some disk to one database)

All datafiles ,oracle home and archives are reside on Hdisk3 and 4 and swapping/paging are on hdisk0 and 1

When system damn slow or hanged even to run any small query or to open any form system reply very late while hdisk0 and 1 (notice here 0 and 1) reached 100% or 95%. Please advice.

As a pdf book I received by Oracle Support find below. In the pdf there are mentioned VMO parameters of RAC or NONRAC database. Find below VMO parameters which are set in our AIX environment. Please Advice.

Pdf: white paper: http://www.oracle.com/technetwork/database/rac-aix-system-stability-131022.pdf
vmo -r -o strict_maxperm
strict_maxperm = 0

vmo -r -o strict_maxclient
strict_maxclient = 1

vmo -r -o lru_file_repage
lru_file_repage = 1

vmo -r -o page_steal_method
page_steal_method = 0

$ vmstat -v
4718592 memory pages
4489294 lruable pages
549172 free pages
8 memory pools
1160530 pinned pages
80.0 maxpin percentage
3.0 minperm percentage
90.0 maxperm percentage
26.8 numperm percentage
1206254 file pages
0.0 compressed percentage
0 compressed pages
26.8 numclient percentage
90.0 maxclient percentage
1206254 client pages
0 remote pageouts scheduled
45532 pending disk I/Os blocked with no pbuf
8365475 paging space I/Os blocked with no psbuf
2484 filesystem I/Os blocked with no fsbuf
57 client filesystem I/Os blocked with no fsbuf
10056 external pager filesystem I/Os blocked with no fsbuf
0 Virtualized Partition Memory Page Faults
0.00 Time resolving virtualized partition memory page faults

FYI there is filesystemio_options database parameter It was set before asynch, Now it is SETALL.
After setting SETALL it improved little bit.
Thanks
#AIX-Forum

Originally posted by: ostost

hdisk0 and hdisk1 are probably 100% busy because the system is short on memory and is paging memory in and out from these disks. Did you look at paging rates you have. For example using vmstat or nmon? If the paging rates are high you would need to increase the memory of the server to improve performance.
#AIX-Forum

Originally posted by: alethad

Sajid1 ,
Do you have this document also? It might be worth looking through just to see if there are any other tidbits you might could benefit from. It does have a few references to running RAC I keep it close and refer to it often. It does get updated once in a while too.

You might see if you DBA can do anything to help tune the Oracle also. I know there is some things they can do because mine does. My co-worker and my DBA also just made a few changes letting Oracle handle the more of the disk I/O instead of AIX OS. I can check and find out exactly what they did.
good luck.

Originally posted by: alethad

I found out we set up CIO. You didn't say how you had your Oracle config'd in OS with raw lv's or jfs2 fs's. We don't use raw lv's hence we have benefited greatly by this.

If you think it will help here's the doc for you.

otherwise maybe someone will have a better suggestion for you.
#AIX-Forum

Originally posted by: sajid1

---I have raised Service request to Oracle Support for this issue and he recommended as below changing VMO parameters into the system and these are the same as you provided pdf book.

Oracle support says it is the issue with memory swapping/paging Please check with IBM to confirm they should be changed?

SR Updated:

“Thank you for the information. this sounds like a memory swapping/paging issue.
The vmo parameters can be checked with IBM.
You can also check the vmo parameters against the white paper: http://www.oracle.com/technetwork/database/rac-aix-system-stability-131022.pdf

Note, this white paper was written for RAC, but the vmo parameter settings in the document are also valid for non-rac. “

In our environment the VMO parameter settings are :

$ vmstat –v
80.0 maxpin percentage
3.0 minperm percentage
90.0 maxperm percentage
90.0 maxclient percentage

vmo -r -o strict_maxperm
strict_maxperm = 0

vmo -r -o strict_maxclient
strict_maxclient = 1

vmo -r -o lru_file_repage
lru_file_repage = 1

vmo -r -o page_steal_method
page_steal_method = 0

Recommended pdf parameters by you and Oracle SR:

vmo -p -o maxperm%=90;
vmo -p -o minperm%=3;
vmo -p -o maxclient%=90;
vmo -p -o strict_maxperm=0;
vmo -p -o strict_maxclient=1;
vmo -p -o lru_file_repage=0;
vmo -r -o page_steal_method=1;
---kindly update me can I change VMO parameter as recommended in pdf as above.
#AIX-Forum

Originally posted by: Holgervk

As you use CIO (check with lsof +fg /your/oracle/files), there is no VMO involved in IO.
Thus, all this minperm/maxperm/vmo stuff wont have much effect.

If you use CIO and your system is paging, SGA is too big.
Or other applications (f.e. backup) that dont use CIO fill up ram with filecache.
#AIX-Forum

Originally posted by: sajid1

FILESYSTEMIO_OPTIONS and DISK_ASYNCH_IO what are use of these parameters into database and what optimal value should be defined.

Many Thanks
#AIX-Forum

Originally posted by: alethad

sajid1,
yes you can change these. You need to check to see which ones are dynamic and which ones require a system reboot. I don't recall right now but be sure before you start.

You can issue the change and let the dynamic ones go ahead and take affect. It should not hurt to do that and then let the other take affect with the restart. I have had to do that in the past myself with either this or other vmo parameters.
Hopefully Oracle support can help you tune the DB but as far as the OS they probably won't be able to help you much.

good luck.
#AIX-Forum