Originally posted by: SystemAdmin

In ANZ, we have three customers running SAP on AIX5L who are reporting IO performance problems, typically manifesting as high IO Wait%. Two customers are using ESS, one is running on a large EMC Symmetrix. The EMC customer also experienced problems on a Clariion system (prompting the move to Symmetrix). I have examined the EMC based customer's problem in some detail and don't believe there is an AIX problem. This customer is using Oracle 9i and, I think SAP 7. I believe one ESS customer is using DB2.

Has anyone else experienced similar problems?
#AIX-Forum

Originally posted by: unycop

En Sap and Oracle9i with Aix 5L and ESS is normal to have waits for IO.
¿The database is in filesystem or in raw device?
if is in filesystem, ¿is the async aio activated in the database and Aix level?
My experience is that when there is a lot of transaction(2000,3000 or more access to disk/sec.) is best raw device.

Is important to distrubute the datafiles of database across the disk, for explample doing a stripping. This is extremaly important for redo logs, in the case of oracle.

Too, is important reduct the io access with a cache of database(sga in oracle) apropiated to the type of access.

My recomendation is to read a lot of notes in SapNet database with recomendation to io problems.
#AIX-Forum

Originally posted by: BruceSpencer

How high is the I/O Wait?

If they are meeting their Service Level Agreement, I would tend to downplay high I/O Wait.

In the past, an I/O Wait 30-40+% was a strong indication of an I/O problem. Today, this is not necessarily the case. The following URL shows why. It shows a real life example where Wait I/O increased after upgrading the server, but keeping the same storage. (The batch time dropped from 60 to 30 minutes, so higher wait I/O wasn't an issue.)

http://www.aixtips.com/AIXtip/iowait.htm

To verify you have a problem, run a [b]filemon[/b] trace and see R/W response times by LV and PV. Response times should be below 10 ms.

Also run a [b]vmstat -v[/b] a couple times during peak. Look for "I/Os blocked" that are growing over time (not the absolute value).

code
vmstat -v
......skipped lines.......
0 pending disk I/Os blocked with no pbuf
2627 paging space I/Os blocked with no psbuf
2740 filesystem I/Os blocked with no fsbuf
46 client filesystem I/Os blocked with no fsbuf
0 external pager filesystem I/Os blocked with no fsbuf
[/code]

If you feel you still have a problem, open a PerfPMR with IBM Support. They can diagnose the performance.


#AIX-Forum

Originally posted by: VirtualGreg

The system interconnect and I/O capability of POWER systems is extremely good. You don't really give any detail on the configuration...

Oracle is prone to hot spots - they recommend I/O layout based on the SAME mantra - Stripe and Mirror Everything. Well that's good for performance, but not for cost. RAID-5 in ESS / DS8000 class disk is good enough 99% of the time.

Check filemon for hot spots - hot LV's, hot files, etc.

I have heard a few performance horror stories if you happen to run with the feature code 6239 FC adapter with firmare 1.90X2 - it was pulled from the support site and if you are running this adapter - downgrade to 1.81X1 available on the techupport download site.
#AIX-Forum

Originally posted by: SystemAdmin

I started this topic because I was asked to assist a customer with a perceived performance problem. I consequently discovered from colleagues that two other customers had superficially similar problems.

Therefore I wondered if this was wider than my three examples - it seems not.

In the case I have investigated in depth (with nmon), waitIO% can be up to 50%. Not exceptional but worthy of note.

The peculiar issue this site has is that they can run the same SQL query under carefully controlled conditions and the run time varies between ~15 secs and 4 minutes. This is against a Symmetrix system. The OS is AIX 5.2, They are using DirectIO. SGA caching is eliminated by restarting Oracle before each test. AFAIK, monitoring the Symmetrix cache hits shows relatively constant hit ratios. Nmon reports ~50% WaitiO in both cases but the quick test does ~1800 IOs/sec while the slow test does 400 IOs/sec. I've been told Symmetrix monitoring shows that it's lightly loaded and doesn't seem to be suffering any interference from other hosts.

So I'm looking for explanations of this erratic behaviour. Any contributions gratefully accepted.

I noted VirtualGreg's comment about 1.90X2 FW and have put that question to the customer.

> The system interconnect and I/O capability of POWER
> systems is extremely good. You don't really give any
> detail on the configuration...
>
> Oracle is prone to hot spots - they recommend I/O
> layout based on the SAME mantra - Stripe and Mirror
> Everything. Well that's good for performance, but
> not for cost. RAID-5 in ESS / DS8000 class disk is
> good enough 99% of the time.
>
> Check filemon for hot spots - hot LV's, hot files,
> etc.
>
> I have heard a few performance horror stories if you
> happen to run with the feature code 6239 FC adapter
> with firmare 1.90X2 - it was pulled from the support
> site and if you are running this adapter - downgrade
> to 1.81X1 available on the techupport download site.

#AIX-Forum