Originally posted by: SystemAdmin

Hi guys,

have found the following issue recently and confused a little bit.

AIX5.3, DS4700, RAID5.
disk read speed difference between raw device and usual one can reach 10 times.

1. lscfg -vl hdisk19

hdisk19 U...L1000000000000 MPIO Other FC SCSI Disk Drive

1. time dd if=/dev/rhdisk19 of=/dev/null bs=1m count=1000

real 0m6.147s
user 0m0.010s
sys 0m0.347s

1. time dd if=/dev/hdisk19 of=/dev/null bs=1m count=1000

real 0m46.370s
user 0m0.160s
sys 0m2.881s

almost 8 times slower. LPAR is under light load, but no heavy I/O. CPU is sufficient. I suppose the only difference between hdisk and rhdisk is LVM layer, so no fcs/scsi tuning may help here. The same tests on DS8k give the same results so it's not a storage issue either.

Looks like I miss something obvious, could you please give me a clue?

Regards,
Alexander
#AIX-Forum

Originally posted by: SystemAdmin

Could you repeat the same test on your hardware maybe?

Thanks,
Alexander
#AIX-Forum

Originally posted by: SystemAdmin

welcome to unix

you are not using lvm in your tests. lvm devices are called lvs (easy to to guess).
read up the chapter on block and character devices.
#AIX-Forum

Originally posted by: SystemAdmin

you are not using lvm in your tests. lvm devices are called lvs (easy to to guess).

My 5.3/6.1 boxes know nothing about lvs devices. Are you sure you don't confuse it with Linux LVM? And I do use LVM for LVM's not only about volume groups, PP, LP and stuff. It's complex infrastructure that involves disks as well.

read up the chapter on block and character devices.

The only major difference as I said earlier is lack of LVM-related issues (buffers, etc) in case of rhdisk*. The devices itself are only interface to real LUN behind them. Character device (rhdisk*) shows the real physical limit for storage LUN behind. And I would like block device to be only a little slower then character one while currently it's 5-10 times slower. If there's a way to do it (cut buffers, etc) - I would like to know it. Of course that can end up in bad performance doing massive parallel i/o, but there're tasks when fair parallel queuing is not needed (backup).
#AIX-Forum

Originally posted by: jvk

It's documented in AIX documentation.

rhdisk Special File
Purpose

Provides raw I/O access to the physical volumes (fixed-disk) device driver.
Description

The rhdisk special file provides raw I/O access and control functions to physical-disk device drivers for physical disks. Raw I/O access is provided through the /dev/rhdisk0, /dev/rhdisk1, ..., character special files.

Direct access to physical disks through block special files should be avoided. Such access can impair performance and also cause data consistency problems between data in the block I/O buffer cache and data in system pages. The /dev/hdisk block special files are reserved for system use in managing file systems, paging devices and logical volumes.
Important is last paragraph above. The /dev/hdisk block is not for application use directly. It is only for the filesystem, paging devices, and logical volume interfaces to use who can deal with synchronous design of block I/O.
The operation with rhdisk# is synchronous..
#AIX-Forum

Originally posted by: jvk

Typo here:
It is only for the filesystem, paging devices, and logical volume interfaces to use who can deal with synchronous asynchronous design of block I/O.
#AIX-Forum

Originally posted by: SystemAdmin

Direct access to physical disks through block special files should be avoided. Such access can impair performance

That makes sense, thanks a lot! And that also proves that block level disk devices are tightly bound with LVM, cache, etc. Though words about "data consistency problems" seem to have meaning only with write access. I cannot imagine the case where reads from hdiskX will crash fs.

OK, let us repeat tests via high level routines. Again, let us test reading through raw device (as it shows the real throughput we can reach on LUN) against reading large file from FS.

AIX5.3, EMC V-MAX, reading 1G of data, storage read cache is flushed between tries.

1. time dd if=/tmp/1/test of=/dev/null bs=1m

real 0m5.902s
user 0m0.009s
sys 0m1.560s

1. time dd if=/dev/rhdiskpower0 of=/dev/null count=1000 bs=1m

real 0m6.556s
user 0m0.009s
sys 0m0.477s
AIX5.3, DS4700, reading 1G of data, storage read cache is flushed between tries.

1. time dd if=/tmp/1234/test of=/dev/null bs=1m

real 0m7.934s
user 0m0.010s
sys 0m2.040s

1. time dd if=/dev/rhdisk82 of=/dev/null bs=1m count=1000

real 0m10.217s
user 0m0.007s
sys 0m0.403s

Good, character device is a little bit slower as it should be. Shame on me for I didn't know IBM forbids to use hdiskX for any manipulating of data (strange though they exist at all).

And thanks again! Case is done.
#AIX-Forum