AIX

Originally posted by: hillanes

Hi everybody.

I have a new AIX53TLl2 in a Power7. We copy around 500GB, (a DB archive) to another AIX53-LABO (power5), we reach data transfer around 15Mb/second. But when we do the same with another AIX Power 5 we reach 30Mb/second.

We use scp to copy files. All tree AIX are connected to the same switch, and have the IP in the same subnet, also the same vlan.

We have to find out why we cannot get the same data transfer

I attach a the output of the no command of the power7

Thanks, best regards.
hillanes

Originally posted by: dukessd

p7 > p5-LABO = 15Mbps
p7 > p5-other = 30Mbps

I'd say you should be looking at the no output on the p5-LABO for the problem because the p7 and the p5-other seem fine.

I'd take the no, netstat/entstat and lsattr -El outputs for associated adapters / interfaces from all three and then play spot-the-difference.

HTH.

Originally posted by: MarkTaylor

>>p5-LABO = 15Mbps
>>p5-other = 30Mbps

Ok, so .. back to basics ..

Paste this info for both p5 systems .. you systems could just be configured differently, or one of the systems could be overloaed, or, if using VIO, you could have a VIO that's maxed out or incorrectly configured .. the list goes on ..

oslevel -s
lparstat -i
lsdev -Ccadapter
lsdev -Ccdisk
lsps -a
vmstat 1 20
vmo -Fa | egrep "lru|max|min"

Are the network adapters virtual or physical ?
Are your disks virtual or physical ?

endstat -d en# | egrep -i "media"
ifconfig -a
lsattr -EHl en#
lsattr -EHl ent#

Rgds
Mark Taylor

Originally posted by: hillanes

Hi:

This is the enviroment:

p7 > p5-LABO = 15Mbps
p5-Other > p5-LABO = 30Mbps
I also attach the output of the commands you asked for.
p7 -- p5_Other -- p5_Labo (in that order)
I cant have a vmo command because I dont have root access.

There is no VIOS.
There are ethernet physical adapters.
We try with physical disk and also with disks of a DS4700 (LUNs)

One diference is that the ports of the Cisco switch in wich p7 is connected, is 1000Full fixed. We assume this could improve the performance but it doesnt.

Thanks for your time

Originally posted by: MarkTaylor

p7 -> p5-LABO = 15Mbps
p5-Other -> p5-LABO = 30Mbps

That's interesting, that rules out .. the p5 config, however, you will want to check which interface each set of traffic is coming in on, beacuse you may be coming in over a different interface from the p7 to the p5, thats where I would look next.

hint .. "route get" or "traceroute" from the p7 to the p5 ..

HTH
Mark Taylor

Originally posted by: MarkTaylor

p7 -> p5-LABO = 15Mbps
p5-Other -> p5-LABO = 30Mbps

That's interesting, that rules out .. the p5 config, however, you will want to check which interface each set of traffic is coming in on, beacuse you may be coming in over a different interface from the p7 to the p5, thats where I would look next.

hint .. "route get" or "traceroute" from the p7 to the p5 ..

HTH
Mark Taylor

Originally posted by: hillanes

Hi.

p7 (en2) -> p5-LABO (en0)= 15Mbps
p5-Other (en4) -> p5-LABO (en0)= 30Mbps
I already check the traceroute between the servers and it if fine.

However,I have found this on some forums:

"There have been some reported problems with FC 5700, 5701, 5706 and 5707 which are the newer Gigabit Ethernet adapters we support in the pSeries systems. The devices have two TCP offload features, large_send (TCP Resegmentation) and chksum_offload (Checksum Offload), that are intended
to be performance features. In some instances the devices work as designed, but there have been issues with some software combinations where the offload features have caused problems. It is not difficult to disable the two features, but you want to make sure you have them either on or off for all devices that used the feature. To disable the large_send and chksum_offload you will detach the IP interfaces, put the device into a defined state, change the device configuration then reinitialized the device and its interfaces. Sometimes it is easier to reboot the server as their may be application issues when IP interfaces are temporarily removed. The example shows how to change one device, ent1, and if you have other devices of the same type you will want to modify them as well."

In my case both parameters (large_send & chksum_offload) are in YES state and it is the only difference between the new p7 and p5-other. I am assuming this could be the reason. What do you thnik about it?

I also note that in networking tunning there are many (TCP/UDP) parameters, I really dont want to change all of them because a wrong tunning may affect my performance seriously.
Remember we use scp to copy. (I guess it is TCP)

Thanks

Originally posted by: MarkTaylor

ok, so, you have ruled out

p5-Other (en4)30Mbps
p5-LABO (en0)30Mbps

But you have not ruled out p7 (en2) ..
p7 (en2)15Mbps

What interface does this traffic go over

p7 --> p5-Other ? both ends ?

How many tests do you run ? are the circumstances the same when you run each test, are the results consistient ? are you ruling out disk access ? i.e. /dev/zero --> /dev/null ?

HTH
Mark Taylor

Originally posted by: hillanes

Hi.

This are the interfaces used by the trafic
p7 (en2) -> p5-LABO (en0)= 15Mbps
p5-Other (en4) -> p5-LABO (en0)= 30Mbps

I made many test, also in different types of disks (internal and LUNS), and the results are the same.
I didn't change the (large_send & chksum_offload) parameters, I only assume that this could be the reason. But not sure because I find it in forums and not in a IBM site.

Do you think I could change&probe.

It is a production machine, I don't have many chance to do laboratory.

Originally posted by: hillanes

Hi.

p7 (en2) -> p5-LABO (en0)= 15Mbps
p5-Other (en4) -> p5-LABO (en0)= 30Mbps

Someone can help with this issue?, Updates or something?
Just for asking, which is the transfer rate I can achieve?

Originally posted by: MurstiMurikka

How much CPU cycles the p7 lpar has compared to sending p5? There is only that much encrypting cpu can do and it might be that the bottleneck is not Your network but cpu. I recently experimented with jumbo_frames on both physical (between frames) and virtual vlans (within frame between lpars).

Suprisingly (hopefully formatting is preserved :) )

SCP FTP
VIRTUAL ~39 MB/sec ~220 MB/sec
PHYSICAL ~39 MB/sec ~90 MB/sec

it would seem that there were only ~39 MB:s worth of data to transfer per second with scp. But only without encryption the true network throughput is visible.

Originally posted by: hillanes

Hi:

p7 (en2) -> p5-LABO (en0)= 15Mbps
p5-Other (en4) -> p5-LABO (en0)= 30Mbps

Finally I make the change (large_send & chksum_offload = NO ), It's very litte better, almost the same.

I didn't think about CPU bottleneck, assuming the P7 is much powerful than a p5. Here is the info about CPU, how can I test if it is CPU? or another kind of bottleneck?
Old p5

System Model: IBM,9117-570
Processor Type: PowerPC_POWER5
Processor Implementation Mode: POWER 5
Processor Version: PV_5
Number Of Processors: 8
Processor Clock Speed: 1502 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
LPAR Info: 2 BME_SRV_1
Memory Size: 31232 MB

Good Memory Size: 31232 MB
Node Name : BME01_SRV_1
Partition Name : BME_SRV_1
Partition Number : 2
Type : Dedicated-SMT
Mode : Capped
Entitled Capacity : 8.00
Partition Group-ID : 32770
Shared Pool ID : -
Online Virtual CPUs : 8
Maximum Virtual CPUs : 8
Minimum Virtual CPUs : 1
Online Memory : 31231 MB
Maximum Memory : 32768 MB
Minimum Memory : 8192 MB
Variable Capacity Weight : -
Minimum Capacity : 1.00
Maximum Capacity : 8.00
Capacity Increment : 1.00
Maximum Physical CPUs in system : 16
Active Physical CPUs in system : 8
Active CPUs in Pool : -
Shared Physical CPUs in system : -
Maximum Capacity of Pool : -
Entitled Capacity of Pool : -
Unallocated Capacity : -
Physical CPU Percentage : 100.00%
Unallocated Weight : -
Desired Virtual CPUs : 8
Desired Memory : 31232 MB
Desired Variable Capacity Weight : -
Desired Capacity : 8.00

And this is the new p7

System Model: IBM,8205-E6B
Processor Type: PowerPC_POWER7
Processor Implementation Mode: POWER 6
Processor Version: PV_6_Compat
Number Of Processors: 6
Processor Clock Speed: 3720 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
LPAR Info: 1 FISA_A
Memory Size: 57344 MB
Good Memory Size: 57344 MB

Node Name : FISA_A
Partition Name : FISA_A
Partition Number : 1
Type : Shared-SMT
Mode : Capped
Entitled Capacity : 3.00
Partition Group-ID : 32769
Shared Pool ID : 0
Online Virtual CPUs : 6
Maximum Virtual CPUs : 6
Minimum Virtual CPUs : 2
Online Memory : 57344 MB
Maximum Memory : 65536 MB
Minimum Memory : 20480 MB
Variable Capacity Weight : 0
Minimum Capacity : 2.00
Maximum Capacity : 6.00
Capacity Increment : 0.01
Maximum Physical CPUs in system : 6
Active Physical CPUs in system : 6
Active CPUs in Pool : 6
Shared Physical CPUs in system : 6
Maximum Capacity of Pool : 600
Entitled Capacity of Pool : 350
Unallocated Capacity : 0.00
Physical CPU Percentage : 50.00%
Unallocated Weight : 0
Desired Virtual CPUs : 6
Desired Memory : 57344 MB
Desired Variable Capacity Weight : 0
Desired Capacity : 3.00
According to MurstiMurikka post, I can achieve around 39MB/s whit SCP

Thanks for your cooperation.

Originally posted by: MurstiMurikka

Have You tried to ftp files from host to host? That way You would know whether the problem is network or the cpu. If the problem is network You can't ftp files much faster than You can scp them. If the bottleneck is cpu then You can ftp files remarkably faster than You can scp them.