Originally posted by: SystemAdmin

Hi all!

Have anybody measured throughput of your link aggregation interfaces?

What i have:

I've tried iperf and spray for measuring.

A number of AIX-es with 802.3ad trunks: different adapters (physical and IVE), different count of members, but almost all they have just about 1Gbit/sec throughput.

And i have just one server, where throughput is adequate to count of memeber interfaces (2x1GE - 230MB/sec).
All attributes on AIX ssystems are equivalent.
It looks like on switches too.
Tried different hash_methods - no effect.

entstat says - all ok.
Switch can see that LACP group is built, but...

What can i do?

For example:

# lsattr -El ent4 adapter_names   ent3,ent1      EtherChannel Adapters                           True alt_addr        0x000000000000 Alternate EtherChannel Address                  True auto_recovery   yes            Enable automatic recovery after failover        True backup_adapter  ent2           Adapter used when whole channel fails           True hash_mode       src_dst_port   Determines how outgoing adapter is chosen       True interval        

short          Determines interval value 

for IEEE 802.3ad mode True mode            8023ad         EtherChannel mode of operation                  True netaddr         192.168.93.1   Address to ping                                 True noloss_failover yes            Enable lossless failover after ping failure     True num_retries     3              Times to retry ping before failing              True retry_time      1              Wait time (in seconds) between pings            True use_alt_addr    no             Enable Alternate EtherChannel Address           True use_jumbo_frame no             Enable Gigabit Ethernet Jumbo Frames            True

Originally posted by: j.gann

depending on the algorithm (hash_mode) for member adapter choice ethernet frames of a single tcp connection (your iperf benchmark e.g.) might all go out through the same member adapter (your src_dst_port algorithm causes this as you can verify with entstat).

cisco type etherchannel (unlike 802.3ad) allows your aix channel to send out frames with a round_robin algorithm (single tcp connection will use all member adapters)

anyway, the switch has its own set of algorithms for member adapter choice (incoming frames from aix perspective). last time I checked, cisco did not offer round_robin.

j.gann

Originally posted by: SystemAdmin

As far as i know, round_robin is a legacy.
Docs recommends to use src_dst_port instead and says that this method provides the best load balancing.

I've tried to use a couple of iperf servers and it's --parallel option - it works good just on only one host.
I don't know why.

So, have You ever tested your PaGP or LACP trunks under AIX?
What's Your results.

And one more question:
I have LACP trunk composed from bold 4 physical 1GE ports bold.
According to entstat output all Actors and Partners are Aggregatable, Active and IN_SYNC,
but...

General Statistics: ------------------- No mbuf Errors: 0 Adapter Reset Count: 0 *bold* Adapter Data Rate: 2000 *bold*

Why is it just 2000, but not 4000?

Originally posted by: SystemAdmin

Sorry, i'm newbie :)

4 phys ports
but

General Statistics:

---

No mbuf Errors: 0
Adapter Reset Count: 0
Adapter Data Rate: 2000

Originally posted by: LeoAN

Is there a requirement to use two ether adapters and do a load balancing? Is one ether adapter not enough for the server? I'm just wondering why one adapter doesn't serve the purpose.

Originally posted by: SystemAdmin

That's not a correct question.
It might by various reasons to have huge network trunks, for backups for example.
I'd like to know why Link aggregation doesn't provide adequate perfomance?

Originally posted by: j.gann

like i wrote, for single tcp connections and named hash_mode you are getting the expected results.
if (repeated) tests with parallel tcp connections show single adapter bandwidth, I'd suggest looking at all involved switchport and host port statistics and config.

Originally posted by: SystemAdmin

May be You talking about default hash method *src_port*?
Your suggestion concerning src_dst_port is wrong.
I've tried to test it from several hosts with different types of applications simultaniously and looked at network load in topas.

Originally posted by: j.gann

well, but your single tcp connection has 1 src and 1 dst port, right? the outgoing adapter will be chosen based on these 2 values. they do not change during the lifetime of the connection.
additional parallel tcp connections will have a statistical chance (due to different src port) to be delivered through a different adapter.

port statistics (entstat -d) should make it more obvious than topas.

nice weekend

Originally posted by: SystemAdmin

Why just one tcp connection?
Look:

# iperf -c somehost --parallel 2 ------------------------------------------------------------ Client connecting to somehost, TCP port 5001 TCP window size:   128 KByte (

default) ------------------------------------------------------------ [  3] local 1.1.1.1 port *49889* connected with 1.1.1.2 port 5001 [  4] local  1.1.1.1 port *49890* connected with 1.1.1.2 port 5001 [ ID] Interval       Transfer     Bandwidth [  4]  0.0-10.0 sec    504 MBytes    422 Mbits/sec [ ID] Interval       Transfer     Bandwidth [  3]  0.0-10.0 sec    488 MBytes    410 Mbits/sec [SUM]  0.0-10.0 sec    992 MBytes    832 Mbits/sec

Here we've opened 2 sockets, and when we running iperf on another hosts simultaneously we have various combinations of sc and dst ports.
Now look at the same AIX oslevel (6100-05-01-1016) on the similar HW (P702):

# iperf -c somehost --parallel 2 ------------------------------------------------------------ Client connecting to somehost, TCP port 5001 TCP window size:   128 KByte (

default) ------------------------------------------------------------ [  4] local 1.1.1.3 port *64944* connected with 1.1.1.2 port 5001 [  3] local 1.1.1.3 port *64945* connected with 1.1.1.2 port 5001 [ ID] Interval       Transfer Bandwidth [  4]  0.0-10.0 sec    768 MBytes    645 Mbits/sec [ ID] Interval       Transfer     Bandwidth [  3]  0.0-10.0 sec    821 MBytes    689 Mbits/sec [SUM]  0.0-10.0 sec  1.55 GBytes  1.33 Gbits/sec

When --parallel is 4 or more, this host shows about 1.9Gbit/sec
BTW, there is 4x1GE LACP on 1.1.1.2, but when we initiate the same test from this somehost to 1.1.1.3, throughput is about 850Mbit/sec...

Originally posted by: j.gann

repeating myself:
if (repeated) tests with parallel tcp connections show single adapter bandwidth, I'd suggest looking at all involved switchport and host port statistics and config.

Originally posted by: SystemAdmin

Ok

# lsattr -El ent4 adapter_names   ent3,ent1      EtherChannel Adapters                           True alt_addr        0xdeaddeaddead Alternate EtherChannel Address                  True auto_recovery   yes            Enable automatic recovery after failover        True backup_adapter  ent2           Adapter used when whole channel fails           True hash_mode       src_dst_port   Determines how outgoing adapter is chosen       True interval        

short          Determines interval value 

for IEEE 802.3ad mode True mode            8023ad         EtherChannel mode of operation                  True netaddr         1.0.0.1        Address to ping                                 True noloss_failover yes            Enable lossless failover after ping failure     True num_retries     3              Times to retry ping before failing True retry_time      1              Wait time (in seconds) between pings            True use_alt_addr    no             Enable Alternate EtherChannel Address           True use_jumbo_frame no             Enable Gigabit Ethernet Jumbo Frames            True

sh run 

int port-channel 2   Building configuration...     Current configuration : 114 bytes   !   

interface Port-channel2   switchport access vlan 111   link state group 1 downstream   spanning-tree portfast   end   sh run 

int g0/8   Building configuration...     Current configuration : 119 bytes   !   

interface GigabitEthernet0/8   switchport access vlan 111   channel-group 2 mode passive   spanning-tree portfast   end     sh run 

int g0/9   Building configuration...     Current configuration : 119 bytes   !   

interface GigabitEthernet0/9   switchport access vlan 111   channel-group 2 mode passive   spanning-tree portfast   end

entstat.out is attached

Originally posted by: j.gann

unless I happen to be on your payroll,
I'd suggest -you- look at these.

Originally posted by: SystemAdmin

Thanks :)

Me and my colleagues have already looked at these and still don't have any ideas :)

So, did you ever tested your own etherchannels?
Have you got adequate perfomance?

Originally posted by: SystemAdmin

So, you suppose, that LACP doesn't distribute packets.
Which stats from entstat you would like to see?
Give me, please, grep mask for example.

Originally posted by: Siddhartha.Sinha

We did aggregate for our NIM master Server. Our network guys did some setting in the network switch, I believe they also did aggregate. I ran couple of instances of this old method and monitored it running nmon and got more than 197MB/Sec which is definitely more than 1Gb ( 128MB). You need to talk to your Network Team and explain them that you need etherchannel aggregate not failover. I don't think any problem with the setting you did as we achieved 197MB/Sec with just basic settings.

ftp targethost
username/passwd...
bin
hash
put "|dd if=/dev/zero bs=32k count=100000" /dev/null