AIX Open Source

 View Only
Expand all | Collapse all

rsync performance problem

  • 1.  rsync performance problem

    Posted Thu December 07, 2023 02:54 PM
    Edited by Mario Fri December 08, 2023 08:41 AM

    AIX 7300-02-01-2346

    rsync 3.2.7-1 rpm

    I have an AIX box that has been slow to transfer files to since I updated to the last TL. It's only rsync that's the issue. I had been getting about 90MB/s uploading to it from a Linux host (Alma 8.9 up to date) but now it's down to 30MB/s. I also transfer from the AIX host to a Linux LPAR on the same managed system, and still only get about 30MB/s.

    I installed iperf3 and did some poking. I'm getting line speed of 100MB/s between all hosts. scp transfers are similarly quick. 

    Recalling that I had to previously downgrade rsync --protocol to 29 I tried that. Lo and behold, 90MB/s. Protocol levels 30 and 31 are slow.

    So I tried directly between the Linux workstation and the Linux LPAR. I get line speed at protocol level 31.

    It appears that something has changed to slow rsync --protocol=31 down on AIX. I haven't put my finger on it yet, but

    1. Can anyone confirm this observation? 
    2. Use --protocol=29 for now as a workaround.

    If you want to see what protocol level is being negotiated, use --debug=PROTO in your rsync command.

    The rsync rpm installed on AIX has a build date of Dec 22 2022.

    Many thanks for any insights or assistance you may be able to offer.



    ------------------------------
    Mario
    ------------------------------



  • 2.  RE: rsync performance problem

    Posted Tue December 12, 2023 06:42 AM
    Edited by Emerson Jose de Lima Tue December 12, 2023 06:43 AM
    Hello Mario, how are you?
    For high-speed networks that can achieve better performance, you could tune these two parameters hstcp=1 and sack=1, where 1 means that you are enabling them and also adjust tcp_sendspace=16777216 and tcp_recvspace=16777216 on the network cards source and destination, but first change the parameters with no -a , before and after no -po tcp_sendspace=16777216 and tcp_recvspace=16777216 , then you adjust it on the network card.
    ifconfig <interface> tcp_sendspace=16777216 and tcp_recvspace=16777216
     
    To adjust the hstcp and sack parameters do the following:
    no -po hstcp=1
    no -po sack=1
    This will make a difference and can reach up to 200MB/s in rsync.

    Ex : rsync : /usr/custom/bin/rsync -aHPx -e 'ssh -T -o Compression=no -x -p 22' --rsync-path=/usr/custom/bin/rsync /home/ root@x.xx.xx.xx:/home/



    ------------------------------
    Emerson Jose de Lima
    ------------------------------

    Original Message


  • 3.  RE: rsync performance problem

    Posted Wed December 13, 2023 04:12 PM

    I've had a good look at this. Thanks for the tuning tips @Emerson Jose de Lima. This does not appear to be a tuning issue.

    The bottom line is that newer versions of openssl (1.1.1 and up) for AIX have very slow md5 performance. I'm measuring a speed decline by a factor of 4 between 1.0.2 and 3.0.10 versions of openssl. It would appear that the new versions are build with xlc, and the old ones with "cc".

    This explains to my satisfaction the performance difference I'm seeing in transferring a single 1GB file of random data across the network from a rhel 8 x86 box to an AIX box. I tested on AIX 7.3 and 7.2. You can toggle the performance by specifying a different checksum algorithm. If you choose md4, sha1, or none, then you get full speed. md5 is the default digest for rsync protocols >= 30.

    I've also been able to sidestep the issue by rebuilding the rsync rpm from source and disabling openssl in the configure stage. In this case, rsync uses its own code to generate md5 and is line speed. That might be a quick fix.

    New versions of rsync are starting to support xxhash, and that's probably something that should be built for AIX. I couldn't get it build for AIX in a way that would let me link it in the rsync build. Oh well.

    For anyone looking for a workaround, use 

    rsync --checksum-choice=sha1

    which was very fast for me using the rsync available in the Linux toolbox.

    Would someone please test this to verify for me?

    rsync-4.2.7 and openssl >= 1.1.1 on AIX >= 7.2. Transfer a 1GB file from Linux to AIX.

    Thanks,

    M.



    ------------------------------
    Mario
    ------------------------------

    Original Message


  • 4.  RE: rsync performance problem

    Posted Wed December 13, 2023 04:44 PM

    I wonder if openssl is not using the hardware crypto engine? I'm on P8's for this test.

    https://www.ibm.com/support/pages/node/6620187



    ------------------------------
    Mario
    ------------------------------

    Original Message


  • 5.  RE: rsync performance problem

    Posted Thu December 14, 2023 06:43 AM
    Mario,

    Kudos on the detailed analysis!

    Have you run "openssl speed" on different systems to compare?

    Thanks.
    On Wed, Dec 13, 2023 at 09:12:16PM +0000, Mario via IBM TechXchange Community wrote:
    > I've had a good look at this. Thanks for the tuning tips @Emerson Jose de Lima <https: community.ibm.com/community/user/network/members/profile?userkey="2beddf31-8983-4cf8-b129-0189bd73c14d">. This does not appear to be a tuning issue.
    >
    >
    > The bottom line is that newer versions of openssl (1.1.1 and up) for AIX have very slow md5 performance. I'm measuring a speed decline by a factor of 4 between 1.0.2 and 3.0.10 versions of openssl. It would appear that the new versions are build with xlc, and the old ones with "cc".
    >
    >
    > This explains to my satisfaction the performance difference I'm seeing in transferring a single 1GB file of random data across the network from a rhel 8 x86 box to an AIX box. I tested on AIX 7.3 and 7.2. You can toggle the performance by specifying a different checksum algorithm. If you choose md4, sha1, or none, then you get full speed. md5 is the default digest for rsync protocols >= 30.
    >
    >
    > I've also been able to sidestep the issue by rebuilding the rsync rpm from source and disabling openssl in the configure stage. In this case, rsync uses its own code to generate md5 and is line speed. That might be a quick fix.
    >
    >
    > New versions of rsync are starting to support xxhash, and that's probably something that should be built for AIX. I couldn't get it build for AIX in a way that would let me link it in the rsync build. Oh well.
    >
    >
    > For anyone looking for a workaround, use
    >
    >
    > rsync --checksum-choice=sha1
    >
    >
    > which was very fast for me using the rsync available in the Linux toolbox.
    >
    >
    > Would someone please test this to verify for me?
    >
    >
    > rsync-4.2.7 and openssl >= 1.1.1 on AIX >= 7.2. Transfer a 1GB file from Linux to AIX.
    >
    >
    > Thanks,
    >
    >
    > M.
    >
    >
    > ------------------------------
    > Mario
    > ------------------------------
    > -------------------------------------------
    > Original Message:
    > Sent: Thu December 07, 2023 02:54 PM
    > From: Mario
    > Subject: rsync performance problem
    >
    >
    > AIX 7300-02-01-2346
    >
    > rsync 3.2.7-1 rpm
    >
    > I have an AIX box that has been slow to transfer files to since I updated to the last TL. It's only rsync that's the issue. I had been getting about 90MB/s uploading to it from a Linux host (Alma 8.9 up to date) but now it's down to 30MB/s. I also transfer from the AIX host to a Linux LPAR on the same managed system, and still only get about 30MB/s.
    >
    > I installed iperf3 and did some poking. I'm getting line speed of 100MB/s between all hosts. scp transfers are similarly quick.
    >
    > Recalling that I had to previously downgrade rsync --protocol to 29 I tried that. Lo and behold, 90MB/s. Protocol levels 30 and 31 are slow.
    >
    > So I tried directly between the Linux workstation and the Linux LPAR. I get line speed at protocol level 31.
    >
    > It appears that something has changed to slow rsync --protocol=31 down on AIX. I haven't put my finger on it yet, but
    >
    > Can anyone confirm this observation?
    > Use --protocol=29 for now as a workaround.
    > If you want to see what protocol level is being negotiated, use --debug=PROTO in your rsync command.
    >
    > The rsync rpm installed on AIX has a build date of Dec 22 2022.
    >
    > Many thanks for any insights or assistance you may be able to offer.
    >
    >
    >
    >
    >
    > ------------------------------
    > Mario
    > ------------------------------
    >
    >
    > Reply to Sender : https://community.ibm.com/community/user/eGroups/PostReply?GroupId=6211&MID=391372&SenderKey=d8845398-87a2-4151-bca3-0186d8d491ed
    >
    > Reply to Discussion : https://community.ibm.com/community/user/eGroups/PostReply?GroupId=6211&MID=391372
    >
    >
    >
    > You are subscribed to "AIX Open Source" as Russell.Adams@AdamsSystems.nl. To change your subscriptions, go to http://community.ibm.com/community/user/preferences?section=Subscriptions. To unsubscribe from this community discussion, go to http://community.ibm.com/HigherLogic/eGroups/Unsubscribe.aspx?UserKey=c23dfccc-9910-40ae-beeb-fdcbced5bf1f&sKey=KeyRemoved&GroupKey=af5415f8-d8d8-4ac4-b3cb-08a8da184054.


    ------------------------------------------------------------------
    Russell Adams Russell.Adams@AdamsSystems.nl
    Principal Consultant Adams Systems Consultancy
    https://adamssystems.nl/


    Original Message


  • 6.  RE: rsync performance problem

    Posted Thu December 14, 2023 09:33 AM
    Edited by Mario Thu December 14, 2023 09:42 AM

    Yep, that's how I know it's 4 times slower. You have to pick an AIX 7.1 box with openssl 1.0.2 on it. You can force an AIX 7.2 box down to 1.0.2, but it breaks many things.

    root@devtest2# oslevel -s
7200-05-07-2346

root@devtest2# openssl version
OpenSSL 3.0.10 1 Aug 2023 (Library: OpenSSL 3.0.10 1 Aug 2023)

root@devtest2# openssl speed md5
Doing md5 for 3s on 16 size blocks: 1635611 md5's in 1.75s
Doing md5 for 3s on 64 size blocks: 1184774 md5's in 1.76s
Doing md5 for 3s on 256 size blocks: 653051 md5's in 1.75s
Doing md5 for 3s on 1024 size blocks: 234110 md5's in 1.74s
Doing md5 for 3s on 8192 size blocks: 33491 md5's in 1.75s
Doing md5 for 3s on 16384 size blocks: 16921 md5's in 1.74s
version: 3.0.10
built on: Wed Aug 30 15:44:58 2023 UTC
options: bn(64,32)
compiler: xlc_r -qpic -q32 -qmaxmem=16384 -qro -qroconst -qthreaded  -DB_ENDIAN -DOPENSSL_PIC -D_THREAD_SAFE -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -DSSL_ALLOW_ADH -DAIXSSL_IBM_VERSION=3.0.10.1001
CPUINFO: N/A
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
md5              14954.16k    43082.69k    95532.03k   137775.08k   156776.16k   159329.69k


root@devtest2# openssl version
OpenSSL 1.0.2u  20 Dec 2019

root@devtest2# openssl speed md5
Doing md5 for 3s on 16 size blocks: 6641787 md5's in 1.77s
Doing md5 for 3s on 64 size blocks: 4679948 md5's in 1.76s
Doing md5 for 3s on 256 size blocks: 2456617 md5's in 1.76s
Doing md5 for 3s on 1024 size blocks: 846475 md5's in 1.76s
Doing md5 for 3s on 8192 size blocks: 118983 md5's in 1.75s
OpenSSL 1.0.2u  20 Dec 2019
built on: reproducible build, date unspecified
options:bn(64,32) rc4(ptr,char) des(idx,cisc,2,long) aes(partial) blowfish(idx)
compiler: cc -I. -I.. -I../include  -DOPENSSL_THREADS -qthreaded -D_THREAD_SAFE -DDSO_DLFCN -DHAVE_DLFCN_H -DSSL_ALLOW_ADH -DAIXSSL_IBM_VERSION=1.0.2.2104 -q32 -O -DB_ENDIAN -qmaxmem=16384 -qro -qroconst -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DVPAES_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md5              60038.75k   170179.93k   357326.11k   492494.55k   556976.42k



    ------------------------------
    Mario
    ------------------------------

    Original Message


  • 7.  RE: rsync performance problem

    Posted Tue January 02, 2024 01:23 PM
      |   view attached

    IBM has released openssl 3.0.10-1002 via MRS which fixes the underlying MD5 performance issue. However rsync 3.2.7-1 from the toolbox doesn't pick up this benefit unless it's rebuilt, which I did from the published SRPM on an AIX 7.2 box.

    @Ayappan P Could I request a new build of rsync? I would also like to see xxhash support appear to keep pace with rsync on various Linux distros. I've built xxhash on AIX 7.2 with minimal fuss and used it on a clean build of rsync. I grabbed xxhash from CentOS 8 source here:

    https://cbs.centos.org/koji/buildinfo?buildID=36604

    I've attached the spec file for xxhash which I got working on AIX 7.2. I sidestepped the doxygen requirement.



    ------------------------------
    Mario
    ------------------------------

    Attachment(s)

    spec
    xxhash.spec   3 KB 1 version
    Original Message


  • 8.  RE: rsync performance problem

    Posted Wed January 03, 2024 03:06 AM