AIX Open Source

Expand all | Collapse all

rsync out of memory

Archive User

Archive UserMon September 02, 2019 09:25 AM

Archive User

Archive UserTue September 03, 2019 09:34 AM

  • 1.  rsync out of memory

    Posted Fri March 09, 2018 12:02 PM

    Originally posted by: blt


    rsync crashes with the following error message
    ERROR: out of memory in flist_expand [sender]
    rsync error: error allocating core memory buffers (code 22) at util2.c (106) [receiver = 3.1.2]
    rsync: [sender] write error: Broken pipe (32)

    the problem seems to arise when a directory is examined with lots of files.
    doing subfolders manually rsyncen works well.
    version used = 3.1.2
    aix = 7100-05-01-1731
    when we use a older version of rsync everything works fine.
    anyone know if there is in issue with version 3.1.2 ?
    is there something to change in the AIX enviroment ?

    i saw there is already version 3.1.3 but not available as a binary.



  • 2.  Re: rsync out of memory

    Posted Wed September 04, 2019 08:38 AM

    Originally posted by: AyappanP


    I am trying to recreate this behavior but couldn't able to do it.

    Just tried with 1400 files but no luck.

    Any way to reproduce this problem is highly appreciated. 

     



  • 3.  Re: rsync out of memory

    Posted Tue October 01, 2019 11:23 AM

    Originally posted by: AyappanP


    Built rsync 64bit binary with proper library path and attaching it here.

    You can try with this binary and let me know. 



  • 4.  Re: rsync out of memory

    Posted Fri March 09, 2018 01:00 PM

    Originally posted by: blt


    sorry,
    forgot to tell: the old version is running on another aix-server (with the same data-set)

    aix-level is there 7100-03-04-1441

     



  • 5.  Re: rsync out of memory

    Posted Fri March 09, 2018 12:48 PM

    Originally posted by: blt


    rpm -qa:
    rsync-3.1.2-2.ppc

    rsync --version
    rsync  version 3.1.2  protocol version 31
    Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
    Web site: http://rsync.samba.org/
    Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, no IPv6, batchfiles, inplace,
    append, ACLs, no xattrs, iconv, no symtimes, no prealloc

    ls -l /opt/freeware/bin/rsync
    lrwxrwxrwx    1 root     system            8 Mar 09 17:07 /opt/freeware/bin/rsync -> rsync_64
    file /opt/freeware/bin/rsync
    /opt/freeware/bin/rsync: 64-bit XCOFF executable or object module
     

    Version that works:

    rpm -qa | grep rsync
    rsync-3.0.9-1

    rsync --version
    rsync  version 3.0.9  protocol version 30
    Copyright (C) 1996-2011 by Andrew Tridgell, Wayne Davison, and others.
    Web site: http://rsync.samba.org/
    Capabilities:
        64-bit files, 32-bit inums, 32-bit timestamps, 64-bit long ints,
        socketpairs, hardlinks, symlinks, no IPv6, batchfiles, inplace,
        append, ACLs, no xattrs, iconv, no symtimes

     



  • 6.  Re: rsync out of memory

    Posted Thu September 05, 2019 02:17 AM

    Originally posted by: ruxi005


    Maybe we could get a Debugging 64-bit binary that gives you a hint how to reproduce it yourself.

     

    We have lots of transmissions using the 64-bit binary and they work fine, but there is also a procedure that reliably fails every single time, regardless which target host is used.

     

    Regards,

     

    Matthias Hofer



  • 7.  Re: rsync out of memory

    Posted Fri March 09, 2018 12:29 PM

    Originally posted by: sangameshm


    Hi Marcel,

    What is the exact version of rsync is installed in your machine ? Please provide "rpm -qa | grep rsync" output.

    rsync-3.1.2-2 by default ships 64-bit version. This version shouldn't be having any memory limitation issues.

     

    # rpm -qa | grep rsync
    rsync-3.1.2-2.ppc
    # ls -l /opt/freeware/bin/rsync
    lrwxrwxrwx 1 root system 8 Mar  9 11:10 /opt/freeware/bin/rsync -> rsync_64
    # file /opt/freeware/bin/rsync_64
    /opt/freeware/bin/rsync_64: 64-bit XCOFF executable or object module not stripped

     

     

    Also let us know the exact version of old rsync which is working ? 

     

    Thanks,

    Sangamesh



  • 8.  Re: rsync out of memory

    Posted Wed October 02, 2019 11:14 AM

    Originally posted by: Edward Davignon


    The new rsync_64 uses libiconv.so.2 from libiconv.a
    The 3.1.3-2 rsync_64 uses shr4_64.o from libiconv.a

    The new rsync_64 works when LIBPATH is unset, which is the recommended setting for LIBPATH on a modern system, unless required by a particular application.

    The new rsync_64 does not work when LIBPATH is set to "/usr/lib:/lib" or "/usr/lib:/lib:/usr/lpp/xlC/lib".  I will need to clean up a few old login scripts and wrapper scripts; which should be done anyways.

    The 3.1.3-2 rsync_32 uses shr4.o from libiconv.a
    What will the new rsync_32 use from libiconv.a?

    Is the change from system libiconv to GNU libiconv the change that is being tested to correct these memory errors in rsync_64?

    What changes to the compiler or build options are being considered for these memory errors in rsync_64?

    I replaced /opt/freeware/bin/rsync_64 with the attached version on a few of my systems, including some that use rsync from cron.

     



  • 9.  Re: rsync out of memory

    Posted Thu October 03, 2019 05:26 AM

    Originally posted by: ruxi005


    I tried out the new temporary compile you supplied, but I have to say that the memory problems are still there

     

    [...]
    sending incremental file list
    sending incremental file list
    ERROR: out of memory in flist_expand [receiver]
    rsync error: error allocating core memory buffers (code 22) at util2.c(105) [receiver=3.1.3]
    rsync: [sender] write error: Broken pipe (32)

     

    I hope you can solve that.

     

    Thank you, and Regards



  • 10.  Re: rsync out of memory

    Posted Tue October 01, 2019 09:07 AM

    Originally posted by: Edward Davignon


    Thank you Ayappan.  3.1.3-3 does not work unless LIBPATH contains /opt/freeware/lib

    Unfortunately I cannot test it on the server that has the problem until LIBPATH does not need to be set.

    $ rsync_32 --version
    Could not load program rsync_32:
            Dependent module /usr/lib/libiconv.a(libiconv.so.2) could not be loaded.
            Member libiconv.so.2 is not found in archive
    $ rsync_64 --version
    Could not load program rsync_64:
            Dependent module /usr/lib/libiconv.a(libiconv.so.2) could not be loaded.
            Member libiconv.so.2 is not found in archive

    Spam prevention did not allow me to include more details.



  • 11.  Re: rsync out of memory

    Posted Fri March 09, 2018 04:44 PM

    Originally posted by: blt


    for your information:

    i have installed 3.0.9 on the machine with aix 7100-05-01-1731 and that seems to work fine also
     



  • 12.  Re: rsync out of memory

    Posted Mon September 02, 2019 09:25 AM

    Originally posted by: ruxi005


    Hi,

     

    We also have the same issue with rsync: 3.1.0-6 does work fine, but newer versions (now 3.1.3-2) does throw that error.

     

    This is well reproducible in our enviroment.

    We would be very happy to get a solution for that. Can you give any hint on what can be done?

     

    Thank you, and Regards,

     

    Matthias Hofer



  • 13.  Re: rsync out of memory

    Posted Tue October 01, 2019 02:46 AM

    Originally posted by: AyappanP


    A new release of rsync  " 3.1.3-3 " is made available in Toolbox . Yum repo is not updated yet. 

    There were some wrong build flags for 64bit build that are fixed with this release. 

    Can you try with this new rpm and see whether the issue is still there ?

    http://public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/RPMS/ppc/rsync/



  • 14.  Re: rsync out of memory

    Posted Tue October 01, 2019 09:50 AM

    Originally posted by: AyappanP


    Okay. You can run like this , so that the LIBPATH is applicable only for rsync .

    LIBPATH=/opt/freeware/lib rsync 



  • 15.  Re: rsync out of memory

    Posted Thu October 03, 2019 04:57 AM

    Originally posted by: AyappanP


    The new rsync (both 64 bit and 32 bit ) uses libiconv library (libiconv.a (libiconv.so.2))coming from libiconv rpm rather than AIX Base libiconv. 

    The Library loading path not properly set during compilation. We will fix it and release a new rpm. 
    The change in libiconv is nothing to do with the memory errors. It's just that going forward opensource packages are linked against libiconv rpm rather than AIX base libiconv. 

    The memory errors could be because of these cflags " -D_FILE_OFFSET_BITS=64  -D_LARGEFILE_SOURCE " used in 64 bit build. 

    These are not required for 64bit build and can adversely affect it. 



  • 16.  Re: rsync out of memory

    Posted Thu October 03, 2019 03:25 PM

    Originally posted by: Edward Davignon


    I was going to see if the binary from 10/1 fixed the problem, but discovered that GNU libiconv is not installed on the server that demonstrates the rsync_64 memory problem.

    I updated my wrapper with a new binary name and it failed with the libiconv error.

    So I include LIBPATH in the wrapper.

    export LIBPATH="/opt/freeware/lib:/usr/lib:/lib"

    But it still failed with the same libiconv error.

    So I checked "rpm -q libiconv" and it was not installed.

    If the new rsync RPM depended on GNU libiconv, I would install GNU libiconv.  But that is another topic.  Is there an open forum that discusses some of the decisions being made for building the AIX Open Source Software?  It is not a simple task to maintain two environments on one system and have them work nicely together (as any product manager or steering committee knows).  I would be interested in hearing if there are any pros and cons I had not heard before.  Knowing the current direction of the project would be nice too.



  • 17.  Re: rsync out of memory

    Posted Thu September 05, 2019 12:52 PM

    Originally posted by: Edward Davignon


    The following command succeeds for me when running the rsync command from the receiving server,

    /opt/freeware/bin/rsync_32 --rsync-path=/opt/freeware/bin/rsync_64 …

    The following command succeeds for me when running the rsync command from the sending server,

    /opt/freeware/bin/rsync_64 --rsync-path=/opt/freeware/bin/rsync_32 …

    Note: by default, the rsync command simply spawns "rsync" on the remote server, which is rsync_64 (in the AIX toolbox rsync-3.1.3-2 and a default path in AIX).

    My problem seems to be in the 64-bit binary on the receiving server.

     



  • 18.  Re: rsync out of memory

    Posted Thu October 24, 2019 02:28 AM

    Originally posted by: AyappanP


    rsync libpath issue is fixed and release 3.1.3-4 is uploaded to Toolbox. 

    http://public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/RPMS/ppc/rsync/rsync-3.1.3-4.aix6.1.ppc.rpm



  • 19.  Re: rsync out of memory

    Posted Thu October 03, 2019 02:46 PM

    Originally posted by: Edward Davignon


    Using this wrapper script on the remote server.

    $ ls -ld /opt/freeware/bin/rsync_wrapper
    -rwxr-xr-x    1 root     system          288 Oct  3 12:08 /opt/freeware/bin/rsync_wrapper
    $ cat /opt/freeware/bin/rsync_wrapper
    #!/bin/ksh93 -p -
    cmd="/opt/freeware/bin/rsync_64"
    log="/tmp/rsync_wrapper.$(date -u +%Y%m%dT%H%M%S%Z).$$"
    print "logfile: $log" > "$log"
    /usr/bin/ulimit -Sa   >> "$log" 2> /dev/null
    /usr/bin/env          >> "$log" 2> /dev/null
    print "exec $cmd $@"  >> "$log" 2> /dev/null
    exec $cmd "$@"


    And running something like this on the sending server.

    cd /tmp
    touch stuff
    rsync --rsync-path=/opt/freeware/bin/rsync_wrapper stuff $remote:/tmp/stuff


    And looking on the remote server like this.

    for f in /tmp/rsync_wrapper.*; do echo "$f:"; cat "$f"; echo; echo =====; done


    I can see that the ulimits are not favorable on the remote server.

    [...]
    time(seconds)        unlimited
    file(blocks)         unlimited
    data(kbytes)         131072
    stack(kbytes)        32768
    memory(kbytes)       32768
    coredump(blocks)     unlimited
    nofiles(descriptors) 2000
    threads(per process) unlimited
    processes(per user)  unlimited

    [...]


    Similarly, they are all unlimited on the sending server.



  • 20.  Re: rsync out of memory

    Posted Thu September 05, 2019 12:31 PM

    Originally posted by: Edward Davignon


    The man page for rsync has many options to increase how verbose it is.  There are many options for info and debug messages.  For example, when I use -vvv or -vv, it seems to fail on a file with a long user name in the owner (and the UIDs are different between the sending and receiving systems).  Note: The --debug=FLIST4 does not behave the same as --debug=FLIST,FLIST,FLIST,FLIST nor as --debug=FLIST --debug=FLIST --debug=FLIST --debug=FLIST

    I am currently looking at rsync options:

    -n -a --info=ALL4 --debug=ALL4

    -n -av --debug=FLIST4

    -n -a --debug=FLIST4

    -n -a --numeric-ids --debug=FLIST4

    When running the rsync command from the sending server, adding the "--numeric-ids" option seems to change the end of the output from:

    recv_file_list done
    [receiver] receiving flist for dir 498
    ERROR: out of memory in flist_expand [receiver]
    rsync error: error allocating core memory buffers (code 22) at util2.c(105) [receiver=3.1.3]
    

    to:

    recv_file_list done
    [receiver] receiving flist for dir 502
    ERROR: out of memory in flist_expand [receiver]
    rsync error: error allocating core memory buffers (code 22) at util2.c(105) [receiver=3.1.3]
    

    When running the rsync command from the receiving server, adding the "--numeric-ids" option seems to change the end of the output from:

    recv_file_list done
    [generator] receiving flist for dir 378
    rsync: [generator] write error: Broken pipe (32)
    

    to:

    rsync: [receiver] write error: Broken pipe (32)
    

    and it does not display the ksh93 shell prompt until I press Enter.

     

    I hesitate to put much more detail, such as hostnames, usernames, uids, gids, and other login credentials, in a public forum.



  • 21.  Re: rsync out of memory

    Posted Tue September 03, 2019 09:34 AM

    Originally posted by: ruxi005


    Hi,

     

    there is one more thing: We could try out different combinations, and it is clear now that all 64-bit versions of rsync don't work with our copying process, however, all (also the newest) 32-bit versions work fine.

     

    Please give us a hint if you need more information to fix the problem.

     

    Regards,

     

    Matthias Hofer



  • 22.  Re: rsync out of memory

    Posted Tue September 03, 2019 10:12 AM

    Originally posted by: Edward Davignon


    As a work around, I added "--protocol=29" to one of our servers that was consistently failing with "ERROR: out of memory in flist_expand [receiver]" "rsync error: error allocating core memory buffers (code 22) at util2.c(105) [receiver=3.1.3]" in rsync-3.1.3-2.ppc

    I read the man page and started experimenting with the protocol version until I lowered it enough to get it to work consistently.  I am not sure what functionality is lost at this protocol level, but seems to transfer the regular UNIX file permissions enough for my application.

    This same rsync job (without the work around) works fine from our other servers.  I also looked at the output of "/usr/bin/ulimit -Sa" and "/usr/bin/env" when run from the same script as the rsync command that was failing.  I did not check what "ssh $remotehost /usr/bin/ulimit -Sa" nor "ssh $remotehost /usr/bin/env" (or similar if not an AIX remote host) would have shown if run from the same script.

    Again, this is a temporary work around.  I would like a permanent fix, but I did not investigate it fully.