AIX Open Source

AIX Open Source

Share your experiences and connect with fellow developers to discover how to build and manage open source software for the AIX operating system

 View Only
Expand all | Collapse all

updatedb & sort filling up /tmp

  • 1.  updatedb & sort filling up /tmp

    Posted Tue March 28, 2023 05:31 AM

    Good morning,

    We have a problem with updatedb, when running sort, fills up /tmp/ with stm-files.

     I realised that updatedb on AIX, due to the search path, runs AIX' sort instead of GNU sort, so tried to run it using GNU sort instead. Helped a little bit but didn't go away.

    Why have I not come across this problem in 25 years of Linux use? The man page for sort on Linux says it puts files in /tmp/ as well, what's the difference?

    Does anyone here run updatedb on AIX? Do you have the same issue? What's the fix?

    Regards,

    Henrik Morsing



    ------------------------------
    Henrik Morsing
    ------------------------------


  • 2.  RE: updatedb & sort filling up /tmp

    Posted Tue March 28, 2023 10:08 AM

    /opt/freeware/bin/updatedb from findutils rpm checks for the sort binary from the aix toolbox...not sure whats your problem.

    excerpt from updatedb shell script:

    if true
    then
        sort="/opt/freeware/bin/sort -z"
        print_option="-print0"
        frcode_options="$frcode_options -0"
    else
        sort="/opt/freeware/bin/sort"
        print_option="-print"
    fi
    



    ------------------------------
    I regret starting this entire conversation
    ------------------------------



  • 3.  RE: updatedb & sort filling up /tmp

    Posted Tue March 28, 2023 10:14 AM

    Thank-you for that, I hadn't even realised updatedb was a shell script, I was looking at the C source code on GitHub! Not sure what I have been looking at now.

    The script on my system says:

        sort="/usr/bin/sort"

    throughout, not sure why it is different. I can try changing it.

    Regards,

    Henrik Morsing



    ------------------------------
    Henrik Morsing
    ------------------------------



  • 4.  RE: updatedb & sort filling up /tmp

    Posted Tue March 28, 2023 10:17 AM

    concerning your /tmp fillup troubles...updatedb runs mktemp somewhere, which uses whatever $TMPDIR is currently set to. maybe you should run

    a cron job to cleanup /tmp regulary.



    ------------------------------
    I regret starting this entire conversation
    ------------------------------



  • 5.  RE: updatedb & sort filling up /tmp

    Posted Tue March 28, 2023 10:31 AM

    It actually does this:

    # Directory to hold intermediate files.
    if test -d /var/tmp; then
      : ${TMPDIR=/var/tmp}
    elif test -d /usr/tmp; then
      : ${TMPDIR=/usr/tmp}
    else
      : ${TMPDIR=/tmp}
    fi
    export TMPDIR

    I have just tested both, and it makes no difference to filling up the filesystem. I know GNU sort is highly customisable, but it is a compromise between disk space, memory/processor and time.

    Regards,

    Henrik Morsing



    ------------------------------
    Henrik Morsing
    ------------------------------



  • 6.  RE: updatedb & sort filling up /tmp

    Posted Tue March 28, 2023 11:25 AM
    Note that this bit of shell won't change the value of TMPDIR if it is already set.

    Is TMPDIR already set, or is there a place you could set it to (temporarily) with enough space?

    -- 
    Stephen Ulmer

    Sent from a mobile device; please excuse auto-correct silliness.





  • 7.  RE: updatedb & sort filling up /tmp

    Posted Wed March 29, 2023 03:20 AM

    Let's just rewind a bit here. 

    My issue is:

    "I have used updatedb for 25 years on Linux and have never had this problem. What is the difference to AIX?"

    I checked the disk space graphs on my home server (Linux (sadly)) running updatedb every night, and they are completely flat. 

    This is the first place I have ever worked where locate on AIX is standard on our builds, which is cool, but there is now talk about scrapping it because of frequent alerting.

    Move it to a filesystem with more space? We don't just have three servers. We need a standard place for this, /tmp or /var/tmp would be correct, and adding space to lots of servers because of this issue will not go down well.

    Regards,

    Henrik Morsing



    ------------------------------
    Henrik Morsing
    ------------------------------



  • 8.  RE: updatedb & sort filling up /tmp

    Posted Wed March 29, 2023 03:47 AM
    Edited by C- -T Wed March 29, 2023 03:47 AM

    i must confess i never  paid much attention to this locate stuff...but the first obvious difference between the updatedb command between linux and aix is that on linux its not a shell script but a binary. just checked on my ubuntu notbook.

    so there might be some hidden "treasures" somewhere,

    ------------------------------
    I regret starting this entire conversation
    ------------------------------



  • 9.  RE: updatedb & sort filling up /tmp

    Posted Wed March 29, 2023 03:52 AM

    i guess this is what the linux guys are using...

    https://github.com/msekletar/mlocate/blob/master/src/updatedb.c

    seems that this is not generating any temporary files unlike the aix script...could not find something related in the code, just direct db creation.



    ------------------------------
    I regret starting this entire conversation
    ------------------------------



  • 10.  RE: updatedb & sort filling up /tmp

    Posted Wed March 29, 2023 03:55 AM

    from the readme, explains a lot:

    About
    =====
    mlocate is a locate/updatedb implementation.  The 'm' stands for "merging":
    updatedb reuses the existing database to avoid rereading most of the file
    system, which makes updatedb faster and does not trash the system caches as
    much.



    ------------------------------
    I regret starting this entire conversation
    ------------------------------



  • 11.  RE: updatedb & sort filling up /tmp

    Posted Wed March 29, 2023 03:54 AM

    Maybe, I did find references to some distributions using a different updatedb, but I doubt that was the case 20 years ago.



    ------------------------------
    Henrik Morsing
    ------------------------------



  • 12.  RE: updatedb & sort filling up /tmp

    Posted Thu March 30, 2023 09:23 AM

    let us forgot linux. What is your problem on AIX and what you want to fix. Linux command may not work on AIX. 

    if you do find / -name xxx (what ever updateDB) and man pages then there is not a proper command for AIX as it exist for Linux. 

    From pure AIX just tell us what is the problem. what you want to fix is it file system space. Then will be able to may be help or some one will have idea.

    what kind of database you are using on AIX since you do - updatedb and it is used for database.



    ------------------------------
    minesh patel
    ------------------------------



  • 13.  RE: updatedb & sort filling up /tmp

    Posted Fri March 31, 2023 02:36 AM

    Don't throw misleading statements like "this may not work or that may not work in AIX". 
    The issue discussed here is all about GNU findutils rpm from AIX Toolbox. 



    ------------------------------
    Ayappan P
    ------------------------------



  • 14.  RE: updatedb & sort filling up /tmp

    Posted Thu March 30, 2023 12:48 PM

    Hi Henrik,

    What is the AIX version and findutils version you have ?

    While you tried to use GNU sort it was creating tmp files with the name something like sort* ?



    ------------------------------
    SANGAMESH
    ------------------------------



  • 15.  RE: updatedb & sort filling up /tmp

    Posted Wed April 12, 2023 03:38 AM

    Hi Sangamesh,

     # rpm -qa | grep findutils
    findutils-4.6.0-1.ppc
    # oslevel -s
    7200-04-02-2028

    Regards,
    Henrik Morsing



    ------------------------------
    Henrik Morsing
    ------------------------------



  • 16.  RE: updatedb & sort filling up /tmp

    Posted Mon April 03, 2023 04:33 AM

    In 20+ years of running different *nix systems, I've never seen a Linux machine the size of some of the AIX ones that I've been running.  Working space for these things matters when scale comes into it.  It's unlikely that your home Linux machine anywhere near as large your work AIX boxes.



    ------------------------------
    Phill Rowbottom
    ------------------------------



  • 17.  RE: updatedb & sort filling up /tmp

    Posted Wed April 12, 2023 03:46 AM

    Hi Phill,

    Just had a play around on one of the AIX Oracle servers alerting this morning. It peaked at using about 600MB for sort files with a total of 3 million files on the system. I would assume the number of files is what matters here.

    My home router has 7 million files and even zooming in on the graph, you can 't see it move, so I'd guess is uses less than 1MB in the process of sorting and building the DB.

    My problem is also partly that running this tool on AIX, prunuefs doesn't work, and my tests indicates that prunepaths doesn't either. This means I can't exclude database filesystems that aren't needed anyway.

    This doesn't change the fact that none of our Linux systems have this problem, but we had three AIX systems alert this morning, nor that my home Linux router with 7 million files only uses a few KBs in this process.

    I will email a bug report in for prunefs and pruinepaths not working on AIX, but for the time being, I can't see any other solution that ditching locate on AIX.

    Regards,
    Henrik Morsing



    ------------------------------
    Henrik Morsing
    ------------------------------



  • 18.  RE: updatedb & sort filling up /tmp

    Posted Wed March 29, 2023 08:40 AM

    Do find / -name updatedb on AIX do you find result? Also do "man updatedb" you get result. You can search AIX commands webpage and see if updatedb is listed or not. It may be there for linux

    https://www.ibm.com/docs/en/tivoli-netcoolimpact/7.1?topic=functions-updatedb

     

    never heard of this for AIX but I am not expert learn some thing new everyday.