AIX

 View Only
Expand all | Collapse all

Can't remove files

  • 1.  Can't remove files

    Posted Thu July 07, 2022 10:22 AM
    Hi All,

    I am trying to remove files from a directory, or even remove the entire directory, but nothing works.
    Server is AIX 7.2

    Here is the situation:
    Directory "backup" contains the following files:

    # cd backup
    # ls
    bwaix2_backup_202204281657.log       bwaix2_backup_202205031657.log
    bwaix2_backup_202204281657.log.gz  bwaix2_backup_202205031657.log.gz
    bwaix2_backup_202204291657.err.gz   bwaix2_backup_202205041700.err.gz
    bwaix2_backup_202204291657.log       bwaix2_backup_202205041700.log
    bwaix2_backup_202204291657.log.gz  bwaix2_backup_202205041700.log.gz
    bwaix2_backup_202204301657.err        bwaix2_backup_202205051700.err.gz
    bwaix2_backup_202204301657.log        bwaix2_backup_202205051700.log
    bwaix2_backup_202204301657.log.gz   bwaix2_backup_202205051700.log.gz
    bwaix2_backup_202205011657.err        bwaix2_backup_202205061700.err
    bwaix2_backup_202205011657.log       bwaix2_backup_202205061700.log
    bwaix2_backup_202205021657.err       bwaix2_backup_202205061700.log.gz
    bwaix2_backup_202205021657.log       bwaix2_backup_202205071700.err
    bwaix2_backup_202205021657.log.gz  bwaix2_backup_202205071700.log
    bwaix2_backup_202205031657.err
    #
    trying to remove any file or all of them:

    # rm *
    rm: bwaix2_backup_202204281657.log: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202204281657.log.gz: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202204291657.err.gz: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202204291657.log: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202204291657.log.gz: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202204301657.err: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202204301657.log: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202204301657.log.gz: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205011657.err: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205011657.log: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205021657.err: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205021657.log: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205021657.log.gz: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205031657.err: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205031657.log: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205031657.log.gz: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205041700.err.gz: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205041700.log: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205041700.log.gz: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205051700.err.gz: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205051700.log: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205051700.log.gz: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205061700.err: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205061700.log: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205061700.log.gz: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205071700.err: A file or directory in the path name does not exist.
    rm: bwaix2_backup_202205071700.log: A file or directory in the path name does not exist.
    #

    Even with force:

    # rm -f *
    # ls
    bwaix2_backup_202204281657.log bwaix2_backup_202205031657.log
    bwaix2_backup_202204281657.log.gz bwaix2_backup_202205031657.log.gz
    bwaix2_backup_202204291657.err.gz bwaix2_backup_202205041700.err.gz
    bwaix2_backup_202204291657.log bwaix2_backup_202205041700.log
    bwaix2_backup_202204291657.log.gz bwaix2_backup_202205041700.log.gz
    bwaix2_backup_202204301657.err bwaix2_backup_202205051700.err.gz
    bwaix2_backup_202204301657.log bwaix2_backup_202205051700.log
    bwaix2_backup_202204301657.log.gz bwaix2_backup_202205051700.log.gz
    bwaix2_backup_202205011657.err bwaix2_backup_202205061700.err
    bwaix2_backup_202205011657.log bwaix2_backup_202205061700.log
    bwaix2_backup_202205021657.err bwaix2_backup_202205061700.log.gz
    bwaix2_backup_202205021657.log bwaix2_backup_202205071700.err
    bwaix2_backup_202205021657.log.gz bwaix2_backup_202205071700.log
    bwaix2_backup_202205031657.err
    #

    Trying to remove by inode:
    # ls -i
    16420 bwaix2_backup_202204281657.log
    16418 bwaix2_backup_202204281657.log.gz
    16427 bwaix2_backup_202204291657.err.gz
    16426 bwaix2_backup_202204291657.log
    16425 bwaix2_backup_202204291657.log.gz
    16424 bwaix2_backup_202204301657.err
    16423 bwaix2_backup_202204301657.log
    16428 bwaix2_backup_202204301657.log.gz
    16430 bwaix2_backup_202205011657.err
    16429 bwaix2_backup_202205011657.log
    16432 bwaix2_backup_202205021657.err
    16431 bwaix2_backup_202205021657.log
    16433 bwaix2_backup_202205021657.log.gz
    16435 bwaix2_backup_202205031657.err
    16434 bwaix2_backup_202205031657.log
    16436 bwaix2_backup_202205031657.log.gz
    16441 bwaix2_backup_202205041700.err.gz
    16440 bwaix2_backup_202205041700.log
    16439 bwaix2_backup_202205041700.log.gz
    16444 bwaix2_backup_202205051700.err.gz
    16443 bwaix2_backup_202205051700.log
    16442 bwaix2_backup_202205051700.log.gz
    16438 bwaix2_backup_202205061700.err
    16437 bwaix2_backup_202205061700.log
    16445 bwaix2_backup_202205061700.log.gz
    16447 bwaix2_backup_202205071700.err
    16446 bwaix2_backup_202205071700.log
    #
    # find . -i 16446 -exec rm {} \;
    find: 0652-019 The status on ./bwaix2_backup_202204281657.log is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202204281657.log.gz is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202204291657.err.gz is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202204291657.log is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202204291657.log.gz is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202204301657.err is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202204301657.log is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202204301657.log.gz is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205011657.err is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205011657.log is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205021657.err is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205021657.log is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205021657.log.gz is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205031657.err is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205031657.log is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205031657.log.gz is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205041700.err.gz is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205041700.log is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205041700.log.gz is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205051700.err.gz is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205051700.log is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205051700.log.gz is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205061700.err is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205061700.log is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205061700.log.gz is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205071700.err is not valid.
    find: 0652-019 The status on ./bwaix2_backup_202205071700.log is not valid.
    #

    Trying to remove the whole directory:
    # cd ..
    # rm -fr backup
    is hang for ever....

    Checking the status of the dirctory:
    # istat backup
    Inode 16387 on device 10/6 Directory
    Protection: rwxr-xr-x
    Owner: 0(root) Group: 0(system)
    Link count: 2 Length 8192 bytes

    Last updated: Thu Jul 7 13:15:40 IDT 2022
    Last modified: Thu Jul 7 13:15:40 IDT 2022
    Last accessed: Thu Jul 7 17:10:32 IDT 2022

    So it might be a filesystem corruption.
    I run fsck on the filesystems but no errors where found.

    I'll appreciate any suggestions.
    Thanks
    Jacob Fried





    ------------------------------
    Yaacov Fried
    ------------------------------


  • 2.  RE: Can't remove files

    Posted Fri July 08, 2022 06:29 AM

    Hi
    At first glance it looks like fs is corrupted. Do you see anything in errpt?
    Other exercises like cp file, read inode, move file/directory , dd if=/dev... of=/dev/null , find -inum ? shows any thing more ?

    Kind Regards,

    Slawek


    ------------------------------
    Slawomir Ksiazek
    ------------------------------



  • 3.  RE: Can't remove files

    Posted Fri July 08, 2022 08:45 AM
    Hi

    Looks like the whole File System is corrupted. I would recomend umounting the File System and running the fsck utility. 

    HTH





  • 4.  RE: Can't remove files

    Posted Sun July 10, 2022 12:16 AM
    Hi.

    I tried umount and fsck, but it run clean

    ------------------------------
    Yaacov Fried
    ------------------------------



  • 5.  RE: Can't remove files

    Posted Sun July 10, 2022 12:12 AM
    Hi,
    Nothing in errpt.
    also, cp, dd and find, don't add anything else then "A file or directory in the path name does not exist"

    ------------------------------
    Yaacov Fried
    ------------------------------



  • 6.  RE: Can't remove files

    Posted Sun July 10, 2022 12:16 AM
    Hi,
    Nothing in errpt.
    Also, cp dd and find don't add info more then "A file or directory in the path name does not exist"

    ------------------------------
    Yaacov Fried
    ------------------------------



  • 7.  RE: Can't remove files

    Posted Fri July 08, 2022 09:19 AM

    So it might be a filesystem corruption.

     

    That would be my first guess, although you said fsck ran clean.  Was the filesystem unmounted when you ran the fsck?  If not-and if unmounting is an option-then try that.

     

    What disk technology is this?  We use non-IBM disk technology and things like this can happen if the underlying disk subsystem

    • goes read-only.
    • returns odd statuses that AIX isn't expecting.

     

    I hope that helps.

     

    Anker Lerret

    Blue Cross Blue Shield of Massachusetts






  • 8.  RE: Can't remove files

    Posted Sun July 10, 2022 12:22 AM
    Hi,
    Of course I run fsck when fs unmounted but it run clean
    The disk is IBM SAS

    Thanks

    ------------------------------
    Yaacov Fried
    ------------------------------



  • 9.  RE: Can't remove files

    Posted Fri July 08, 2022 10:00 AM
    What kind of file system is this? What does the output of the mount command show? If you post the output, please only list the file system that has backup in in it.

    And for this I also have to ask: Did you open a support ticket with IBM?

    ------------------------------
    Grover Davidson
    ------------------------------



  • 10.  RE: Can't remove files

    Posted Fri July 08, 2022 10:07 AM
    Also check the permissions on the mount point UNDER it




  • 11.  RE: Can't remove files

    Posted Sun July 10, 2022 12:25 AM
    Hi,

    It's JFS2
    and mount command output looks OK to me:
    /dev/fslv01 /app jfs2 Jul 07 17:53 rw,log=/dev/loglv00

    ------------------------------
    Yaacov Fried
    ------------------------------



  • 12.  RE: Can't remove files

    IBM Champion
    Posted Mon July 11, 2022 11:15 AM
    As @Tom McGivern asked:  Check the permissions of the "app" directory of the root filesystem (not the root directory of the app filesystem).  It should be root/system 755​

    ------------------------------
    José Pina Coelho
    IT Specialist at Kyndryl
    ------------------------------



  • 13.  RE: Can't remove files

    Posted Mon July 11, 2022 11:25 AM
    Thank you,, but  I double checked the permissions of the entire tree.
    definitely, it is not a permission problem, apparently some kind of filesystem/directory  corruption which cannot be repaired by fsck.
    I opened a support case, hoping for a solution.

    ------------------------------
    Yaacov Fried
    ------------------------------



  • 14.  RE: Can't remove files

    Posted Mon July 11, 2022 12:31 PM
    Hi

    Please keep us posted. I've never seen this before.

    TIA


     


    César Daniel Delgado Ponce.

    Sistemas Operativos Seguridad Distribuidos (6181)
    +58 212 503 0619
    Twitter:@MercantilBanco
    YouTube: Mercantil Banco







  • 15.  RE: Can't remove files

    Posted Tue July 12, 2022 10:21 AM
    Have you tried using utility such as Filezilla or Winscp to delete selected files.  In my case Filezilla was able to remove the files where from command line  the "rm" command would hang. 

    Also make sure your user account in /etc/security/limits for filesize is set to -1.

    On occasion I've experienced this before; however, usually when there are too many files vs size of file and the various "ls" commands fail.  Last time >300K files in a directory and what was odd was old server could list the files but after tarball restore on new AIX 7.2 server ls and various commands would fail which is odd.

    Doubt my suggestion will help but worth a try.  Keep us posted.

    ------------------------------
    Ron Arms
    ------------------------------



  • 16.  RE: Can't remove files

    Posted Tue July 12, 2022 11:01 AM
    Hi,

    There are about few dozens files in that directory, each no more than 10MB.
    But I'll give it a try (set to unlimited fsize and using winscp.

    Thanks.

    ------------------------------
    Yaacov Fried
    ------------------------------



  • 17.  RE: Can't remove files

    Posted Tue July 12, 2022 11:19 AM
    I don't believe limits would apply here
    The issue you bring up about "ls" and too many files has more to do with stack and data and the number of pages allocated by the loader.  And that can be increased as well, but, again, not applicable here.

    I've not read if the underlying mount point permissions have been checked yet.