Open Source Development

Power Open Source Development

Explore the open source tools and capabilities for building and deploying modern applications on IBM Power platforms including AIX, IBM i, and Linux.


#Power


#Power


#Servers
#Opensource
 View Only
Back to discussions
Expand all | Collapse all

Issue with Bash read builtin in latest release

  • 1.  Issue with Bash read builtin in latest release

    Posted 04/19/21 08:22 PM

    In the latest release of Bash on the toolbox website (5.0 Patch 18) I am running into a problem with one of my scripts and have isolated the issue.  I did downgrade to the previous version of Bash from the toolbox (5.0, no patches) and it works fine.

    What is happening is when I run a command (such as 'ls -al /etc') with a while loop and read the line it works fine sometimes but sometimes it does not.  When it does not work fine it hangs once it reaches the end of the input that is being fed into the loop, it should see the EOF and automatically terminate but it does not.  This previously worked fine 100% of the time.
    I trussed the output and this is what we are getting when we reach the end of the line:
         kfcntl(2, F_GETFL, 0x0FFFFFFFFFFFFFE8)          = 67110914
         kioctl(0, 22528, 0x0000000000000000, 0x0000000000000000) Err#25 ENOTTY
         lseek(0, 0, 1)                                  Err#29 ESPIPE
         kread(0, "\t", 1)               (sleeping...)

    When it is working the kread function does not sleep, it just moves onto the next iteration in the while loop.

    Here is a sample piece of code to recreate the issue:
          SGLIST=("testfile1" "testfile1" "testfile1" "testfile1") 

          for sg in ${SGLIST[@]}
          do
                  while read line
                  do
                          echo $line
                  done < <(ls -al /etc 2>&1)
           done
    This particular sample code reliably fails every time.  The piece of code I have written fails with the same basic setup one out of every four times or so.  

    Any thoughts on this?

    Thanks






    ------------------------------
    Mike Whitton
    ------------------------------

    #AIXOpenSource


  • 2.  RE: Issue with Bash read builtin in latest release

    Posted 04/20/21 12:57 PM

    I downloaded the official GNU release, patched it to version 5.0.18, the same release IBM is distributing, and it is working fine.  The GCC compilers and GNU Make were used to build this package.

    There is possibly something wrong with the version that is being distributed on the IBM AIX Linux Toolbox website.

    Thanks

    ------------------------------
    Mike Whitton
    ------------------------------



  • 3.  RE: Issue with Bash read builtin in latest release

    Posted 04/21/21 03:29 AM

    Strange, running your script, it doesn't end, it stands on the last read:
    <root@t0201:/home/00000217/testdir>./testhg2.sh
    SGLIST=("testfile1" "testfile1" "testfile1" "testfile1")
    + SGLIST=("testfile1" "testfile1" "testfile1" "testfile1")

    for sg in ${SGLIST[@]}
    do
    while read line
    do
    echo "$line"
    done < <(ls -al ./ 2>&1)
    done
    + for sg in ${SGLIST[@]}
    + read line
    ++ ls -al ./
    + echo 'total 32'
    total 32
    + read line
    + echo 'drwxr-xr-x 2 root system 256 Apr 21 09:12 .'
    drwxr-xr-x 2 root system 256 Apr 21 09:12 .
    + read line
    + echo 'drwxr-xr-x 7 00000217 staff 4096 Apr 21 09:04 ..'
    drwxr-xr-x 7 00000217 staff 4096 Apr 21 09:04 ..
    + read line
    + echo '-rwxr-xr-x 1 root system 179 Apr 21 09:15 testhg2.sh'
    -rwxr-xr-x 1 root system 179 Apr 21 09:15 testhg2.sh
    + read line
    + echo '-rwxr-xr-x 1 root system 378 Apr 21 09:26 testhga.sh'
    -rwxr-xr-x 1 root system 378 Apr 21 09:26 testhga.sh
    + read line
    + echo '-rwxr-xr-x 1 root system 381 Apr 21 09:12 testhgb.sh'
    -rwxr-xr-x 1 root system 381 Apr 21 09:12 testhgb.sh
    + read line

    Now I modified it slightly and it works (run it 50 times):

    <root@t0201:/home/00000217/testdir>cat testhga.sh
    !/bin/bash
    echo "bash version: ${BASH_VERSINFO}"
    rpm -qa |grep bash


    SGLIST=("testfile1" "testfile1" "testfile1" "testfile1")
    i=0
    for sg in ${SGLIST[@]}
    do
    let i=i+1
    echo "Loop $i"
    j=0
    while read line
    do
    let j=j+1
    echo "Loop $i MYLINE: $line"
    done < <(ls -al ./ 2>&1)
    done
    <root@t0201:/home/00000217/testdir>kkk
    <root@t0201:/home/00000217/testdir>./testhga.sh
    ./testhga.sh: line 1: !/bin/bash: No such file or directory
    bash version: 5
    bash-5.0.18-1.ppc
    Loop 1
    Loop 1 MYLINE: total 32
    Loop 1 MYLINE: drwxr-xr-x 2 root system 256 Apr 21 09:12 .
    Loop 1 MYLINE: drwxr-xr-x 7 00000217 staff 4096 Apr 21 09:04 ..
    Loop 1 MYLINE: -rwxr-xr-x 1 root system 179 Apr 21 09:15 testhg2.sh
    Loop 1 MYLINE: -rwxr-xr-x 1 root system 378 Apr 21 09:26 testhga.sh
    Loop 1 MYLINE: -rwxr-xr-x 1 root system 381 Apr 21 09:12 testhgb.sh
    Loop 2
    Loop 2 MYLINE: total 32
    Loop 2 MYLINE: drwxr-xr-x 2 root system 256 Apr 21 09:12 .
    Loop 2 MYLINE: drwxr-xr-x 7 00000217 staff 4096 Apr 21 09:04 ..
    Loop 2 MYLINE: -rwxr-xr-x 1 root system 179 Apr 21 09:15 testhg2.sh
    Loop 2 MYLINE: -rwxr-xr-x 1 root system 378 Apr 21 09:26 testhga.sh
    Loop 2 MYLINE: -rwxr-xr-x 1 root system 381 Apr 21 09:12 testhgb.sh
    Loop 3
    Loop 3 MYLINE: total 32
    Loop 3 MYLINE: drwxr-xr-x 2 root system 256 Apr 21 09:12 .
    Loop 3 MYLINE: drwxr-xr-x 7 00000217 staff 4096 Apr 21 09:04 ..
    Loop 3 MYLINE: -rwxr-xr-x 1 root system 179 Apr 21 09:15 testhg2.sh
    Loop 3 MYLINE: -rwxr-xr-x 1 root system 378 Apr 21 09:26 testhga.sh
    Loop 3 MYLINE: -rwxr-xr-x 1 root system 381 Apr 21 09:12 testhgb.sh
    Loop 4
    Loop 4 MYLINE: total 32
    Loop 4 MYLINE: drwxr-xr-x 2 root system 256 Apr 21 09:12 .
    Loop 4 MYLINE: drwxr-xr-x 7 00000217 staff 4096 Apr 21 09:04 ..
    Loop 4 MYLINE: -rwxr-xr-x 1 root system 179 Apr 21 09:15 testhg2.sh
    Loop 4 MYLINE: -rwxr-xr-x 1 root system 378 Apr 21 09:26 testhga.sh
    Loop 4 MYLINE: -rwxr-xr-x 1 root system 381 Apr 21 09:12 testhgb.sh
    <root@t0201:/home/00000217/testdir>





    ------------------------------
    Achim Haag
    ------------------------------

    Original Message


  • 4.  RE: Issue with Bash read builtin in latest release

    Posted 04/21/21 09:23 AM
    Thanks for taking the time to look into this.

    I have also been communicating with one of the maintainers of bash and he feels that the way patch 17 was implemented in the IBM version is causing the problem, he feels that the FIFO's are not being reaped properly after being written to.

    btw, I was able to recompile the mainline bash binaries with all of the patches upto 18 applied and it works on the same AIX server without issue.  So I do believe this issue exists with the distribution of bash released by IBM...

    Thanks


    ------------------------------
    Mike Whitton
    ------------------------------

    Original Message


  • 5.  RE: Issue with Bash read builtin in latest release

    Posted 04/21/21 12:44 PM
    Thank you Mike and Achim for reporting and looking into issue and also working with community.
    We will look into this.

    ------------------------------
    SANKET RATHI
    ------------------------------

    Original Message


  • 6.  RE: Issue with Bash read builtin in latest release

    Posted 04/23/21 02:07 AM
    It seems bash-5.1 has changed the ways FIFOs are implemented, hence the patch which is causing this issue is no more required.
    We will be to updating bash to 5.1 which fixes this issue and let you know once it's available on AIX Toolbox.


    ------------------------------
    SANGAMESH
    ------------------------------

    Original Message


  • 7.  RE: Issue with Bash read builtin in latest release

    Posted 05/17/21 05:26 AM
    Edited by Achim Haag 05/17/21 05:32 AM
    I don't know if it's another behaviour of the same issue, but today, I noticed that "while read" results aren't given out of the loop in bash 5.0.18.
    Here my example:

    >cat hga.mailsend.table
    # This is a comment line
    # Now an address line:
    donald.duck@duckburg.de
    mickey.mouse@nowhere.com


    >cat hga.sh
    #!/bin/bash
    echo "My bash version is ${BASH_VERSION}"
    emma="will it be changed ?"
    hugo="initial value before loop"
    anna="initial value before loop"
    cat hga.mailsend.table | while read myrec ; do
    echo "myrec: $myrec"
    #
    emma=$myrec
    echo "inside loop: emma=$emma"
    #
    hugo="$hugo + $myrec"
    echo "inside loop: hugo=[$hugo]"
    #
    anna=$(echo "$hugo + $myrec")
    echo "inside loop: anna=[$anna]"
    done
    #
    echo "loopend: emma=[$emma]"
    echo "loopend: hugo=[$hugo]"
    echo "loopend: anna=[$anna]"



    The result is:

    >./hga.sh
    My bash version is 5.0.18(1)-release
    myrec: # This is a comment line
    inside loop: emma=# This is a comment line
    inside loop: hugo=[initial value before loop + # This is a comment line]
    inside loop: anna=[initial value before loop + # This is a comment line + # This is a comment line]
    myrec: # Now an address line:
    inside loop: emma=# Now an address line:
    inside loop: hugo=[initial value before loop + # This is a comment line + # Now an address line:]
    inside loop: anna=[initial value before loop + # This is a comment line + # Now an address line: + # Now an address line:]
    myrec: donald.duck@duckburg.de
    inside loop: emma=donald.duck@duckburg.de
    inside loop: hugo=[initial value before loop + # This is a comment line + # Now an address line: + donald.duck@duckburg.de]
    inside loop: anna=[initial value before loop + # This is a comment line + # Now an address line: + donald.duck@duckburg.de + donald.duck@duckburg.de]
    myrec: mickey.mouse@nowhere.com
    inside loop: emma=mickey.mouse@nowhere.com
    inside loop: hugo=[initial value before loop + # This is a comment line + # Now an address line: + donald.duck@duckburg.de + mickey.mouse@nowhere.com]
    inside loop: anna=[initial value before loop + # This is a comment line + # Now an address line: + donald.duck@duckburg.de + mickey.mouse@nowhere.com + mickey.mouse@nowhere.com]
    loopend: emma=[will it be changed ?]
    loopend: hugo=[initial value before loop]
    loopend: anna=[initial value before loop]

    So, the read loop works correct, no hangup, but the values set in the loop are reset after "done".


    If I change the while-pipe to "for myrec in $(cat hga.mailsend.table); do" , I get the variable contents outside of the loop, but as "for" split by words, it doesn't make sense.

    It seems not related to file I/O, for example:

    cat hga.endless.sh
    #!/bin/bash
    ctr=0
    while hugo="init" ; do
      echo "before: $hugo"
      hugo="exit"
      echo "after: $hugo"
      let ctr=ctr+1
      [[ $ctr -gt 10 ]] && exit
    done
    echo "The end is near..."

    Result :

    ./hga.endless.sh
    before: init
    after: exit
    before: init
    after: exit
    before: init
    after: exit
    before: init
    after: exit
    before: init
    after: exit
    before: init
    after: exit
    before: init
    after: exit
    before: init
    after: exit
    before: init
    after: exit
    before: init
    after: exit
    before: init
    after: exit
    ...aborted by ctrl-c...


    ------------------------------
    Achim Haag
    ------------------------------

    Original Message


  • 8.  RE: Issue with Bash read builtin in latest release

    Posted 05/17/21 08:28 AM
    Further test: I ran the following script in some of our systems, most with bash 5.0.18, one with 5.0.1:

    #!/bin/bash
    echo "My bash is ${BASH_VERSION}"
    set -x -v
    ctr=0
    ###cat $0 | while read myline ; do
    while [[ $ctr -lt 2 ]] ; do
      let ctr=ctr+1
      echo "ctr: $ctr"
      let berta=ctr+10
      echo "berta: $bertbertaa"
    done
    echo "final ctr : $ctr"
    echo "final berta : $berta"

    In both bash versions, if the "cat ... while" is active, the loop set variables (ctr, berta) are empty after "done",
    if the "while [[]]" is active, the loop set variables have their correct contents.

    I see the initial problem of this topic (halt after last read) only in AIX bash 5.0.18,
    but the problem of a "while read" not keeping the values of in-loop-set variables after "done" seems to exist in 5.0.1 and 5.0.18.

    Maybe I made a mistake and somebody else could verify the erasure of variables after a "while read" loop ?

    ------------------------------
    Achim Haag
    ------------------------------

    Original Message


  • 9.  RE: Issue with Bash read builtin in latest release

    Posted 05/17/21 11:19 AM
    Sorry, my fault. bash pipe with while seems to run a subshell, therefore variables set inside the while aren't saved cross the "done".

    I found the solution as a "here string":

    while read MAILADR; do
      SENDTO="${SENDTO}${MAILADR},"
    done <<< $(grep -v -E "^#|^$|^ $" $ADRFILE)

    In advance, it seems to avoid the above mentioned "stop" after the last read (run it 20 times without problems).

    ------------------------------
    Achim Haag
    ------------------------------

    Original Message


  • 10.  RE: Issue with Bash read builtin in latest release

    Posted 05/19/21 03:27 AM
    Thanks for update.

    ------------------------------
    SANKET RATHI
    ------------------------------

    Original Message


  • 11.  RE: Issue with Bash read builtin in latest release

    Posted 05/24/21 01:40 AM
    I met the same problem. My similar testcase is:

    #!/bin/bash -x

    for ((i=1; i<=100; i++))
     do
      while read line
       do
        echo "$line"
       done < <( echo xxx$i )
     done

    Finding a solution brought me here. I'm glad it's already known. Thank you for your investigation. I'm looking forward to the rebuilt version 5.0.18 or the new 5.1 in the AIX toolbox repository.

    ------------------------------
    Ondřej Rajmon
    ------------------------------

    Original Message


  • 12.  RE: Issue with Bash read builtin in latest release

    Posted 06/21/21 01:04 AM
    Bash 5.1.4 is now available on AIX toolbox.

    ------------------------------
    SANKET RATHI
    ------------------------------

    Original Message


  • 13.  RE: Issue with Bash read builtin in latest release

    Posted 06/18/21 12:48 PM
    I have downloaded and tested the latest Bash 5.1.4 release from the Aix Toolbox Repository and it appears to be working fine.

    Thanks for addressing this issue.

    ------------------------------
    Mike Whitton
    ------------------------------

    Original Message


Related Content