Originally posted by: hanya
Hey guys,
We've been having recurring bash_64 process core dumps on our servers running AIX 7.1 after updating to bash 4.4-3 from the AIX Toolbox for Linux. It was fine when we were on bash 4.2-3 before that.
We opened a case with IBM Support and here is what they came back with (in blue) after debugging one of the core files:
david.gray (IBM)
25 Mar 2019 02:11 PM
It appears bash is looping in code that is setting terminal attributes in response to a signal, perhaps a SIGWINCH or something similar, and it loops in this forever until it runs out of stack.
itcaix16 $ dbx -p /=./ ./usr/opt/freeware/bin/bash_64 ./home/fcroot/core
Type 'help' for help.
warning: The core file is not a fullcore. Some info may
not be available.
[using memory image in ./home/fcroot/core]
reading symbolic information ...warning: Unable to access the stab file. Some info may not be available
warning: no source compiled with -g
Segmentation fault in __ioctl at 0x9000000000336d4
0x9000000000336d4 (__ioctl+0xd4) e8410028 ld r2,0x28(r1)
ioctl(0x0, 0x540300005403, 0x11001b4c0, 0x800000000000d032, 0x3b68, 0x0, 0xf1000a0150839800, 0x8000000000001032) at 0x900000000033dac
tcsetattr(??, ??, ??) at 0x90000000017f834
_set_tty_settings(0x0, 0x11001b4c0) at 0x1000d79a8
set_tty_settings(0x0, 0x11001b4c0) at 0x1000d7a84
rl_deprep_terminal() at 0x1000d8268
rl_cleanup_after_signal() at 0x1000d699c
...
rl_cleanup_after_signal() at 0x1000d699c
_rl_handle_signal(0x100000001) at 0x1000d6ea8
_rl_signal_handler(0x100000001) at 0x1000d7010
_rl_release_sigint() at 0x1000d7100
rl_deprep_terminal() at 0x1000d8274
(dbx) proc rlimit
rlimit name: rlimit_cur rlimit_max (units)
RLIMIT_CPU: (unlimited) (unlimited) sec
RLIMIT_FSIZE: (unlimited) (unlimited) bytes
RLIMIT_DATA: (unlimited) (unlimited) bytes
RLIMIT_STACK: 33554432 4294967296 bytes
RLIMIT_CORE: 1073741312 (unlimited) bytes
RLIMIT_RSS: 33554432 (unlimited) bytes
RLIMIT_AS: (unlimited) (unlimited) bytes
RLIMIT_NOFILE: 2000 (unlimited) descriptors
RLIMIT_THREADS: (unlimited) (unlimited) per process
RLIMIT_NPROC: (unlimited) (unlimited) per user
We tried increasing the stack ulimit as David Gray, the case engineer, had advised and it did not make a difference in the number or frequency of the core dumps.
We then tried upgrading to bash 5.0-1, the latest version available in the AIX Toolbox for Linux Applications and also updated from AIX 7100-05-02 to 7100-05-04. None of these changes made a difference.
I opened another case with IBM but they said there is nothing further they can do and pointed me to this forum. I would greatly appreciate any help to figure out what might be causing this issue and how to resolve it.
Here is the oslevel and bash level, and a recent core dump error from one of the servers. I also attached the core file to this thread.
paris(root)/home/root#>oslevel -s
7100-05-04-1914
paris(root)/home/root#>lslpp -L bash
Fileset Level State Type Description (Uninstaller)
----------------------------------------------------------------------------
bash 5.0-1 C R The GNU Bourne Again shell
(bash) version 5.0 (/bin/rpm)
---------------------------------------------------------------------------
LABEL: CORE_DUMP
IDENTIFIER: A924A5FC
Date/Time: Mon Aug 12 13:31:34 EDT 2019
Sequence Number: 2398
Machine Id: 00FAD7524C00
Node Id: paris
Class: S
Type: PERM
WPAR: Global
Resource Name: SYSPROC
Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED
Probable Causes
SOFTWARE PROGRAM
User Causes
USER GENERATED SIGNAL
Recommended Actions
CORRECT THEN RETRY
Failure Causes
SOFTWARE PROGRAM
Recommended Actions
RERUN THE APPLICATION PROGRAM
IF PROBLEM PERSISTS THEN DO THE FOLLOWING
CONTACT APPROPRIATE SERVICE REPRESENTATIVE
Detail Data
SIGNAL NUMBER
11
USER'S PROCESS ID:
32112722
FILE SYSTEM SERIAL NUMBER
7
INODE NUMBER
12288
CORE FILE NAME
/home/fcroot/core
PROGRAM NAME
bash_64
STACK EXECUTION DISABLED
0
COME FROM ADDRESS REGISTER
PROCESSOR ID
hw_fru_id: N/A
hw_cpu_id: N/A
ADDITIONAL INFORMATION
Unable to generate symptom string.
Too many stack elements.
---------------------------------------------------------------------------
Please let me know if I should provide any other info. : )
Thanks!
Lavanya Herbert
Unix Administrator