AIX

AIX

Connect with fellow AIX users and experts to gain knowledge, share insights, and solve problems.


#Power
 View Only
  • 1.  Command is respawning too rapidly

    Posted Wed July 06, 2011 12:42 PM

    Originally posted by: SystemAdmin


    AIX 5.3
    oslevel -r responds: 5300-05
    Model: 9131-52A

    Has been running fine for years. Just recently, we have begun generating multiple "respawn too rapid" errors in the error log as below:


    LABEL: INIT_RAPID
    IDENTIFIER: 3A30359F

    Date/Time: Wed Jul 6 06:27:16 PDT 2011
    Sequence Number: 11511
    Machine Id: 00047500D700
    Node Id: nwwinc
    Class: S
    Type: TEMP
    Resource Name: init

    Description
    SOFTWARE PROGRAM ERROR

    Probable Causes
    SOFTWARE PROGRAM

    User Causes
    PERFORMANCE DEGRADED

    Recommended Actions
    REVIEW DETAILED DATA

    Detail Data
    SOFTWARE ERROR CODE
    Command is respawning too rapidly. Check for possible errors.
    COMMAND
    id: agd "/usr/appgen/agdaemon.sh > /home/sysadmin/agd.out 2>&1"


    The only changes to the system or environment are switching all the PCs to Win7.

    This program has been running reliably on this and previous levels of AIX plus this and previous RISC systems for some time. It provides a client login to an accounting application on AIX 5.3 from an ethernet local WAN made of Cisco 831-K9 routers. There are no serial terminals to create this problem from a TTYHOG overrun. There are no recent AIX updates. There is no significant error output sent to the error output location.

    There seems to be no AIX troubleshooting information that really explains what causes "respawning too rapidly" errors.

    Any ideas or referrals to decent technical documentation for "respawn too rapid" errors would be appreciated.
    #AIX-Forum


  • 2.  Re: Command is respawning too rapidly

    Posted Thu July 07, 2011 10:50 AM

    Originally posted by: shargus


    "respawning too rapidly" basically means exactly that - the process dies, init respawns it, repeat... When this happens too many times in certain period of time (5 minutes, I think), init quits trying and issues the message.

    Try running the command from a root shell. I'll bet you'll get a more informative message.
    You may have to disable it in inittab first.
    #AIX-Forum


  • 3.  Re: Command is respawning too rapidly

    Posted Thu July 07, 2011 08:47 PM

    Originally posted by: SystemAdmin


    THX for the response. Starting from the command line may reveal something, so I appreciate the suggestion.

    What I am trying to find out is; can I "nice" it to a lower priority or something to reduce the rapid respawn. It has run fine for years. This is not the only command that I have seen become a respawn problem on this and other systems. As such, there should be some priority lowering or respawn delay that would cut it down. Maybe I should embed a "sleep 2" in the shell that invokes it. Do you have any idea if that might help??
    #AIX-Forum


  • 4.  Re: Command is respawning too rapidly

    Posted Fri July 08, 2011 04:05 PM

    Originally posted by: shargus


    According to the man page for init,

    
    If the init command finds that it is continuously running an entry in the /etc/inittab file (more than five times in 225 seconds), it assumes that an error in the entry command string exists. It then prints an error message to the console and logs an error in the system error log. After the message is sent, the entry does not run 
    
    for 60 seconds. If the error continues to occur, the command will respawn the entry only five times every 240 seconds. The init command continues to assume an error occurred until the command does not respond five times in the interval, or until it receives a signal from a user. The init command logs an error 
    
    for only the first occurrence of the error.
    


    So, there is no way to change the behavior of init. Adding a "sleep 2" won't change anything, either (will still respawn 5 times in 225 seconds). You COULD sleep for a really long time (4 minutes should do the trick), but it still doesn't fix the underlying problem - your process starts, dies, then four minutes later it restarts, then dies again...

    Renicing the process - doesn't really matter, unless it's eating up a significant amount of CPU time.

    You didn't mention it in your first post, but have you checked the agd.out log file?

    I still think your best bet is to start the process up from a root shell, then see what happens when it dies.
    #AIX-Forum


  • 5.  Re: Command is respawning too rapidly

    Posted Fri July 08, 2011 06:32 PM

    Originally posted by: SystemAdmin


    THX, again, for the response.

    The results for agd.out are typically this:

    1. cat agd.out
    server: can't bind local address
    Using default port (8188)

    The process being respawned is bound to port 8188 to perform login to the accounting application. Other users are likely
    logged on at the time of the respawning and already utilizing a port 8188 connection. I have no way to have the script
    clear the port (8188) prior to the attempt to bind it.

    This has become an issue since all the PCs on the network (and they are the devices logging in via this port) converted to
    Win7. I prefer not to tell a customer "Oh well, it's Windows' fault...live with it". On the other hand, the AIX servers have
    run reliably for over 10 years running this client login process. Though diags show nothing, I can only guess an intermittent
    ethernet problem if it's the AIX box.
    #AIX-Forum