AIX

 View Only
Expand all | Collapse all

rsh not see acknowlegement from a crontab execution

  • 1.  rsh not see acknowlegement from a crontab execution

    Posted Tue December 21, 2021 08:58 AM

    I will try to make this short and quick.  Sorry, I have not used this site in many years as I was moved off AIX duties.  But I have been called back for this issue and I need some insights.

    We have an AIX 5300-12 server that runs a lot of scripts in root's crontab.  Some of these scripts interact with files lists on this AIX box as well as its twin in our EMEA site.  All of a sudden, as of last week (12/11/2021) the jobs started hanging.  If you look at "ps -ef" you see various "rsh EMEA-Server <Some command to run on the EMEA server, like "mv some file" or "touch a file".  And you see the script (the parent process) that was called in cron.  They are just sitting there.

    Turns out that the rsh process is the first one in each of the scripts.  BUT if you go to the EMEA server, the action has been received and accomplished (be it move a file or touch a file or whatever) it is done and no longer in the "ps -ef" on the EMEA AIX server.

    If you come back to the USA AIX system and do a "kill -1" against the PID of the hung rsh command, the parrent program moves on but hangs again if there is a new "rsh" to EMEA?

    But get this.... If I run the same script that is scheduled in root's crontab, but run it at the command line.... it runs through, including all the "rsh" to EMEA and completes?

    So, what changed?  We had some WAN/LAN changes on 12/11/2021 that beefed up the encryption between USA and EMEA sites.  But our Linux and Windows servers do not have issues.  And, again, these AIX scripts (kshell and perl scripts) run fine at the command line of the USA AIX server??

    Tried other AIX servers on both sides, same hang up.
    Tried the same command in crontab, between two AIX 5.3 servers both in USA, It WORKS?
    Only if fired up from cron and only if it goes from servers in one site to a server in the other site.

    Any insights would be appreciated.

    (Oh, Yes, I know SSH would be much better and safer than RSH.  But I have to work, for now, with what I have.)

    Kind regards,

    Chris Baker



    ------------------------------
    Christopher Baker
    ------------------------------


  • 2.  RE: rsh not see acknowlegement from a crontab execution

    Posted Tue December 21, 2021 10:44 AM
    Doesn't rsh also make a connection back to the original system? I wonder if that's part of the problem. I wonder what you'd see running "tcpdump" or "iptrace"/"ipreport" on both sides.

    ------------------------------
    Dave Marquardt
    Sr. Software Engineer
    IBM
    Austin TX
    ------------------------------



  • 3.  RE: rsh not see acknowlegement from a crontab execution

    Posted 12 days ago
    Dave,  Thank ouo for responding to my issue.
    I agree with you observation.  But the question that still remains is "Why does this script (the one with the multiple rsh's in it) run at the command line without issue or hanging, but hangs on the first, and all, rsh's when it is run by the cron process via a scheduled line in the root crontab?

    ------------------------------
    Christopher Baker
    ------------------------------



  • 4.  RE: rsh not see acknowlegement from a crontab execution

    Posted 12 days ago
    So you're running these scripts on the command line when logged in as root?

    I suspect the hang is on the server side. The rsh client connects to the rsh server, which is supposed to connect back to the client. Maybe running "iptrace" or "tcpdump" to capture the TCP traffic would help.

    ------------------------------
    Dave Marquardt
    Sr. Software Engineer
    IBM
    Austin TX
    ------------------------------



  • 5.  RE: rsh not see acknowlegement from a crontab execution

    Posted 9 days ago
    @Dave Marquardt,

    I am saying that I can manually run the same script from the command line as root and they run fine.
    If I attempt to run them from root's crontab, they hang on the rsh ​commands.

    Looking at the remote system, I see the rsh command arrive, it is acted upon and completed and it goes away.  But the local system still shows the rsh ​command running and there it hangs.  If I do a "kill -1" to that stuck rsh process on the local box, the script progresses until another rsh to the remote system is reached, where it runs on the remote system successfully, but hangs on the local box.  Note that "rcp" to the remote box works and does not hang?
    And, to be clear, all this works if I run the same shell script from the command line.  
    And, all this worked from cron until mid-December when the connection between these two systems was upgraded to new network hardware and an encrypted tunnel.

    Oh, we did do tcpdumps and our network team evaluated the dumps and could not find the cause of the hang.

    ------------------------------
    Christopher Baker
    ------------------------------



  • 6.  RE: rsh not see acknowlegement from a crontab execution

    Posted 9 days ago

    Christopher,  did you try to run with a closed stdin ?

    It's one thing that is frequently forgotten is that stuff run from cron doesn't have a terminal descriptor on stdin, and that sometimes causes problems (and rsh/ssh are know to have issues with that).



    ------------------------------
    José Pina Coelho
    IT Specialist at Kyndryl
    ------------------------------



  • 7.  RE: rsh not see acknowlegement from a crontab execution

    Posted 9 days ago
    @José Pina Coelho

    I am not sure what you mean by "a closed stdin"?
    I am redirecting stdout and stderr to log files.
    There is no "input" required for this script to run.

    Please kindly provide more information.
    ​​​

    ------------------------------
    Christopher Baker
    ------------------------------



  • 8.  RE: rsh not see acknowlegement from a crontab execution

    Posted 9 days ago

    On an interactive session, try to run the script like this:

    (/path/to/script/file > stdout 2> stderr)

    Other things to look into:

    - Make sure that the rsh command is called with '-n' (local stdin not passed to remote command).

    - You can avoid having a separate connection for stderr by calling rsh with '-a'. 



    ------------------------------
    José Pina Coelho
    IT Specialist at Kyndryl
    ------------------------------



  • 9.  RE: rsh not see acknowlegement from a crontab execution

    Posted 9 days ago
    @José Pina Coelho

    Jose,
    Thank you for your information.  We currently have the network encrypted tunnel set back to the previous hardware so that we could get around this issue.  When we get a chance to reset the network in a few days, I will attempt so see if these options correct the issue we were seeing.

    In the meantime. I keep focusing on the fact that all this worked before the network changes/updates.  And, that it still works as long as I do not run the scripts from cron?   I know that scripts run as root in cron do not inherit the same environment as root does outside of cron.  I wonder if there is something I can do, define, run from the crontab entry before the running of the shell script that would set the env to be as it would be at the command line and thus get these scripts to run and succeed with the rsh commands? ​

    ------------------------------
    Christopher Baker
    ------------------------------



  • 10.  RE: rsh not see acknowlegement from a crontab execution

    Posted 8 days ago

    >   I know that scripts run as root in cron do not inherit the same environment as root does outside of cron.

    There are two things that are different in a cronjob:

    - It's not an interactive session.  (ex. "tty -s" will set an RC != 0 because there's no TTY associated with STDIN)

    - It's not a login shell (No sourcing of /etc/profile, $HOME/.profile, and $ENV)

    Capture the environment from the script, at the head of the script add "env | sort > ~root/script.env.out".

    Run manually, move script.env.out to manual.out, then wait for it to run from cron and move script.env.out to cron.out

    Check the differences: diff -u manual.out cron.out 

    If there is any variable that the script needs, it should be declared on the script.



    ------------------------------
    José Pina Coelho
    IT Specialist at Kyndryl
    ------------------------------



  • 11.  RE: rsh not see acknowlegement from a crontab execution

    Posted 8 days ago
    You might use procstack to see where the rsh process is hanging.

    ------------------------------
    Dave Marquardt
    Sr. Software Engineer
    IBM
    Austin TX
    ------------------------------



  • 12.  RE: rsh not see acknowlegement from a crontab execution

    Posted 8 days ago
    I would ensure you have the target servers info in the servers /etc/hosts file and the same on the target server....ensure the servers info is in the targets /etc/hosts file.  Sometimes these hangs occur because the client cant answer back to the server....I know it worked before....but something might have changed and this will ensure its not an answer back issue of not knowing where to go.  ( should also verify you resolve with local hosts files first in /etc/netsvc.conf )

    Second....Ive found that sometimes in server to client communication the connection / command wont work unless you use the "terminal option"...we do this using ssh, not sure if its available in rsh....but in ssh I sometimes have to do in this way:
    ssh -t <target_server> "commands to run in qoutes"    

    You might see if there is a similar option to use in rsh, or better would be to just switch to ssh.

    -Paul

    ------------------------------
    Paul Queen
    ------------------------------