AIX Open Source

 View Only
Expand all | Collapse all

Reboot AIX server with ansible playbook failed

  • 1.  Reboot AIX server with ansible playbook failed

    Posted Wed July 19, 2023 09:10 AM

    Hi Team,

    We are trying to reboot AIX server with ansible playbook, server is rebooted but playbook never exited and it got stuck. Could you please check the below playbook and let me know if I missed anything.

    ---

    - hosts: server_name

      gather_facts: false

      tasks:

       - name: "Reboot a machine"

         reboot:

           pre_reboot_delay: 0

           post_reboot_delay: 0

           connect_timeout: 10

           reboot_timeout: 300

    Regards

    Manoj



    ------------------------------
    Manoj Kumar
    ------------------------------


  • 2.  RE: Reboot AIX server with ansible playbook failed

    Posted Wed July 19, 2023 09:41 AM

    Is this playbook from power_aix collection from Ansible Galaxy or this is your custom playbook?

    For Ansible playbook and module on AIX from Ansible Galaxy the right forum is https://github.com/IBM/ansible-power-aix/issues

    But I don't think they will be able to help in custom module/playbook.

     

    Thanks,

     

    Sanket Rathi

     




    Original Message


  • 3.  RE: Reboot AIX server with ansible playbook failed

    Posted Wed July 19, 2023 10:26 AM

    Hi Sanket,

    We just create this playbook for testing the reboot of AIX servers. Refer little bit by Redhat reboot playbook. Do we have the playbook available for reboot in AIX?

    Please guide if you have the playbook ready for reboot in AIX. We will try to use that.



    ------------------------------
    Manoj Kumar
    ------------------------------

    Original Message


  • 4.  RE: Reboot AIX server with ansible playbook failed

    IBM Champion
    Posted Thu July 20, 2023 04:05 AM

    Manoj,

    Hi, I've run the (IBM module) re-boot below as part of an AIX Gold image process.

    From memory I had to play around with the pre- and post- settings so that it worked for me.

    #########################
#                       #
# LPAR re-boot          #
#                       #
#########################
    - name: Re-boot AIX LPAR
      reboot:
        pre_reboot_delay: 5
        post_reboot_delay: 10
        connect_timeout: 30
        reboot_timeout: 180

    Thanks, Steve



    ------------------------------
    Steve Munday
    AIX, IBM i, HMC, PowerVM
    ------------------------------

    Original Message


  • 5.  RE: Reboot AIX server with ansible playbook failed

    Posted Thu July 20, 2023 04:42 AM

    Thanks Steve for share the update.

    I also tried in the same way but my playbook is still hanging and not exiting but server has been rebooted.

    Here is my playbook. Am I missing something here?

    ---
    - hosts: host_name
      gather_facts: false
      tasks:
        - name: Re-boot AIX LPAR
          reboot:
            pre_reboot_delay: 5
            post_reboot_delay: 10
            connect_timeout: 30
            reboot_timeout: 180

    server status

    uptime
      04:38AM   up 7 mins,  1 user,  load average: 2.33, 1.96, 1.05

    Playbook status

     ansible-playbook reboothost4.yml

    PLAY [host_name] ***************************************************************************************************************************************************************************************************************************

    TASK [Re-boot AIX LPAR] ********************************************************************************************************************************************************************************************************************* 



    ------------------------------
    Manoj Kumar
    ------------------------------

    Original Message


  • 6.  RE: Reboot AIX server with ansible playbook failed

    IBM Champion
    Posted Thu July 20, 2023 05:18 AM

    Manoj,

    Hi, I re-checked some (very) old notes and found the below image which is showing the Playbook times out even though the LPAR does successfully re-boot.  As you can see, the Playbook does "end" unlike yours which is hanging (for some reason).  Also note that I am running my Playbook within AWX (hosted on AIX thanks to @Andrey Klyachkin)

    I've Emailed two members of the IBM/Ansible module team to see if they can assist.

    Thanks, Steve



    ------------------------------
    Steve Munday
    AIX, IBM i, HMC, PowerVM
    ------------------------------

    Original Message


  • 7.  RE: Reboot AIX server with ansible playbook failed

    IBM Champion
    Posted Thu July 20, 2023 05:26 AM

    Manoj,

    Checking here https://github.com/IBM/ansible-power-aix/issues/191 has the following within it, have you checked your ansible.cfg parms?

    Thanks, Steve



    ------------------------------
    Steve Munday
    AIX, IBM i, HMC, PowerVM
    ------------------------------

    Original Message


  • 8.  RE: Reboot AIX server with ansible playbook failed

    Posted Thu July 20, 2023 06:13 AM

    Thanks Steve. Really great help. 

    We made ssh_connection change in ansible configuration file and server is rebooted and also playbook finished once server reboot completed. Do we understand the meaning of all those attributes which we are using in ssh_connection command? Can we find useful information about all those attributes which is used in this ssh connection string.

    Regards

    Manoj



    ------------------------------
    Manoj Kumar
    ------------------------------

    Original Message


  • 9.  RE: Reboot AIX server with ansible playbook failed

    IBM Champion
    Posted Thu July 20, 2023 06:46 AM

    Manoj,

    I did a little searching and found this https://man.openbsd.org/ssh_config.5 which might help.

    Thanks, Steve



    ------------------------------
    Steve Munday
    AIX, IBM i, HMC, PowerVM
    ------------------------------

    Original Message


  • 10.  RE: Reboot AIX server with ansible playbook failed

    Posted Thu July 20, 2023 07:24 AM

    Thanks Steve, it was really great help. I will review this article.



    ------------------------------
    Manoj Kumar
    ------------------------------

    Original Message


  • 11.  RE: Reboot AIX server with ansible playbook failed

    Posted Thu July 20, 2023 12:01 PM

    Not really related to the ansible piece, but to kick off a disconnected reboot I've often done:

    # echo '/usr/sbin/shutdown -Fr' | at now

    This way it returns immediately to the command prompt and I can log off before the reboot kicks me off.



    ------------------------------
    Glen Corneau
    ------------------------------

    Original Message


  • 12.  RE: Reboot AIX server with ansible playbook failed

    Posted Thu July 20, 2023 02:06 PM
    On Thu, Jul 20, 2023 at 04:01:19PM +0000, Glen Corneau via IBM TechXchange Community wrote:
    > Not really related to the ansible piece, but to kick off a disconnected reboot I've often done:
    >
    > # echo '/usr/sbin/shutdown -Fr' | at now

    I have to lecture everyone about this...

    Never never never use shutdown -F! Can I get fireworks and sparkles?

    The -F flag for shutdown means FORCE down, not fast down. If your
    giant database doesn't come down in 60 seconds, -F will kill -9 it
    resulting in a crashed database. I've had multiple customers cause HA
    failovers (killed clstrmgrES) and many needless integrity checks of
    crashed DBs due to this.

    Regular shutdown is fine. It waits and sends repeated kill -15 until
    everything exits cleanly. Only use -F if you're already shutting down,
    and it isn't coming down and appears hung.

    I think IBM training got lazy years ago and taught "shutdown -Fr"
    instead of "shutdown -rl now". No one adds -F to mean force, they just
    forget to use "now" instead of the default delayed shutdown.

    If you're paying attention, -l means log the shutdown. You do check
    your boot and console alog's after booting? You want to keep records
    of what happened? Then log your shutdown.



    ------------------------------------------------------------------
    Russell Adams Russell.Adams@AdamsSystems.nl
    Principal Consultant Adams Systems Consultancy
    https://adamssystems.nl/


    Original Message


  • 13.  RE: Reboot AIX server with ansible playbook failed

    Posted Thu July 20, 2023 02:53 PM

    Thanks @Russell Adams for the "real-world" knowledge!  I will admit that 99.99% of AIX reboots I've kicked off were always in my lab, with 0% having any real-life clustering (other than maybe GPFS) and/or real applications!

    I concur that you need to heed this warning!  ;-)   

    <insert fireworks and sparklers here>



    ------------------------------
    Glen Corneau
    ------------------------------

    Original Message


  • 14.  RE: Reboot AIX server with ansible playbook failed

    Posted Fri July 21, 2023 03:01 AM

    > I think IBM training got lazy years ago and taught
    > "shutdown -Fr" instead of "shutdown -rl now"

    I'd rather say that people tend to forget that "shutdown" system command is to be used to shutdown the system. That means you are supposed to shutdown your applications before shutting down the operating system, and if you don't then you are supposed to accept the consequences.



    ------------------------------
    Lech Szychowski
    ------------------------------

    Original Message


  • 15.  RE: Reboot AIX server with ansible playbook failed

    Posted Fri July 21, 2023 05:47 AM

    Thanks @Russell Adams @Glen Corneau@Lech Szychowski to be involved in this discussion thread. We always get to knoe useful information when we connect with group discussions.@Glen Corneau

    We also used to shutdown -Fr now command or sometime shutdown -Fr. I believe this is the better command shutdown -r now? Correct me if I am wrong.

    But sometime this command didn't bring down the server if it is heavily utilized server specially DB server. More than 30 minutes to bring down complete server.



    ------------------------------
    Manoj Kumar
    ------------------------------

    Original Message


  • 16.  RE: Reboot AIX server with ansible playbook failed

    Posted Fri July 21, 2023 06:01 AM

    > I believe this is the better command shutdown -r now? Correct me if I am wrong.

    As I wrote a moment ago, i believe proper way to shutdown the system is to shutdown applications/services first, then shutdown the system itself.

    > But sometime this command didn't bring down the server if it is heavily utilized server specially DB server.

    This is exactly what I am taking about. You should first shudown the DB and then, when DB is already down, proceeed to shut the system down.

    > More than 30 minutes to bring down complete server.

    How much time out of this half an hour was spent on shutiing down the system itself, and how much on other shutdown tasks?



    ------------------------------
    Lech Szychowski
    ------------------------------

    Original Message


  • 17.  RE: Reboot AIX server with ansible playbook failed

    Posted Fri July 21, 2023 07:01 AM

    > This is exactly what I am taking about. You should first shudown the DB and then, when DB is already down, proceeed to shut the system down.

    Now a days everything is automated and DB/app shutdown/startup also included with server rc script. Once server bring down, first it brings the app down then DB down then it goes for system down. Same happened when it came up. All process startup automatically. 



    ------------------------------
    Manoj Kumar
    ------------------------------

    Original Message


  • 18.  RE: Reboot AIX server with ansible playbook failed

    Posted Tue April 16, 2024 01:47 AM

    Hi Russell, I was updating some IBM AIX training material today. I made sure to change the shutdown command examples from -F to -rl now.

    ;)

    Thanks!



    ------------------------------
    Chris Gibson
    ------------------------------

    Original Message


  • 19.  RE: Reboot AIX server with ansible playbook failed

    Posted Tue April 16, 2024 03:28 AM
    On Tue, Apr 16, 2024 at 05:47:13AM +0000, Chris Gibson via IBM TechXchange Community wrote:
    > Hi Russell, I was updating some IBM AIX training material today. I made sure to change the shutdown command examples from -F to -rl now.

    Brilliant! Thanks for your attention to detail.

    ------------------------------------------------------------------
    Russell Adams Russell.Adams@AdamsSystems.nl
    Principal Consultant Adams Systems Consultancy
    https://adamssystems.nl/


    Original Message


  • 20.  RE: Reboot AIX server with ansible playbook failed

    Posted Wed April 17, 2024 09:08 AM

    I hope the IBM documentation being updated includes the AIX man pages, as I've been believing this for the last 40 years:

     -F
                Does a fast shutdown, bypassing the messages to other users and bringing the system down as quickly as possible. The +Time [
                Message ] options are ignored if the -F flag is specified.



    ------------------------------
    Mark Skinner
    ------------------------------

    Original Message


  • 21.  RE: Reboot AIX server with ansible playbook failed

    Posted Wed April 17, 2024 09:14 AM
    On Wed, Apr 17, 2024 at 01:07:38PM +0000, Mark Skinner via IBM TechXchange Community wrote:
    > I hope the IBM documentation being updated includes the AIX man
    > pages, as I've been believing this for the last 40 years:
    >
    >
    > -F
    > Does a fast shutdown, bypassing the messages to other users and bringing the system down as quickly as possible. The +Time [
    > Message ] options are ignored if the -F flag is specified.

    Nice catch! Notice it says "bringing the system down as quickly as
    possible", which is where kill -9 comes into play. A fast or forced
    shutdown in this case will break large applications.

    As I said, my experience is most admins that used -F did so because it
    didn't require a time to be specified (ie: now).

    ------------------------------------------------------------------
    Russell Adams Russell.Adams@AdamsSystems.nl
    Principal Consultant Adams Systems Consultancy
    https://adamssystems.nl/


    Original Message


  • 22.  RE: Reboot AIX server with ansible playbook failed

    Posted Wed May 29, 2024 01:49 AM

    hi,Steve Munday

    I tried to add ssh parameters to the ansible.cfg file as you did, but SSH still hung after execution. I am executing yml file to add -vvv, check the execution process found the following screenshot. Is there any other way?



    ------------------------------
    De Quan Qu
    ------------------------------

    Original Message


  • 23.  RE: Reboot AIX server with ansible playbook failed

    Posted Wed May 29, 2024 02:11 AM

    Is that a (semicolon) ; in front of the ssh_args line? If it is, try removing it.

    This is what I have in my /etc/ansible/ansible.cfg file. It works for me i.e. the AIX reboot works.

    [ssh_connection]
    ssh_args = -o ForwardAgent=yes -o ControlPersist=30m -o ServerAliveInterval=45 -o ServerAliveCountMax=10



    ------------------------------
    Chris Gibson
    ------------------------------

    Original Message


  • 24.  RE: Reboot AIX server with ansible playbook failed

    Posted Wed May 29, 2024 02:28 AM

    The same goes for removing the semicolon



    ------------------------------
    De Quan Qu
    ------------------------------

    Original Message


  • 25.  RE: Reboot AIX server with ansible playbook failed

    Posted Wed May 29, 2024 02:33 AM

    Can you share your ansible.cfg file?



    ------------------------------
    Chris Gibson
    ------------------------------

    Original Message


  • 26.  RE: Reboot AIX server with ansible playbook failed

    Posted Wed May 29, 2024 02:44 AM
      |   view attached

    yes, has been upload attachment



    ------------------------------
    De Quan Qu
    ------------------------------

    Attachment(s)

    cfg
    ansible.cfg   38 KB 1 version
    Original Message


  • 27.  RE: Reboot AIX server with ansible playbook failed

    Posted Wed May 29, 2024 03:30 AM
    Edited by Chris Gibson Wed May 29, 2024 03:30 AM

    OK, try removing your current ssh_args entries and placing them under the [defaults] section, at the very top of the ansible.cfg file. Please remove the semicolon. For example:

    [defaults]
    # (boolean) By default Ansible will issue a warning when received from a task action (module or action plugin)
    # These warnings can be silenced by adjusting this setting to False.
    ;action_warnings=True

    host_key_checking = False
    #interpreter_python = /usr/bin/python3
    interpreter_python = /opt/freeware/bin/python3

    [ssh_connection]
    ssh_args = -o ForwardAgent=yes -o ControlPersist=30m -o ServerAliveInterval=45 -o ServerAliveCountMax=10



    ------------------------------
    Chris Gibson
    ------------------------------

    Original Message


  • 28.  RE: Reboot AIX server with ansible playbook failed

    Posted Wed May 29, 2024 04:04 AM

    Wow, great. I moved to default the way you did it. reboot is now in effect. Thank you very much for your help.



    ------------------------------
    De Quan Qu
    ------------------------------

    Original Message