PowerVC

PowerVC

Connect, learn, share, and engage with IBM Power.

 View Only

  • 1.  PowerVC 2.0.3 upgrade failure

    Posted Sun July 03, 2022 03:01 PM

    I am updating our Dev PowerVC from 2.0.2.1 to 2.0.3 finally, but encountered some issues.

     

    Opsmgr update went OK

     

    ./update_opsmgr.sh

    2022-07-03T21:14:50.92+03:00 [INFO]:[INSTALL_POWERVC_OPSMGR] POWERVC OPSMGR PACKAGES : sshpass python3-netaddr rsync python3-oslo-i18n python3-powervc-opsmgr powervc-opsmgr

    2022-07-03T21:15:15.13+03:00 [INFO]:[INSTALL_POWERVC_OPSMGR] PowerVC Opsmgr packages installed successfully.

     

     

    But then the actual PowerVC upgrade fails to quite strange error:

     

    powervc-opsmgr update -c powervc-dev -n power-pvc-d05

     

    Sunday 03 July 2022  21:54:25 +0300 (0:00:00.080)       0:00:09.879 *********** 

    fatal: [10.241.93.172]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 102, in <module>\n  File \"<stdin>\", line 94, in _ansiballz_main\n  File \"<stdin>\", line 40, in invoke_module\n  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_process_password_dict_payload_4funbfg4/ansible_process_password_dict_payload.zip/ansible/modules/process_password_dict.py\", line 79, in <module>\n  File \"/tmp/ansible_process_password_dict_payload_4funbfg4/ansible_process_password_dict_payload.zip/ansible/modules/process_password_dict.py\", line 75, in main\n  File \"/tmp/ansible_process_password_dict_payload_4funbfg4/ansible_process_password_dict_payload.zip/ansible/modules/process_password_dict.py\", line 71, in process_password_dict_module\n  File \"/tmp/ansible_process_password_dict_payload_4funbfg4/ansible_process_password_dict_payload.zip/ansible/module_utils/basic.py\", line 2175, in exit_json\n  File \"/tmp/ansible_process_password_dict_payload_4funbfg4/ansible_process_password_dict_payload.zip/ansible/module_utils/basic.py\", line 2168, in _return_formatted\n  File \"/tmp/ansible_process_password_dict_payload_4funbfg4/ansible_process_password_dict_payload.zip/ansible/module_utils/basic.py\", line 426, in remove_values\n  File \"/tmp/ansible_process_password_dict_payload_4funbfg4/ansible_process_password_dict_payload.zip/ansible/module_utils/basic.py\", line 403, in _remove_values_conditions\nTypeError: Value of unknown type: <class 'UnicodeDecodeError'>, 'utf-8' codec can't decode byte 0x9b in position 0: invalid start byte\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr f

    TASK [pvc_utils : decrypt strings in input_password_dict] **********************

    An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: Value of unknown type: <class 'UnicodeDecodeError'>, 'utf-8' codec can't decode byte 0x9b in position 0: invalid start byte

    fatal: [10.241.93.172]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 102, in <module>\n  File \"<stdin>\", line 94, in _ansiballz_main\n  File \"<stdin>\", line 40, in invoke_module\n  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_process_password_dict_payload_4funbfg4/ansible_process_password_dict_payload.zip/ansible/modules/process_password_dict.py\", line 79, in <module>\n  File \"/tmp/ansible_process_password_dict_payload_4funbfg4/ansible_process_password_dict_payload.zip/ansible/modules/process_password_dict.py\", line 75, in main\n  File \"/tmp/ansible_process_password_dict_payload_4funbfg4/ansible_process_password_dict_payload.zip/ansible/modules/process_password_dict.py\", line 71, in process_password_dict_module\n  File \"/tmp/ansible_process_password_dict_payload_4funbfg4/ansible_process_password_dict_payload.zip/ansible/module_utils/basic.py\", line 2175, in exit_json\n  File \"/tmp/ansible_process_password_dict_payload_4funbfg4/ansible_process_password_dict_payload.zip/ansible/module_utils/basic.py\", line 2168, in _return_formatted\n  File \"/tmp/ansible_process_password_dict_payload_4funbfg4/ansible_process_password_dict_payload.zip/ansible/module_utils/basic.py\", line 426, in remove_values\n  File \"/tmp/ansible_process_password_dict_payload_4funbfg4/ansible_process_password_dict_payload.zip/ansible/module_utils/basic.py\", line 403, in _remove_values_conditions\nTypeError: Value of unknown type: <class 'UnicodeDecodeError'>, 'utf-8' codec can't decode byte 0x9b in position 0: invalid start byte\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

    Sunday 03 July 2022  21:54:29 +0300 (0:00:03.729)       0:00:13.609 *********** 

    Sunday 03 July 2022  21:54:29 +0300 (0:00:00.070)       0:00:13.679 *********** 

     Anyone encountering same? 

    Br,
    tommi



    ------------------------------
    Tommi Sihvo, Lead Service Architect
    TietoEVRY, Compute Services
    email tommi.sihvo@tietoevry.com mobile +358 (0)40 5180 Finland
    ------------------------------


  • 2.  RE: PowerVC 2.0.3 upgrade failure

    Posted Wed July 06, 2022 01:17 AM
    Please try below steps on the node where update is being triggered. 


    Step 1:
    cp /opt/ibm/powervc_opsmgr/ansible/inventory/<clustername>/facts.d/powervc.fact /opt/ibm/powervc_opsmgr/ansible/inventory/<clustername>/facts.d/powervc.fact_backup

    Step 2:
    Remove following passwords from /opt/ibm/powervc_opsmgr/ansible/inventory/<clustername>/facts.d/powervc.fact 
    'keystone_service_user_password'
    'ceilometer'
     'swift'
     'ttv' # TTV
      ttv_service_user_password'
     'ttv-validation'

    Step 3:
    Retry update powervc-opsmgr update -c <clustername> -n <node_ip>

    ------------------------------
    Vijayendra Radhakrishna
    ------------------------------

    Original Message


  • 3.  RE: PowerVC 2.0.3 upgrade failure

    Posted Wed July 06, 2022 02:32 AM
    Hi,

    Many Thanks for the tip; The upgrade continued OK after performing the steps.

    However, it ended to failure after a while for 2nd error: :(

    TASK [{{ pvc_backup_task_names.create_dbdump_directory }}] *********************

    changed: [10.241.93.172]

    Wednesday 06 July 2022  09:26:19 +0300 (0:00:00.425)       0:04:13.036 ******** 

    fatal: [10.241.93.172]: FAILED! => {"changed": false, "msg": "unable to find /root/.my.cnf. Exception message: (2003, \"Can't connect to MySQL server on 'localhost' ([Errno 2] No such file or directory)\")"}

    TASK [{{ pvc_backup_task_names.take_mysqldump_for_backup }}] *******************

    fatal: [10.241.93.172]: FAILED! => {"changed": false, "msg": "unable to find /root/.my.cnf. Exception message: (2003, \"Can't connect to MySQL server on 'localhost' ([Errno 2] No such file or directory)\")"}

    Wednesday 06 July 2022  09:26:19 +0300 (0:00:00.461)       0:04:13.498 ******** 

    Wednesday 06 July 2022  09:26:19 +0300 (0:00:00.103)       0:04:13.601 ******** 


    Any idea how to continue from this error? 

    Thanks,
    Br,
    tommi

    ------------------------------
    Tommi Sihvo, Lead Service Architect
    TietoEVRY, Compute Services
    email tommi.sihvo@tietoevry.com mobile +358 (0)40 5180 Finland
    ------------------------------

    Original Message


  • 4.  RE: PowerVC 2.0.3 upgrade failure

    Posted Wed July 06, 2022 02:58 AM
    Can you please retry update one more time see if it goes fine?

    Please share the crm_mon -1 output if possible

    ------------------------------
    Vijayendra Radhakrishna
    ------------------------------

    Original Message


  • 5.  RE: PowerVC 2.0.3 upgrade failure

    Posted Wed July 06, 2022 03:02 AM
    Hello,

    cp /etc/my.cnf /root/.my.cnf < Helped on the 2nd error ..now update is continuing OK

    However, I started thinking that symbolic link might be safer (thinking about the next update, in case /root/.my.cnf file is not updated properly to be intact with /etc/my.cnf ?

    ------------------------------
    Tommi Sihvo, Lead Service Architect
    TietoEVRY, Compute Services
    email tommi.sihvo@tietoevry.com mobile +358 (0)40 5180 Finland
    ------------------------------

    Original Message


  • 6.  RE: PowerVC 2.0.3 upgrade failure

    Posted Wed July 06, 2022 03:50 AM
    above symbolic link should be fine.

    ------------------------------
    Vijayendra Radhakrishna
    ------------------------------

    Original Message


  • 7.  RE: PowerVC 2.0.3 upgrade failure

    Posted Wed July 06, 2022 07:45 AM
    Hi,

    Proceeded from the previous error with the file copy / symbolic link...but entered next error; MariaDB not starting properly.

    Jul 06 11:19:54 power-pvc-d05.tieto.com systemd[1]: Starting MariaDB 10.5 database server...

    Jul 06 11:19:54 power-pvc-d05.tieto.com mariadb-check-socket[2360146]: Socket file /var/lib/mysql/mysql.sock exists.

    Jul 06 11:19:54 power-pvc-d05.tieto.com mariadb-check-socket[2360146]: Is another MariaDB daemon already running with the same unix socket?

    Jul 06 11:19:54 power-pvc-d05.tieto.com mariadb-check-socket[2360146]: Please, stop the process using the socket /var/lib/mysql/mysql.sock or remove the file manually to start the service.

    Jul 06 11:19:54 power-pvc-d05.tieto.com systemd[1]: mariadb.service: Control process exited, code=exited status=1

    Jul 06 11:19:54 power-pvc-d05.tieto.com systemd[1]: mariadb.service: Failed with result 'exit-code'.

    Jul 06 11:19:54 power-pvc-d05.tieto.com systemd[1]: Failed to start MariaDB 10.5 database server.

    Some processes seem to prevent the start: 

    root@power-pvc-d05 ~]# ps -ef |grep -i maria

    root     2347413       1  0 11:13 ?        00:00:00 runuser - mysql -s /bin/sh -c /usr/bin/mysqld_safe --defaults-file=/etc/my.cnf     --pid-file=/var/lib/mysql/mysql.pid     --socket=/var/lib/mysql/mysql.sock     --datadir=/var/opt/ibm/powervc/db     --log-error=/var/log/mariadb/mariadb.log     --open-files-limit=655350     --wsrep-cluster-address=gcomm:// >/dev/null 2>&1

    mysql    2347427 2347413  0 11:13 ?        00:00:00 -sh -c /usr/bin/mysqld_safe --defaults-file=/etc/my.cnf     --pid-file=/var/lib/mysql/mysql.pid     --socket=/var/lib/mysql/mysql.sock     --datadir=/var/opt/ibm/powervc/db     --log-error=/var/log/mariadb/mariadb.log     --open-files-limit=655350     --wsrep-cluster-address=gcomm:// >/dev/null 2>&1

    mysql    2347442 2347427  0 11:13 ?        00:00:00 /bin/sh /usr/bin/mysqld_safe --defaults-file=/etc/my.cnf --pid-file=/var/lib/mysql/mysql.pid --socket=/var/lib/mysql/mysql.sock --datadir=/var/opt/ibm/powervc/db --log-error=/var/log/mariadb/mariadb.log --open-files-limit=655350 --wsrep-cluster-address=gcomm://

    mysql    2347831 2347442  0 11:13 ?        00:00:00 /usr/libexec/mariadbd --defaults-file=/etc/my.cnf --basedir=/usr --datadir=/var/opt/ibm/powervc/db --plugin-dir=/usr/lib64/mariadb/plugin --wsrep_on=0 --wsrep_provider=/usr/lib64/galera/libgalera_smm.so --wsrep_on=ON --wsrep_provider=/usr/lib64/galera/libgalera_smm.so --wsrep-cluster-address=gcomm:// --log-error=/var/log/mariadb/mariadb.log --open-files-limit=655350 --pid-file=/var/lib/mysql/mysql.pid --socket=/var/lib/mysql/mysql.sock --port=50110 --wsrep_start_position=f3724054-5a77-11ec-9c23-5f002e7233c1:9047295

    root     2365270 2040591  0 11:22 pts/0    00:00:00 grep --color=auto -i maria

     
    Updated the Support case with new findings + fresh PowerVC Diag logs.



    ------------------------------
    Tommi Sihvo, Lead Service Architect
    TietoEVRY, Compute Services
    email tommi.sihvo@tietoevry.com mobile +358 (0)40 5180 Finland
    ------------------------------

    Original Message


Related Content