IBM Z and LinuxONE IBM Z

While trying to list APF Datasets, I am getting this error .

Hi,

It works for me if I set the environment like in the example below. However you need to have the PTF installed which provides support the PYTHONSTDINENCODING variable. See here for more details --> https://community.ibm.com/community/user/ibmz-and-linuxone/blogs/demetrios-dimatos1/2023/04/21/ibm-ansible-zos-core-150-has-released

---
- name: List APF
  hosts: all
  gather_facts: False

  vars:
    PYZ: /usr/lpp/python
    ZOAU: /usr/lpp/zoau

    environment_vars:
      _BPXK_AUTOCVT: "ON"
      ZOAU_HOME: "{{ ZOAU }}"
      PYTHONPATH: "{{ ZOAU }}/lib"
      LIBPATH: "{{ ZOAU }}/lib:{{ PYZ }}/lib:/lib:/usr/lib:."
      PATH: "{{ ZOAU }}/bin:{{ PYZ }}/bin:/bin:/var/bin"
      _CEE_RUNOPTS: "FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
      _TAG_REDIR_ERR: "txt"
      _TAG_REDIR_IN: "txt"
      _TAG_REDIR_OUT: "txt"
      LANG: "C"
      PYTHONSTDINENCODING: "cp1047"

  environment: "{{ environment_vars }}"

  tasks:
  - name: List APF
    ibm.ibm_zos_core.zos_apf:
      operation: list
    register: myapf

...

While trying to list APF Datasets, I am getting this error .

Hi Oliver,

Thanks for your response but still receiving the same error.

This is my playbook

zos_ping works .

zos_apf fails with same error as reported earlier

TASK [List APF] ******************************************************************************************************************************************************************************************************************
task path: /root/myenv/newt.yml:25
Using module file /root/.ansible/collections/ansible_collections/ibm/ibm_zos_core/plugins/modules/zos_apf.py
Pipelining is enabled.
<db2a.xxx.com> ESTABLISH SSH CONNECTION FOR USER: mvsmkj
<db2a.xxx.com> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o 'User="mvsmkj"' -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/0494b81ceb"' db2a.xxx.com '/bin/sh -c '"'"'_BPXK_AUTOCVT=ON ZOAU_HOME=/shrd/zoau1.2.4 PYTHONPATH=/shrd/zoau1.2.4/lib LIBPATH=/shrd/zoau1.2.4/lib:/shrd/cyp/usr/lpp/IBM/cyp/v3r10/pyz/lib:/lib:/usr/lib:. PATH=/shrd/zoau1.2.4/bin:/shrd/cyp/usr/lpp/IBM/cyp/v3r10/pyz/bin:/bin:/var/bin PYTHONSTDINENCODING=cp1047 /shrd/cyp/usr/lpp/IBM/cyp/v3r10/pyz/bin/python3 && sleep 0'"'"''
<db2a.xxx.com> (1, b'', b"SyntaxError: Non-UTF-8 code starting with '\\xa2' in file <stdin> on line 2, but no encoding declared; see https://python.org/dev/peps/pep-0263/ for details\n")
<db2a.xxx.com> Failed to connect to the host via ssh: SyntaxError: Non-UTF-8 code starting with '\xa2' in file <stdin> on line 2, but no encoding declared; see https://python.org/dev/peps/pep-0263/ for details
fatal: [db2a.xxx.com]: FAILED! => {
    "changed": false,
    "module_stderr": "SyntaxError: Non-UTF-8 code starting with '\\xa2' in file <stdin> on line 2, but no encoding declared; see https://python.org/dev/peps/pep-0263/ for details\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

PLAY RECAP ***********************************************************************************************************************************************************************************************************************
db2a.xxx.com               : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

configuration changed since I first reported this

zoaversion is 1.2.4  earlier it was 1.2.1

python 3.10 (on z/os) 

Ansible Version

ansible             8.3.0
ansible-core        2.15.3

Collection                    Version
----------------------------- -------
ibm.ibm_zos_core              1.6.0  

From sysprog got to know Python on zos is not SMPE maintained , it is received pax.z form so could not list applied maintenance . 

If I run same play as exercise on zos xplorer it works , there the zo au version 1.2 and python is 3.9 .

Any suggestions will be helpful.

Regards,

Manoj K Jadwani

------------------------------
Manoj Jadwani
Product Manager
BMC Software

Original Message:
Sent: Fri September 08, 2023 02:17 AM
From: Oliver Stadler
Subject: Ansible zos core collection module zos_apf failure .

Hi,

It works for me if I set the environment like in the example below. However you need to have the PTF installed which provides support the PYTHONSTDINENCODING variable. See here for more details --> https://community.ibm.com/community/user/ibmz-and-linuxone/blogs/demetrios-dimatos1/2023/04/21/ibm-ansible-zos-core-150-has-released

---
- name: List APF
  hosts: all
  gather_facts: False

  vars:
    PYZ: /usr/lpp/python
    ZOAU: /usr/lpp/zoau

    environment_vars:
      _BPXK_AUTOCVT: "ON"
      ZOAU_HOME: "{{ ZOAU }}"
      PYTHONPATH: "{{ ZOAU }}/lib"
      LIBPATH: "{{ ZOAU }}/lib:{{ PYZ }}/lib:/lib:/usr/lib:."
      PATH: "{{ ZOAU }}/bin:{{ PYZ }}/bin:/bin:/var/bin"
      _CEE_RUNOPTS: "FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
      _TAG_REDIR_ERR: "txt"
      _TAG_REDIR_IN: "txt"
      _TAG_REDIR_OUT: "txt"
      LANG: "C"
      PYTHONSTDINENCODING: "cp1047"

  environment: "{{ environment_vars }}"

  tasks:
  - name: List APF
    ibm.ibm_zos_core.zos_apf:
      operation: list
    register: myapf

...

While trying to list APF Datasets, I am getting this error .

Hi Manoj, this appears this will require diagnosis, if you are able to, I suggest you engage the community who maintains the modules; you can start with a discussion or open an bug issue. The community has tooling that allows for bugs and discussions to be tracked, this forum is better for non diagnostic questions. Git issues and discussions are triaged weekly and then identified when they can be addressed, otherwise there is S&S available through Ansible Automation Platform. 

Back to your question, if you review the community issues, you will see these 3 issues touching on the same problem which is your query to `list` is probably returning a character outside of the UTF-8 encoding meaning Python can't not understand what that character is position 2. 

See:

• [Bug] [zos_job_submit] 'utf-8' codec can't decode byte 0xdd error on zos_job_submit module #677
• [Bug] zos_operator_action_query reported UnicodeDecodeError when the outstanding messages contain special characters #776
• 'utf-8' codec can't decode byte 0xbb in position 0: invalid start byte" #972

See also recent update to known issues:
known_issues:
   - Several modules have reported UTF8 decoding errors when interacting with results
     that contain non-printable UTF8 characters in the response. This occurs when
     a module receives content that does not correspond to a UTF-8 value.
     These include modules `zos_job_submit`, `zos_job_output`,
     `zos_operator_action_query` but are not limited to this list.
     This will be addressed in `ibm_zos_core` version 1.10.0-beta.1.
     Each case is unique, some options to work around the error are below.
     - Specify that the ASA assembler option be enabled to instruct the assembler
       to use ANSI control characters instead of machine code control characters.
     - Add `ignore_errors:true` to the playbook task so the task error will not
       fail the playbook.
     - If the error is resulting from a batch job, add `ignore_errors:true` to the
       task and capture the output into a variable and extract the job ID with a
       regular expression and then use `zos_job_output` to display the DD without
       the non-printable character such as the DD `JESMSGLG`.
       (https://github.com/ansible-collections/ibm_zos_core/issues/677)
       (https://github.com/ansible-collections/ibm_zos_core/issues/776)
       (https://github.com/ansible-collections/ibm_zos_core/issues/972)
   - With later versions of `ansible-core` used with `ibm_zos_core` collection a
     warning has started to appear "Module "ansible.builtin.command" returned non UTF-8 data in the JSON response"
     that is currently being reviewed. There are no recommendations at this point.
     (https://github.com/ansible-collections/ibm_zos_core/issues/983)

Hi Oliver,

Thanks for your response but still receiving the same error.

This is my playbook

zos_ping works .

zos_apf fails with same error as reported earlier

TASK [List APF] ******************************************************************************************************************************************************************************************************************
task path: /root/myenv/newt.yml:25
Using module file /root/.ansible/collections/ansible_collections/ibm/ibm_zos_core/plugins/modules/zos_apf.py
Pipelining is enabled.
<db2a.xxx.com> ESTABLISH SSH CONNECTION FOR USER: mvsmkj
<db2a.xxx.com> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o 'User="mvsmkj"' -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/0494b81ceb"' db2a.xxx.com '/bin/sh -c '"'"'_BPXK_AUTOCVT=ON ZOAU_HOME=/shrd/zoau1.2.4 PYTHONPATH=/shrd/zoau1.2.4/lib LIBPATH=/shrd/zoau1.2.4/lib:/shrd/cyp/usr/lpp/IBM/cyp/v3r10/pyz/lib:/lib:/usr/lib:. PATH=/shrd/zoau1.2.4/bin:/shrd/cyp/usr/lpp/IBM/cyp/v3r10/pyz/bin:/bin:/var/bin PYTHONSTDINENCODING=cp1047 /shrd/cyp/usr/lpp/IBM/cyp/v3r10/pyz/bin/python3 && sleep 0'"'"''
<db2a.xxx.com> (1, b'', b"SyntaxError: Non-UTF-8 code starting with '\\xa2' in file <stdin> on line 2, but no encoding declared; see https://python.org/dev/peps/pep-0263/ for details\n")
<db2a.xxx.com> Failed to connect to the host via ssh: SyntaxError: Non-UTF-8 code starting with '\xa2' in file <stdin> on line 2, but no encoding declared; see https://python.org/dev/peps/pep-0263/ for details
fatal: [db2a.xxx.com]: FAILED! => {
    "changed": false,
    "module_stderr": "SyntaxError: Non-UTF-8 code starting with '\\xa2' in file <stdin> on line 2, but no encoding declared; see https://python.org/dev/peps/pep-0263/ for details\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

PLAY RECAP ***********************************************************************************************************************************************************************************************************************
db2a.xxx.com               : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

configuration changed since I first reported this

zoaversion is 1.2.4  earlier it was 1.2.1

python 3.10 (on z/os) 

Ansible Version

ansible             8.3.0
ansible-core        2.15.3

Collection                    Version
----------------------------- -------
ibm.ibm_zos_core              1.6.0  

From sysprog got to know Python on zos is not SMPE maintained , it is received pax.z form so could not list applied maintenance . 

If I run same play as exercise on zos xplorer it works , there the zo au version 1.2 and python is 3.9 .

Any suggestions will be helpful.

Regards,

Manoj K Jadwani

------------------------------
Manoj Jadwani
Product Manager
BMC Software

Original Message:
Sent: Fri September 08, 2023 02:17 AM
From: Oliver Stadler
Subject: Ansible zos core collection module zos_apf failure .

Hi,

It works for me if I set the environment like in the example below. However you need to have the PTF installed which provides support the PYTHONSTDINENCODING variable. See here for more details --> https://community.ibm.com/community/user/ibmz-and-linuxone/blogs/demetrios-dimatos1/2023/04/21/ibm-ansible-zos-core-150-has-released

---
- name: List APF
  hosts: all
  gather_facts: False

  vars:
    PYZ: /usr/lpp/python
    ZOAU: /usr/lpp/zoau

    environment_vars:
      _BPXK_AUTOCVT: "ON"
      ZOAU_HOME: "{{ ZOAU }}"
      PYTHONPATH: "{{ ZOAU }}/lib"
      LIBPATH: "{{ ZOAU }}/lib:{{ PYZ }}/lib:/lib:/usr/lib:."
      PATH: "{{ ZOAU }}/bin:{{ PYZ }}/bin:/bin:/var/bin"
      _CEE_RUNOPTS: "FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
      _TAG_REDIR_ERR: "txt"
      _TAG_REDIR_IN: "txt"
      _TAG_REDIR_OUT: "txt"
      LANG: "C"
      PYTHONSTDINENCODING: "cp1047"

  environment: "{{ environment_vars }}"

  tasks:
  - name: List APF
    ibm.ibm_zos_core.zos_apf:
      operation: list
    register: myapf

...

While trying to list APF Datasets, I am getting this error .

ThankYou Demetri , I was able to figure this out . It was due to mismatch of release of Python and zosAU running on our system. 

Once the compatibility was fixed was able to use the new PYTHONSTDINENCODING environment variable.

On the issue forum , I suggested idea also for this , there should be some form of verification script to be available when Python on z/OS is upgraded a compatibility should be checked each time with other pieces /library etc.  so that overall  Ansible for z/OS framework should not be broken.

Hi Manoj, this appears this will require diagnosis, if you are able to, I suggest you engage the community who maintains the modules; you can start with a discussion or open an bug issue. The community has tooling that allows for bugs and discussions to be tracked, this forum is better for non diagnostic questions. Git issues and discussions are triaged weekly and then identified when they can be addressed, otherwise there is S&S available through Ansible Automation Platform. 

Back to your question, if you review the community issues, you will see these 3 issues touching on the same problem which is your query to `list` is probably returning a character outside of the UTF-8 encoding meaning Python can't not understand what that character is position 2. 

See:

• [Bug] [zos_job_submit] 'utf-8' codec can't decode byte 0xdd error on zos_job_submit module #677
• [Bug] zos_operator_action_query reported UnicodeDecodeError when the outstanding messages contain special characters #776
• 'utf-8' codec can't decode byte 0xbb in position 0: invalid start byte" #972

See also recent update to known issues:
known_issues:
   - Several modules have reported UTF8 decoding errors when interacting with results
     that contain non-printable UTF8 characters in the response. This occurs when
     a module receives content that does not correspond to a UTF-8 value.
     These include modules `zos_job_submit`, `zos_job_output`,
     `zos_operator_action_query` but are not limited to this list.
     This will be addressed in `ibm_zos_core` version 1.10.0-beta.1.
     Each case is unique, some options to work around the error are below.
     - Specify that the ASA assembler option be enabled to instruct the assembler
       to use ANSI control characters instead of machine code control characters.
     - Add `ignore_errors:true` to the playbook task so the task error will not
       fail the playbook.
     - If the error is resulting from a batch job, add `ignore_errors:true` to the
       task and capture the output into a variable and extract the job ID with a
       regular expression and then use `zos_job_output` to display the DD without
       the non-printable character such as the DD `JESMSGLG`.
       (https://github.com/ansible-collections/ibm_zos_core/issues/677)
       (https://github.com/ansible-collections/ibm_zos_core/issues/776)
       (https://github.com/ansible-collections/ibm_zos_core/issues/972)
   - With later versions of `ansible-core` used with `ibm_zos_core` collection a
     warning has started to appear "Module "ansible.builtin.command" returned non UTF-8 data in the JSON response"
     that is currently being reviewed. There are no recommendations at this point.
     (https://github.com/ansible-collections/ibm_zos_core/issues/983)

Hi Oliver,

Thanks for your response but still receiving the same error.

This is my playbook

zos_ping works .

zos_apf fails with same error as reported earlier

TASK [List APF] ******************************************************************************************************************************************************************************************************************
task path: /root/myenv/newt.yml:25
Using module file /root/.ansible/collections/ansible_collections/ibm/ibm_zos_core/plugins/modules/zos_apf.py
Pipelining is enabled.
<db2a.xxx.com> ESTABLISH SSH CONNECTION FOR USER: mvsmkj
<db2a.xxx.com> SSH: EXEC sshpass -d12 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o 'User="mvsmkj"' -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/0494b81ceb"' db2a.xxx.com '/bin/sh -c '"'"'_BPXK_AUTOCVT=ON ZOAU_HOME=/shrd/zoau1.2.4 PYTHONPATH=/shrd/zoau1.2.4/lib LIBPATH=/shrd/zoau1.2.4/lib:/shrd/cyp/usr/lpp/IBM/cyp/v3r10/pyz/lib:/lib:/usr/lib:. PATH=/shrd/zoau1.2.4/bin:/shrd/cyp/usr/lpp/IBM/cyp/v3r10/pyz/bin:/bin:/var/bin PYTHONSTDINENCODING=cp1047 /shrd/cyp/usr/lpp/IBM/cyp/v3r10/pyz/bin/python3 && sleep 0'"'"''
<db2a.xxx.com> (1, b'', b"SyntaxError: Non-UTF-8 code starting with '\\xa2' in file <stdin> on line 2, but no encoding declared; see https://python.org/dev/peps/pep-0263/ for details\n")
<db2a.xxx.com> Failed to connect to the host via ssh: SyntaxError: Non-UTF-8 code starting with '\xa2' in file <stdin> on line 2, but no encoding declared; see https://python.org/dev/peps/pep-0263/ for details
fatal: [db2a.xxx.com]: FAILED! => {
    "changed": false,
    "module_stderr": "SyntaxError: Non-UTF-8 code starting with '\\xa2' in file <stdin> on line 2, but no encoding declared; see https://python.org/dev/peps/pep-0263/ for details\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

PLAY RECAP ***********************************************************************************************************************************************************************************************************************
db2a.xxx.com               : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

configuration changed since I first reported this

zoaversion is 1.2.4  earlier it was 1.2.1

python 3.10 (on z/os) 

Ansible Version

ansible             8.3.0
ansible-core        2.15.3

Collection                    Version
----------------------------- -------
ibm.ibm_zos_core              1.6.0  

From sysprog got to know Python on zos is not SMPE maintained , it is received pax.z form so could not list applied maintenance . 

If I run same play as exercise on zos xplorer it works , there the zo au version 1.2 and python is 3.9 .

Any suggestions will be helpful.

Regards,

Manoj K Jadwani

------------------------------
Manoj Jadwani
Product Manager
BMC Software

Original Message:
Sent: Fri September 08, 2023 02:17 AM
From: Oliver Stadler
Subject: Ansible zos core collection module zos_apf failure .

Hi,

It works for me if I set the environment like in the example below. However you need to have the PTF installed which provides support the PYTHONSTDINENCODING variable. See here for more details --> https://community.ibm.com/community/user/ibmz-and-linuxone/blogs/demetrios-dimatos1/2023/04/21/ibm-ansible-zos-core-150-has-released

---
- name: List APF
  hosts: all
  gather_facts: False

  vars:
    PYZ: /usr/lpp/python
    ZOAU: /usr/lpp/zoau

    environment_vars:
      _BPXK_AUTOCVT: "ON"
      ZOAU_HOME: "{{ ZOAU }}"
      PYTHONPATH: "{{ ZOAU }}/lib"
      LIBPATH: "{{ ZOAU }}/lib:{{ PYZ }}/lib:/lib:/usr/lib:."
      PATH: "{{ ZOAU }}/bin:{{ PYZ }}/bin:/bin:/var/bin"
      _CEE_RUNOPTS: "FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
      _TAG_REDIR_ERR: "txt"
      _TAG_REDIR_IN: "txt"
      _TAG_REDIR_OUT: "txt"
      LANG: "C"
      PYTHONSTDINENCODING: "cp1047"

  environment: "{{ environment_vars }}"

  tasks:
  - name: List APF
    ibm.ibm_zos_core.zos_apf:
      operation: list
    register: myapf

...

While trying to list APF Datasets, I am getting this error .