IBM Z and LinuxONE IBM Z

Hello Oliver,
Looking at this snippet and from what you have shared in that you have from:to set to IBM-1047

  - name: Copy JCL without encoding    ibm.ibm_zos_core.zos_copy:      src: iefbr14.jcl      dest: "{{ my_seq }}"      backup: False      force: True

It seems that the JCL source `iefbr14.jcl` on the controller (eg your laptop) is encoded in EBCDIC (IBM-1047) so when you don't specify an encoding what the `zos_copy` module does is take the `locale`of the controller which in most cases is either `UTF8` or `ISO8859-1` and uses that as the `from` and when the module runs on the managed node (z/OS) it takes the `locale` there and sets it as the `to` which in most cases is `IBM-1047`.

Why `iefbr14.jcl` ends up unreadable is because the module has used incorrect inputs/oputput (from/to) for its encoding when you don't specify. We rely on these user inputs because its very difficult to detect and correctly asses a files encoding reading a files Byte Order Mark (BOM) and has a high probability of inaccuracy in doing so.

In your case you have a few options, if the file is already in the desired encoding on your controller eg IBM-1047, you can continue to set the FROM an TO encoding options as IBM-1047 for both so that the  module does not perform any encoding conversion or you can instruct the module the file is binary and should prevent any conversion but I don't recommend you use the binary trick. Your best option is to have the file on the controller in the controllers encoding or continue to do what you are doing.

In this article under section "Plugins and modules" I cover the locale and architecture a bit if you are interested.
We try to monitor all sources and this one was overlooked, feel free to also use our repositories discussions if you are permitted to access it.

Hello Oliver,
Looking at this snippet and from what you have shared in that you have from:to set to IBM-1047

  - name: Copy JCL without encoding    ibm.ibm_zos_core.zos_copy:      src: iefbr14.jcl      dest: "{{ my_seq }}"      backup: False      force: True

It seems that the JCL source `iefbr14.jcl` on the controller (eg your laptop) is encoded in EBCDIC (IBM-1047) so when you don't specify an encoding what the `zos_copy` module does is take the `locale`of the controller which in most cases is either `UTF8` or `ISO8859-1` and uses that as the `from` and when the module runs on the managed node (z/OS) it takes the `locale` there and sets it as the `to` which in most cases is `IBM-1047`.

Why `iefbr14.jcl` ends up unreadable is because the module has used incorrect inputs/oputput (from/to) for its encoding when you don't specify. We rely on these user inputs because its very difficult to detect and correctly asses a files encoding reading a files Byte Order Mark (BOM) and has a high probability of inaccuracy in doing so.

In your case you have a few options, if the file is already in the desired encoding on your controller eg IBM-1047, you can continue to set the FROM an TO encoding options as IBM-1047 for both so that the  module does not perform any encoding conversion or you can instruct the module the file is binary and should prevent any conversion but I don't recommend you use the binary trick. Your best option is to have the file on the controller in the controllers encoding or continue to do what you are doing.

In this article under section "Plugins and modules" I cover the locale and architecture a bit if you are interested.
We try to monitor all sources and this one was overlooked, feel free to also use our repositories discussions if you are permitted to access it.

ANSIBLE_DEBUG=1 ansible-playbook -i inventory your-playbook.yml -vvvv​

I have seen this happen when the controller can't find the action plugin (zos_copy has both , action and modules - see my blog for more on this), and I can usually tell from the log when I see a None/Null passed in as the controllers locale.  You can trouble shoot this by adding the path in your `ansible.cfg` , the entry would look similar to this:

[defaults]action_plugins=/Users/ddimatos/.ansible/collections/ansible_collections/ibm/ibm_zos_core/plugins/action

The other time, but its slight a different error (we are working on this) is with IBM Enterprise Python 3.10 or later but you are using 3.9 according to what Ansible discovered. You might try to not rely on that discovery and tell Ansible what python to use with property `ansible_python_interpreter`, as I can't tell without a log if it really did use 3.9, you can do that from your inventory with an entry that looks like:

zsystem:  hosts:    zvm:      ansible_host: <some host>      ansible_user: <some user>      ansible_python_interpreter: /python2/usr/lpp/IBM/cyp/v3r9/pyz/bin/python3.9

Hello Oliver,
Looking at this snippet and from what you have shared in that you have from:to set to IBM-1047

  - name: Copy JCL without encoding    ibm.ibm_zos_core.zos_copy:      src: iefbr14.jcl      dest: "{{ my_seq }}"      backup: False      force: True

It seems that the JCL source `iefbr14.jcl` on the controller (eg your laptop) is encoded in EBCDIC (IBM-1047) so when you don't specify an encoding what the `zos_copy` module does is take the `locale`of the controller which in most cases is either `UTF8` or `ISO8859-1` and uses that as the `from` and when the module runs on the managed node (z/OS) it takes the `locale` there and sets it as the `to` which in most cases is `IBM-1047`.

Why `iefbr14.jcl` ends up unreadable is because the module has used incorrect inputs/oputput (from/to) for its encoding when you don't specify. We rely on these user inputs because its very difficult to detect and correctly asses a files encoding reading a files Byte Order Mark (BOM) and has a high probability of inaccuracy in doing so.

In your case you have a few options, if the file is already in the desired encoding on your controller eg IBM-1047, you can continue to set the FROM an TO encoding options as IBM-1047 for both so that the  module does not perform any encoding conversion or you can instruct the module the file is binary and should prevent any conversion but I don't recommend you use the binary trick. Your best option is to have the file on the controller in the controllers encoding or continue to do what you are doing.

In this article under section "Plugins and modules" I cover the locale and architecture a bit if you are interested.
We try to monitor all sources and this one was overlooked, feel free to also use our repositories discussions if you are permitted to access it.

ANSIBLE_DEBUG=1 ansible-playbook -i inventory your-playbook.yml -vvvv​

I have seen this happen when the controller can't find the action plugin (zos_copy has both , action and modules - see my blog for more on this), and I can usually tell from the log when I see a None/Null passed in as the controllers locale.  You can trouble shoot this by adding the path in your `ansible.cfg` , the entry would look similar to this:

[defaults]action_plugins=/Users/ddimatos/.ansible/collections/ansible_collections/ibm/ibm_zos_core/plugins/action

The other time, but its slight a different error (we are working on this) is with IBM Enterprise Python 3.10 or later but you are using 3.9 according to what Ansible discovered. You might try to not rely on that discovery and tell Ansible what python to use with property `ansible_python_interpreter`, as I can't tell without a log if it really did use 3.9, you can do that from your inventory with an entry that looks like:

zsystem:  hosts:    zvm:      ansible_host: <some host>      ansible_user: <some user>      ansible_python_interpreter: /python2/usr/lpp/IBM/cyp/v3r9/pyz/bin/python3.9

Hello Oliver,
Looking at this snippet and from what you have shared in that you have from:to set to IBM-1047

  - name: Copy JCL without encoding    ibm.ibm_zos_core.zos_copy:      src: iefbr14.jcl      dest: "{{ my_seq }}"      backup: False      force: True

It seems that the JCL source `iefbr14.jcl` on the controller (eg your laptop) is encoded in EBCDIC (IBM-1047) so when you don't specify an encoding what the `zos_copy` module does is take the `locale`of the controller which in most cases is either `UTF8` or `ISO8859-1` and uses that as the `from` and when the module runs on the managed node (z/OS) it takes the `locale` there and sets it as the `to` which in most cases is `IBM-1047`.

Why `iefbr14.jcl` ends up unreadable is because the module has used incorrect inputs/oputput (from/to) for its encoding when you don't specify. We rely on these user inputs because its very difficult to detect and correctly asses a files encoding reading a files Byte Order Mark (BOM) and has a high probability of inaccuracy in doing so.

In your case you have a few options, if the file is already in the desired encoding on your controller eg IBM-1047, you can continue to set the FROM an TO encoding options as IBM-1047 for both so that the  module does not perform any encoding conversion or you can instruct the module the file is binary and should prevent any conversion but I don't recommend you use the binary trick. Your best option is to have the file on the controller in the controllers encoding or continue to do what you are doing.

In this article under section "Plugins and modules" I cover the locale and architecture a bit if you are interested.
We try to monitor all sources and this one was overlooked, feel free to also use our repositories discussions if you are permitted to access it.

ANSIBLE_DEBUG=1 ansible-playbook -i inventory your-playbook.yml -vvvv​

I have seen this happen when the controller can't find the action plugin (zos_copy has both , action and modules - see my blog for more on this), and I can usually tell from the log when I see a None/Null passed in as the controllers locale.  You can trouble shoot this by adding the path in your `ansible.cfg` , the entry would look similar to this:

[defaults]action_plugins=/Users/ddimatos/.ansible/collections/ansible_collections/ibm/ibm_zos_core/plugins/action

The other time, but its slight a different error (we are working on this) is with IBM Enterprise Python 3.10 or later but you are using 3.9 according to what Ansible discovered. You might try to not rely on that discovery and tell Ansible what python to use with property `ansible_python_interpreter`, as I can't tell without a log if it really did use 3.9, you can do that from your inventory with an entry that looks like:

zsystem:  hosts:    zvm:      ansible_host: <some host>      ansible_user: <some user>      ansible_python_interpreter: /python2/usr/lpp/IBM/cyp/v3r9/pyz/bin/python3.9

Hello Oliver,
Looking at this snippet and from what you have shared in that you have from:to set to IBM-1047

  - name: Copy JCL without encoding    ibm.ibm_zos_core.zos_copy:      src: iefbr14.jcl      dest: "{{ my_seq }}"      backup: False      force: True

It seems that the JCL source `iefbr14.jcl` on the controller (eg your laptop) is encoded in EBCDIC (IBM-1047) so when you don't specify an encoding what the `zos_copy` module does is take the `locale`of the controller which in most cases is either `UTF8` or `ISO8859-1` and uses that as the `from` and when the module runs on the managed node (z/OS) it takes the `locale` there and sets it as the `to` which in most cases is `IBM-1047`.

Why `iefbr14.jcl` ends up unreadable is because the module has used incorrect inputs/oputput (from/to) for its encoding when you don't specify. We rely on these user inputs because its very difficult to detect and correctly asses a files encoding reading a files Byte Order Mark (BOM) and has a high probability of inaccuracy in doing so.

In your case you have a few options, if the file is already in the desired encoding on your controller eg IBM-1047, you can continue to set the FROM an TO encoding options as IBM-1047 for both so that the  module does not perform any encoding conversion or you can instruct the module the file is binary and should prevent any conversion but I don't recommend you use the binary trick. Your best option is to have the file on the controller in the controllers encoding or continue to do what you are doing.

In this article under section "Plugins and modules" I cover the locale and architecture a bit if you are interested.
We try to monitor all sources and this one was overlooked, feel free to also use our repositories discussions if you are permitted to access it.

Hello Oliver, actually you had my name correct, full pronunciation is `Demetrios`, I updated it to the shortened `Demetri` the other day, no problems here on my end how you refer to me :) .

I did a quick pass and noticed something, it appears the code is either detecting IBM-037 or your setting it to IBM-037 (but I think you are setting it to IBM-1047) , i have to look a the code to follow the path and where the swap is happening.
- Which version of the IBM z/OS core collection are you using, eg

cat /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/MANIFEST.json | grep version

- What does USS command on the managed node (one you are connecting to) yield: `locale -c charmap`, eg I get:

# locale -c charmapIBM-1047

The log snippet that has me questioning the code is:

{"changed": true, "src": "iefbr14.jcl", "dest": "Tmyuser.TEST4.SEQ", "ds_type": "PS", "dest_exists": true, "backup_name": null, "invocation": {"module_args": {"is_pds": false, "src_member": false, "size": 194, "local_charset": "UTF-8", "dest": "Tmyuser.TEST4.SEQ", "is_src_dir": false, "is_uss": false, "temp_path": "/tmp/ansible-zos-copy-payload-D230206-T181246", "src": "iefbr14.jcl", "copy_member": false, "is_mvs_dest": true, "is_binary": false, "backup": false, "local_follow": true, "remote_src": false, "ignore_sftp_stderr": false, "validate": false, "force": false, "follow": false, "unsafe_writes": false, "encoding": {"from": "UTF-8", "to": "IBM-037"}, "content": null, "backup_name": null, "sftp_port": null, "volume": null, "dest_data_set": null, "mode": null, "owner": null, "group": null, "seuser": null, "serole": null, "selevel": null, "setype": null, "attributes": null, "regexp": null, "delimiter": null, "directory_mode": null, "from_encoding": "UTF-8", "to_encoding": "IBM-1047"}}}

The last thing that caught my eye which ill need some time to review is the plugin search path differs from your install path but it appears Ansible figured it out but still I want to look into, your install path is /home/tmyuser/.ansible/collections, Ansible searched paths `/usr/share/ansible/plugins/action:/usr/lib/python2.7/site-packages/ansible/plugins/action` and found the zos_copy action plugin, so I am not sure if the locale confusion has anything to do with this yet.

Loading ActionModule 'ansible_collections.ibm.ibm_zos_core.plugins.action.zos_copy' from /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action/zos_copy.py (searched paths: /usr/share/ansible/plugins/action:/usr/lib/python2.7/site-packages/ansible/plugins/action) (found_in_cache=False, class_only=True)

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Mon February 06, 2023 12:32 PM
From: Oliver Stadler
Subject: zos_copy module encoding question

And I just realized that I spelt your first name wrongly in my previous posts. Sorry for that but I got that now :-)

------------------------------
Oliver Stadler

Original Message:
Sent: Mon February 06, 2023 12:29 PM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetri,

I added the action_plugins path but it didn't make a difference.

Python version on the target server is 3.9. 

/usr/lpp/python/bin/python --version
Python 3.9.14

The config I used was as follows:

[tmyuser@controller ansible-zos]$ ansible-config dump --only-changed
ANSIBLE_PIPELINING(/home/tmyuser/ansible-zos/ansible.cfg) = True
COLLECTIONS_PATHS(/home/tmyuser/ansible-zos/ansible.cfg) = [u'/home/tmyuser/colltest', u'/usr/share/ansible/collections', u'/home/tmyuser/.ansible/collections', u'/usr/share/ansible/collections', u'/home/tmyuser/ansible_collections']
DEFAULT_ACTION_PLUGIN_PATH(/home/tmyuser/ansible-zos/ansible.cfg) = [u'/home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action/~/.ansible/plugins/action', u'/usr/share/ansible/plugins/action']

I also ran the playbook with the debugging enabled and verbose. Output is attached in playbook_output_clean.txt.

Kind regards,
Oliver

------------------------------
Oliver Stadler

Original Message:
Sent: Mon February 06, 2023 02:15 AM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hi Oliver,  I figured that was going to be the case with GitHub, your not the only client who has that policy, if you have Ansible Automation Hub entitlement you could go through L2, otherwise I don't mind working here.  For some reason I don't get notifications for this thread so I have to come back and review if there are updates.

Anyhow, thank you for the diagnosis, I guess I assumed too much without a log, there are a couple of things I can think of that might cause this but without the output from

ANSIBLE_DEBUG=1 ansible-playbook -i inventory your-playbook.yml -vvvv​

I have seen this happen when the controller can't find the action plugin (zos_copy has both , action and modules - see my blog for more on this), and I can usually tell from the log when I see a None/Null passed in as the controllers locale.  You can trouble shoot this by adding the path in your `ansible.cfg` , the entry would look similar to this:

[defaults]action_plugins=/Users/ddimatos/.ansible/collections/ansible_collections/ibm/ibm_zos_core/plugins/action

The other time, but its slight a different error (we are working on this) is with IBM Enterprise Python 3.10 or later but you are using 3.9 according to what Ansible discovered. You might try to not rely on that discovery and tell Ansible what python to use with property `ansible_python_interpreter`, as I can't tell without a log if it really did use 3.9, you can do that from your inventory with an entry that looks like:

zsystem:  hosts:    zvm:      ansible_host: <some host>      ansible_user: <some user>      ansible_python_interpreter: /python2/usr/lpp/IBM/cyp/v3r9/pyz/bin/python3.9

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Wed February 01, 2023 11:36 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetrios,

Thanks a lot for the feedback. I actually wanted to place the question directly in the Github discussion forum but unfortunately we are not allowed to due to company policy.

I checked the iefbr14.cjl source file on the controller and it is definitely not  EBCDIC (IBM-1047). 

$ file iefbr14.jcl
iefbr14.jcl: ASCII text

I also doublechecked the actual hex codes of some of the characters via the od command and they are in fact ASCII encoded.

When I look at the payload file which is copied during the execution of the task into /tmp (e.g. /tmp/ansible-zos-copy-payload-D230201-T155244) that looks fine (using -vvv to see the verbose output) . The file is not tagged and if I do a cat on it the file content is displayed correctly. So it looks like it gets converted later on. 

I wonder if we have something configured differently on Unix System Services side on our end that might cause this.

Kind regards,
Oliver



------------------------------
Oliver Stadler

Original Message:
Sent: Sun January 29, 2023 12:19 AM
From: DEMETRIOS DIMATOS
Subject: zos_copy module encoding question

Hello Oliver,
Looking at this snippet and from what you have shared in that you have from:to set to IBM-1047

  - name: Copy JCL without encoding    ibm.ibm_zos_core.zos_copy:      src: iefbr14.jcl      dest: "{{ my_seq }}"      backup: False      force: True

It seems that the JCL source `iefbr14.jcl` on the controller (eg your laptop) is encoded in EBCDIC (IBM-1047) so when you don't specify an encoding what the `zos_copy` module does is take the `locale`of the controller which in most cases is either `UTF8` or `ISO8859-1` and uses that as the `from` and when the module runs on the managed node (z/OS) it takes the `locale` there and sets it as the `to` which in most cases is `IBM-1047`.

Why `iefbr14.jcl` ends up unreadable is because the module has used incorrect inputs/oputput (from/to) for its encoding when you don't specify. We rely on these user inputs because its very difficult to detect and correctly asses a files encoding reading a files Byte Order Mark (BOM) and has a high probability of inaccuracy in doing so.

In your case you have a few options, if the file is already in the desired encoding on your controller eg IBM-1047, you can continue to set the FROM an TO encoding options as IBM-1047 for both so that the  module does not perform any encoding conversion or you can instruct the module the file is binary and should prevent any conversion but I don't recommend you use the binary trick. Your best option is to have the file on the controller in the controllers encoding or continue to do what you are doing.

In this article under section "Plugins and modules" I cover the locale and architecture a bit if you are interested.
We try to monitor all sources and this one was overlooked, feel free to also use our repositories discussions if you are permitted to access it.

------------------------------
DEMETRIOS DIMATOS
IBM zOS Ansible Core Senior Technical Lead
IBM zOS Ansible Core Senior Technical Lead
San Jose CA

Original Message:
Sent: Fri January 06, 2023 10:42 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi,

I have another encoding related question. When I copy an ASCII file using the zos_copy module it works fine if I specify IBM-1047 both in the from and to argument. If I don't specify an encoding the file arrives corrupted. Specifying the encoding it no big deal but it's somewhat confusing that I have to specify IBM-1047 as "from" encoding when copying an ASCII file.

This is the file I use as input
(az1) [myuser@myhost ansible-zos]$ cat iefbr14.jcl
//MYUSERA JOB (123),'LOCAL',CLASS=A,MSGCLASS=S,MSGLEVEL=(1,1),
// NOTIFY=&SYSUID
//*
//STEP001 EXEC PGM=IEFBR14
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSDUMP DD SYSOUT=*
//*

The following playbook shows the behaviour.

(az1) [myuser@myhost ansible-zos]$ cat zos_copy_example.yml
---
- name: Working with datasets
  hosts: all
  gather_facts: False
  vars:
  - my_seq: MYUSER.TEST3.SEQ

  tasks:
  - name: Create {{ my_seq }}
    ibm.ibm_zos_core.zos_data_set:
      name: "{{ my_seq }}"
      type: seq
      record_format: VB
      record_length: 255

  - name: Copy JCL with encoding
    ibm.ibm_zos_core.zos_copy:
      src: iefbr14.jcl
      dest: "{{ my_seq }}"
      backup: False
      force: True
    encoding:
      to: IBM-1047
      from: IBM-1047

  - name: Submit the job
    ibm.ibm_zos_core.zos_job_submit:
      location: DATA_SET
      src: "{{ my_seq }}"

  - name: Copy JCL without encoding
    ibm.ibm_zos_core.zos_copy:
      src: iefbr14.jcl
      dest: "{{ my_seq }}"
      backup: False
      force: True

  - name: Submit the job
    ibm.ibm_zos_core.zos_job_submit:
      location: DATA_SET
      src: "{{ my_seq }}"
...

When I run it I get the following result.

(az1) [myuser@myhost ansible-zos]$ ansible-playbook -i inventory -l mysys zos_copy_example.yml
[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the controller starting with Ansible 2.12. Current version: 3.6.8 (default, Aug 13 2020, 07:46:32) [GCC 4.8.5 20150623 (Red Hat
4.8.5-39)]. This feature will be removed from ansible-core in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [Working with datasets] ************************************************************************************************************************************************************************

TASK [Create MYUSER.TEST3.SEQ] *********************************************************************************************************************************************************************
[WARNING]: Platform os/390 on host mysys is using the discovered Python interpreter at /usr/lpp/python/bin/python3.9, but future installation of another Python interpreter could change
the meaning of that path. See https://docs.ansible.com/ansible-core/2.11/reference_appendices/interpreter_discovery.html for more information.
ok: [mysys]

TASK [Copy JCL with encoding] ***********************************************************************************************************************************************************************
changed: [mysys]

TASK [Submit the job] *******************************************************************************************************************************************************************************
changed: [mysys]

TASK [Copy JCL without encoding] ********************************************************************************************************************************************************************
changed: [mysys]

TASK [Submit the job] *******************************************************************************************************************************************************************************
fatal: [mysys]: FAILED! => {"changed": false, "msg": "SubmitJCLError('Submit job failed: Stderr :FSUMB408 submit: data set MYUSER.TEST3.SEQ Not accepted by JES.\\n')"}

PLAY RECAP ******************************************************************************************************************************************************************************************
mysys: ok=4 changed=3 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

------------------------------
Oliver Stadler
------------------------------

Hello Oliver, actually you had my name correct, full pronunciation is `Demetrios`, I updated it to the shortened `Demetri` the other day, no problems here on my end how you refer to me :) .

I did a quick pass and noticed something, it appears the code is either detecting IBM-037 or your setting it to IBM-037 (but I think you are setting it to IBM-1047) , i have to look a the code to follow the path and where the swap is happening.
- Which version of the IBM z/OS core collection are you using, eg

cat /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/MANIFEST.json | grep version

- What does USS command on the managed node (one you are connecting to) yield: `locale -c charmap`, eg I get:

# locale -c charmapIBM-1047

The log snippet that has me questioning the code is:

{"changed": true, "src": "iefbr14.jcl", "dest": "Tmyuser.TEST4.SEQ", "ds_type": "PS", "dest_exists": true, "backup_name": null, "invocation": {"module_args": {"is_pds": false, "src_member": false, "size": 194, "local_charset": "UTF-8", "dest": "Tmyuser.TEST4.SEQ", "is_src_dir": false, "is_uss": false, "temp_path": "/tmp/ansible-zos-copy-payload-D230206-T181246", "src": "iefbr14.jcl", "copy_member": false, "is_mvs_dest": true, "is_binary": false, "backup": false, "local_follow": true, "remote_src": false, "ignore_sftp_stderr": false, "validate": false, "force": false, "follow": false, "unsafe_writes": false, "encoding": {"from": "UTF-8", "to": "IBM-037"}, "content": null, "backup_name": null, "sftp_port": null, "volume": null, "dest_data_set": null, "mode": null, "owner": null, "group": null, "seuser": null, "serole": null, "selevel": null, "setype": null, "attributes": null, "regexp": null, "delimiter": null, "directory_mode": null, "from_encoding": "UTF-8", "to_encoding": "IBM-1047"}}}

The last thing that caught my eye which ill need some time to review is the plugin search path differs from your install path but it appears Ansible figured it out but still I want to look into, your install path is /home/tmyuser/.ansible/collections, Ansible searched paths `/usr/share/ansible/plugins/action:/usr/lib/python2.7/site-packages/ansible/plugins/action` and found the zos_copy action plugin, so I am not sure if the locale confusion has anything to do with this yet.

Loading ActionModule 'ansible_collections.ibm.ibm_zos_core.plugins.action.zos_copy' from /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action/zos_copy.py (searched paths: /usr/share/ansible/plugins/action:/usr/lib/python2.7/site-packages/ansible/plugins/action) (found_in_cache=False, class_only=True)

ANSIBLE_DEBUG=1 ansible-playbook -i inventory your-playbook.yml -vvvv​

I have seen this happen when the controller can't find the action plugin (zos_copy has both , action and modules - see my blog for more on this), and I can usually tell from the log when I see a None/Null passed in as the controllers locale.  You can trouble shoot this by adding the path in your `ansible.cfg` , the entry would look similar to this:

[defaults]action_plugins=/Users/ddimatos/.ansible/collections/ansible_collections/ibm/ibm_zos_core/plugins/action

The other time, but its slight a different error (we are working on this) is with IBM Enterprise Python 3.10 or later but you are using 3.9 according to what Ansible discovered. You might try to not rely on that discovery and tell Ansible what python to use with property `ansible_python_interpreter`, as I can't tell without a log if it really did use 3.9, you can do that from your inventory with an entry that looks like:

zsystem:  hosts:    zvm:      ansible_host: <some host>      ansible_user: <some user>      ansible_python_interpreter: /python2/usr/lpp/IBM/cyp/v3r9/pyz/bin/python3.9

Hello Oliver,
Looking at this snippet and from what you have shared in that you have from:to set to IBM-1047

  - name: Copy JCL without encoding    ibm.ibm_zos_core.zos_copy:      src: iefbr14.jcl      dest: "{{ my_seq }}"      backup: False      force: True

It seems that the JCL source `iefbr14.jcl` on the controller (eg your laptop) is encoded in EBCDIC (IBM-1047) so when you don't specify an encoding what the `zos_copy` module does is take the `locale`of the controller which in most cases is either `UTF8` or `ISO8859-1` and uses that as the `from` and when the module runs on the managed node (z/OS) it takes the `locale` there and sets it as the `to` which in most cases is `IBM-1047`.

Why `iefbr14.jcl` ends up unreadable is because the module has used incorrect inputs/oputput (from/to) for its encoding when you don't specify. We rely on these user inputs because its very difficult to detect and correctly asses a files encoding reading a files Byte Order Mark (BOM) and has a high probability of inaccuracy in doing so.

In your case you have a few options, if the file is already in the desired encoding on your controller eg IBM-1047, you can continue to set the FROM an TO encoding options as IBM-1047 for both so that the  module does not perform any encoding conversion or you can instruct the module the file is binary and should prevent any conversion but I don't recommend you use the binary trick. Your best option is to have the file on the controller in the controllers encoding or continue to do what you are doing.

In this article under section "Plugins and modules" I cover the locale and architecture a bit if you are interested.
We try to monitor all sources and this one was overlooked, feel free to also use our repositories discussions if you are permitted to access it.

Hello Oliver, it looks like things are good with regard to the plugin path and discovery from what I can see in your log. I found in our code where we are setting a `IBM-037`, I think it was an oversight given the slight variation in vars were `DEFAULT_EBCDIC_USS_CHARSET` vs `DEFAULT_EBCDIC_MVS_CHARSET`. 

I can't tell from your last message if its working for you now because it seems like you are sharing the expected HEX and the actual HEX which are the same and thus you are good now.

If you this is working for you then I will put the investigation of our use of `DEFAULT_EBCDIC_USS_CHARSET` in the backlog.

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Tue February 07, 2023 04:00 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetri,

zos collection version is 1.4

[tmyuser@mycontroller ansible-zos]$ cat /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/MANIFEST.json | grep version
"version": "1.4.0",

Locale on USS side is IBM-1047

[tmyuser@mycontroller ansible-zos]$ ansible -i inventory myplex1 -m command -a 'locale -c charmap'
[WARNING]: Platform os/390 on host mysys1.com is using the discovered Python interpreter at /usr/lpp/python/bin/python, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.
mysys1.com | CHANGED | rc=0 >>
IBM-1047

I noticed that there was actually a typo in the plugin path. I cleaned that up. Config looks like this now:

[myuser@mycontroller ansible-zos]$ ansible-config dump --only-changed
ANSIBLE_PIPELINING(/home/myuser/ansible-zos/ansible.cfg) = True
COLLECTIONS_PATHS(/home/myuser/ansible-zos/ansible.cfg) = [u'/home/myuser/colltest', u'/usr/share/ansible/collections']
DEFAULT_ACTION_PLUGIN_PATH(/home/myuser/ansible-zos/ansible.cfg) = [u'/home/myuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action', u'/usr/share/ansible/plugins/action']

I also rerun the playbook with this config and attached the output (playbook_output3.txt)

It looks like the actual conversion which is done is from ASCII to IBM-1047.

The target dataset in hex looks like this:

Command ===> Scroll ===> CSR
********************************* Top of Data **********************************


------------------------------------------------------------------------------
aa..@@..M.k}.Á.kÓb.k........~M.k.aa@@@@@@@......aa\.aa.....@...............aa...
88337732D39D3639E83933333333AD39388777777733333388E38833333733333333333333388333
11FFCCF04F20F5F2E2F2FFFFFFFF14F2D11CCCCCCCFFFFFD110D11FFFFFCFFFFFFFFFFFFFFD11FFF
------------------------------------------------------------------------------
******************************** Bottom of Data ********************************

That corresponds to the output I get when I run an iconv on the file being copied.
Tmyuser@TR01:/home/tmyuser ==> iconv -f UTF-8 -t IBM-1047 iefbr14.jcl | od -x
0000000000 8181 3F3F 7C7C 3F20 D43F 92D0 3F65 3F92
0000000020 EE82 3F92 3F3F 3F3F 3F3F 3F3F A1D4 3F92
0000000040 3D81 817C 7C7C 7C7C 7C7C 3F3F 3F3F 3F3D
0000000060 8181 E03D 8181 3F3F 3F3F 3F7C 3F3F 3F3F
0000000100 3F3F 3F3F 3F3F 3F3F 3F3F 3D81 813F 3F3F
0000000120 7C3F 7C3F 3F3F 3D81 813F 3F3F 7C7C 3F7C
0000000140 3F3F 3F3D 8181 E03D
0000000150



------------------------------
Oliver Stadler

Original Message:
Sent: Mon February 06, 2023 05:43 PM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hello Oliver, actually you had my name correct, full pronunciation is `Demetrios`, I updated it to the shortened `Demetri` the other day, no problems here on my end how you refer to me :) .

I did a quick pass and noticed something, it appears the code is either detecting IBM-037 or your setting it to IBM-037 (but I think you are setting it to IBM-1047) , i have to look a the code to follow the path and where the swap is happening.
- Which version of the IBM z/OS core collection are you using, eg

cat /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/MANIFEST.json | grep version

- What does USS command on the managed node (one you are connecting to) yield: `locale -c charmap`, eg I get:

# locale -c charmapIBM-1047

The log snippet that has me questioning the code is:

{"changed": true, "src": "iefbr14.jcl", "dest": "Tmyuser.TEST4.SEQ", "ds_type": "PS", "dest_exists": true, "backup_name": null, "invocation": {"module_args": {"is_pds": false, "src_member": false, "size": 194, "local_charset": "UTF-8", "dest": "Tmyuser.TEST4.SEQ", "is_src_dir": false, "is_uss": false, "temp_path": "/tmp/ansible-zos-copy-payload-D230206-T181246", "src": "iefbr14.jcl", "copy_member": false, "is_mvs_dest": true, "is_binary": false, "backup": false, "local_follow": true, "remote_src": false, "ignore_sftp_stderr": false, "validate": false, "force": false, "follow": false, "unsafe_writes": false, "encoding": {"from": "UTF-8", "to": "IBM-037"}, "content": null, "backup_name": null, "sftp_port": null, "volume": null, "dest_data_set": null, "mode": null, "owner": null, "group": null, "seuser": null, "serole": null, "selevel": null, "setype": null, "attributes": null, "regexp": null, "delimiter": null, "directory_mode": null, "from_encoding": "UTF-8", "to_encoding": "IBM-1047"}}}

The last thing that caught my eye which ill need some time to review is the plugin search path differs from your install path but it appears Ansible figured it out but still I want to look into, your install path is /home/tmyuser/.ansible/collections, Ansible searched paths `/usr/share/ansible/plugins/action:/usr/lib/python2.7/site-packages/ansible/plugins/action` and found the zos_copy action plugin, so I am not sure if the locale confusion has anything to do with this yet.

Loading ActionModule 'ansible_collections.ibm.ibm_zos_core.plugins.action.zos_copy' from /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action/zos_copy.py (searched paths: /usr/share/ansible/plugins/action:/usr/lib/python2.7/site-packages/ansible/plugins/action) (found_in_cache=False, class_only=True)

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Mon February 06, 2023 12:32 PM
From: Oliver Stadler
Subject: zos_copy module encoding question

And I just realized that I spelt your first name wrongly in my previous posts. Sorry for that but I got that now :-)

------------------------------
Oliver Stadler

Original Message:
Sent: Mon February 06, 2023 12:29 PM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetri,

I added the action_plugins path but it didn't make a difference.

Python version on the target server is 3.9. 

/usr/lpp/python/bin/python --version
Python 3.9.14

The config I used was as follows:

[tmyuser@controller ansible-zos]$ ansible-config dump --only-changed
ANSIBLE_PIPELINING(/home/tmyuser/ansible-zos/ansible.cfg) = True
COLLECTIONS_PATHS(/home/tmyuser/ansible-zos/ansible.cfg) = [u'/home/tmyuser/colltest', u'/usr/share/ansible/collections', u'/home/tmyuser/.ansible/collections', u'/usr/share/ansible/collections', u'/home/tmyuser/ansible_collections']
DEFAULT_ACTION_PLUGIN_PATH(/home/tmyuser/ansible-zos/ansible.cfg) = [u'/home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action/~/.ansible/plugins/action', u'/usr/share/ansible/plugins/action']

I also ran the playbook with the debugging enabled and verbose. Output is attached in playbook_output_clean.txt.

Kind regards,
Oliver

------------------------------
Oliver Stadler

Original Message:
Sent: Mon February 06, 2023 02:15 AM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hi Oliver,  I figured that was going to be the case with GitHub, your not the only client who has that policy, if you have Ansible Automation Hub entitlement you could go through L2, otherwise I don't mind working here.  For some reason I don't get notifications for this thread so I have to come back and review if there are updates.

Anyhow, thank you for the diagnosis, I guess I assumed too much without a log, there are a couple of things I can think of that might cause this but without the output from

ANSIBLE_DEBUG=1 ansible-playbook -i inventory your-playbook.yml -vvvv​

I have seen this happen when the controller can't find the action plugin (zos_copy has both , action and modules - see my blog for more on this), and I can usually tell from the log when I see a None/Null passed in as the controllers locale.  You can trouble shoot this by adding the path in your `ansible.cfg` , the entry would look similar to this:

[defaults]action_plugins=/Users/ddimatos/.ansible/collections/ansible_collections/ibm/ibm_zos_core/plugins/action

The other time, but its slight a different error (we are working on this) is with IBM Enterprise Python 3.10 or later but you are using 3.9 according to what Ansible discovered. You might try to not rely on that discovery and tell Ansible what python to use with property `ansible_python_interpreter`, as I can't tell without a log if it really did use 3.9, you can do that from your inventory with an entry that looks like:

zsystem:  hosts:    zvm:      ansible_host: <some host>      ansible_user: <some user>      ansible_python_interpreter: /python2/usr/lpp/IBM/cyp/v3r9/pyz/bin/python3.9

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Wed February 01, 2023 11:36 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetrios,

Thanks a lot for the feedback. I actually wanted to place the question directly in the Github discussion forum but unfortunately we are not allowed to due to company policy.

I checked the iefbr14.cjl source file on the controller and it is definitely not  EBCDIC (IBM-1047). 

$ file iefbr14.jcl
iefbr14.jcl: ASCII text

I also doublechecked the actual hex codes of some of the characters via the od command and they are in fact ASCII encoded.

When I look at the payload file which is copied during the execution of the task into /tmp (e.g. /tmp/ansible-zos-copy-payload-D230201-T155244) that looks fine (using -vvv to see the verbose output) . The file is not tagged and if I do a cat on it the file content is displayed correctly. So it looks like it gets converted later on. 

I wonder if we have something configured differently on Unix System Services side on our end that might cause this.

Kind regards,
Oliver



------------------------------
Oliver Stadler

Original Message:
Sent: Sun January 29, 2023 12:19 AM
From: DEMETRIOS DIMATOS
Subject: zos_copy module encoding question

Hello Oliver,
Looking at this snippet and from what you have shared in that you have from:to set to IBM-1047

  - name: Copy JCL without encoding    ibm.ibm_zos_core.zos_copy:      src: iefbr14.jcl      dest: "{{ my_seq }}"      backup: False      force: True

It seems that the JCL source `iefbr14.jcl` on the controller (eg your laptop) is encoded in EBCDIC (IBM-1047) so when you don't specify an encoding what the `zos_copy` module does is take the `locale`of the controller which in most cases is either `UTF8` or `ISO8859-1` and uses that as the `from` and when the module runs on the managed node (z/OS) it takes the `locale` there and sets it as the `to` which in most cases is `IBM-1047`.

Why `iefbr14.jcl` ends up unreadable is because the module has used incorrect inputs/oputput (from/to) for its encoding when you don't specify. We rely on these user inputs because its very difficult to detect and correctly asses a files encoding reading a files Byte Order Mark (BOM) and has a high probability of inaccuracy in doing so.

In your case you have a few options, if the file is already in the desired encoding on your controller eg IBM-1047, you can continue to set the FROM an TO encoding options as IBM-1047 for both so that the  module does not perform any encoding conversion or you can instruct the module the file is binary and should prevent any conversion but I don't recommend you use the binary trick. Your best option is to have the file on the controller in the controllers encoding or continue to do what you are doing.

In this article under section "Plugins and modules" I cover the locale and architecture a bit if you are interested.
We try to monitor all sources and this one was overlooked, feel free to also use our repositories discussions if you are permitted to access it.

------------------------------
DEMETRIOS DIMATOS
IBM zOS Ansible Core Senior Technical Lead
IBM zOS Ansible Core Senior Technical Lead
San Jose CA

Original Message:
Sent: Fri January 06, 2023 10:42 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi,

I have another encoding related question. When I copy an ASCII file using the zos_copy module it works fine if I specify IBM-1047 both in the from and to argument. If I don't specify an encoding the file arrives corrupted. Specifying the encoding it no big deal but it's somewhat confusing that I have to specify IBM-1047 as "from" encoding when copying an ASCII file.

This is the file I use as input
(az1) [myuser@myhost ansible-zos]$ cat iefbr14.jcl
//MYUSERA JOB (123),'LOCAL',CLASS=A,MSGCLASS=S,MSGLEVEL=(1,1),
// NOTIFY=&SYSUID
//*
//STEP001 EXEC PGM=IEFBR14
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSDUMP DD SYSOUT=*
//*

The following playbook shows the behaviour.

(az1) [myuser@myhost ansible-zos]$ cat zos_copy_example.yml
---
- name: Working with datasets
  hosts: all
  gather_facts: False
  vars:
  - my_seq: MYUSER.TEST3.SEQ

  tasks:
  - name: Create {{ my_seq }}
    ibm.ibm_zos_core.zos_data_set:
      name: "{{ my_seq }}"
      type: seq
      record_format: VB
      record_length: 255

  - name: Copy JCL with encoding
    ibm.ibm_zos_core.zos_copy:
      src: iefbr14.jcl
      dest: "{{ my_seq }}"
      backup: False
      force: True
    encoding:
      to: IBM-1047
      from: IBM-1047

  - name: Submit the job
    ibm.ibm_zos_core.zos_job_submit:
      location: DATA_SET
      src: "{{ my_seq }}"

  - name: Copy JCL without encoding
    ibm.ibm_zos_core.zos_copy:
      src: iefbr14.jcl
      dest: "{{ my_seq }}"
      backup: False
      force: True

  - name: Submit the job
    ibm.ibm_zos_core.zos_job_submit:
      location: DATA_SET
      src: "{{ my_seq }}"
...

When I run it I get the following result.

(az1) [myuser@myhost ansible-zos]$ ansible-playbook -i inventory -l mysys zos_copy_example.yml
[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the controller starting with Ansible 2.12. Current version: 3.6.8 (default, Aug 13 2020, 07:46:32) [GCC 4.8.5 20150623 (Red Hat
4.8.5-39)]. This feature will be removed from ansible-core in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [Working with datasets] ************************************************************************************************************************************************************************

TASK [Create MYUSER.TEST3.SEQ] *********************************************************************************************************************************************************************
[WARNING]: Platform os/390 on host mysys is using the discovered Python interpreter at /usr/lpp/python/bin/python3.9, but future installation of another Python interpreter could change
the meaning of that path. See https://docs.ansible.com/ansible-core/2.11/reference_appendices/interpreter_discovery.html for more information.
ok: [mysys]

TASK [Copy JCL with encoding] ***********************************************************************************************************************************************************************
changed: [mysys]

TASK [Submit the job] *******************************************************************************************************************************************************************************
changed: [mysys]

TASK [Copy JCL without encoding] ********************************************************************************************************************************************************************
changed: [mysys]

TASK [Submit the job] *******************************************************************************************************************************************************************************
fatal: [mysys]: FAILED! => {"changed": false, "msg": "SubmitJCLError('Submit job failed: Stderr :FSUMB408 submit: data set MYUSER.TEST3.SEQ Not accepted by JES.\\n')"}

PLAY RECAP ******************************************************************************************************************************************************************************************
mysys: ok=4 changed=3 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

------------------------------
Oliver Stadler
------------------------------

Hello Oliver, it looks like things are good with regard to the plugin path and discovery from what I can see in your log. I found in our code where we are setting a `IBM-037`, I think it was an oversight given the slight variation in vars were `DEFAULT_EBCDIC_USS_CHARSET` vs `DEFAULT_EBCDIC_MVS_CHARSET`. 

I can't tell from your last message if its working for you now because it seems like you are sharing the expected HEX and the actual HEX which are the same and thus you are good now.

If you this is working for you then I will put the investigation of our use of `DEFAULT_EBCDIC_USS_CHARSET` in the backlog.

Hello Oliver, actually you had my name correct, full pronunciation is `Demetrios`, I updated it to the shortened `Demetri` the other day, no problems here on my end how you refer to me :) .

I did a quick pass and noticed something, it appears the code is either detecting IBM-037 or your setting it to IBM-037 (but I think you are setting it to IBM-1047) , i have to look a the code to follow the path and where the swap is happening.
- Which version of the IBM z/OS core collection are you using, eg

cat /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/MANIFEST.json | grep version

- What does USS command on the managed node (one you are connecting to) yield: `locale -c charmap`, eg I get:

# locale -c charmapIBM-1047

The log snippet that has me questioning the code is:

{"changed": true, "src": "iefbr14.jcl", "dest": "Tmyuser.TEST4.SEQ", "ds_type": "PS", "dest_exists": true, "backup_name": null, "invocation": {"module_args": {"is_pds": false, "src_member": false, "size": 194, "local_charset": "UTF-8", "dest": "Tmyuser.TEST4.SEQ", "is_src_dir": false, "is_uss": false, "temp_path": "/tmp/ansible-zos-copy-payload-D230206-T181246", "src": "iefbr14.jcl", "copy_member": false, "is_mvs_dest": true, "is_binary": false, "backup": false, "local_follow": true, "remote_src": false, "ignore_sftp_stderr": false, "validate": false, "force": false, "follow": false, "unsafe_writes": false, "encoding": {"from": "UTF-8", "to": "IBM-037"}, "content": null, "backup_name": null, "sftp_port": null, "volume": null, "dest_data_set": null, "mode": null, "owner": null, "group": null, "seuser": null, "serole": null, "selevel": null, "setype": null, "attributes": null, "regexp": null, "delimiter": null, "directory_mode": null, "from_encoding": "UTF-8", "to_encoding": "IBM-1047"}}}

The last thing that caught my eye which ill need some time to review is the plugin search path differs from your install path but it appears Ansible figured it out but still I want to look into, your install path is /home/tmyuser/.ansible/collections, Ansible searched paths `/usr/share/ansible/plugins/action:/usr/lib/python2.7/site-packages/ansible/plugins/action` and found the zos_copy action plugin, so I am not sure if the locale confusion has anything to do with this yet.

Loading ActionModule 'ansible_collections.ibm.ibm_zos_core.plugins.action.zos_copy' from /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action/zos_copy.py (searched paths: /usr/share/ansible/plugins/action:/usr/lib/python2.7/site-packages/ansible/plugins/action) (found_in_cache=False, class_only=True)

ANSIBLE_DEBUG=1 ansible-playbook -i inventory your-playbook.yml -vvvv​

I have seen this happen when the controller can't find the action plugin (zos_copy has both , action and modules - see my blog for more on this), and I can usually tell from the log when I see a None/Null passed in as the controllers locale.  You can trouble shoot this by adding the path in your `ansible.cfg` , the entry would look similar to this:

[defaults]action_plugins=/Users/ddimatos/.ansible/collections/ansible_collections/ibm/ibm_zos_core/plugins/action

The other time, but its slight a different error (we are working on this) is with IBM Enterprise Python 3.10 or later but you are using 3.9 according to what Ansible discovered. You might try to not rely on that discovery and tell Ansible what python to use with property `ansible_python_interpreter`, as I can't tell without a log if it really did use 3.9, you can do that from your inventory with an entry that looks like:

zsystem:  hosts:    zvm:      ansible_host: <some host>      ansible_user: <some user>      ansible_python_interpreter: /python2/usr/lpp/IBM/cyp/v3r9/pyz/bin/python3.9

Hello Oliver,
Looking at this snippet and from what you have shared in that you have from:to set to IBM-1047

  - name: Copy JCL without encoding    ibm.ibm_zos_core.zos_copy:      src: iefbr14.jcl      dest: "{{ my_seq }}"      backup: False      force: True

It seems that the JCL source `iefbr14.jcl` on the controller (eg your laptop) is encoded in EBCDIC (IBM-1047) so when you don't specify an encoding what the `zos_copy` module does is take the `locale`of the controller which in most cases is either `UTF8` or `ISO8859-1` and uses that as the `from` and when the module runs on the managed node (z/OS) it takes the `locale` there and sets it as the `to` which in most cases is `IBM-1047`.

Why `iefbr14.jcl` ends up unreadable is because the module has used incorrect inputs/oputput (from/to) for its encoding when you don't specify. We rely on these user inputs because its very difficult to detect and correctly asses a files encoding reading a files Byte Order Mark (BOM) and has a high probability of inaccuracy in doing so.

In your case you have a few options, if the file is already in the desired encoding on your controller eg IBM-1047, you can continue to set the FROM an TO encoding options as IBM-1047 for both so that the  module does not perform any encoding conversion or you can instruct the module the file is binary and should prevent any conversion but I don't recommend you use the binary trick. Your best option is to have the file on the controller in the controllers encoding or continue to do what you are doing.

In this article under section "Plugins and modules" I cover the locale and architecture a bit if you are interested.
We try to monitor all sources and this one was overlooked, feel free to also use our repositories discussions if you are permitted to access it.

Hi Oliver, 

I must have not understood 2 messages back, thank you for clarifying. I will try to reproduce this, this is the only way for me to track where the inconsistency is happening, thank you for clearly explaning this helps me quiet a bit. I will get back to you shortly. 

I also went ahead and created a UTF-8 encoded version of the same ISO-88591 content and they match:
UTF-8: 2f2f 5435 3037 3632 3241 204a 4f42 2028  for content //T507622A JOB (

You could try the `file -I` command to check the the encoding source:

$ file -I utf-8.txt
utf-8.txt: text/plain; charset=utf-8
$ file -I ascii.txt
ascii.txt: text/plain; charset=us-ascii

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Wed February 08, 2023 01:59 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetri,

No it's not yet working. Below end to end what I see happening.

On the controller the original file looks ASCII like this (just looking at the first line to limit the output)

$ head -1 iefbr14.jcl
//T507622A JOB (123),'LOCAL',CLASS=A,MSGCLASS=S,MSGLEVEL=(1,1),

First two characters are the forward slashes ASCII encoded as hex 2f 2f 
$ head -1 iefbr14.jcl | od -x
0000000 2f2f 5435 3037 3632 3241 2020 4a4f 4220
0000020 2831 3233 292c 274c 4f43 414c 272c 434c
0000040 4153 533d 412c 4d53 4743 4c41 5353 3d53
0000060 2c4d 5347 4c45 5645 4c3d 2831 2c31 292c
0000100 0a00

When I look at the payload file that Ansible creates when doing the copy to the USS system the files is converted to EBCDIC. First two characters are now hex 61 61 which corresponds to the forward slashes in EBCDIC.

head -1 /tmp/ansible-zos-copy-payload-D230208-T073216 | od -x
0000000000 6161 E3F5 F0F7 F6F2 F2C1 4040 D1D6 C240
0000000020 4DF1 F2F3 5D6B 7DD3 D6C3 C1D3 7D6B C3D3
0000000040 C1E2 E27E C16B D4E2 C7C3 D3C1 E2E2 7EE2
0000000060 6BD4 E2C7 D3C5 E5C5 D37E 4DF1 6BF1 5D6B
0000000100 1500
0000000101
0000101

However in the target data set the arriving content looks like this. The first two characters are encoded as hex 81.

********************************* Top of Data **********************************

-----------------------------------------------------------------------------
aa..@@..M.k}.Á.kÓb.k........~M.k.aa@@@@@@@......aa\.aa.....@...............aa...
88337732D39D3639E83933333333AD39388777777733333388E38833333733333333333333388333
11FFCCF04F20F5F2E2F2FFFFFFFF14F2D11CCCCCCCFFFFFD110D11FFFFFCFFFFFFFFFFFFFFD11FFF
------------------------------------------------------------------------------
******************************** Bottom of Data ********************************

Which is what I get if I convert the already EBCDIC content as UTF-8 to IBM-1047.

head -1 /tmp/ansible-zos-copy-payload-D230208-T073216 | iconv -f UTF-8 -t IBM-1047 | od -x
0000000000 8181 3F3F 7C7C 3F20 D43F 92D0 3F65 3F92
0000000020 EE82 3F92 3F3F 3F3F 3F3F 3F3F A1D4 3F92
0000000040 3D00
0000000041


Kind regards,
Oliver

------------------------------
Oliver Stadler

Original Message:
Sent: Tue February 07, 2023 04:47 PM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hello Oliver, it looks like things are good with regard to the plugin path and discovery from what I can see in your log. I found in our code where we are setting a `IBM-037`, I think it was an oversight given the slight variation in vars were `DEFAULT_EBCDIC_USS_CHARSET` vs `DEFAULT_EBCDIC_MVS_CHARSET`. 

I can't tell from your last message if its working for you now because it seems like you are sharing the expected HEX and the actual HEX which are the same and thus you are good now.

If you this is working for you then I will put the investigation of our use of `DEFAULT_EBCDIC_USS_CHARSET` in the backlog.

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Tue February 07, 2023 04:00 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetri,

zos collection version is 1.4

[tmyuser@mycontroller ansible-zos]$ cat /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/MANIFEST.json | grep version
"version": "1.4.0",

Locale on USS side is IBM-1047

[tmyuser@mycontroller ansible-zos]$ ansible -i inventory myplex1 -m command -a 'locale -c charmap'
[WARNING]: Platform os/390 on host mysys1.com is using the discovered Python interpreter at /usr/lpp/python/bin/python, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.
mysys1.com | CHANGED | rc=0 >>
IBM-1047

I noticed that there was actually a typo in the plugin path. I cleaned that up. Config looks like this now:

[myuser@mycontroller ansible-zos]$ ansible-config dump --only-changed
ANSIBLE_PIPELINING(/home/myuser/ansible-zos/ansible.cfg) = True
COLLECTIONS_PATHS(/home/myuser/ansible-zos/ansible.cfg) = [u'/home/myuser/colltest', u'/usr/share/ansible/collections']
DEFAULT_ACTION_PLUGIN_PATH(/home/myuser/ansible-zos/ansible.cfg) = [u'/home/myuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action', u'/usr/share/ansible/plugins/action']

I also rerun the playbook with this config and attached the output (playbook_output3.txt)

It looks like the actual conversion which is done is from ASCII to IBM-1047.

The target dataset in hex looks like this:

Command ===> Scroll ===> CSR
********************************* Top of Data **********************************


------------------------------------------------------------------------------
aa..@@..M.k}.Á.kÓb.k........~M.k.aa@@@@@@@......aa\.aa.....@...............aa...
88337732D39D3639E83933333333AD39388777777733333388E38833333733333333333333388333
11FFCCF04F20F5F2E2F2FFFFFFFF14F2D11CCCCCCCFFFFFD110D11FFFFFCFFFFFFFFFFFFFFD11FFF
------------------------------------------------------------------------------
******************************** Bottom of Data ********************************

That corresponds to the output I get when I run an iconv on the file being copied.
Tmyuser@TR01:/home/tmyuser ==> iconv -f UTF-8 -t IBM-1047 iefbr14.jcl | od -x
0000000000 8181 3F3F 7C7C 3F20 D43F 92D0 3F65 3F92
0000000020 EE82 3F92 3F3F 3F3F 3F3F 3F3F A1D4 3F92
0000000040 3D81 817C 7C7C 7C7C 7C7C 3F3F 3F3F 3F3D
0000000060 8181 E03D 8181 3F3F 3F3F 3F7C 3F3F 3F3F
0000000100 3F3F 3F3F 3F3F 3F3F 3F3F 3D81 813F 3F3F
0000000120 7C3F 7C3F 3F3F 3D81 813F 3F3F 7C7C 3F7C
0000000140 3F3F 3F3D 8181 E03D
0000000150



------------------------------
Oliver Stadler

Original Message:
Sent: Mon February 06, 2023 05:43 PM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hello Oliver, actually you had my name correct, full pronunciation is `Demetrios`, I updated it to the shortened `Demetri` the other day, no problems here on my end how you refer to me :) .

I did a quick pass and noticed something, it appears the code is either detecting IBM-037 or your setting it to IBM-037 (but I think you are setting it to IBM-1047) , i have to look a the code to follow the path and where the swap is happening.
- Which version of the IBM z/OS core collection are you using, eg

cat /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/MANIFEST.json | grep version

- What does USS command on the managed node (one you are connecting to) yield: `locale -c charmap`, eg I get:

# locale -c charmapIBM-1047

The log snippet that has me questioning the code is:

{"changed": true, "src": "iefbr14.jcl", "dest": "Tmyuser.TEST4.SEQ", "ds_type": "PS", "dest_exists": true, "backup_name": null, "invocation": {"module_args": {"is_pds": false, "src_member": false, "size": 194, "local_charset": "UTF-8", "dest": "Tmyuser.TEST4.SEQ", "is_src_dir": false, "is_uss": false, "temp_path": "/tmp/ansible-zos-copy-payload-D230206-T181246", "src": "iefbr14.jcl", "copy_member": false, "is_mvs_dest": true, "is_binary": false, "backup": false, "local_follow": true, "remote_src": false, "ignore_sftp_stderr": false, "validate": false, "force": false, "follow": false, "unsafe_writes": false, "encoding": {"from": "UTF-8", "to": "IBM-037"}, "content": null, "backup_name": null, "sftp_port": null, "volume": null, "dest_data_set": null, "mode": null, "owner": null, "group": null, "seuser": null, "serole": null, "selevel": null, "setype": null, "attributes": null, "regexp": null, "delimiter": null, "directory_mode": null, "from_encoding": "UTF-8", "to_encoding": "IBM-1047"}}}

The last thing that caught my eye which ill need some time to review is the plugin search path differs from your install path but it appears Ansible figured it out but still I want to look into, your install path is /home/tmyuser/.ansible/collections, Ansible searched paths `/usr/share/ansible/plugins/action:/usr/lib/python2.7/site-packages/ansible/plugins/action` and found the zos_copy action plugin, so I am not sure if the locale confusion has anything to do with this yet.

Loading ActionModule 'ansible_collections.ibm.ibm_zos_core.plugins.action.zos_copy' from /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action/zos_copy.py (searched paths: /usr/share/ansible/plugins/action:/usr/lib/python2.7/site-packages/ansible/plugins/action) (found_in_cache=False, class_only=True)

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Mon February 06, 2023 12:32 PM
From: Oliver Stadler
Subject: zos_copy module encoding question

And I just realized that I spelt your first name wrongly in my previous posts. Sorry for that but I got that now :-)

------------------------------
Oliver Stadler

Original Message:
Sent: Mon February 06, 2023 12:29 PM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetri,

I added the action_plugins path but it didn't make a difference.

Python version on the target server is 3.9. 

/usr/lpp/python/bin/python --version
Python 3.9.14

The config I used was as follows:

[tmyuser@controller ansible-zos]$ ansible-config dump --only-changed
ANSIBLE_PIPELINING(/home/tmyuser/ansible-zos/ansible.cfg) = True
COLLECTIONS_PATHS(/home/tmyuser/ansible-zos/ansible.cfg) = [u'/home/tmyuser/colltest', u'/usr/share/ansible/collections', u'/home/tmyuser/.ansible/collections', u'/usr/share/ansible/collections', u'/home/tmyuser/ansible_collections']
DEFAULT_ACTION_PLUGIN_PATH(/home/tmyuser/ansible-zos/ansible.cfg) = [u'/home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action/~/.ansible/plugins/action', u'/usr/share/ansible/plugins/action']

I also ran the playbook with the debugging enabled and verbose. Output is attached in playbook_output_clean.txt.

Kind regards,
Oliver

------------------------------
Oliver Stadler

Original Message:
Sent: Mon February 06, 2023 02:15 AM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hi Oliver,  I figured that was going to be the case with GitHub, your not the only client who has that policy, if you have Ansible Automation Hub entitlement you could go through L2, otherwise I don't mind working here.  For some reason I don't get notifications for this thread so I have to come back and review if there are updates.

Anyhow, thank you for the diagnosis, I guess I assumed too much without a log, there are a couple of things I can think of that might cause this but without the output from

ANSIBLE_DEBUG=1 ansible-playbook -i inventory your-playbook.yml -vvvv​

I have seen this happen when the controller can't find the action plugin (zos_copy has both , action and modules - see my blog for more on this), and I can usually tell from the log when I see a None/Null passed in as the controllers locale.  You can trouble shoot this by adding the path in your `ansible.cfg` , the entry would look similar to this:

[defaults]action_plugins=/Users/ddimatos/.ansible/collections/ansible_collections/ibm/ibm_zos_core/plugins/action

The other time, but its slight a different error (we are working on this) is with IBM Enterprise Python 3.10 or later but you are using 3.9 according to what Ansible discovered. You might try to not rely on that discovery and tell Ansible what python to use with property `ansible_python_interpreter`, as I can't tell without a log if it really did use 3.9, you can do that from your inventory with an entry that looks like:

zsystem:  hosts:    zvm:      ansible_host: <some host>      ansible_user: <some user>      ansible_python_interpreter: /python2/usr/lpp/IBM/cyp/v3r9/pyz/bin/python3.9

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Wed February 01, 2023 11:36 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetrios,

Thanks a lot for the feedback. I actually wanted to place the question directly in the Github discussion forum but unfortunately we are not allowed to due to company policy.

I checked the iefbr14.cjl source file on the controller and it is definitely not  EBCDIC (IBM-1047). 

$ file iefbr14.jcl
iefbr14.jcl: ASCII text

I also doublechecked the actual hex codes of some of the characters via the od command and they are in fact ASCII encoded.

When I look at the payload file which is copied during the execution of the task into /tmp (e.g. /tmp/ansible-zos-copy-payload-D230201-T155244) that looks fine (using -vvv to see the verbose output) . The file is not tagged and if I do a cat on it the file content is displayed correctly. So it looks like it gets converted later on. 

I wonder if we have something configured differently on Unix System Services side on our end that might cause this.

Kind regards,
Oliver



------------------------------
Oliver Stadler

Original Message:
Sent: Sun January 29, 2023 12:19 AM
From: DEMETRIOS DIMATOS
Subject: zos_copy module encoding question

Hello Oliver,
Looking at this snippet and from what you have shared in that you have from:to set to IBM-1047

  - name: Copy JCL without encoding    ibm.ibm_zos_core.zos_copy:      src: iefbr14.jcl      dest: "{{ my_seq }}"      backup: False      force: True

It seems that the JCL source `iefbr14.jcl` on the controller (eg your laptop) is encoded in EBCDIC (IBM-1047) so when you don't specify an encoding what the `zos_copy` module does is take the `locale`of the controller which in most cases is either `UTF8` or `ISO8859-1` and uses that as the `from` and when the module runs on the managed node (z/OS) it takes the `locale` there and sets it as the `to` which in most cases is `IBM-1047`.

Why `iefbr14.jcl` ends up unreadable is because the module has used incorrect inputs/oputput (from/to) for its encoding when you don't specify. We rely on these user inputs because its very difficult to detect and correctly asses a files encoding reading a files Byte Order Mark (BOM) and has a high probability of inaccuracy in doing so.

In your case you have a few options, if the file is already in the desired encoding on your controller eg IBM-1047, you can continue to set the FROM an TO encoding options as IBM-1047 for both so that the  module does not perform any encoding conversion or you can instruct the module the file is binary and should prevent any conversion but I don't recommend you use the binary trick. Your best option is to have the file on the controller in the controllers encoding or continue to do what you are doing.

In this article under section "Plugins and modules" I cover the locale and architecture a bit if you are interested.
We try to monitor all sources and this one was overlooked, feel free to also use our repositories discussions if you are permitted to access it.

------------------------------
DEMETRIOS DIMATOS
IBM zOS Ansible Core Senior Technical Lead
IBM zOS Ansible Core Senior Technical Lead
San Jose CA

Original Message:
Sent: Fri January 06, 2023 10:42 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi,

I have another encoding related question. When I copy an ASCII file using the zos_copy module it works fine if I specify IBM-1047 both in the from and to argument. If I don't specify an encoding the file arrives corrupted. Specifying the encoding it no big deal but it's somewhat confusing that I have to specify IBM-1047 as "from" encoding when copying an ASCII file.

This is the file I use as input
(az1) [myuser@myhost ansible-zos]$ cat iefbr14.jcl
//MYUSERA JOB (123),'LOCAL',CLASS=A,MSGCLASS=S,MSGLEVEL=(1,1),
// NOTIFY=&SYSUID
//*
//STEP001 EXEC PGM=IEFBR14
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSDUMP DD SYSOUT=*
//*

The following playbook shows the behaviour.

(az1) [myuser@myhost ansible-zos]$ cat zos_copy_example.yml
---
- name: Working with datasets
  hosts: all
  gather_facts: False
  vars:
  - my_seq: MYUSER.TEST3.SEQ

  tasks:
  - name: Create {{ my_seq }}
    ibm.ibm_zos_core.zos_data_set:
      name: "{{ my_seq }}"
      type: seq
      record_format: VB
      record_length: 255

  - name: Copy JCL with encoding
    ibm.ibm_zos_core.zos_copy:
      src: iefbr14.jcl
      dest: "{{ my_seq }}"
      backup: False
      force: True
    encoding:
      to: IBM-1047
      from: IBM-1047

  - name: Submit the job
    ibm.ibm_zos_core.zos_job_submit:
      location: DATA_SET
      src: "{{ my_seq }}"

  - name: Copy JCL without encoding
    ibm.ibm_zos_core.zos_copy:
      src: iefbr14.jcl
      dest: "{{ my_seq }}"
      backup: False
      force: True

  - name: Submit the job
    ibm.ibm_zos_core.zos_job_submit:
      location: DATA_SET
      src: "{{ my_seq }}"
...

When I run it I get the following result.

(az1) [myuser@myhost ansible-zos]$ ansible-playbook -i inventory -l mysys zos_copy_example.yml
[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the controller starting with Ansible 2.12. Current version: 3.6.8 (default, Aug 13 2020, 07:46:32) [GCC 4.8.5 20150623 (Red Hat
4.8.5-39)]. This feature will be removed from ansible-core in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [Working with datasets] ************************************************************************************************************************************************************************

TASK [Create MYUSER.TEST3.SEQ] *********************************************************************************************************************************************************************
[WARNING]: Platform os/390 on host mysys is using the discovered Python interpreter at /usr/lpp/python/bin/python3.9, but future installation of another Python interpreter could change
the meaning of that path. See https://docs.ansible.com/ansible-core/2.11/reference_appendices/interpreter_discovery.html for more information.
ok: [mysys]

TASK [Copy JCL with encoding] ***********************************************************************************************************************************************************************
changed: [mysys]

TASK [Submit the job] *******************************************************************************************************************************************************************************
changed: [mysys]

TASK [Copy JCL without encoding] ********************************************************************************************************************************************************************
changed: [mysys]

TASK [Submit the job] *******************************************************************************************************************************************************************************
fatal: [mysys]: FAILED! => {"changed": false, "msg": "SubmitJCLError('Submit job failed: Stderr :FSUMB408 submit: data set MYUSER.TEST3.SEQ Not accepted by JES.\\n')"}

PLAY RECAP ******************************************************************************************************************************************************************************************
mysys: ok=4 changed=3 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

------------------------------
Oliver Stadler
------------------------------

Hello Oliver,

Here is what I have so far:

1. Created file ascii.txt with this content:
   1. //T507622A JOB (123),'LOCAL',CLASS=A,MSGCLASS=S,MSGLEVEL=(1,1),(ansible-doc
   2. With hex values:

      1. 0000000 2f2f 5435 3037 3632 3241 204a 4f42 2028  //T507622A JOB (0000010 3132 3329 2c27 4c4f 4341 4c27 2c43 4c41  123),'LOCAL',CLA0000020 5353 3d41 2c4d 5347 434c 4153 533d 532c  SS=A,MSGCLASS=S,0000030 4d53 474c 4556 454c 3d28 312c 3129 2c0a  MSGLEVEL=(1,1),.

2. With playbook:

   1. ---- hosts: zvm  gather_facts: false  vars:    - SEQ: "ANSIBLE.CONTENT.SEQ"  environment: "{{ environment_vars }}"  collections:    - ibm.ibm_zos_core  tasks:    - name: Create/replace the PDS      zos_data_set:        name: "{{ SEQ }}"        type: seq        space_primary: 5        space_type: M        record_format: fb        record_length: 80        block_size: 3120        replace: true      register: result    - name: Copy local file to PDS      zos_copy:        src: "{{playbook_dir}}/files/ascii.txt"        dest: "{{ SEQ }}"        remote_src: false        encoding:           from: ISO8859-1           to: IBM-1047      register: result    - name: Result of copying local directory      debug:        msg: "{{ result }}"

3. And on z/OS when I look at the sequential data set content in Hex I have:

   1. //T507622A JOB (123),'LOCAL',CLASS=A,MSGCLASS=S,MSGLEVEL=(1,1),                 66EFFFFFFC4DDC44FFF567DDCCD76CDCEE7C6DECCDCEE7E6DECDCECD74F6F5644444444444444444113507622101620D123DBD36313DB33122E1B42733122E2B42735553ED1B1DB00000000000000000

Looking at the last bullet (3) hex content and picking random characters they correspond to code page IBM-1047:

• / is HEX 61
• L is HEX D3
• ( is HEX 4D

What I did notice in your ASCII content (assuming ISO-8859-1) , partway through some of the characters don't seem to correspond to ISO-8859-1. Lets review this portion  //T507622A JOB (.

Text:        //T507622A JOB (

Mine:      2f2f 5435 3037 3632 3241 204a 4f42 2028

Yours:    2f2f 5435 3037 3632 3241 2020 4a4f 4220

HEX values begin to not align , my:

• 4a vs 20
• 4f vs 4a
• 42 vs 4f

It appears we don't have the same data to work with. Could you send me your ASCII content, feel free to email me or attach it here but  I wonder if that is the source of the issue where the conversion goes wrong because source may not be consistent with ISO-8859-1.

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Fri February 10, 2023 10:41 PM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hi Oliver, 

I must have not understood 2 messages back, thank you for clarifying. I will try to reproduce this, this is the only way for me to track where the inconsistency is happening, thank you for clearly explaning this helps me quiet a bit. I will get back to you shortly. 

I also went ahead and created a UTF-8 encoded version of the same ISO-88591 content and they match:
UTF-8: 2f2f 5435 3037 3632 3241 204a 4f42 2028  for content //T507622A JOB (

You could try the `file -I` command to check the the encoding source:

$ file -I utf-8.txtutf-8.txt: text/plain; charset=utf-8$ file -I ascii.txtascii.txt: text/plain; charset=us-ascii

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Wed February 08, 2023 01:59 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetri,

No it's not yet working. Below end to end what I see happening.

On the controller the original file looks ASCII like this (just looking at the first line to limit the output)

$ head -1 iefbr14.jcl
//T507622A JOB (123),'LOCAL',CLASS=A,MSGCLASS=S,MSGLEVEL=(1,1),

First two characters are the forward slashes ASCII encoded as hex 2f 2f 
$ head -1 iefbr14.jcl | od -x
0000000 2f2f 5435 3037 3632 3241 2020 4a4f 4220
0000020 2831 3233 292c 274c 4f43 414c 272c 434c
0000040 4153 533d 412c 4d53 4743 4c41 5353 3d53
0000060 2c4d 5347 4c45 5645 4c3d 2831 2c31 292c
0000100 0a00

When I look at the payload file that Ansible creates when doing the copy to the USS system the files is converted to EBCDIC. First two characters are now hex 61 61 which corresponds to the forward slashes in EBCDIC.

head -1 /tmp/ansible-zos-copy-payload-D230208-T073216 | od -x
0000000000 6161 E3F5 F0F7 F6F2 F2C1 4040 D1D6 C240
0000000020 4DF1 F2F3 5D6B 7DD3 D6C3 C1D3 7D6B C3D3
0000000040 C1E2 E27E C16B D4E2 C7C3 D3C1 E2E2 7EE2
0000000060 6BD4 E2C7 D3C5 E5C5 D37E 4DF1 6BF1 5D6B
0000000100 1500
0000000101
0000101

However in the target data set the arriving content looks like this. The first two characters are encoded as hex 81.

********************************* Top of Data **********************************

-----------------------------------------------------------------------------
aa..@@..M.k}.Á.kÓb.k........~M.k.aa@@@@@@@......aa\.aa.....@...............aa...
88337732D39D3639E83933333333AD39388777777733333388E38833333733333333333333388333
11FFCCF04F20F5F2E2F2FFFFFFFF14F2D11CCCCCCCFFFFFD110D11FFFFFCFFFFFFFFFFFFFFD11FFF
------------------------------------------------------------------------------
******************************** Bottom of Data ********************************

Which is what I get if I convert the already EBCDIC content as UTF-8 to IBM-1047.

head -1 /tmp/ansible-zos-copy-payload-D230208-T073216 | iconv -f UTF-8 -t IBM-1047 | od -x
0000000000 8181 3F3F 7C7C 3F20 D43F 92D0 3F65 3F92
0000000020 EE82 3F92 3F3F 3F3F 3F3F 3F3F A1D4 3F92
0000000040 3D00
0000000041


Kind regards,
Oliver

------------------------------
Oliver Stadler

Original Message:
Sent: Tue February 07, 2023 04:47 PM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hello Oliver, it looks like things are good with regard to the plugin path and discovery from what I can see in your log. I found in our code where we are setting a `IBM-037`, I think it was an oversight given the slight variation in vars were `DEFAULT_EBCDIC_USS_CHARSET` vs `DEFAULT_EBCDIC_MVS_CHARSET`. 

I can't tell from your last message if its working for you now because it seems like you are sharing the expected HEX and the actual HEX which are the same and thus you are good now.

If you this is working for you then I will put the investigation of our use of `DEFAULT_EBCDIC_USS_CHARSET` in the backlog.

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Tue February 07, 2023 04:00 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetri,

zos collection version is 1.4

[tmyuser@mycontroller ansible-zos]$ cat /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/MANIFEST.json | grep version
"version": "1.4.0",

Locale on USS side is IBM-1047

[tmyuser@mycontroller ansible-zos]$ ansible -i inventory myplex1 -m command -a 'locale -c charmap'
[WARNING]: Platform os/390 on host mysys1.com is using the discovered Python interpreter at /usr/lpp/python/bin/python, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.
mysys1.com | CHANGED | rc=0 >>
IBM-1047

I noticed that there was actually a typo in the plugin path. I cleaned that up. Config looks like this now:

[myuser@mycontroller ansible-zos]$ ansible-config dump --only-changed
ANSIBLE_PIPELINING(/home/myuser/ansible-zos/ansible.cfg) = True
COLLECTIONS_PATHS(/home/myuser/ansible-zos/ansible.cfg) = [u'/home/myuser/colltest', u'/usr/share/ansible/collections']
DEFAULT_ACTION_PLUGIN_PATH(/home/myuser/ansible-zos/ansible.cfg) = [u'/home/myuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action', u'/usr/share/ansible/plugins/action']

I also rerun the playbook with this config and attached the output (playbook_output3.txt)

It looks like the actual conversion which is done is from ASCII to IBM-1047.

The target dataset in hex looks like this:

Command ===> Scroll ===> CSR
********************************* Top of Data **********************************


------------------------------------------------------------------------------
aa..@@..M.k}.Á.kÓb.k........~M.k.aa@@@@@@@......aa\.aa.....@...............aa...
88337732D39D3639E83933333333AD39388777777733333388E38833333733333333333333388333
11FFCCF04F20F5F2E2F2FFFFFFFF14F2D11CCCCCCCFFFFFD110D11FFFFFCFFFFFFFFFFFFFFD11FFF
------------------------------------------------------------------------------
******************************** Bottom of Data ********************************

That corresponds to the output I get when I run an iconv on the file being copied.
Tmyuser@TR01:/home/tmyuser ==> iconv -f UTF-8 -t IBM-1047 iefbr14.jcl | od -x
0000000000 8181 3F3F 7C7C 3F20 D43F 92D0 3F65 3F92
0000000020 EE82 3F92 3F3F 3F3F 3F3F 3F3F A1D4 3F92
0000000040 3D81 817C 7C7C 7C7C 7C7C 3F3F 3F3F 3F3D
0000000060 8181 E03D 8181 3F3F 3F3F 3F7C 3F3F 3F3F
0000000100 3F3F 3F3F 3F3F 3F3F 3F3F 3D81 813F 3F3F
0000000120 7C3F 7C3F 3F3F 3D81 813F 3F3F 7C7C 3F7C
0000000140 3F3F 3F3D 8181 E03D
0000000150



------------------------------
Oliver Stadler

Original Message:
Sent: Mon February 06, 2023 05:43 PM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hello Oliver, actually you had my name correct, full pronunciation is `Demetrios`, I updated it to the shortened `Demetri` the other day, no problems here on my end how you refer to me :) .

I did a quick pass and noticed something, it appears the code is either detecting IBM-037 or your setting it to IBM-037 (but I think you are setting it to IBM-1047) , i have to look a the code to follow the path and where the swap is happening.
- Which version of the IBM z/OS core collection are you using, eg

cat /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/MANIFEST.json | grep version

- What does USS command on the managed node (one you are connecting to) yield: `locale -c charmap`, eg I get:

# locale -c charmapIBM-1047

The log snippet that has me questioning the code is:

{"changed": true, "src": "iefbr14.jcl", "dest": "Tmyuser.TEST4.SEQ", "ds_type": "PS", "dest_exists": true, "backup_name": null, "invocation": {"module_args": {"is_pds": false, "src_member": false, "size": 194, "local_charset": "UTF-8", "dest": "Tmyuser.TEST4.SEQ", "is_src_dir": false, "is_uss": false, "temp_path": "/tmp/ansible-zos-copy-payload-D230206-T181246", "src": "iefbr14.jcl", "copy_member": false, "is_mvs_dest": true, "is_binary": false, "backup": false, "local_follow": true, "remote_src": false, "ignore_sftp_stderr": false, "validate": false, "force": false, "follow": false, "unsafe_writes": false, "encoding": {"from": "UTF-8", "to": "IBM-037"}, "content": null, "backup_name": null, "sftp_port": null, "volume": null, "dest_data_set": null, "mode": null, "owner": null, "group": null, "seuser": null, "serole": null, "selevel": null, "setype": null, "attributes": null, "regexp": null, "delimiter": null, "directory_mode": null, "from_encoding": "UTF-8", "to_encoding": "IBM-1047"}}}

The last thing that caught my eye which ill need some time to review is the plugin search path differs from your install path but it appears Ansible figured it out but still I want to look into, your install path is /home/tmyuser/.ansible/collections, Ansible searched paths `/usr/share/ansible/plugins/action:/usr/lib/python2.7/site-packages/ansible/plugins/action` and found the zos_copy action plugin, so I am not sure if the locale confusion has anything to do with this yet.

Loading ActionModule 'ansible_collections.ibm.ibm_zos_core.plugins.action.zos_copy' from /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action/zos_copy.py (searched paths: /usr/share/ansible/plugins/action:/usr/lib/python2.7/site-packages/ansible/plugins/action) (found_in_cache=False, class_only=True)

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Mon February 06, 2023 12:32 PM
From: Oliver Stadler
Subject: zos_copy module encoding question

And I just realized that I spelt your first name wrongly in my previous posts. Sorry for that but I got that now :-)

------------------------------
Oliver Stadler

Original Message:
Sent: Mon February 06, 2023 12:29 PM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetri,

I added the action_plugins path but it didn't make a difference.

Python version on the target server is 3.9. 

/usr/lpp/python/bin/python --version
Python 3.9.14

The config I used was as follows:

[tmyuser@controller ansible-zos]$ ansible-config dump --only-changed
ANSIBLE_PIPELINING(/home/tmyuser/ansible-zos/ansible.cfg) = True
COLLECTIONS_PATHS(/home/tmyuser/ansible-zos/ansible.cfg) = [u'/home/tmyuser/colltest', u'/usr/share/ansible/collections', u'/home/tmyuser/.ansible/collections', u'/usr/share/ansible/collections', u'/home/tmyuser/ansible_collections']
DEFAULT_ACTION_PLUGIN_PATH(/home/tmyuser/ansible-zos/ansible.cfg) = [u'/home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action/~/.ansible/plugins/action', u'/usr/share/ansible/plugins/action']

I also ran the playbook with the debugging enabled and verbose. Output is attached in playbook_output_clean.txt.

Kind regards,
Oliver

------------------------------
Oliver Stadler

Original Message:
Sent: Mon February 06, 2023 02:15 AM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hi Oliver,  I figured that was going to be the case with GitHub, your not the only client who has that policy, if you have Ansible Automation Hub entitlement you could go through L2, otherwise I don't mind working here.  For some reason I don't get notifications for this thread so I have to come back and review if there are updates.

Anyhow, thank you for the diagnosis, I guess I assumed too much without a log, there are a couple of things I can think of that might cause this but without the output from

ANSIBLE_DEBUG=1 ansible-playbook -i inventory your-playbook.yml -vvvv​

I have seen this happen when the controller can't find the action plugin (zos_copy has both , action and modules - see my blog for more on this), and I can usually tell from the log when I see a None/Null passed in as the controllers locale.  You can trouble shoot this by adding the path in your `ansible.cfg` , the entry would look similar to this:

[defaults]action_plugins=/Users/ddimatos/.ansible/collections/ansible_collections/ibm/ibm_zos_core/plugins/action

The other time, but its slight a different error (we are working on this) is with IBM Enterprise Python 3.10 or later but you are using 3.9 according to what Ansible discovered. You might try to not rely on that discovery and tell Ansible what python to use with property `ansible_python_interpreter`, as I can't tell without a log if it really did use 3.9, you can do that from your inventory with an entry that looks like:

zsystem:  hosts:    zvm:      ansible_host: <some host>      ansible_user: <some user>      ansible_python_interpreter: /python2/usr/lpp/IBM/cyp/v3r9/pyz/bin/python3.9

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Wed February 01, 2023 11:36 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetrios,

Thanks a lot for the feedback. I actually wanted to place the question directly in the Github discussion forum but unfortunately we are not allowed to due to company policy.

I checked the iefbr14.cjl source file on the controller and it is definitely not  EBCDIC (IBM-1047). 

$ file iefbr14.jcl
iefbr14.jcl: ASCII text

I also doublechecked the actual hex codes of some of the characters via the od command and they are in fact ASCII encoded.

When I look at the payload file which is copied during the execution of the task into /tmp (e.g. /tmp/ansible-zos-copy-payload-D230201-T155244) that looks fine (using -vvv to see the verbose output) . The file is not tagged and if I do a cat on it the file content is displayed correctly. So it looks like it gets converted later on. 

I wonder if we have something configured differently on Unix System Services side on our end that might cause this.

Kind regards,
Oliver



------------------------------
Oliver Stadler

Original Message:
Sent: Sun January 29, 2023 12:19 AM
From: DEMETRIOS DIMATOS
Subject: zos_copy module encoding question

Hello Oliver,
Looking at this snippet and from what you have shared in that you have from:to set to IBM-1047

  - name: Copy JCL without encoding    ibm.ibm_zos_core.zos_copy:      src: iefbr14.jcl      dest: "{{ my_seq }}"      backup: False      force: True

It seems that the JCL source `iefbr14.jcl` on the controller (eg your laptop) is encoded in EBCDIC (IBM-1047) so when you don't specify an encoding what the `zos_copy` module does is take the `locale`of the controller which in most cases is either `UTF8` or `ISO8859-1` and uses that as the `from` and when the module runs on the managed node (z/OS) it takes the `locale` there and sets it as the `to` which in most cases is `IBM-1047`.

Why `iefbr14.jcl` ends up unreadable is because the module has used incorrect inputs/oputput (from/to) for its encoding when you don't specify. We rely on these user inputs because its very difficult to detect and correctly asses a files encoding reading a files Byte Order Mark (BOM) and has a high probability of inaccuracy in doing so.

In your case you have a few options, if the file is already in the desired encoding on your controller eg IBM-1047, you can continue to set the FROM an TO encoding options as IBM-1047 for both so that the  module does not perform any encoding conversion or you can instruct the module the file is binary and should prevent any conversion but I don't recommend you use the binary trick. Your best option is to have the file on the controller in the controllers encoding or continue to do what you are doing.

In this article under section "Plugins and modules" I cover the locale and architecture a bit if you are interested.
We try to monitor all sources and this one was overlooked, feel free to also use our repositories discussions if you are permitted to access it.

------------------------------
DEMETRIOS DIMATOS
IBM zOS Ansible Core Senior Technical Lead
IBM zOS Ansible Core Senior Technical Lead
San Jose CA

Original Message:
Sent: Fri January 06, 2023 10:42 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi,

I have another encoding related question. When I copy an ASCII file using the zos_copy module it works fine if I specify IBM-1047 both in the from and to argument. If I don't specify an encoding the file arrives corrupted. Specifying the encoding it no big deal but it's somewhat confusing that I have to specify IBM-1047 as "from" encoding when copying an ASCII file.

This is the file I use as input
(az1) [myuser@myhost ansible-zos]$ cat iefbr14.jcl
//MYUSERA JOB (123),'LOCAL',CLASS=A,MSGCLASS=S,MSGLEVEL=(1,1),
// NOTIFY=&SYSUID
//*
//STEP001 EXEC PGM=IEFBR14
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSDUMP DD SYSOUT=*
//*

The following playbook shows the behaviour.

(az1) [myuser@myhost ansible-zos]$ cat zos_copy_example.yml
---
- name: Working with datasets
  hosts: all
  gather_facts: False
  vars:
  - my_seq: MYUSER.TEST3.SEQ

  tasks:
  - name: Create {{ my_seq }}
    ibm.ibm_zos_core.zos_data_set:
      name: "{{ my_seq }}"
      type: seq
      record_format: VB
      record_length: 255

  - name: Copy JCL with encoding
    ibm.ibm_zos_core.zos_copy:
      src: iefbr14.jcl
      dest: "{{ my_seq }}"
      backup: False
      force: True
    encoding:
      to: IBM-1047
      from: IBM-1047

  - name: Submit the job
    ibm.ibm_zos_core.zos_job_submit:
      location: DATA_SET
      src: "{{ my_seq }}"

  - name: Copy JCL without encoding
    ibm.ibm_zos_core.zos_copy:
      src: iefbr14.jcl
      dest: "{{ my_seq }}"
      backup: False
      force: True

  - name: Submit the job
    ibm.ibm_zos_core.zos_job_submit:
      location: DATA_SET
      src: "{{ my_seq }}"
...

When I run it I get the following result.

(az1) [myuser@myhost ansible-zos]$ ansible-playbook -i inventory -l mysys zos_copy_example.yml
[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the controller starting with Ansible 2.12. Current version: 3.6.8 (default, Aug 13 2020, 07:46:32) [GCC 4.8.5 20150623 (Red Hat
4.8.5-39)]. This feature will be removed from ansible-core in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [Working with datasets] ************************************************************************************************************************************************************************

TASK [Create MYUSER.TEST3.SEQ] *********************************************************************************************************************************************************************
[WARNING]: Platform os/390 on host mysys is using the discovered Python interpreter at /usr/lpp/python/bin/python3.9, but future installation of another Python interpreter could change
the meaning of that path. See https://docs.ansible.com/ansible-core/2.11/reference_appendices/interpreter_discovery.html for more information.
ok: [mysys]

TASK [Copy JCL with encoding] ***********************************************************************************************************************************************************************
changed: [mysys]

TASK [Submit the job] *******************************************************************************************************************************************************************************
changed: [mysys]

TASK [Copy JCL without encoding] ********************************************************************************************************************************************************************
changed: [mysys]

TASK [Submit the job] *******************************************************************************************************************************************************************************
fatal: [mysys]: FAILED! => {"changed": false, "msg": "SubmitJCLError('Submit job failed: Stderr :FSUMB408 submit: data set MYUSER.TEST3.SEQ Not accepted by JES.\\n')"}

PLAY RECAP ******************************************************************************************************************************************************************************************
mysys: ok=4 changed=3 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

------------------------------
Oliver Stadler
------------------------------

Hello Oliver,

Here is what I have so far:

1. Created file ascii.txt with this content:
   1. //T507622A JOB (123),'LOCAL',CLASS=A,MSGCLASS=S,MSGLEVEL=(1,1),(ansible-doc
   2. With hex values:

      1. 0000000 2f2f 5435 3037 3632 3241 204a 4f42 2028  //T507622A JOB (0000010 3132 3329 2c27 4c4f 4341 4c27 2c43 4c41  123),'LOCAL',CLA0000020 5353 3d41 2c4d 5347 434c 4153 533d 532c  SS=A,MSGCLASS=S,0000030 4d53 474c 4556 454c 3d28 312c 3129 2c0a  MSGLEVEL=(1,1),.

2. With playbook:

   1. ---- hosts: zvm  gather_facts: false  vars:    - SEQ: "ANSIBLE.CONTENT.SEQ"  environment: "{{ environment_vars }}"  collections:    - ibm.ibm_zos_core  tasks:    - name: Create/replace the PDS      zos_data_set:        name: "{{ SEQ }}"        type: seq        space_primary: 5        space_type: M        record_format: fb        record_length: 80        block_size: 3120        replace: true      register: result    - name: Copy local file to PDS      zos_copy:        src: "{{playbook_dir}}/files/ascii.txt"        dest: "{{ SEQ }}"        remote_src: false        encoding:           from: ISO8859-1           to: IBM-1047      register: result    - name: Result of copying local directory      debug:        msg: "{{ result }}"

3. And on z/OS when I look at the sequential data set content in Hex I have:

   1. //T507622A JOB (123),'LOCAL',CLASS=A,MSGCLASS=S,MSGLEVEL=(1,1),                 66EFFFFFFC4DDC44FFF567DDCCD76CDCEE7C6DECCDCEE7E6DECDCECD74F6F5644444444444444444113507622101620D123DBD36313DB33122E1B42733122E2B42735553ED1B1DB00000000000000000

Looking at the last bullet (3) hex content and picking random characters they correspond to code page IBM-1047:

• / is HEX 61
• L is HEX D3
• ( is HEX 4D

What I did notice in your ASCII content (assuming ISO-8859-1) , partway through some of the characters don't seem to correspond to ISO-8859-1. Lets review this portion  //T507622A JOB (.

Text:        //T507622A JOB (

Mine:      2f2f 5435 3037 3632 3241 204a 4f42 2028

Yours:    2f2f 5435 3037 3632 3241 2020 4a4f 4220

HEX values begin to not align , my:

• 4a vs 20
• 4f vs 4a
• 42 vs 4f

It appears we don't have the same data to work with. Could you send me your ASCII content, feel free to email me or attach it here but  I wonder if that is the source of the issue where the conversion goes wrong because source may not be consistent with ISO-8859-1.

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Fri February 10, 2023 10:41 PM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hi Oliver, 

I must have not understood 2 messages back, thank you for clarifying. I will try to reproduce this, this is the only way for me to track where the inconsistency is happening, thank you for clearly explaning this helps me quiet a bit. I will get back to you shortly. 

I also went ahead and created a UTF-8 encoded version of the same ISO-88591 content and they match:
UTF-8: 2f2f 5435 3037 3632 3241 204a 4f42 2028  for content //T507622A JOB (

You could try the `file -I` command to check the the encoding source:

$ file -I utf-8.txtutf-8.txt: text/plain; charset=utf-8$ file -I ascii.txtascii.txt: text/plain; charset=us-ascii

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Wed February 08, 2023 01:59 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetri,

No it's not yet working. Below end to end what I see happening.

On the controller the original file looks ASCII like this (just looking at the first line to limit the output)

$ head -1 iefbr14.jcl
//T507622A JOB (123),'LOCAL',CLASS=A,MSGCLASS=S,MSGLEVEL=(1,1),

First two characters are the forward slashes ASCII encoded as hex 2f 2f 
$ head -1 iefbr14.jcl | od -x
0000000 2f2f 5435 3037 3632 3241 2020 4a4f 4220
0000020 2831 3233 292c 274c 4f43 414c 272c 434c
0000040 4153 533d 412c 4d53 4743 4c41 5353 3d53
0000060 2c4d 5347 4c45 5645 4c3d 2831 2c31 292c
0000100 0a00

When I look at the payload file that Ansible creates when doing the copy to the USS system the files is converted to EBCDIC. First two characters are now hex 61 61 which corresponds to the forward slashes in EBCDIC.

head -1 /tmp/ansible-zos-copy-payload-D230208-T073216 | od -x
0000000000 6161 E3F5 F0F7 F6F2 F2C1 4040 D1D6 C240
0000000020 4DF1 F2F3 5D6B 7DD3 D6C3 C1D3 7D6B C3D3
0000000040 C1E2 E27E C16B D4E2 C7C3 D3C1 E2E2 7EE2
0000000060 6BD4 E2C7 D3C5 E5C5 D37E 4DF1 6BF1 5D6B
0000000100 1500
0000000101
0000101

However in the target data set the arriving content looks like this. The first two characters are encoded as hex 81.

********************************* Top of Data **********************************

-----------------------------------------------------------------------------
aa..@@..M.k}.Á.kÓb.k........~M.k.aa@@@@@@@......aa\.aa.....@...............aa...
88337732D39D3639E83933333333AD39388777777733333388E38833333733333333333333388333
11FFCCF04F20F5F2E2F2FFFFFFFF14F2D11CCCCCCCFFFFFD110D11FFFFFCFFFFFFFFFFFFFFD11FFF
------------------------------------------------------------------------------
******************************** Bottom of Data ********************************

Which is what I get if I convert the already EBCDIC content as UTF-8 to IBM-1047.

head -1 /tmp/ansible-zos-copy-payload-D230208-T073216 | iconv -f UTF-8 -t IBM-1047 | od -x
0000000000 8181 3F3F 7C7C 3F20 D43F 92D0 3F65 3F92
0000000020 EE82 3F92 3F3F 3F3F 3F3F 3F3F A1D4 3F92
0000000040 3D00
0000000041


Kind regards,
Oliver

------------------------------
Oliver Stadler

Original Message:
Sent: Tue February 07, 2023 04:47 PM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hello Oliver, it looks like things are good with regard to the plugin path and discovery from what I can see in your log. I found in our code where we are setting a `IBM-037`, I think it was an oversight given the slight variation in vars were `DEFAULT_EBCDIC_USS_CHARSET` vs `DEFAULT_EBCDIC_MVS_CHARSET`. 

I can't tell from your last message if its working for you now because it seems like you are sharing the expected HEX and the actual HEX which are the same and thus you are good now.

If you this is working for you then I will put the investigation of our use of `DEFAULT_EBCDIC_USS_CHARSET` in the backlog.

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Tue February 07, 2023 04:00 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetri,

zos collection version is 1.4

[tmyuser@mycontroller ansible-zos]$ cat /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/MANIFEST.json | grep version
"version": "1.4.0",

Locale on USS side is IBM-1047

[tmyuser@mycontroller ansible-zos]$ ansible -i inventory myplex1 -m command -a 'locale -c charmap'
[WARNING]: Platform os/390 on host mysys1.com is using the discovered Python interpreter at /usr/lpp/python/bin/python, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.
mysys1.com | CHANGED | rc=0 >>
IBM-1047

I noticed that there was actually a typo in the plugin path. I cleaned that up. Config looks like this now:

[myuser@mycontroller ansible-zos]$ ansible-config dump --only-changed
ANSIBLE_PIPELINING(/home/myuser/ansible-zos/ansible.cfg) = True
COLLECTIONS_PATHS(/home/myuser/ansible-zos/ansible.cfg) = [u'/home/myuser/colltest', u'/usr/share/ansible/collections']
DEFAULT_ACTION_PLUGIN_PATH(/home/myuser/ansible-zos/ansible.cfg) = [u'/home/myuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action', u'/usr/share/ansible/plugins/action']

I also rerun the playbook with this config and attached the output (playbook_output3.txt)

It looks like the actual conversion which is done is from ASCII to IBM-1047.

The target dataset in hex looks like this:

Command ===> Scroll ===> CSR
********************************* Top of Data **********************************


------------------------------------------------------------------------------
aa..@@..M.k}.Á.kÓb.k........~M.k.aa@@@@@@@......aa\.aa.....@...............aa...
88337732D39D3639E83933333333AD39388777777733333388E38833333733333333333333388333
11FFCCF04F20F5F2E2F2FFFFFFFF14F2D11CCCCCCCFFFFFD110D11FFFFFCFFFFFFFFFFFFFFD11FFF
------------------------------------------------------------------------------
******************************** Bottom of Data ********************************

That corresponds to the output I get when I run an iconv on the file being copied.
Tmyuser@TR01:/home/tmyuser ==> iconv -f UTF-8 -t IBM-1047 iefbr14.jcl | od -x
0000000000 8181 3F3F 7C7C 3F20 D43F 92D0 3F65 3F92
0000000020 EE82 3F92 3F3F 3F3F 3F3F 3F3F A1D4 3F92
0000000040 3D81 817C 7C7C 7C7C 7C7C 3F3F 3F3F 3F3D
0000000060 8181 E03D 8181 3F3F 3F3F 3F7C 3F3F 3F3F
0000000100 3F3F 3F3F 3F3F 3F3F 3F3F 3D81 813F 3F3F
0000000120 7C3F 7C3F 3F3F 3D81 813F 3F3F 7C7C 3F7C
0000000140 3F3F 3F3D 8181 E03D
0000000150



------------------------------
Oliver Stadler

Original Message:
Sent: Mon February 06, 2023 05:43 PM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hello Oliver, actually you had my name correct, full pronunciation is `Demetrios`, I updated it to the shortened `Demetri` the other day, no problems here on my end how you refer to me :) .

I did a quick pass and noticed something, it appears the code is either detecting IBM-037 or your setting it to IBM-037 (but I think you are setting it to IBM-1047) , i have to look a the code to follow the path and where the swap is happening.
- Which version of the IBM z/OS core collection are you using, eg

cat /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/MANIFEST.json | grep version

- What does USS command on the managed node (one you are connecting to) yield: `locale -c charmap`, eg I get:

# locale -c charmapIBM-1047

The log snippet that has me questioning the code is:

{"changed": true, "src": "iefbr14.jcl", "dest": "Tmyuser.TEST4.SEQ", "ds_type": "PS", "dest_exists": true, "backup_name": null, "invocation": {"module_args": {"is_pds": false, "src_member": false, "size": 194, "local_charset": "UTF-8", "dest": "Tmyuser.TEST4.SEQ", "is_src_dir": false, "is_uss": false, "temp_path": "/tmp/ansible-zos-copy-payload-D230206-T181246", "src": "iefbr14.jcl", "copy_member": false, "is_mvs_dest": true, "is_binary": false, "backup": false, "local_follow": true, "remote_src": false, "ignore_sftp_stderr": false, "validate": false, "force": false, "follow": false, "unsafe_writes": false, "encoding": {"from": "UTF-8", "to": "IBM-037"}, "content": null, "backup_name": null, "sftp_port": null, "volume": null, "dest_data_set": null, "mode": null, "owner": null, "group": null, "seuser": null, "serole": null, "selevel": null, "setype": null, "attributes": null, "regexp": null, "delimiter": null, "directory_mode": null, "from_encoding": "UTF-8", "to_encoding": "IBM-1047"}}}

The last thing that caught my eye which ill need some time to review is the plugin search path differs from your install path but it appears Ansible figured it out but still I want to look into, your install path is /home/tmyuser/.ansible/collections, Ansible searched paths `/usr/share/ansible/plugins/action:/usr/lib/python2.7/site-packages/ansible/plugins/action` and found the zos_copy action plugin, so I am not sure if the locale confusion has anything to do with this yet.

Loading ActionModule 'ansible_collections.ibm.ibm_zos_core.plugins.action.zos_copy' from /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action/zos_copy.py (searched paths: /usr/share/ansible/plugins/action:/usr/lib/python2.7/site-packages/ansible/plugins/action) (found_in_cache=False, class_only=True)

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Mon February 06, 2023 12:32 PM
From: Oliver Stadler
Subject: zos_copy module encoding question

And I just realized that I spelt your first name wrongly in my previous posts. Sorry for that but I got that now :-)

------------------------------
Oliver Stadler

Original Message:
Sent: Mon February 06, 2023 12:29 PM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetri,

I added the action_plugins path but it didn't make a difference.

Python version on the target server is 3.9. 

/usr/lpp/python/bin/python --version
Python 3.9.14

The config I used was as follows:

[tmyuser@controller ansible-zos]$ ansible-config dump --only-changed
ANSIBLE_PIPELINING(/home/tmyuser/ansible-zos/ansible.cfg) = True
COLLECTIONS_PATHS(/home/tmyuser/ansible-zos/ansible.cfg) = [u'/home/tmyuser/colltest', u'/usr/share/ansible/collections', u'/home/tmyuser/.ansible/collections', u'/usr/share/ansible/collections', u'/home/tmyuser/ansible_collections']
DEFAULT_ACTION_PLUGIN_PATH(/home/tmyuser/ansible-zos/ansible.cfg) = [u'/home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action/~/.ansible/plugins/action', u'/usr/share/ansible/plugins/action']

I also ran the playbook with the debugging enabled and verbose. Output is attached in playbook_output_clean.txt.

Kind regards,
Oliver

------------------------------
Oliver Stadler

Original Message:
Sent: Mon February 06, 2023 02:15 AM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hi Oliver,  I figured that was going to be the case with GitHub, your not the only client who has that policy, if you have Ansible Automation Hub entitlement you could go through L2, otherwise I don't mind working here.  For some reason I don't get notifications for this thread so I have to come back and review if there are updates.

Anyhow, thank you for the diagnosis, I guess I assumed too much without a log, there are a couple of things I can think of that might cause this but without the output from

ANSIBLE_DEBUG=1 ansible-playbook -i inventory your-playbook.yml -vvvv​

I have seen this happen when the controller can't find the action plugin (zos_copy has both , action and modules - see my blog for more on this), and I can usually tell from the log when I see a None/Null passed in as the controllers locale.  You can trouble shoot this by adding the path in your `ansible.cfg` , the entry would look similar to this:

[defaults]action_plugins=/Users/ddimatos/.ansible/collections/ansible_collections/ibm/ibm_zos_core/plugins/action

The other time, but its slight a different error (we are working on this) is with IBM Enterprise Python 3.10 or later but you are using 3.9 according to what Ansible discovered. You might try to not rely on that discovery and tell Ansible what python to use with property `ansible_python_interpreter`, as I can't tell without a log if it really did use 3.9, you can do that from your inventory with an entry that looks like:

zsystem:  hosts:    zvm:      ansible_host: <some host>      ansible_user: <some user>      ansible_python_interpreter: /python2/usr/lpp/IBM/cyp/v3r9/pyz/bin/python3.9

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Wed February 01, 2023 11:36 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetrios,

Thanks a lot for the feedback. I actually wanted to place the question directly in the Github discussion forum but unfortunately we are not allowed to due to company policy.

I checked the iefbr14.cjl source file on the controller and it is definitely not  EBCDIC (IBM-1047). 

$ file iefbr14.jcl
iefbr14.jcl: ASCII text

I also doublechecked the actual hex codes of some of the characters via the od command and they are in fact ASCII encoded.

When I look at the payload file which is copied during the execution of the task into /tmp (e.g. /tmp/ansible-zos-copy-payload-D230201-T155244) that looks fine (using -vvv to see the verbose output) . The file is not tagged and if I do a cat on it the file content is displayed correctly. So it looks like it gets converted later on. 

I wonder if we have something configured differently on Unix System Services side on our end that might cause this.

Kind regards,
Oliver



------------------------------
Oliver Stadler

Original Message:
Sent: Sun January 29, 2023 12:19 AM
From: DEMETRIOS DIMATOS
Subject: zos_copy module encoding question

Hello Oliver,
Looking at this snippet and from what you have shared in that you have from:to set to IBM-1047

  - name: Copy JCL without encoding    ibm.ibm_zos_core.zos_copy:      src: iefbr14.jcl      dest: "{{ my_seq }}"      backup: False      force: True

It seems that the JCL source `iefbr14.jcl` on the controller (eg your laptop) is encoded in EBCDIC (IBM-1047) so when you don't specify an encoding what the `zos_copy` module does is take the `locale`of the controller which in most cases is either `UTF8` or `ISO8859-1` and uses that as the `from` and when the module runs on the managed node (z/OS) it takes the `locale` there and sets it as the `to` which in most cases is `IBM-1047`.

Why `iefbr14.jcl` ends up unreadable is because the module has used incorrect inputs/oputput (from/to) for its encoding when you don't specify. We rely on these user inputs because its very difficult to detect and correctly asses a files encoding reading a files Byte Order Mark (BOM) and has a high probability of inaccuracy in doing so.

In your case you have a few options, if the file is already in the desired encoding on your controller eg IBM-1047, you can continue to set the FROM an TO encoding options as IBM-1047 for both so that the  module does not perform any encoding conversion or you can instruct the module the file is binary and should prevent any conversion but I don't recommend you use the binary trick. Your best option is to have the file on the controller in the controllers encoding or continue to do what you are doing.

In this article under section "Plugins and modules" I cover the locale and architecture a bit if you are interested.
We try to monitor all sources and this one was overlooked, feel free to also use our repositories discussions if you are permitted to access it.

------------------------------
DEMETRIOS DIMATOS
IBM zOS Ansible Core Senior Technical Lead
IBM zOS Ansible Core Senior Technical Lead
San Jose CA

Original Message:
Sent: Fri January 06, 2023 10:42 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi,

I have another encoding related question. When I copy an ASCII file using the zos_copy module it works fine if I specify IBM-1047 both in the from and to argument. If I don't specify an encoding the file arrives corrupted. Specifying the encoding it no big deal but it's somewhat confusing that I have to specify IBM-1047 as "from" encoding when copying an ASCII file.

This is the file I use as input
(az1) [myuser@myhost ansible-zos]$ cat iefbr14.jcl
//MYUSERA JOB (123),'LOCAL',CLASS=A,MSGCLASS=S,MSGLEVEL=(1,1),
// NOTIFY=&SYSUID
//*
//STEP001 EXEC PGM=IEFBR14
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSDUMP DD SYSOUT=*
//*

The following playbook shows the behaviour.

(az1) [myuser@myhost ansible-zos]$ cat zos_copy_example.yml
---
- name: Working with datasets
  hosts: all
  gather_facts: False
  vars:
  - my_seq: MYUSER.TEST3.SEQ

  tasks:
  - name: Create {{ my_seq }}
    ibm.ibm_zos_core.zos_data_set:
      name: "{{ my_seq }}"
      type: seq
      record_format: VB
      record_length: 255

  - name: Copy JCL with encoding
    ibm.ibm_zos_core.zos_copy:
      src: iefbr14.jcl
      dest: "{{ my_seq }}"
      backup: False
      force: True
    encoding:
      to: IBM-1047
      from: IBM-1047

  - name: Submit the job
    ibm.ibm_zos_core.zos_job_submit:
      location: DATA_SET
      src: "{{ my_seq }}"

  - name: Copy JCL without encoding
    ibm.ibm_zos_core.zos_copy:
      src: iefbr14.jcl
      dest: "{{ my_seq }}"
      backup: False
      force: True

  - name: Submit the job
    ibm.ibm_zos_core.zos_job_submit:
      location: DATA_SET
      src: "{{ my_seq }}"
...

When I run it I get the following result.

(az1) [myuser@myhost ansible-zos]$ ansible-playbook -i inventory -l mysys zos_copy_example.yml
[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the controller starting with Ansible 2.12. Current version: 3.6.8 (default, Aug 13 2020, 07:46:32) [GCC 4.8.5 20150623 (Red Hat
4.8.5-39)]. This feature will be removed from ansible-core in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [Working with datasets] ************************************************************************************************************************************************************************

TASK [Create MYUSER.TEST3.SEQ] *********************************************************************************************************************************************************************
[WARNING]: Platform os/390 on host mysys is using the discovered Python interpreter at /usr/lpp/python/bin/python3.9, but future installation of another Python interpreter could change
the meaning of that path. See https://docs.ansible.com/ansible-core/2.11/reference_appendices/interpreter_discovery.html for more information.
ok: [mysys]

TASK [Copy JCL with encoding] ***********************************************************************************************************************************************************************
changed: [mysys]

TASK [Submit the job] *******************************************************************************************************************************************************************************
changed: [mysys]

TASK [Copy JCL without encoding] ********************************************************************************************************************************************************************
changed: [mysys]

TASK [Submit the job] *******************************************************************************************************************************************************************************
fatal: [mysys]: FAILED! => {"changed": false, "msg": "SubmitJCLError('Submit job failed: Stderr :FSUMB408 submit: data set MYUSER.TEST3.SEQ Not accepted by JES.\\n')"}

PLAY RECAP ******************************************************************************************************************************************************************************************
mysys: ok=4 changed=3 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

------------------------------
Oliver Stadler
------------------------------

Hi Demetri,

You are right the hex codes differ somewhat because in one case there were two blanks after the jobname and in other case not.

Therefore I reproduced the problem using an input file that matches exactly with yours (same hex codes). It looks like this:

$ od -x files/ascii.txt
0000000 2f2f 5435 3037 3632 3241 204a 4f42 2028
0000020 3132 3329 2c27 4c4f 4341 4c27 2c43 4c41
0000040 5353 3d41 2c4d 5347 434c 4153 533d 532c
0000060 4d53 474c 4556 454c 3d28 312c 3129 2c0a
0000100

It is identified as us-ascii:

$ file -i files/ascii.txt
files/ascii.txt: text/plain; charset=us-ascii

Then I took your playbook and executed it. I just commented the line "environment: "{{ environment_vars }}" because it caused an "undefined variable" error.

Result of playbook:

TASK [Result of copying local directory] ******************************************************************
ok: [mysys] => {
    "msg": {
        "changed": true,
        "dest": "MYUSER.CONTENT.SEQ",
        "failed": false,
        "is_binary": false,
        "src": "/home/myuser/ansible-zos/files/ascii.txt"
    }
}

PLAY RECAP ************************************************************************************************
mysys               : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The content of the dataset looks like this:
********************************* Top of Data **********************************

 ------------------------------------------------------------------------------
aaãõð÷öòòÁ@ÑÖÂ@Mñòó¨k}ÓÖÃÁÓ}kÃÓÁââ~ÁkÔâÇÃÓÁââ~âkÔâÇÓÅåÅÓ~Mñkñ¨k.
884C8ECCC676E67D4CCB9DEE66ED96E644A69E466E644A49E46E646EAD494B934444444444444444
116FC1CDD5C9C2C49DEB20EC65E026E522152B286E522122B28E777E14929B2D0000000000000000
 ------------------------------------------------------------------------------
******************************** Bottom of Data ********************************

Kind regards,

Oliver

------------------------------
Oliver Stadler

Original Message:
Sent: Sat February 11, 2023 12:59 AM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hello Oliver,

Here is what I have so far:

1. Created file ascii.txt with this content:
   1. //T507622A JOB (123),'LOCAL',CLASS=A,MSGCLASS=S,MSGLEVEL=(1,1),(ansible-doc
   2. With hex values:

      1. 0000000 2f2f 5435 3037 3632 3241 204a 4f42 2028  //T507622A JOB (0000010 3132 3329 2c27 4c4f 4341 4c27 2c43 4c41  123),'LOCAL',CLA0000020 5353 3d41 2c4d 5347 434c 4153 533d 532c  SS=A,MSGCLASS=S,0000030 4d53 474c 4556 454c 3d28 312c 3129 2c0a  MSGLEVEL=(1,1),.

2. With playbook:

   1. ---- hosts: zvm  gather_facts: false  vars:    - SEQ: "ANSIBLE.CONTENT.SEQ"  environment: "{{ environment_vars }}"  collections:    - ibm.ibm_zos_core  tasks:    - name: Create/replace the PDS      zos_data_set:        name: "{{ SEQ }}"        type: seq        space_primary: 5        space_type: M        record_format: fb        record_length: 80        block_size: 3120        replace: true      register: result    - name: Copy local file to PDS      zos_copy:        src: "{{playbook_dir}}/files/ascii.txt"        dest: "{{ SEQ }}"        remote_src: false        encoding:           from: ISO8859-1           to: IBM-1047      register: result    - name: Result of copying local directory      debug:        msg: "{{ result }}"

3. And on z/OS when I look at the sequential data set content in Hex I have:

   1. //T507622A JOB (123),'LOCAL',CLASS=A,MSGCLASS=S,MSGLEVEL=(1,1),                 66EFFFFFFC4DDC44FFF567DDCCD76CDCEE7C6DECCDCEE7E6DECDCECD74F6F5644444444444444444113507622101620D123DBD36313DB33122E1B42733122E2B42735553ED1B1DB00000000000000000

Looking at the last bullet (3) hex content and picking random characters they correspond to code page IBM-1047:

• / is HEX 61
• L is HEX D3
• ( is HEX 4D

What I did notice in your ASCII content (assuming ISO-8859-1) , partway through some of the characters don't seem to correspond to ISO-8859-1. Lets review this portion  //T507622A JOB (.

Text:        //T507622A JOB (

Mine:      2f2f 5435 3037 3632 3241 204a 4f42 2028

Yours:    2f2f 5435 3037 3632 3241 2020 4a4f 4220

HEX values begin to not align , my:

• 4a vs 20
• 4f vs 4a
• 42 vs 4f

It appears we don't have the same data to work with. Could you send me your ASCII content, feel free to email me or attach it here but  I wonder if that is the source of the issue where the conversion goes wrong because source may not be consistent with ISO-8859-1.

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Fri February 10, 2023 10:41 PM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hi Oliver, 

I must have not understood 2 messages back, thank you for clarifying. I will try to reproduce this, this is the only way for me to track where the inconsistency is happening, thank you for clearly explaning this helps me quiet a bit. I will get back to you shortly. 

I also went ahead and created a UTF-8 encoded version of the same ISO-88591 content and they match:
UTF-8: 2f2f 5435 3037 3632 3241 204a 4f42 2028  for content //T507622A JOB (

You could try the `file -I` command to check the the encoding source:

$ file -I utf-8.txtutf-8.txt: text/plain; charset=utf-8$ file -I ascii.txtascii.txt: text/plain; charset=us-ascii

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Wed February 08, 2023 01:59 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetri,

No it's not yet working. Below end to end what I see happening.

On the controller the original file looks ASCII like this (just looking at the first line to limit the output)

$ head -1 iefbr14.jcl
//T507622A JOB (123),'LOCAL',CLASS=A,MSGCLASS=S,MSGLEVEL=(1,1),

First two characters are the forward slashes ASCII encoded as hex 2f 2f 
$ head -1 iefbr14.jcl | od -x
0000000 2f2f 5435 3037 3632 3241 2020 4a4f 4220
0000020 2831 3233 292c 274c 4f43 414c 272c 434c
0000040 4153 533d 412c 4d53 4743 4c41 5353 3d53
0000060 2c4d 5347 4c45 5645 4c3d 2831 2c31 292c
0000100 0a00

When I look at the payload file that Ansible creates when doing the copy to the USS system the files is converted to EBCDIC. First two characters are now hex 61 61 which corresponds to the forward slashes in EBCDIC.

head -1 /tmp/ansible-zos-copy-payload-D230208-T073216 | od -x
0000000000 6161 E3F5 F0F7 F6F2 F2C1 4040 D1D6 C240
0000000020 4DF1 F2F3 5D6B 7DD3 D6C3 C1D3 7D6B C3D3
0000000040 C1E2 E27E C16B D4E2 C7C3 D3C1 E2E2 7EE2
0000000060 6BD4 E2C7 D3C5 E5C5 D37E 4DF1 6BF1 5D6B
0000000100 1500
0000000101
0000101

However in the target data set the arriving content looks like this. The first two characters are encoded as hex 81.

********************************* Top of Data **********************************

-----------------------------------------------------------------------------
aa..@@..M.k}.Á.kÓb.k........~M.k.aa@@@@@@@......aa\.aa.....@...............aa...
88337732D39D3639E83933333333AD39388777777733333388E38833333733333333333333388333
11FFCCF04F20F5F2E2F2FFFFFFFF14F2D11CCCCCCCFFFFFD110D11FFFFFCFFFFFFFFFFFFFFD11FFF
------------------------------------------------------------------------------
******************************** Bottom of Data ********************************

Which is what I get if I convert the already EBCDIC content as UTF-8 to IBM-1047.

head -1 /tmp/ansible-zos-copy-payload-D230208-T073216 | iconv -f UTF-8 -t IBM-1047 | od -x
0000000000 8181 3F3F 7C7C 3F20 D43F 92D0 3F65 3F92
0000000020 EE82 3F92 3F3F 3F3F 3F3F 3F3F A1D4 3F92
0000000040 3D00
0000000041


Kind regards,
Oliver

------------------------------
Oliver Stadler

Original Message:
Sent: Tue February 07, 2023 04:47 PM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hello Oliver, it looks like things are good with regard to the plugin path and discovery from what I can see in your log. I found in our code where we are setting a `IBM-037`, I think it was an oversight given the slight variation in vars were `DEFAULT_EBCDIC_USS_CHARSET` vs `DEFAULT_EBCDIC_MVS_CHARSET`. 

I can't tell from your last message if its working for you now because it seems like you are sharing the expected HEX and the actual HEX which are the same and thus you are good now.

If you this is working for you then I will put the investigation of our use of `DEFAULT_EBCDIC_USS_CHARSET` in the backlog.

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Tue February 07, 2023 04:00 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetri,

zos collection version is 1.4

[tmyuser@mycontroller ansible-zos]$ cat /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/MANIFEST.json | grep version
"version": "1.4.0",

Locale on USS side is IBM-1047

[tmyuser@mycontroller ansible-zos]$ ansible -i inventory myplex1 -m command -a 'locale -c charmap'
[WARNING]: Platform os/390 on host mysys1.com is using the discovered Python interpreter at /usr/lpp/python/bin/python, but future installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.
mysys1.com | CHANGED | rc=0 >>
IBM-1047

I noticed that there was actually a typo in the plugin path. I cleaned that up. Config looks like this now:

[myuser@mycontroller ansible-zos]$ ansible-config dump --only-changed
ANSIBLE_PIPELINING(/home/myuser/ansible-zos/ansible.cfg) = True
COLLECTIONS_PATHS(/home/myuser/ansible-zos/ansible.cfg) = [u'/home/myuser/colltest', u'/usr/share/ansible/collections']
DEFAULT_ACTION_PLUGIN_PATH(/home/myuser/ansible-zos/ansible.cfg) = [u'/home/myuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action', u'/usr/share/ansible/plugins/action']

I also rerun the playbook with this config and attached the output (playbook_output3.txt)

It looks like the actual conversion which is done is from ASCII to IBM-1047.

The target dataset in hex looks like this:

Command ===> Scroll ===> CSR
********************************* Top of Data **********************************


------------------------------------------------------------------------------
aa..@@..M.k}.Á.kÓb.k........~M.k.aa@@@@@@@......aa\.aa.....@...............aa...
88337732D39D3639E83933333333AD39388777777733333388E38833333733333333333333388333
11FFCCF04F20F5F2E2F2FFFFFFFF14F2D11CCCCCCCFFFFFD110D11FFFFFCFFFFFFFFFFFFFFD11FFF
------------------------------------------------------------------------------
******************************** Bottom of Data ********************************

That corresponds to the output I get when I run an iconv on the file being copied.
Tmyuser@TR01:/home/tmyuser ==> iconv -f UTF-8 -t IBM-1047 iefbr14.jcl | od -x
0000000000 8181 3F3F 7C7C 3F20 D43F 92D0 3F65 3F92
0000000020 EE82 3F92 3F3F 3F3F 3F3F 3F3F A1D4 3F92
0000000040 3D81 817C 7C7C 7C7C 7C7C 3F3F 3F3F 3F3D
0000000060 8181 E03D 8181 3F3F 3F3F 3F7C 3F3F 3F3F
0000000100 3F3F 3F3F 3F3F 3F3F 3F3F 3D81 813F 3F3F
0000000120 7C3F 7C3F 3F3F 3D81 813F 3F3F 7C7C 3F7C
0000000140 3F3F 3F3D 8181 E03D
0000000150



------------------------------
Oliver Stadler

Original Message:
Sent: Mon February 06, 2023 05:43 PM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hello Oliver, actually you had my name correct, full pronunciation is `Demetrios`, I updated it to the shortened `Demetri` the other day, no problems here on my end how you refer to me :) .

I did a quick pass and noticed something, it appears the code is either detecting IBM-037 or your setting it to IBM-037 (but I think you are setting it to IBM-1047) , i have to look a the code to follow the path and where the swap is happening.
- Which version of the IBM z/OS core collection are you using, eg

cat /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/MANIFEST.json | grep version

- What does USS command on the managed node (one you are connecting to) yield: `locale -c charmap`, eg I get:

# locale -c charmapIBM-1047

The log snippet that has me questioning the code is:

{"changed": true, "src": "iefbr14.jcl", "dest": "Tmyuser.TEST4.SEQ", "ds_type": "PS", "dest_exists": true, "backup_name": null, "invocation": {"module_args": {"is_pds": false, "src_member": false, "size": 194, "local_charset": "UTF-8", "dest": "Tmyuser.TEST4.SEQ", "is_src_dir": false, "is_uss": false, "temp_path": "/tmp/ansible-zos-copy-payload-D230206-T181246", "src": "iefbr14.jcl", "copy_member": false, "is_mvs_dest": true, "is_binary": false, "backup": false, "local_follow": true, "remote_src": false, "ignore_sftp_stderr": false, "validate": false, "force": false, "follow": false, "unsafe_writes": false, "encoding": {"from": "UTF-8", "to": "IBM-037"}, "content": null, "backup_name": null, "sftp_port": null, "volume": null, "dest_data_set": null, "mode": null, "owner": null, "group": null, "seuser": null, "serole": null, "selevel": null, "setype": null, "attributes": null, "regexp": null, "delimiter": null, "directory_mode": null, "from_encoding": "UTF-8", "to_encoding": "IBM-1047"}}}

The last thing that caught my eye which ill need some time to review is the plugin search path differs from your install path but it appears Ansible figured it out but still I want to look into, your install path is /home/tmyuser/.ansible/collections, Ansible searched paths `/usr/share/ansible/plugins/action:/usr/lib/python2.7/site-packages/ansible/plugins/action` and found the zos_copy action plugin, so I am not sure if the locale confusion has anything to do with this yet.

Loading ActionModule 'ansible_collections.ibm.ibm_zos_core.plugins.action.zos_copy' from /home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action/zos_copy.py (searched paths: /usr/share/ansible/plugins/action:/usr/lib/python2.7/site-packages/ansible/plugins/action) (found_in_cache=False, class_only=True)

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Mon February 06, 2023 12:32 PM
From: Oliver Stadler
Subject: zos_copy module encoding question

And I just realized that I spelt your first name wrongly in my previous posts. Sorry for that but I got that now :-)

------------------------------
Oliver Stadler

Original Message:
Sent: Mon February 06, 2023 12:29 PM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetri,

I added the action_plugins path but it didn't make a difference.

Python version on the target server is 3.9. 

/usr/lpp/python/bin/python --version
Python 3.9.14

The config I used was as follows:

[tmyuser@controller ansible-zos]$ ansible-config dump --only-changed
ANSIBLE_PIPELINING(/home/tmyuser/ansible-zos/ansible.cfg) = True
COLLECTIONS_PATHS(/home/tmyuser/ansible-zos/ansible.cfg) = [u'/home/tmyuser/colltest', u'/usr/share/ansible/collections', u'/home/tmyuser/.ansible/collections', u'/usr/share/ansible/collections', u'/home/tmyuser/ansible_collections']
DEFAULT_ACTION_PLUGIN_PATH(/home/tmyuser/ansible-zos/ansible.cfg) = [u'/home/tmyuser/colltest/ansible_collections/ibm/ibm_zos_core/plugins/action/~/.ansible/plugins/action', u'/usr/share/ansible/plugins/action']

I also ran the playbook with the debugging enabled and verbose. Output is attached in playbook_output_clean.txt.

Kind regards,
Oliver

------------------------------
Oliver Stadler

Original Message:
Sent: Mon February 06, 2023 02:15 AM
From: Demetri Dimatos
Subject: zos_copy module encoding question

Hi Oliver,  I figured that was going to be the case with GitHub, your not the only client who has that policy, if you have Ansible Automation Hub entitlement you could go through L2, otherwise I don't mind working here.  For some reason I don't get notifications for this thread so I have to come back and review if there are updates.

Anyhow, thank you for the diagnosis, I guess I assumed too much without a log, there are a couple of things I can think of that might cause this but without the output from

ANSIBLE_DEBUG=1 ansible-playbook -i inventory your-playbook.yml -vvvv​

I have seen this happen when the controller can't find the action plugin (zos_copy has both , action and modules - see my blog for more on this), and I can usually tell from the log when I see a None/Null passed in as the controllers locale.  You can trouble shoot this by adding the path in your `ansible.cfg` , the entry would look similar to this:

[defaults]action_plugins=/Users/ddimatos/.ansible/collections/ansible_collections/ibm/ibm_zos_core/plugins/action

The other time, but its slight a different error (we are working on this) is with IBM Enterprise Python 3.10 or later but you are using 3.9 according to what Ansible discovered. You might try to not rely on that discovery and tell Ansible what python to use with property `ansible_python_interpreter`, as I can't tell without a log if it really did use 3.9, you can do that from your inventory with an entry that looks like:

zsystem:  hosts:    zvm:      ansible_host: <some host>      ansible_user: <some user>      ansible_python_interpreter: /python2/usr/lpp/IBM/cyp/v3r9/pyz/bin/python3.9

------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose CA

Original Message:
Sent: Wed February 01, 2023 11:36 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi Demetrios,

Thanks a lot for the feedback. I actually wanted to place the question directly in the Github discussion forum but unfortunately we are not allowed to due to company policy.

I checked the iefbr14.cjl source file on the controller and it is definitely not  EBCDIC (IBM-1047). 

$ file iefbr14.jcl
iefbr14.jcl: ASCII text

I also doublechecked the actual hex codes of some of the characters via the od command and they are in fact ASCII encoded.

When I look at the payload file which is copied during the execution of the task into /tmp (e.g. /tmp/ansible-zos-copy-payload-D230201-T155244) that looks fine (using -vvv to see the verbose output) . The file is not tagged and if I do a cat on it the file content is displayed correctly. So it looks like it gets converted later on. 

I wonder if we have something configured differently on Unix System Services side on our end that might cause this.

Kind regards,
Oliver



------------------------------
Oliver Stadler

Original Message:
Sent: Sun January 29, 2023 12:19 AM
From: DEMETRIOS DIMATOS
Subject: zos_copy module encoding question

Hello Oliver,
Looking at this snippet and from what you have shared in that you have from:to set to IBM-1047

  - name: Copy JCL without encoding    ibm.ibm_zos_core.zos_copy:      src: iefbr14.jcl      dest: "{{ my_seq }}"      backup: False      force: True

It seems that the JCL source `iefbr14.jcl` on the controller (eg your laptop) is encoded in EBCDIC (IBM-1047) so when you don't specify an encoding what the `zos_copy` module does is take the `locale`of the controller which in most cases is either `UTF8` or `ISO8859-1` and uses that as the `from` and when the module runs on the managed node (z/OS) it takes the `locale` there and sets it as the `to` which in most cases is `IBM-1047`.

Why `iefbr14.jcl` ends up unreadable is because the module has used incorrect inputs/oputput (from/to) for its encoding when you don't specify. We rely on these user inputs because its very difficult to detect and correctly asses a files encoding reading a files Byte Order Mark (BOM) and has a high probability of inaccuracy in doing so.

In your case you have a few options, if the file is already in the desired encoding on your controller eg IBM-1047, you can continue to set the FROM an TO encoding options as IBM-1047 for both so that the  module does not perform any encoding conversion or you can instruct the module the file is binary and should prevent any conversion but I don't recommend you use the binary trick. Your best option is to have the file on the controller in the controllers encoding or continue to do what you are doing.

In this article under section "Plugins and modules" I cover the locale and architecture a bit if you are interested.
We try to monitor all sources and this one was overlooked, feel free to also use our repositories discussions if you are permitted to access it.

------------------------------
DEMETRIOS DIMATOS
IBM zOS Ansible Core Senior Technical Lead
IBM zOS Ansible Core Senior Technical Lead
San Jose CA

Original Message:
Sent: Fri January 06, 2023 10:42 AM
From: Oliver Stadler
Subject: zos_copy module encoding question

Hi,

I have another encoding related question. When I copy an ASCII file using the zos_copy module it works fine if I specify IBM-1047 both in the from and to argument. If I don't specify an encoding the file arrives corrupted. Specifying the encoding it no big deal but it's somewhat confusing that I have to specify IBM-1047 as "from" encoding when copying an ASCII file.

This is the file I use as input
(az1) [myuser@myhost ansible-zos]$ cat iefbr14.jcl
//MYUSERA JOB (123),'LOCAL',CLASS=A,MSGCLASS=S,MSGLEVEL=(1,1),
// NOTIFY=&SYSUID
//*
//STEP001 EXEC PGM=IEFBR14
//SYSPRINT DD SYSOUT=*
//SYSOUT DD SYSOUT=*
//SYSDUMP DD SYSOUT=*
//*

The following playbook shows the behaviour.

(az1) [myuser@myhost ansible-zos]$ cat zos_copy_example.yml
---
- name: Working with datasets
  hosts: all
  gather_facts: False
  vars:
  - my_seq: MYUSER.TEST3.SEQ

  tasks:
  - name: Create {{ my_seq }}
    ibm.ibm_zos_core.zos_data_set:
      name: "{{ my_seq }}"
      type: seq
      record_format: VB
      record_length: 255

  - name: Copy JCL with encoding
    ibm.ibm_zos_core.zos_copy:
      src: iefbr14.jcl
      dest: "{{ my_seq }}"
      backup: False
      force: True
    encoding:
      to: IBM-1047
      from: IBM-1047

  - name: Submit the job
    ibm.ibm_zos_core.zos_job_submit:
      location: DATA_SET
      src: "{{ my_seq }}"

  - name: Copy JCL without encoding
    ibm.ibm_zos_core.zos_copy:
      src: iefbr14.jcl
      dest: "{{ my_seq }}"
      backup: False
      force: True

  - name: Submit the job
    ibm.ibm_zos_core.zos_job_submit:
      location: DATA_SET
      src: "{{ my_seq }}"
...

When I run it I get the following result.

(az1) [myuser@myhost ansible-zos]$ ansible-playbook -i inventory -l mysys zos_copy_example.yml
[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the controller starting with Ansible 2.12. Current version: 3.6.8 (default, Aug 13 2020, 07:46:32) [GCC 4.8.5 20150623 (Red Hat
4.8.5-39)]. This feature will be removed from ansible-core in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [Working with datasets] ************************************************************************************************************************************************************************

TASK [Create MYUSER.TEST3.SEQ] *********************************************************************************************************************************************************************
[WARNING]: Platform os/390 on host mysys is using the discovered Python interpreter at /usr/lpp/python/bin/python3.9, but future installation of another Python interpreter could change
the meaning of that path. See https://docs.ansible.com/ansible-core/2.11/reference_appendices/interpreter_discovery.html for more information.
ok: [mysys]

TASK [Copy JCL with encoding] ***********************************************************************************************************************************************************************
changed: [mysys]

TASK [Submit the job] *******************************************************************************************************************************************************************************
changed: [mysys]

TASK [Copy JCL without encoding] ********************************************************************************************************************************************************************
changed: [mysys]

TASK [Submit the job] *******************************************************************************************************************************************************************************
fatal: [mysys]: FAILED! => {"changed": false, "msg": "SubmitJCLError('Submit job failed: Stderr :FSUMB408 submit: data set MYUSER.TEST3.SEQ Not accepted by JES.\\n')"}

PLAY RECAP ******************************************************************************************************************************************************************************************
mysys: ok=4 changed=3 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

------------------------------
Oliver Stadler
------------------------------