Thank you Demetrios! That was very helpful for me to understand what exactly is going on here.
Surrounding my replace step with zos_encode steps that convert the file from IBM-1047 to ISO8859-1 and backwards helped.
It would be great to have an improvement here as soon as possible. Replacing strings in files and datasets is something we need very often.
Original Message:
Sent: Fri August 23, 2024 12:04 PM
From: Demetri Dimatos
Subject: Using ansible.builtin.replace for Unix System Services files
Hi Martin Weiß,
Since there are no details I am going to assume the content you are replacing is encoded in EBCDIC.
There is some actions you must take before using ansible.builtin modules that interact with files on z/OS USS that are not encoded in UTF-8. It is subtle but it is in the documentation:
Much of Ansible assumes that all encoded text is UTF-8. At some point, if there is demand for other encodings we may change that, but for now it is safe to assume that bytes are UTF-8.
The Ansible team at IBM is considering expanding the mission beyond the Z collections which not only handle the encoding issues but also support MVS, here is a rough uncommitted plan:
- Create a 'zos_replace' module that works with z/OS USS and z/OS data sets. (roadmap item - no approximate date but I would estimate a year or so as other modules are already queued)
- Supply updated documentation to the community regarding encodings and additional operations needed when using community modules (currently a work in progress)
- Expand our mission Ansible mission to include the community modules (discussions in progress)
On to the work around, which for this, I recommend you use the zos_encode module but in the event you don't have access to it and to better depict the issue, I will use shell commands.
$ cat test.1047AAAAaaaaBBBBbbbb# See that it is tagged as IBM-1047, yet we need a UTF-8 file and tagged appropriately so lets make a copy.$ ls -laT test*m IBM-1047 T=off -rw-r--r-- 1 BPXROOT OMVSGRP 20 Aug 23 08:09 test.1047# Make a copy and covert in one line command$ iconv -f IBM-1047 -t UTF-8 test.1047 > test.utf8# Tag the UTF-8 file as it should be.$ chtag -t -c ISO8859-1 test.utf8# Everything is tagged accordingly$ ls -laT test*m IBM-1047 T=off -rw-r--r-- 1 BPXROOT OMVSGRP 20 Aug 23 08:09 test.1047t ISO8859-1 T=on -rw-r--r-- 1 BPXROOT OMVSGRP 20 Aug 23 08:12 test.utf8# Dump the hex just to demonstrate this, HEX 0xC1 in EBCDIC is 'A' (correct)$ od -x test.10470000000000 C1C1 C1C1 1581 8181 8115 C2C2 C2C2 15820000000020 8282 82150000000024# Dump the hex just to demonstrate this, HEX 0x11 in UTF-8 is 'A' (correct)$ od -x test.utf80000000000 4141 4141 0A61 6161 610A 4242 4242 0A620000000020 6262 620A
Now run the playbook using the ansible.builtin.replace module:
---- hosts: zvm gather_facts: false environment: "{{ environment_vars }}" tasks: - name: Replace text ansible.builtin.replace: path: /tmp/test.utf8 regexp: 'AAAA' replace: 'CCCC'
Adhoc results:
venv-2.16) ansible-playbook -i inventory ansible-builtin-replace.ymlPLAY [zvm] *******************************************************TASK [Replace text] **********************************************ok: [zvm]PLAY RECAP *******************************************************************zvm : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Now lets look a what happened on z/OS USS:
# Oh oh, whats going on? Nothing, the community module does not know about z/OS tags so it edited the UTF8 module and in doing so the tag was lost, lets fix that.$ cat test.utf8����////��������$ chtag -t -c ISO8859-1 test.utf8$ ls -laT test*m IBM-1047 T=off -rw-r--r-- 1 BPXROOT OMVSGRP 20 Aug 23 08:09 test.1047t ISO8859-1 T=on -rw-r--r-- 1 BPXROOT OMVSGRP 20 Aug 23 08:17 test.utf8# Now you can see my replace of 'AAAA' has completed with 'CCCC'$ cat test.utf8CCCCaaaaBBBBbbbb# Still not sure, lets dump the hex, UTF-8 0X43 is 'C', success !!!$ od -x test.utf80000000000 4343 4343 0A61 6161 610A 4242 4242 0A620000000020 6262 620A0000000024
Now what should I do with the UTF-8 file, that is up to you but my guess is you probably want it back in EBCDIC, earlier I stated that you probably should use the `zos_encode` module, so instead of using the chtag, I will demonstrate the module.
Why use zos_encode? It does so much more than just encode USS files, it handles directories, z/OS data sets, offers backups, etc, have a look at the docs.
---- hosts: zvm gather_facts: false environment: "{{ environment_vars }}" tasks: - name: Convert file encoding from to ISO8859-1 to IBM-1047 for the same file zos_encode: src: /tmp/test.utf8 encoding: from: UTF-8 to: IBM-1047
Result:
# Here you can see the `zos_encode` module tagged your file and converted the encoding from UTF8 to IBM-1047.$ ls -laT test*m IBM-1047 T=off -rw-r--r-- 1 BPXROOT OMVSGRP 20 Aug 23 08:09 test.1047t IBM-1047 T=on -rw-r--r-- 1 BPXROOT OMVSGRP 20 Aug 23 08:57 test.utf8# Lets make sure$ cat test.utf8CCCCaaaaBBBBbbbb# Still not sure, lets dump the hex, EBCDIC 0xC3 is 'C' , success !!!$ od -x test.utf80000000000 C3C3 C3C3 1581 8181 8115 C2C2 C2C2 15820000000020 8282 82150000000024
While this seems like quite a bit to do, its quite simple if you use the `zos_encode` module before and after the `ansible.builtin.replace` module, else you can rely on the community `command`, `shell` , `raw` modules to run the shell commands accordingly.
Demetrios Dimatos
------------------------------
Demetri Dimatos
IBM zOS Ansible Core Senior Technical Lead
IBM
San Jose
Original Message:
Sent: Fri August 23, 2024 05:25 AM
From: Martin Weiß
Subject: Using ansible.builtin.replace for Unix System Services files
Does anyone know if the ansible.builtin.replace should generally work with USS files? I was not able to use it in our installation. Are there any special requirements?
There is no error message but it doesnt replace any content.
------------------------------
Martin Weiß
------------------------------