Ansible for IBM Z

Hi Mark,

The first thing that I notice here is that the directive for choosing how to use elevated permissions is become_method: su 

 Using become:method:su wouldn't be valid YAML

You could also use ansible_become_method=su in the inventory.

Thanks Ian.

Sorry - the 'become:method: su' was a typo - that I copied a pasted to generate the same typo twice :-)

It should have read 'become_method: su' - which is valid, and is where it tried to switch to 'root'

I have just tried your suggestion of using ansible_become_method in the inventory, but that also result in it trying to switch to 'root'.

I also tried using ansible_become_user in the inventory, but the behaviour is the same a using become_user in the playbook.

Thanks for the suggestion, but it's not working for me.

Hello Mark, 

Before we try to recreate or look at source, could you clarify a few things?
Lets assume the OMVS Segment is `ibmuser` and your Ansible user is `ibmuser`. 

1) For clarity, when you don't set `become_user:` it defaults to root?

2) For clarity, when you set `become_user:ibmuser` user access is not being elevated? 

If have understood both 1 & 2 correctly, can you clarify this point "I do not want to switch to an alternate id - I am elevating the id that is logged on":
3) How are you elevating the ID without expecting an alternate ID switch when using `su`?  Unlike `sudo` (available via Open Enterprise Foundation) where you can do that, the `su` documentation states "If you do not specify a user ID, su changes your authorization to that of the superuser" (see below):

If you do not specify a user ID, su changes your authorization to that of the superuser. The resulting MVS user ID is the user ID specified in BPXPRMxx by the SUPERUSER statement, and the default for that user ID is BPXROOT. The installation might have specified a different user ID as the default user ID for the SUPERUSER statement.

Which is how my system behaves:

# whoami
BPXROOT
# su USRT001
FSUM5019 Enter the password for USRt001:
$ whoami
USRT001
$ su
# whoami
BPXROOT

A quick glance of Ansibles code looks to me like its using an empty string when `become_user` is unset, we would need to look closer at the source and also collect a log from you but before any of that begins to understand why `root` was passed.

I am looking to understand bullet (3) because if the does not need to switch to an alternate ID that which has additional needed privileges, then why escalate at all, seems the user would have what they need. 

Hello Demetri,

First to clarify your 2 points:

1. That is correct - if I don't set 'become_user' it defaults to 'root'
2. That is also correct - but I have since also tried setting 'become_user' to BPXROOT - see details below.

I can see that my wording "I do not want to switch to an alternate id - I am elevating the id that is logged on"  is not totally clear. What I was trying to say is that on the 'su' command I don't provide an alternate id - but as you say the id does in fact switch - like in your case - to BPXROOT.

My system does behave exactly as you suggest - and switches to BPXROOT when su in invoked.

If I set become_user: "BPXROOT", I get a timeout as it is waiting for a password: "Timeout (12s) waiting for privilege escalation prompt:" 

If I also set become_exe: "su -s" I get the following: "FSUM5027 su: User is not a surrogate of \"BPXROOT\".\r\n"

Thanks

Hi Mark, thanks for the clarity.

For the first point, "if I don't set 'become_user' it defaults to 'root'" , I have asked the team to look into the sources default behavior; and then we can revisit this part of the discussion.

For the "If I set become_user: "BPXROOT"", I get a timeout as it is waiting for a password: "Timeout (12s) waiting for privilege escalation prompt:" , I believe this is because the community source is expecting a different default prompt (this is something we looking into changing), the default prompt ansible-core is expecting is `password:` yet  z/OS always prompts with `FSUM5019 Enter the password for ......`, so ansilbe-core waits for the prompt `password:` and never sees it resulting in a timeout. 

The solution for now (until we work to change the default for z/OS) is to instruct ansible-core to interpret the z/OS privilege escalation prompt.

There are several ways you can do this which I have covered in this blog, its long because I explain them all including vaulting.
The shortest path to success (not the recommended) is to insert this into your playbook:

  vars: 
    ansible_su_prompt_l10n:
     - FSUM5019 Enter the password for BPXROOT

and set:
become_user: BPXROOT

This should work. I recommend looking over the blog in detail, it will provide all the options. 

Thanks Demetri.

This is not currently working for me, but I will take a more thorough look at the blog (next week). I am still getting the Timeout.

I am also not sure we can use the vault because I do not know the password for BPXROOT - and there is no way our security team will ever release that to anyone. BPXROOT can only be used by users with access to the appropriate RACF group, using the su command.

I am definitely not a deep USS guy and I didn't stay in a Holiday Inn Express last night (although I'm in a Marriott property tonight).

Isn't this something that might better be solved with SUDO?

Scott

Mark, 
After your last comment "there is no way our security team will ever release that to anyone. BPXROOT can only be used by users with access to the appropriate RACF group, using the su command" , this made me maybe realize that your user , lets call them `ibmuser` has been defined with a profile (BPX.SRV.target_userid) in the SURROGAT class so no password is needed. In other words, the only way to `su` is your security team has allowed this for your user and its done without a password, is this correct? 

If the above is true ,I bet you still get the prompt "FSUM5019 Enter the password for BPXROOT" but in your case you just press the 'enter' key  (without password) and you are now escalated in shell, is this correct? 

If all the above is true, can you try in shell to `su -s` and see if no prompt "FSUM5019 Enter the password for BPXROOT" occurs, my guess is that the prompt will disappear and you will be escalated in shell.  

If all the above is true, then (without having tested this), I see two options for Ansible:
Option (1): add to your become directives this one become_flags: "-s"

Option (2): instead of become_flags: "-s", emulate the `enter` key stroke with a `\n` and do that by using become_password='\n'

Hope that helps Mark.

Scott,

`sudo` could help but we have not fully explored that as its not part of the base OS and instead its part of IBM Open Enterprise Foundation for z/OS . Some of our clients have gotten `sudo` to work, we do plan to eventually look into this; I think the issue really is that the prompt is not being responded to and I have not tried `sudo` on z/OS to know if this issue will sill exist. I provided options on how to handle when no password is required in my comment before this.

Hi Demetri,

The user is not defined as a SURROGAT. If the user has access to BPX.SUPERUSER the user can issue su and have their access elevated/switched to BPXROOT without being prompted for a password. See Superusers in z/OS UNIX .

I did try your suggestion and that confirms SURROGATE access is not in place as the following errors is returned: FSUM5027 su: User is not a surrogate of \"BPXROOT\"

I also tried become_pass and that returns 'FSUM5017 su: User not authorized to switch to \"BPXROOT\".  Password incorrect.'

Thanks for all the suggestions so far.

Hi @Mark Turner,

I have an set up an ID with access to BPX.SUPERUSER, lets call them USRT001 for now, in shell you can see I am NOT prompted for a password:

$: ssh usrt001@zvm1234.ibm.com
usrt001@zvm1234.ibm.com's password:

$ whoami
USRT001

$ su
$ whoami
BPXROOT

My playbook connects to the z/OS managed node as USRT001, the playbook is interactive, I will share the playbook at the bottom of this response.

The options used are below, even though I am not prompted in shell as you can see above, Ansible seems to require a prompt to move forward, the team can look into this later, but for now, the work around is to FORCE a prompt for your user even though the user does NOT require a prompt in shell and then I responded to that prompt with what emulates an "ENTER" key by using "\n".

ansible_become_user: <Terminal prompt, for security purposes I am not sharing this> 
ansible_become_method: su
ansible_become_password: "\n"
ansible_su_prompt_l10n: FSUM5019 Enter the password for {{ansible_become_user}}

Execution of the playbook:
ansible-playbook -i inventory zos-priv-escal-test.yaml -k  <--- using -k to prompt for USRT001's password, you may not need this if you have it configured elsewhere.
SSH password: <--- Prompt from playbook, enter USRT001's password, you may not need this if you have configured this elsewhere.
Enter the privilege escalation user ID?:  <--- This is the ID you want to become/escalate to, basically who ever is defined as root that corresponds to BPXROOT, I did not share this for security reasons.

Playbook contents, configure the hosts before using, I start off as USRT001 and am escalated to BPXROOT by responding to the password prompt with a "\n".

---
- hosts: zvm
  gather_facts: no
  environment: "{{ environment_vars }}"

  vars_prompt:
    - name: ansible_become_user
      prompt: Enter the privilege escalation user ID?

  vars:
    ansible_become_method: su
    ansible_become_password: "\n"
    ansible_su_prompt_l10n: FSUM5019 Enter the password for {{ansible_become_user}}

  tasks:
  - name: "Query the user ID BEFORE escalated privileges on the managed z/OS node."
    command: whoami
    become: false
    register: result

  - name: "Print the user ID BEFORE escalated privileges on the managed z/OS node."
    debug:
      var: result

  - name: "Query the user ID AFTER escalated privileges on the managed z/OS node."
    ansible.builtin.raw: whoami
    become: true
    register: result

  - name: "Print user ID AFTER escalated privileges on the managed z/OS node."
    debug:
      var: result

Results:

PLAY [zvm] *******************************************************************************
TASK [Query the user ID BEFORE escalated privileges on the managed z/OS node.] 
*******************************************************************************************
changed: [zvm]

TASK [Print the user ID BEFORE escalated privileges on the managed z/OS node.] 
*******************************************************************************************
ok: [zvm] =>
    result:
        changed: true
        cmd:
        - whoami
        delta: '0:00:00.064233'
        end: '2025-04-20 05:56:27.093046'
        failed: false
        msg: ''
        rc: 0
        start: '2025-04-20 05:56:27.028813'
        stderr: ''
        stderr_lines: []
        stdout: USRT001
        stdout_lines:
        - USRT001

TASK [Query the user ID AFTER escalated privileges on the managed z/OS node.] 
********************************************************************************************
changed: [zvm]

TASK [Print user ID AFTER escalated privileges on the managed z/OS node.] 
*******************************************************************************************
ok: [zvm] =>
    result:
        changed: true
        failed: false
        rc: 0
        stderr: |-
            Shared connection to zvm1234.ibm.com closed.
        stderr_lines:
        - Shared connection to zvm1234.ibm.com closed.
        stdout: |4-
            BPXROOT
        stdout_lines:
        - ''
        - BPXROOT

PLAY RECAP 
*******************************************************************************************
zvm                        : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Hi Demetri,

We use the full Ansible Automation Platform and have to invoke playbooks/workflows using the UI or using APIs. We can prompt for input before a playbook runs, but I don't think we can prompt mid-playbook. I did try your playbook, but it just hangs - I assume because it is waiting a response to the password prompt.

We do also have a Dev Spaces implementation with IBM Wazi, and that has Ansible built in. So for a test I have also tried running your scenario below from that environment. However, I get an error that says I need to install sshpass on the host. I am not currently sure how you do that for zOS.

Thanks.

Hi @Mark Turner,

You should be able make the playbook non-interactive so it can run in Ansible Automation Platform (AAP) , I have not tried in AAP, if time permits I will do so and the only reason I made it interactive was to demonstrate the work around. 

To make it non-interactive change the playbook from: 

  vars_prompt:
    - name: ansible_become_user
      prompt: Enter the privilege escalation user ID?

  vars:
    ansible_become_method: su
    ansible_become_password: "\n"
    ansible_su_prompt_l10n: FSUM5019 Enter the password for {{ansible_become_user}}

To: (by removing the vars_prompt section and adding ansible_become_user to vars):

  vars:
    ansible_become_method: su
    ansible_become_password: "\n"
    ansible_su_prompt_l10n: FSUM5019 Enter the password for {{ansible_become_user}}
    ansible_become_user: <who you want to become on z/OS> 

There are better places to set such variables in Ansible Automation Platofrm, for example you can put them in 'hosts' details. 

With regard to sshpass , if Dev Spaces are set up to communicate to the z/OS managed node with SSH Keys (likely they are) , you can run the playbook without the -k, then it will default to SSH keys and not prompt for a password nor require sshpass. 

Hi Demetri,

I hadn't given my ansible user access to BPX.SRV.target_userid in the SURROGAT class. Once that access had been provided the escalation is working. 

Well, my shell steps are now working, but I am also running a zos_copy step. That seems to hangs and never completes - although an 'Escalation succeeded' message does appear. Do you know if the zos collection modules are supposed to be able to run elevated?

Thanks for all your help.

Hi @Mark Turner

Module zos_copy should be able to handle privilege escalation but it is failing to do so. While it is not documented, the general consensus is that modules should have parity with the community as much as possible. There could be a number of reasons why this might have regressed, without going into depth, we have a new team that focuses purely on Ansible proper for engine changes. 

You noted being an Ansible Automation Platform user, might I suggest you open a support case with Red Hat who will triage this and if determined by Red Hat that it belongs to IBM they will instruct you to create a support case with IBM, when doing so, open it against ZOAU in the IBM support portal for now.

[Update: The below issue has been updated (10/24) , I was able to reproduce the privilege escalation hanging]
I would also request that you add the Git Hub issue I created for this issue, which you can find here  (https://github.com/ansible-collections/ibm_zos_core/issues/2062). 

If you are unable to open a case with Red Hat, you can follow the Git Hub issue above, the team triages issues every week, triage can take some time. 

For reference on how to get support, you can review the documentation here.

I hope this helps steer this issue in the right direction, feel free to reach out for further questions. 

Hi Demetri,

I have looked at your GitHub issue and I did initially also get the module not found error for zoautil_py. I managed to fix that by adding the PYTHONPATH environment variable as documented here - https://www.ibm.com/docs/en/zoau/1.3.0?topic=installing-zoau#pythonpath-environment-variable-method.

Maybe you are missing the same? 

Interestingly, it is a bit random whether the zos_copy module works - and when it does it is VERY slow. I left my playbook running all morning the morning, and found that it had actually copied 2 of the 7 files I was trying to copy (in a loop). But it seems to be very hit and miss. For now I have simply switched to using Unix cp commands and everything works. I will raise a case on the zos_copy issue though.

FYI, my zos_copy step that 'hung' was as follows. This works if I run it non-elevated, but I can't write the real dest location when not elevated. At least I can now move forwards using standard cp.

Thanks

Hi @Mark Turner,

You were correct, I guess working into the late night can have this effect, what had happened is the mount for ZOAU had been renamed from 1.3.4.0 to 1.3.4 and I had not noticed nor thought to look since 2 days ago the same config was operational. Our tooling manages about 20 ZOAU versions and over a 12 Python releases so we can easily recreate issues, so sometimes things sneak by me. 

I am still trying some other options, but I believe I have been able to recreate, when escalating with `zos_copy` it seems to hang at this point.

 78512 1745530955.20625: become_success: (source=stdout, state=awaiting_escalation): 'BECOME-SUCCESS-lslwcoxhqthxxnqvvrzuiurxjrzazoaj'
Escalation succeeded

As far as the performance, there is no reason it should take that long, I will come back with my update about that after I try your `zos_copy` loop snippet.

Interestingly only a couple of days ago, I wrote a loop to copy shared dasd to minidisks and was curious how long this would take, note that these zvm's I am use are really not tuned at all, the are purposely configured to not have large virtual storage, 1 GPU, no zipp/zapp , etc. Each iteration of the loop is copying 3 data sets from shared DASD to a minidisk, so all in all, this task below catalogs the data sets first on each zvm, then copies them to the minidisk, so 24 catalogs and 24 copies in just over 4 minutes (during the copy operation, the destination data sets don't exist, so they are created and cataloged by zos_copy per the precedence rules outlined in the modules documentation, just wanted to point that out as well). 

For that copy operation I described above here is what the task was:

  tasks:
    - name: Catalog data sets from shared volser to zvm.
      zos_data_set:
        name: "{{src_hlq}}.{{ item }}"
        state: cataloged
        volumes: "EDSDMP"
      register: result
      loop: "{{ data_set_names }}"

    - name: Copy binaries from shared volser to private volser.
      zos_copy:
        src: "{{src_hlq}}.{{ item }}"
        dest: "{{dest_hlq}}.{{ item }}"
        remote_src: true
        force: true
        volume: 'ANSI01'
        is_binary: true
      register: result
      loop: "{{ data_set_names }}"

    - debug:
       var: result

Here was the command:

(venv-2.18) time ansible-playbook -i zvm1.svl.ibm.com,zvm2.svl.ibm.com,zvm3.svl.ibm.com,zvm4.svl.ibm.com,zvm5.svl.ibm.com,zvm6.svl.ibm.com,zvm7.svl.ibm.com,zvm8.svl.ibm.com zvm_copy_data_sets.yml

Here was the result, a little over 4 minutes (real 4m27.494s):

PLAY RECAP ***************************************************************************************************************************************************************
zvm1.svl.ibm.com  : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
zvm2.svl.ibm.com  : ok=4    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
zvm3.svl.ibm.com  : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
zvm4.svl.ibm.com  : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
zvm5.svl.ibm.com  : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
zvm6.svl.ibm.com  : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
zvm7.svl.ibm.com  : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
zvm8.svl.ibm.com  : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0


real	4m27.494s
user	0m6.163s
sys	0m7.945s

Hi Demetri,

Your test does hang at exactly the same point as mine. It hangs for a single file copy as well - so it's nothing to do with the loop.

I do also have my local Dev Spaces install of Ansible working and I am able to recreate the problem there also.

@Mark Turner, 
Regarding the performance, I looked over your Ansible task and see no reason for the slow down unless it is someplace in the Ansible or z/OS configs. I copy 16 RELFILES and  8 SMPMCS, catalog, create data sets and copy in just over 4 minutes on very slow zVMs. 

I would suggest you open two zos_copy cases to Red Hat, one for the hanging escalation and reference the Git Hub issue, and one for the zos_copy performance to see if we can find the bottleneck. Red Hat will review the case, evaluate their portion of the code and then likely ask you to open a case with IBM and provide them the IBM support case so that a collaboration can be created between IBM and RH. When you go to create the case to IBM, please open it against ZOAU for the time being. 

Hi Demitri,

I have run some tests without elevation and agree that there are no performance issues when running like that. So I suspect the performance problem relates to the elevated access, and so I think only 1 case is required.

I have asked our Ansible support team to open the case with RedHat as I don't seem to be able to do so. If RedHat's response is for us to raise a case with IBM then I will be able to do that.

Thanks

Hi @Mark Turner,

I ran more tests looking for performance issues, what I did was copy 3 - 20 MB text files from my controller (MacBook) to z/OS, I did this from a remote location so there is some latency for not being on the LAN. What I found was that with privilege escalation the cost was about twice as much as with out. I ran my test multiple times, average time to copy the 3 - 20 MB files to z/OS USS without privilege escalation was approximately 20 seconds, while with privilege escalation it as a bit over 40 seconds.

I was not able to replicate what you experienced with performance and think we should look into where the added overhead is. I can't say where the time is spent, I have some ideas but would leave this to @Ketan Kelkar and team to diagnose further. 

I will watch for any support cases, this has been a good discussion and uncovered a few areas we need to look into, feel free to reach to us, thank you. 

With privilege escalation:

Tue Apr 29 21:55:25 PDT 2025
BECOME password:

PLAY [all] ***************************************************************************************************************************************************************

TASK [Copy 20 mb files from controller to zos uss] ***********************************************************************************************************************
changed: [zvm1.ibm.com] => (item=/tmp/20-1-mb.txt)
changed: [zvm1.ibm.com] => (item=/tmp/20-2-mb.txt)
changed: [zvm1.ibm.com] => (item=/tmp/20-3-mb.txt)

PLAY RECAP ***************************************************************************************************************************************************************
zvm1.ibm.com  : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

real	0m44.935s
user	0m1.147s
sys	0m1.007s

Without privilege escalation: 

Tue Apr 29 21:56:57 PDT 2025
TASK [Copy 20 mb files from controller to zos uss] ***********************************************************************************************************************
changed: [zvm1.ibm.com] => (item=/tmp/20-1-mb.txt)
changed: [zvm1.ibm.com] => (item=/tmp/20-2-mb.txt)
changed: [zvm1.ibm.com] => (item=/tmp/20-3-mb.txt)

PLAY RECAP ***************************************************************************************************************************************************************
zvm1.ibm.com  : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

real	0m19.022s
user	0m0.779s
sys	0m0.658s

Noteworthy playbook source:

  vars:
    file_names:
      - /tmp/20-1-mb.txt
      - /tmp/20-2-mb.txt
      - /tmp/20-3-mb.txt

  tasks:
    - name: Copy 20 mb files from controller to zos uss
      ansible.builtin.copy:
        src: /tmp/20-mb.txt
        dest: "{{ item }}"
        remote_src: false
        force: true
        mode: '0755'
      become: true
      loop: "{{ file_names }}"

Hi Demetri,

We have opened the case with RedHat to cover the escalation hang in zos_copy (#04131134).

However I wanted to run some performance tests myself before raising the performance one, and I have just completed those tests.

In my scenario I copied an 87MB file 5 times per execution. The file was already on zos and so there was no network overhead to send the file. I ran tests with the ansible builtin copy both elevated and then not elevated. And I also ran zos_copy not elevated.

The builtin copy with and without elevation were broadly very similar - 7 or 8 seconds across the several tests I ran. But zos_copy not elevated was twice that at approx 16 seconds. So zos_copy appears to definitely be slower than standard copy. I suspect there is more logic in zos_copy as it supports different file types (uss and traditional zos), but I wouldn't expect it to be twice as slow?

Hi @Mark Turner, 

I was not able to reply until today.

The Ansible task I ran was without privilege escalation, it copied a 100 MB file on z/OS USS to z/OS USS to avoid any network overhead. I did this for both ibm_zos_core.zos_copy and ansible.builtin.copy and found that ibm_zos_core.zos_copy was slightly more than twice as slow. 

Before I get into the specifics, allow me to highlight some differences between the two modules as to why the performance might be slower in zos_copy (feel free to review a deep dive blog on the zos_copy module architecture).

1) The zos_copy modules source code is 6 times larger than community copy modules source, and that does not account for the imports we have shared between modules which could easily double this number.

2)  zos_copy invokes our custom arg parser after Ansible's arg parser, our parser validates all the options to ensure conventions are as they should be for z/OS, in other words, we prefer to fail fast rather than the module execute on z/OS only to find out something like volser has to many characters or is not following naming conventions. 

3)  zos_copy provides many features that does come with a price, for example, if the destination is a data set and it does not exist, we invoke our precedence rules and try to create it, if the source is a data set we extract its attributes and create the data set, if the source is a USS file we figure out the longest line, size , is a binary, etc so we can create the right destination and so on. 

vars:
    src_file_names: "/tmp/100mb.txt"
    tgt_file_names: "/tmp/dup-100mb"

  tasks:
    - name: Make 5 copies of 100mb file on USS
      ibm.ibm_zos_core.zos_copy:
        src: "{{ src_file_names }}"
        dest: "{{ tgt_file_names }}-{{ item }}.txt"
        remote_src: true
        force: true
        mode: '0644'
      with_sequence: count=5

    - name: Make 5 copies of 100mb file on USS
      ansible.builtin.copy:
        src: "{{ src_file_names }}"
        dest: "{{ tgt_file_names }}-{{ item }}.txt"
        remote_src: true
        force: true
        mode: '0644'
      with_sequence: count=5

-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                           1            397           2002           1866
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                           1             78            293            299
-------------------------------------------------------------------------------

Hi @Demetri Dimatos,

For info I have open a GitHub issue as suggested - https://github.com/ansible-collections/ibm_zos_core/issues/2084.

Thanks