File and Object Storage

Benefits and implementation of IBM Spectrum Scale™ sudo wrappers

By Nils Haustein posted Thu December 17, 2020 10:16 AM

  

Thanks to Felipe Knop and Sandeep Ramesh for their valuable feedback and contributions.

In this blog article I would like to share my experience configuring IBM Spectrum Scale™ with sudo wrappers. I will start with a short introduction of sudo and explain the objective of sudo wrappers in the context of a Spectrum Scale cluster. Afterwards I will disclose an approach to implement the four-eye principle which relies on two distinct users controlling each other for secure administration. At the end I will demonstrate how to configure and use sudo wrappers in a Spectrum Scale cluster. I assume that you are familiar with the administration of a IBM Spectrum Scale cluster.

 In IBM Spectrum Scale version 5.1 a major security improvement has been implemented regarding sudo wrappers. It is no longer required to allow remote copy commands and the echo-command in the sudo-configuration. These changes have been adopted in this blog article (see section Configure sudo).

What is sudo

Sudo is a program that allows computer users to run programs with the security privileges of another user, by default the superuser (commonly the root user). The name “sudo” originally stood for "superuser do", but sudo is no longer limited to running commands with the privileges of root. It can also run commands with privileges of other (restricted) users.

 The commands a user can run with the security privileges of another user are configured in the sudo configuration file /etc/sudoers (also called sudoer-file). It is recommended to use the command visudo to edit this file because it does a syntax check after saving and closing it. The sudoer-file allows to configure privileges for running commands, including enabling commands only from the invoking terminal; requiring a password per user or group; requiring re-entry of a password every time or never requiring a password at all for a particular command line. It can also be configured to permit passing arguments or multiple commands.

 A simple explanation of the syntax to configure privileges of a user is shown in the following clause:

%user host=(as-user) PASSWD: passwd-cmds NOPASSWD: nopasswd-cmds

Here is a brief explanation of this clause:

  • The user is the name of the user or group configured in the operating system for these privileges.
  • The host is the host name on this clause applies for. When the host name is set to ALL, then the clause applies to all hosts.
  • The as-user names another user whose privileges are used by the user to run commands. This means the user named at the beginning of the line can execute commands as another user. When this is set to ALL the user can run commands for all other users configured in the operating system of the server.
  • The token PASSWD indicates that the following commands (passwd-cmds) can be executed by the user (as the other user) whereby the user must enter his password. The passwd-cmds is a comma separated list of the commands the user is allowed to execute with password. The commands shall include the path and file name (e.g. /usr/bin/visudo).
  • The token NOPASSWD indicates that the following commands (nopasswd-cmds) can be executed by the user whereby the user must NOT enter his password. Likewise, the nopasswd-cmds is a comma separated list of the commands the user is allowed to execute without password. The commands shall include the path and file name.

 

In the following example the user admin is allowed to run the all mm-commands with password and the command mmremote without password.

%admin ALL=(ALL) PASSWD: /usr/lpp/mmfs/bin/mm* NOPASSWD: /usr/lpp/mmfs/bin/mmremote

It is also possible to disallow commands by adding a “!” in front of the command. In the following example the user admin is not allowed to run the command mmfsadm:

%admin ALL=(ALL) !/usr/lpp/mmfs/bin/mmfsadm

Instead of spelling out every command that is allowed or forbidden in the clause, so called command aliases can be created. The following example shows a command alias for some networking commands:

## Networking
Cmnd_Alias NET = /sbin/route, /sbin/ifconfig, /bin/ping, …

This alias can now be embedded into a clause defining user privileges. In the example below the user is allowed to run the above networking commands requiring his password:

%admin ALL=(ALL) PASSWD: NET

In general, the sudoer-file can include multiple clauses for a given user. Be aware, however, that the file is interpreted from top to bottom. If there is a clause that allows a command and later on there is a clause that disallows a command for given user, the latter one becomes effective.

Another important feature in sudo are the logging capabilities. Sudo can be configured to log all commands that a given named user runs in the sudo context. In the following example all commands (input and output) the user admin runs with sudo are logged to the directory /var/log/sudo-io/admin:

%admin ALL=(root) PASSWD: LOG_INPUT: LOG_OUTPUT: cmds
Defaults iolog_dir=/var/log/sudo-io/%{user}

Alternatively, the clauses Defaults logdir=directory or Defaults syslog=auth can be used to enforce logging.

For more information about the features and syntax in sudo refer to the sudo man pages:

https://www.sudo.ws/man/sudoers.man.html#EXAMPLES  

 

What are Spectrum Scale sudo wrappers

Spectrum Scale does not have its own user management for administrative users using the command line interface (CLI). CLI users are authenticated by the operating system and have privileges to perform commands according to user configuration in the operating system. By default, the Spectrum Scale cluster requires root privileges for administration.

In order to eliminate the need for direct root access Spectrum Scale allows to configure and enable so called sudo wrappers. With sudo wrappers commands are executed under the named user with root privileges but no longer as root user itself. Sudo wrappers rely on a sudo configuration that allows the named user to run certain commands with root privileges.

The key advantages of Spectrum Scale sudo wrappers are:

  • Eliminates the need for the root user to allow password-less ssh-access to other nodes. 
  • Allow the use of named users to administer the cluster. Thereby privileges of an administrative user can be limited to a defined set of commands by leveraging sudo definitions.
  • Allows to permanently disable the root login via ssh (PermitRootLogin No).
  • The authorized key file of the root (/root/.ssh/authorized_keys) user is no longer required for Spectrum Scale cluster nodes.
  • Allow logging of all commands executed by a user in the sudo context using the sudo logging features.

In contrast, sudo wrappers do not prevent entirely that a named user cannot elevate its privileges. Therefore, additional operational processes are required to assure that that the sudo definitions cannot be changed without permission and without being audited. In addition, sudo in general does not allow managing access permissions to files, only for commands. File access permission needs to be regulated using file system permissions and Access Control Lists (ACL).

With the current implementation (Spectrum Scale version 5.1) sudo wrappers do not support the following Spectrum Scale functions:

  • Installation toolkit (command: spectrumscale)
  • Spectrum Scale call home
  • With Windows nodes in the cluster
  • With clustered NFS (cNFS)

Configuration and Operations guidance

Users administering a Spectrum Scale cluster should not be able to tamper any data stored in the file system. Therefore, it might be appropriate to allow only commands (on a Spectrum Scale and operating system level) that are required for normal operations executed by administrative users. In special cases the privileges of these users can be temporarily increased by another user. This calls for the implementation of the four-eye principle where the administrative user requires the authorization of another user (security administrator) to perform a special task. 

For the implementation of the four-eye principle create two user groups and assign named users to each of these groups in the operating system. In this example one group is named gpfsadmin and another group is named secadmin. The following principles should apply:

  • Users in the group gpfsadmin should be able to administer Spectrum Scale and Spectrum Archive during normal operations.
  • Users in the group secadmin should be able to manage the privileges of the users in the group gpfsadmin, for example to temporarily allow additional commands in case of problems. For this purpose, the user in the group secadmin is allowed to change the sudo-configuration.

The four-eye principle allows users of the group gpfsadmin to execute a limited set of commands that are required for normal operations. If a user of the group gpfsadmin requires more privileges (e.g. due to a problem) he can contact a user of the group secadmin. The user of the group secadmin can now grant the user of the group gpfsadmin further privileges temporary by changing the sudo definitions.

Configure Spectrum Scale sudo wrappers according to the instructions in the knowledge center (Link). Allow users in group gpfsadmin to run commands that are required to administer the Spectrum Scale cluster. Find an example of the sudo configuration in section Configure sudo. In general, consider the following guidance

  • Users in the group gpfsadmin should not be able to run all commands; instead limit this to the Spectrum Scale commands that are required.
  • Users in the group gpfsadmin should not be able to obtain full root permission without being authorized by a user of the group secadmin.
  • Users in the group gpfsadmin should not be able to manage sudo configuration.
  • Users in the group gpfsadmin should not be able to change the time or time related configurations in the operating system.
  • Do not alter the set of commands for the group gpfsadmin in the sudo definitions that have to run without password.
  • Allow users in the group secadmin to change the sudo configuration (visudo or the like). This allows the user of the secadmin group to temporarily allow certain commands to the user in the gpfsadmin group.
  • Further allow the user in group secadmin to copy the sudo-configuration to the other (admin) nodes.
  • Configure logging of all commands run by all user groups in the sudo context (e.g. by configuring the sudo parameters logfile and / or iolog_dir in the sudoer-file). These logs shall be immediately sent to a remote log server and kept for the retention period of the files.

     Users in group secadmin can easily elevate their privileges because they control sudo. Persons with this privilege should be trusted. In addition, operational processes can be established that audit the involvement of these users and their activities on the system. Consider implementing a process that only allows users of the secadmin group to access a cluster node when approved by a user of the gpfsadmin group.

    The Spectrum Scale GUI has a separate user management. It is recommended to use the same user-names in the GUI and the CLI for the same user. In addition, the roles of the user in GUI should match the roles the user has in CLI.

     

    Configuring sudo wrappers with four eye principle

    Now I will show how to implement sudo wrappers in a Spectrum Scale cluster. Use this as a high level guidance and always consult the Spectrum Scale knowledge center for the exact and most current procedure (Link). I do not guarantee that this works for any environment.

    In this example we have three cluster nodes named g1_node1, g1_node2 and g1_node3 running Red Hat Enterprise Linux version 7. All cluster nodes are administrative nodes which means that Spectrum Scale commands can be executed on any of these nodes.

     

    Configure user and groups

    We will create a group gpfsadmin and add user admin to this. This user - and all other users that might be added to this group - are Spectrum Scale administrators, able to run all mm-commands (with some exceptions). In addition, we configure a group secadmin and add user secco to this. This user can change the sudo configuration and temporarily grant additional privileges to the admin user using the visudo command.

    Note: Many of these steps must be executed on all cluster nodes and still require direct root login.

    Create group gpfsadmin and user admin on all nodes (as root user). The group and user ID (1001) are arbitrarily chosen; you can use different IDs. Just make sure that these IDs are not used by other users and groups (file /etc/passwd):

    # groupadd -g 1001 gpfsadmin
    # useradd -c 'gpfs admin' -g gpfsadmin -m -u 1001 admin

    Set password for user admin on all nodes (as root user):

    # passwd admin

     

    Create group secadmin and user secco on all nodes (as root user):

    # groupadd -g 1002 secadmin
    # useradd -c 'security admin' -g secadmin -m -u 1002 secco

    Set password for user secco on all nodes for new user (as root user):

    # passwd secco

     

    When required, generate the ssh-key pair for the root user on all nodes (as root user). Do not enter a passphrase when asked, but instead press enter a couple of times:

    # ssh-keygen

     

    Copy the public key of root user to the authorized_keys file of the admin user. This must be done from all admin nodes to all other nodes, including the admin node itself. In our example all three nodes are admin nodes, so we have to do this on all nodes (as root user):

    # ssh-copy-id admin@g1_node1
    # ssh-copy-id admin@g1_node2
    # ssh-copy-id admin@g1_node3

     The root user must be able to perform password-less ssh-communication with the remote nodes as user admin (not root).Test password-less ssh-authentication for the new user admin from all admin nodes to all other nodes, including the admin node itself:

    # ssh admin@g1_node1
    # ssh admin@g1_node2
    # ssh admin@g1_node3

     The two steps above do not have to be done for the user secco because this user will not do cluster administration.

     

    Add the Spectrum Scale command path to the profile for the user admin and secco on all nodes (login as admin or secco user respectively):

    # su - admin
    # vi .bash_profile
      Export PATH=$PATH:/usr/lpp/mmfs/bin

     

    Optionally you can create a key pair for the admin user and exchange this from all nodes to all other nodes. This enables the admin user to access other nodes seamlessly without using passwords. Perform the following steps on all nodes:

    1. Login as admin user and create the ssh-key pair, do not enter a passphrase, but instead press enter a couple of times:
      # ssh-keygen 
    2. Copy the public ssh key of the admin user to all other nodes including the node itsel
      # ssh-copy-id <node-name>
    3. Test the ssh-connection to be password less with all other nodes including the node itself:
      # ssh <node-name>

     You have now prepared the users and groups for the sudo configuration. Continue to the next section.

     

    Configure sudo

    IBM Spectrum Scale provides an example of the sudoer-file that can be used as template to implement sudo-wrappers. This file is located on a Spectrum Scale cluster node in /usr/lpp/mmfs/samples/sudoers.sample. Save the current sudoer-file and copy the sudoers.sample file to /etc/sudoers on all nodes (as root user):

    # mv /etc/sudoers /etc/sudoers.backup
    # cp /usr/lpp/mmfs/samples/sudoers.sample  /etc/sudoers

    If the current sudoer-file (/etc/sudoers.backup) was configured already then re-add the definitions to the new sudoer-file (/etc/sudoers).

    Make the necessary adjustments to the sudoer-file on all nodes (as root user). In the example clauses below users in the group gpfsadmin can run all mm-commands, with some exceptions. The users in group secadmin can run the command visudo and scp. In addition, we configure logging for all commands issued from users in the sudo context. To edit the sudoer-file run this command and it will open your default editor (vi):

    # visudo

    Find below different section that should be added or updated in the sudoer-file:

     

    Preserve environment variable associated with Spectrum Scale:

    #### Preserve GPFS env vars
    Defaults    env_keep += "MMMODE environmentType GPFS_rshPath GPFS_rcpPath mmScriptTrace GPFSCMDPORTRANGE GPFS_CIM_MSG_FORMAT"

    Adding the GPFS command path to the secure_path allows to run GPFS commands without specifying the full path of the command:

    #### add the GPFS command path to the secure path
    Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/lpp/mmfs/bin

    Allow members of the group gpfsadmin to run all mm-commands, whereby the user has enter his password. Notice the NOPASSWD-clause, where we specify which commands can be run without entering the password. In IBM Spectrum Scale version 5.1 and above the NOPASSWD-clause must only contain the mmremote-command.

    #### Allow members of gpfsadmin to run all mm-commands commands requiring the user password
    %gpfsadmin ALL=(ALL) PASSWD: LOG_INPUT: LOG_OUTPUT: /usr/lpp/mmfs/bin/mm*, NOPASSWD: LOG_INPUT: LOG_OUTPUT: /usr/lpp/mmfs/bin/mmremote

    Please notice that for IBM Spectrum Scale version below 5.1 and higher than 4.2, the NOPASSWD clause must contain additional programs such as: /usr/lpp/mmfs/bin/mmremote, /usr/bin/scp, /bin/echo, /usr/lpp/mmfs/bin/mmsdrrestore

    Allow members of the group secadmin to run the command visudo and scp:

    #### Allow members of the secadmin group to run visudo
    %secadmin ALL=(ALL) PASSWD: LOG_INPUT: LOG_OUTPUT: /usr/sbin/visudo, /usr/bin/scp

    Finally, enter the path for the logs where all activities from users are recorded:

    #### Add logging
    Defaults iolog_dir=/var/log/sudo-io/%{user}

     

    When the sudoer-file has been updated, distribute all the changes to the other nodes. This can be done using ssh (requires root permissions). For example,  if the adjustments have been done on g1_node1, you can copy the sudoer-file to node 2 and 3 as shown below:

    # scp /etc/sudoers g1_node2:/etc/sudoers
    # scp /etc/sudoers g1_node3:/etc/sudoers

     

    On each admin node run the sudo wrappers test as admin user. These tests must be run without errors. Do not continue if a single test fails:

    # su - admin
    # sudo /usr/lpp/mmfs/bin/mmcommon test sshwrap g1_node1
    # sudo /usr/lpp/mmfs/bin/mmcommon test sshwrap g1_node2
    # sudo /usr/lpp/mmfs/bin/mmcommon test sshwrap g1_node3
    # sudo /usr/lpp/mmfs/bin/mmcommon test scpwrap g1_node1
    # sudo /usr/lpp/mmfs/bin/mmcommon test scpwrap g1_node2
    # sudo /usr/lpp/mmfs/bin/mmcommon test scpwrap g1_node3

     

    Enable and test sudo wrappers

    If the above test completes without errors you can configure sudo wrappers in the cluster as user admin.

    # su - admin
    # sudo mmchcluster --use-sudo-wrappers [--sudo-user admin]

    The option --sudo-user specifies the name of a user that can perform mm-commands, i.e. a user in group gpfsadmin. This setting is important for the Spectrum Scale GUI. Not all version of IBM Spectrum Scale may allow this option. See further below, how to specify the sudo-user for the GUI:

    When the Spectrum Scale GUI is used, the name of the sudo-user allowed to perform GPFS commands must be provided. Depending on the Spectrum Scale version, there are different methods:

    With Spectrum Scale version below 5.0.0: on all nodes running the GUI open the file /usr/lpp/mmfs/gui/conf/gpfsgui.properties and change: GPFS_ADMIN=root to the user-name that you configured as the sudo user (admin):

    GPFS_ADMIN=admin

    Then restart the GUI:

    # systemctl restart gpfsgui

     

    With Spectrum Scale version 5.0 run the following command to set the sudo-user name:

     # sudo mmchconfig sudoUser=admin

     

    With IBM Spectrum Scale version 5.1 and above the sudo-user name is provided when enabling sudo:

    # sudo mmchcluster --use-sudo-wrappers --sudo-user admin

     

    Now perform some tests by logging on with the user admin and running mm-commands. Do not only run list kind of commands, also run command that perform changes on the cluster. You have to run commands using sudo:

    # sudo mmchfs 

    fsname


    # sudo mmcrfileset

    fsname fsetname …

     

    Also test the privileges of the user secadmin. For example, allowing or disallowing certain commands for the user in the gpfsadmin group. Logon with the user secco and edit the sudoer-file to your test scenario:

    # sudo visudo

     

    Whenever the user in the secadmin group makes changes to the sudo-configuration on one node then the sudo-configuration must be distributed to all other (admin) nodes in the cluster using the command:

    # sudo scp /etc/sudoers g1_node2:/etc/sudoers
    # sudo scp /etc/sudoers g1_node3:/etc/sudoers

    Disable root login

    Optionally you disable direct root login on all cluster nodes.

    Attention: Before you disable root login make sure that the user of the secadmin group can effectively change the sudo configuration in order to re-enable root login when required.

    This requires root privileges, so logon as root user or allow another user to become root:

    # vi /etc/ssh/sshd_config
    PermitRootLogin No

    Then restart the ssh-daemon:

    # systemctl restart ssh

     

    Operations with sudo wrappers

    During normal operations the user in the group gpfsadmin login with their named user credentials (e.g. admin) and perform commands using sudo as prefix

    # sudo 

    mm-command

     

    There are some GPFS command that have the potential to allow a malicious user to elevate their privileges. To counter this, you can exclude certain mm-commands. The example below first allows to user to run all mm-commands and second disallows certain commands:

    #### Allow members of gpfsadmin to run all mm-commands commands requiring the user password
    %gpfsadmin ALL=(ALL) PASSWD: LOG_INPUT: LOG_OUTPUT: /usr/lpp/mmfs/bin/mm*, NOPASSWD: LOG_INPUT: LOG_OUTPUT: /usr/lpp/mmfs/bin/mmremote

    #### disallow certain mm-commands for members of the gpfsadmin group
    %gpfsadmin ALL=(ALL) LOG_INPUT: LOG_OUTPUT: !/usr/lpp/mmfs/bin/mmfsadm, !/usr/lpp/mmfs/bin/mmaddcallback, !/usr/lpp/mmfs/bin/mmchcluster, !/usr/lpp/mmfs/bin/mmauth, !/usr/lpp/mmfs/bin/mmdel*, !/usr/lpp/mmfs/bin/mmchconfig

    There might be cases where the user of the group gpfsadmin requires further privileges. Let us assume a fileset needs to be deleted. In the sudo clauses above we have prohibited the admin user to run the mmchconfig commands. In order to temporarily allow the admin user to run the mmchconfig command, the secco user has to be contacted to enable this command in the sudo configuration. The secco user logs to one admin node and enables the mmchconfig command by removing it from the commands that are disallowed:

    # visudo

     

    Allow the mmchconfig command by removing it from the list of disallowed commands:

    #### disallow certain mm-commands for members of the gpfsadmin group
    %gpfsadmin ALL=(ALL) LOG_INPUT: LOG_OUTPUT: !/usr/lpp/mmfs/bin/mmfsadm, !/usr/lpp/mmfs/bin/mmaddcallback, !/usr/lpp/mmfs/bin/mmchcluster, !/usr/lpp/mmfs/bin/mmauth, !/usr/lpp/mmfs/bin/mmdel*

     

    Whenever changes to the sudoer-file have been done by the user of the secadmin group on one node, the sudo-configuration has to be distributed to all other (admin) nodes. This is done by the user of the secadmin group (secco), because this user is allowed to execute the scp command:

    # sudo scp /etc/sudoers g1_node2:/etc/sudoers
    # sudo scp /etc/sudoers g1_node3:/etc/sudoers

     

    Now the admin user can perform the required operations. All operations done in the sudo context can be logged. When the user is done he contacts the user secco in order to disallow the command.

    Summary

    Spectrum Scale sudo wrappers eliminate the need to allow password-less ssh-communication between the cluster nodes. Furthermore, they allow to enforce named user login and tracing of their command activities in the sudo context. Thus, the root user login can be eliminated. However, sudo wrappers do not entirely prevent experienced users from increasing their privileges.

    The setup of sudo wrappers requires a few steps and must be done with care. Especially the testing of sudo wrappers must be done thoughtfully. For the distribution of the sudoer-file federation tools make life easier because they allow to change one instance of the sudoer-file and distribute this to all nodes.

    Security by design is one of the key design principles in the Spectrum Scale development. Given the more than 20-year history of Spectrum Scale (alias GPFS) there is some work to do in order to improve security in the context of a Spectrum Scale cluster.

     

    Disclaimer

    This document provides guidance for certain configuration and operational aspects. IBM does not guarantee that this guidance complies with laws or regulation. In order to obtain a compliance assessment an independent auditor has to be engaged by the client. IBM cannot be made liable for any findings or violations of laws and regulations.

    The software nature of the solution may allow malicious hackers to exploit the system and elevate privileges of users. The use of Spectrum Scale sudo wrappers does not completely assure that there is no way to escape the sudo environment and elevate privileges of users. In addition, it might be possible to use certain undocumented Spectrum Scale commands to exploit the system and elevate privileges of users. IBM cannot be made liable for any damage resulting of the use of exploits.

    The guidance given herein does not imply warranty that the commands given, or their intention satisfies the purpose. IBM cannot be made liable upon damage caused by any of the commands or guidance.

    0 comments
    2 views

    Permalink