Introduction
IBM Spectrum Scale™, based on technology from IBM General Parallel File System (hereinafter referred to as IBM Spectrum Scale or GPFS™), is a high performance software defined file management solution that simplifies data management, scalable to petabytes of data and billion of files, and delivers high performance access to data from multiple servers.
IBM Spectrum Scale offers many advanced features for data security such as encryption, cipherList setting to secure the file system communication between the nodes, file immutability, and sudo-based secure administration. This series provides an overview of the IBM Spectrum Scale sudo-based secure administration as well as logging of the administrative commands.
Sudo based Secure Administration
The secure administration of the IBM Spectrum Scale components requires that the administration commands, generally known as mm commands, must be executed at the root level. Some of the administrative commands requires "root" login to the Spectrum scale nodes. By leveraging the sudo [1] program as well as the Spectrum Scale sudo wrappers [3], the Spectrum Scale administration can be performed securely using a non-root ID. The root user on a Spectrum Scale administration node still needs to be able to log in to all nodes in the cluster as the non-root ID, without being prompted for a password.
The IBM Spectrum Scale sudo wrappers which enables Spectrum Scale administrative operations to be securely performed using a non-root user is based on “GPFS and Remote shell” [2].
The sudo wrappers setup is detailed in the Spectrum Scale Administration and Programming Reference Guide [3]. At the high-level, the setup involves the following tasks:
- Configure sudo
- Configuring the Spectrum Scale cluster to use sudo wrapper scripts
Configuring sudo
Administrator should install the sudo program on the IBM Spectrum Scale node(s) prior to performing the following steps:
- Create user and group with sudo privileges to run the Spectrum Scale administration commands. In this blog, we use the non-root username “gpfsadmin” and group “gpfs”.
- Allow password-less access to root user from any Spectrum Scale administration node to issue commands on all nodes with the user ID gpfsadmin.
- Add the following lines to /etc/sudoers, in each node of the Spectrum Scale cluster
# Preserve GPFS environment variables:
Defaults env_keep += "MMMODE environmentType GPFS_rshPath GPFS_rcpPath mmScriptTrace GPFSCMDPORTRANGE GPFS_CIM_MSG_FORMAT"
# Allow members of the gpfs group to run all commands but only selected commands without a password:
%gpfs ALL=(ALL) PASSWD: ALL, NOPASSWD: /usr/lpp/mmfs/bin/mmremote, /usr/bin/scp, /bin/echo, /usr/lpp/mmfs/bin/mmsdrrestore
# Disable requiretty for group gpfs:
Defaults:%gpfs !requiretty
Configuring the IBM Spectrum Scale cluster to use sudo wrapper scripts
The system administrator must configure the IBM Spectrum Scale™ cluster to call the sudo wrapper scripts sshwrap and scpwrap to run the IBM Spectrum Scale administration commands. The sudo wrapper can be configured using the mmcrcluster command or the mmchcluster command with the --use-sudo-wrapper option.
Perform the following steps to configure a new IBM Spectrum Scale cluster to call the sudo wrapper scripts:
- To configure a new Spectrum Scale cluster to call the sudo wrapper scripts, log in with the non-root user ID (example uses gpfsadmin) and issue the mmcrcluster command with the --use-sudo-wrapper option, as shown in the following example:
$ whoami
gpfsadmin
$ sudo /usr/lpp/mmfs/bin/mmcrcluster -N mmcrcluster-node.lst --ccr-enable --use-sudo-wrapper -C gpfsSudoTest -c mmcrcluster-config.lst
mmcrcluster: Performing preliminary node verification ...
mmcrcluster: Processing quorum and other critical nodes ...
mmcrcluster: Finalizing the cluster data structures ...
mmcrcluster: Processing user configuration file mmcrcluster-config.lst
mmcrcluster: Command successfully completed
mmcrcluster: Warning: Not all nodes have proper GPFS license designations.
Use the mmchlicense command to designate licenses as needed.
mmcrcluster: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
- To verify that the Spectrum Scale cluster is using sudo wrappers, issue the mmlscluster command and verify that the "Remote shell command" and "Remote file copy command" attribute values are set to "sudo wrapper in use" as shown in the following example:
$ mmlscluster
GPFS cluster information
========================
GPFS cluster name: gpfsSudoTest.c25m3n02-ib
GPFS cluster id: 6226082987153929739
GPFS UID domain: gpfsSudoTest.c25m3n02-ib
Remote shell command: sudo wrapper in use
Remote file copy command: sudo wrapper in use
Repository type: CCR
Node Daemon node name IP address Admin node name Designation
-------------------------------------------------------------------
1 c25m3n02-ib 10.168.3.12 c25m3n02-ib quorum-manager
2 c25m3n03-ib 10.168.3.13 c25m3n03-ib quorum-manager
3 c25m3n04-ib 10.168.3.14 c25m3n04-ib quorum
- The rest of the IBM Spectrum Scale cluster administrative setup commands such as designating node license, starting Spectrum Scale daemons on nodes, creating Network Shared Disks (NSD), create file-system, and mounting the file-system can be performed using sudo. The following section provides sample output from some of the administrative setup procedures.
#Start Spectrum Scale service on all the nodes and verify that the state is "active"
$ sudo /usr/lpp/mmfs/bin/mmgetstate -a
Node number Node name GPFS state
------------------------------------------
1 c25m3n02-ib active
2 c25m3n03-ib active
3 c25m3n04-ib active
#Create the NSD using mmcrnsd.
sudo /usr/lpp/mmfs/bin/mmcrnsd -F nsdStanzaFile
# Create and format the file-system using mmcrfs.
$ sudo /usr/lpp/mmfs/bin/mmcrfs gpfs1a -F nsdStanzaFile -j scatter -B 2M -m 1 -M 2 -r 1 -R 2 -T /mnt/gpfs1a
The following disks of gpfs1a will be formatted on node c25m3n02.gpfs.net:
DMD_NSD01: size 1714936 MB
DMD_NSD02: size 1714936 MB
DMD_NSD03: size 1714936 MB
DMD_NSD04: size 1714936 MB
DMD_NSD05: size 1714936 MB
Formatting file system ...
Disks up to size 18 TB can be added to storage pool system.
Creating Inode File
Creating Allocation Maps
Creating Log Files
Clearing Inode Allocation Map
Clearing Block Allocation Map
Formatting Allocation Map for storage pool system
Completed creation of file system /dev/gpfs1a.
mmcrfs: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
#Mount the file-system and verify the file-system is mounted successfully
$ sudo /usr/lpp/mmfs/bin/mmmount all -a
Tue Jun 21 18:03:27 EDT 2016: mmmount: Mounting file systems ...
$ sudo /usr/lpp/mmfs/bin/mmlsmount all -L
File system gpfs1a is mounted on 3 nodes:
10.168.3.12 c25m3n02-ib
10.168.3.14 c25m3n04-ib
10.168.3.13 c25m3n03-ib
IBM Spectrum Scale cluster administration command logging
The IBM Spectrum Scale sudo commands along with the issuer's user name will be logged in system log files (for example, /var/log/secure in RHEL Linux OS) and this can be useful for auditing purposes. The sample output below provides the commands logged during the Spectrum Scale cluster setup using sudo:
Jun 21 17:47:54 c25m3n02 sudo: gpfsadmin : TTY=pts/0 ; PWD=/home/gpfsadmin/gpfs/config ; USER=root ; COMMAND=/usr/lpp/mmfs/bin/mmcrcluster -N mmcrcluster-node.lst --ccr-enable --use-
sudo-wrapper -C gpfsSudoTest -c mmcrcluster-config.lst
Jun 21 17:47:55 c25m3n02 sudo: gpfsadmin : TTY=unknown ; PWD=/home/gpfsadmin ; USER=root ; ENV=LANG=en_US.UTF-8 LC_ALL= LC_COLLATE= LC_TYPE= LC_MONETARY= LC_NUMERIC= LC_TIME=
LC_MESSAGES= MMMODE=lc environmentType=lc2 GPFS_rshPath=/usr/lpp/mmfs/bin/sshwrap GPFS_rcpPath=/usr/lpp/mmfs/bin/scpwrap mmScriptTrace= GPFSCMDPORTRANGE= GPFS_CIM_MSG_FORMAT= ;
COMMAND=/usr/lpp/mmfs/bin/mmremote verifyRsh mmcrcluster:verifyRsh:28812
Jun 21 17:47:55 c25m3n02 sudo: gpfsadmin : TTY=unknown ; PWD=/home/gpfsadmin ; USER=root ; COMMAND=/usr/bin/scp -p -t /var/mmfs/tmp/rcpRemoteTestFile.mmcrcluster.28812
Jun 21 17:47:55 c25m3n02 sudo: gpfsadmin : TTY=unknown ; PWD=/home/gpfsadmin ; USER=root ; COMMAND=/usr/bin/scp -p -f /var/mmfs/tmp/rcpRemoteTestFile.mmcrcluster.28812
Jun 21 17:47:57 c25m3n02 sudo: gpfsadmin : TTY=unknown ; PWD=/home/gpfsadmin ; USER=root ; COMMAND=/usr/bin/scp -p -t /var/mmfs/tmp/sdrfsGen0.mmcrcluster.28812
Jun 21 17:47:57 c25m3n02 sudo: gpfsadmin : TTY=unknown ; PWD=/home/gpfsadmin ; USER=root ; COMMAND=/usr/bin/scp -p -t /var/mmfs/ssl/stage/genkeyData1
Jun 21 17:47:57 c25m3n02 sudo: gpfsadmin : TTY=unknown ; PWD=/home/gpfsadmin ; USER=root ; ENV=LANG=en_US.UTF-8 LC_ALL= LC_COLLATE= LC_TYPE= LC_MONETARY= LC_NUMERIC= LC_TIME=
LC_MESSAGES= MMMODE=lc environmentType= GPFS_rshPath=/usr/lpp/mmfs/bin/sshwrap GPFS_rcpPath=/usr/lpp/mmfs/bin/scpwrap mmScriptTrace= GPFSCMDPORTRANGE= GPFS_CIM_MSG_FORMAT= ;
COMMAND=/usr/lpp/mmfs/bin/mmremote checkNewClusterNode2 lc/setupServer /var/mmfs/tmp/sdrfsGen0.mmcrcluster.28812 26930 1 24723 1=c25m3n02-ib:1191,2=c25m3n03-ib:1191,3=c25m3n04-ib:1191
1191 1
Jun 21 17:50:42 c25m3n02 sudo: gpfsadmin : TTY=pts/0 ; PWD=/home/gpfsadmin/gpfs/config ; USER=root ; COMMAND=/usr/lpp/mmfs/bin/mmchlicense server --accept -N nodes.lst
Jun 21 17:53:27 c25m3n02 sudo: gpfsadmin : TTY=pts/0 ; PWD=/home/gpfsadmin/gpfs/config ; USER=root ; COMMAND=/usr/lpp/mmfs/bin/mmstartup -a
Jun 21 17:53:28 c25m3n02 sudo: gpfsadmin : TTY=unknown ; PWD=/home/gpfsadmin ; USER=root ; ENV=LANG=en_US.UTF-8 LC_ALL= LC_COLLATE= LC_TYPE= LC_MONETARY= LC_NUMERIC= LC_TIME=
LC_MESSAGES= MMMODE=lc environmentType=lc2 GPFS_rshPath=/usr/lpp/mmfs/bin/sshwrap GPFS_rcpPath=/usr/lpp/mmfs/bin/scpwrap mmScriptTrace= GPFSCMDPORTRANGE=0 GPFS_CIM_MSG_FORMAT= ;
COMMAND=/usr/lpp/mmfs/bin/mmremote startSubsys
Jun 21 17:56:43 c25m3n02 sudo: gpfsadmin : TTY=pts/0 ; PWD=/home/gpfsadmin/gpfs/config ; USER=root ; COMMAND=/usr/lpp/mmfs/bin/mmgetstate -a
Jun 21 17:56:44 c25m3n02 sudo: gpfsadmin : TTY=unknown ; PWD=/home/gpfsadmin ; USER=root ; ENV=LANG=en_US.UTF-8 LC_ALL= LC_COLLATE= LC_TYPE= LC_MONETARY= LC_NUMERIC= LC_TIME=
LC_MESSAGES= MMMODE=lc environmentType=lc2 GPFS_rshPath=/usr/lpp/mmfs/bin/sshwrap GPFS_rcpPath=/usr/lpp/mmfs/bin/scpwrap mmScriptTrace= GPFSCMDPORTRANGE=0 GPFS_CIM_MSG_FORMAT= ;
COMMAND=/usr/lpp/mmfs/bin/mmremote mmGetState
Jun 21 18:01:14 c25m3n02 sudo: gpfsadmin : TTY=pts/0 ; PWD=/home/gpfsadmin/gpfs/config ; USER=root ; COMMAND=/usr/lpp/mmfs/bin/mmcrnsd -F nsdStanzaFile
Jun 21 18:01:58 c25m3n02 sudo: gpfsadmin : TTY=pts/0 ; PWD=/home/gpfsadmin/gpfs/config ; USER=root ; COMMAND=/usr/lpp/mmfs/bin/mmlsnsd
Jun 21 18:02:57 c25m3n02 sudo: gpfsadmin : TTY=pts/0 ; PWD=/home/gpfsadmin/gpfs/config ; USER=root ; COMMAND=/usr/lpp/mmfs/bin/mmcrfs gpfs1a -F nsdStanzaFile -j scatter -B 2M -m 1 -M 2
-r 1 -R 2 -T /mnt/gpfs1a
Jun 21 18:03:09 c25m3n02 sudo: gpfsadmin : TTY=unknown ; PWD=/home/gpfsadmin ; USER=root ; COMMAND=/usr/bin/scp -p -t /var/mmfs/tmp/diskfile.mmcommon.9300
Jun 21 18:03:09 c25m3n02 sudo: gpfsadmin : TTY=unknown ; PWD=/home/gpfsadmin ; USER=root ; ENV=LANG=en_US.UTF-8 LC_ALL= LC_COLLATE= LC_TYPE= LC_MONETARY= LC_NUMERIC= LC_TIME=
LC_MESSAGES= MMMODE=lc environmentType=lc2 GPFS_rshPath=/usr/lpp/mmfs/bin/sshwrap GPFS_rcpPath=/usr/lpp/mmfs/bin/scpwrap mmScriptTrace= GPFSCMDPORTRANGE=0 GPFS_CIM_MSG_FORMAT= ;
COMMAND=/usr/lpp/mmfs/bin/mmremote getLocalNsdData2 c25m3n02-ib /var/mmfs/tmp/diskfile.mmcommon.9300 --gpt -F /var mmfs/tmp/diskfile.mmcommon.9300
Jun 21 18:03:27 c25m3n02 sudo: gpfsadmin : TTY=pts/0 ; PWD=/home/gpfsadmin/gpfs/config ; USER=root ; COMMAND=/usr/lpp/mmfs/bin/mmmount all -a
Jun 21 18:03:28 c25m3n02 sudo: gpfsadmin : TTY=unknown ; PWD=/home/gpfsadmin ; USER=root ; ENV=LANG=en_US.UTF-8 LC_ALL= LC_COLLATE= LC_TYPE= LC_MONETARY= LC_NUMERIC= LC_TIME=
LC_MESSAGES= MMMODE=lc environmentType=lc2 GPFS_rshPath=/usr/lpp/mmfs/bin/sshwrap GPFS_rcpPath=/usr/lpp/mmfs/bin/scpwrap mmScriptTrace= GPFSCMDPORTRANGE=0 GPFS_CIM_MSG_FORMAT= ;
COMMAND=/usr/lpp/mmfs/bin/mmremote mountFileSystems all DEFAULT DEFAULT DEFAULT
Jun 21 18:03:53 c25m3n02 sudo: gpfsadmin : TTY=pts/0 ; PWD=/home/gpfsadmin/gpfs/config ; USER=root ; COMMAND=/usr/lpp/mmfs/bin/mmlsmount all -L
Summary
IBM Spectrum scale is a complete software defined storage solution that delivers simplicity, scalability, high-speed access to data, and supports advanced storage management features such as compression, tiering, replication, and encryption. The sudo-based administration enables to securely administer the IBM Spectrum Scale components using non-root user name as well as logging of the administrative operations performed.
References
#Softwaredefinedstorage#IBMSpectrumScale