With the advent of Docker container support in IBM Spectrum LSF 10.1, it’s become much easier to build and maintain environments for containerized workloads. This blog will explorer using LSF Application Profiles with static Docker container options and then how to add dynamic Docker container options via scripting. More specifically, you will learn how to dynamically add the user's secondary LINUX user groups as Docker container run time options.
Prerequisites
To start with you will need the following:
| Component |
Version |
Edition |
| IBM Spectrum LSF |
10.1.0.3+ |
Standard Edition or Suite |
| Docker |
17.03+ |
Community or Enterprise Edition |
| Docker Engine |
1.12+ |
|
Verify your Docker Engine version with this command:
$ docker version | grep API
LSF Configuration
This blog assumes you have installed IBM Spectrum LSF and Docker and both are up and running on nodes in your cluster.
Prepare LSF to run jobs in Docker containers:
Follow steps in this URL: https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_docker/lsf_docker_prepare.html
Configure a test LSF Application Profile
- Create a new LSF Application Profile
Next, we define a Docker application profile in the lsb.applications configuration file. The lsb.application file is located in $LSF_ENVDIR/lsbatch/<your_cluster_name>/configdir.
Begin Application
NAME = dockerapp1
CONTAINER = docker[image(ubuntu:latest) options(--rm --net=host --ipc=host ) \
-v /etc/passwd:/etc/passwd \
-v /etc/group:/etc/group \
starter(root)] \
DESCRIPTION = Test Docker Application Profile 1
End Application
Note, the above example assumes you are using only local passwords and groups.
- Reconfigure the LSF Master batch daemon:
To add or modify LSF Application Profiles reconfigure or restart the LSF Master Batch daemon on the LSF Master node as the LSF Administrator.
$ badmin reconfig
Additional information is available here: https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_docker/lsf_docker_config.html
Running a Test Job
Use the bsub -app option to tell LSF to run a job using a particular application. In our case the container is called “dockerapp1“, so run something like this:
$ bsub -app dockerapp1 -I echo Hello World
Job <760> is submitted to default queue <interactive>.
<<Waiting for dispatch …>>
<<Starting on gauss02>>
Hello World
$
Note, if the Docker image specified in the Application Profile is not on the LSF compute node, it will be pulled down to the compute node. Please consider setting up on your own Docker Registry so as to pull Docker images from your local Docker Registry. Here is an example job that outputs the version of Ubuntu:
$ bsub -app dockerapp1 -I cat /etc/lsb-release
Job <761> is submitted to default queue <interactive>.
<<Waiting for dispatch …>>
<<Starting on gauss02>>
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION=”Ubuntu 18.04 LTS”
$
Create a script to generate the Docker options
Create the script below and call the script docker_options.sh. You will need to place the script on every LSF node in the same directory path or ideally place the script into a shared directory path that is accessible across all LSF nodes. For this example, we will assume the shared directory path is /shared_directory_path/scripts. Additionally, the script must be executable by the LSF Administrator user, which is typically lsfadmin.
Refer to LSF documentation for more details: https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_config_ref/lsb.applications.5.html
/shared_directory_path/scripts/docker_options.sh
#!/bin/sh
#
# Example script to add Docker run options to an LSF job
#
# This script outputs the LSF job’s user’s groups.
#
# Example output when user has groups of 1100 and 1200:
# –group-add 1100 –group-add 1200
#
# Options should be written to standard output on a single line
#
IDCMD=”id $LSFUSER”
# Add groups
for GID2 in `$IDCMD -G`
do
echo -n ” –group-add $GID2″
done
# add newline to standard output
echo
Configure a second test LSF Application Profile with the Docker options script
- Add the LSF Application Profile below to lsb.applications file.
Begin Application
NAME = dockerapp2
CONTAINER = docker[image(ubuntu:latest) options(--rm --net=host --ipc=host ) \
-v /etc/passwd:/etc/passwd \
-v /etc/group:/etc/group \
-v /shared_directory_path/scripts:/shared_directory_path/scripts \
@/shared_directory_path/scripts/docker_options.sh \
starter(root)] \
DESCRIPTION = Test Docker Application Profile 2
End Application
Please change the /shared_directory_path/scripts above to the location where you placed the docker_options.sh script.
- Reconfigure the LSF Master batch daemon:
To add or modify LSF Application Profiles reconfigure or restart the LSF Master Batch daemon on the LSF Master node as the LSF Administrator.
$ badmin reconfig
Test the solution
First, you will need to login as a user that belongs to more than one group and then you can perform the tests below. Here is an example testing the two different application profiles created: Verify your user id has multiple groups.
$ id -G 1100 1200 $
The first application profile should only be aware of your first group.
$ bsub -app dockerapp1 -I id -G
Job <762> is submitted to default queue <interactive>.
<<Waiting for dispatch …>>
<<Starting on gauss02>>
1100
$
The second application profile should show all your groups.
$ bsub -app dockerapp2 -I id -G
Job <763> is submitted to default queue <interactive>.
<<Waiting for dispatch …>>
<<Starting on gauss02>>
1100 1200
$
Conclusion
You have seen how to create a simple LSF Application Profile with static Docker container options and then create another LSF Application Profile to add dynamic Docker container options at run time via a script.
#SpectrumComputingGroup