This blog builds on my previous article on Compiling Open MPI with IBM Spectrum LSF in a Docker container image and extends the concept to OpenFOAM for both X86_64 and IBM Power server platforms. Since 2016 LSF 10.1 has been providing deep container integration, which has made it easier to build and maintain environments running containerized workloads. Additionally, LSF integrates with most MPI implementations, including Open MPI, via an adaptable and scalable distributed application framework. LSF job submissions are extremely flexible including affinity, topology directives and other resources requirements which are integrated with Open MPI. This blog will explorer building a custom Docker container for running OpenFOAM jobs in parallel using LSF and Open MPI.
Prerequisites
This blog assumes you have installed IBM Spectrum LSF on X86_64 or Power Little Endian platform (or linux3.10-glibc2.17-ppc64le) and Docker is up and running on nodes in your cluster. To start with you will need the following:
| Component |
Version |
Edition |
|
| Red Hat Linux Server |
7.6 or higher |
Enterprise |
| IBM Spectrum LSF |
10.1.0.8+ |
Standard Edition or Suite |
| Docker |
17.03+ |
Community or Enterprise Edition |
| Docker Engine |
1.12+ |
|
Verify your Docker Engine version with this command:
$ docker version | grep VERSION
Build a customer OpenFOAM container with Open MPI compile with LSF
Login as a user with the ability to run docker commands. The steps below assume your working directory (pwd) will remain the same through out the steps below.
Prepare minimal LSF files for Open MPI compile
The goal is to prepare the minimal files from your LSF environment necessary to compile Open MPI with LSF inside a Docker container. Copy the script below and paste into a file called mktmplsf.sh. This script will generate a directory called "lsf" with the LSF libraries, include files and configuration file. The files in the "lsf" directory will be used in the next step.
Get a copy of
mktmplsf.sh from github.
Here are the steps to run the script and see the directories and files created:
$ chmod +x mktmplsf.sh
$ ./mktmplsf.sh
$ ls -R lsf
lsf:
10.1 conf
lsf/10.1:
include lib
lsf/10.1/include:
lsf
lsf/10.1/include/lsf:
lsbatch.h lsf.h
lsf/10.1/lib:
libbat.a libbat.so liblsf.a liblsf.so
lsf/conf:
lsf.conf
$
Create a Dockerfile
Get a copy of
Dockerfile from github.
Build a new Docker container
Use the command below to create the new container. It will take many minutes to perform all the steps to create the new container image, which will be called "openfoam/openfoam:v1912". Note, both the Dockerfile file and lsf directory should be in your current working directory.
# ls Dockerfile
Dockerfile
# ls lsf
10.1 conf
# docker build -t openfoam/openfoam:v1912 .
Sending build context to Docker daemon 3.804 GB
Step 1/41 : FROM ubuntu:bionic
---> 172193eb06e7
…
…
…
Removing intermediate container 5b5a5467074e
Step 41/41 : WORKDIR /opt
—> ef0732ed88a7
Removing intermediate container fd79ce38c0e0
Successfully built ef0732ed88a7
#
Now, run the
docker images command and your new container image should be there unless the
docker build command failed.
# docker images | grep openfoamopenfoam/openfoam v1912 ef0732ed88a7 5 minutes ago 6.68 GB
#
You can repeat the above
docker build process on every Docker enabled compute node in your LSF cluster or you can use other methods such as publishing the container to your internal Docker Registry or use the
docker save and the
docker load commands.
Setting up LSF with Docker
1). Prepare IBM Spectrum LSF to run jobs in Docker container by following these steps: LSF docker integration instruction.
2). Configure LSF Docker Application profile for the new Docker container image by adding the following lines to the end of lsb.applications file:
Begin Application
NAME = openfoam
DESCRIPTION = Example OpenFOAM application
CONTAINER = docker[image(openfoam/openfoam:v1912) \
options(--rm --net=host --ipc=host --cap-add=SYS_PTRACE \
-v JOB_REPOSITORY_TOP:JOB_REPOSITORY_TOP \
@JOB_REPOSITORY_TOP/scripts/dockerPasswd.sh \
) starter(root) ]
EXEC_DRIVER = context[user(lsfadmin)] \
starter[LSF_SERVERDIR/docker-starter.py] \
controller[LSF_SERVERDIR/docker-control.py] \
monitor[LSF_SERVERDIR/docker-monitor.py]
End Application
These additional changes are required to the above application profile:
a). Find a shared directory for all computer nodes, and replace JOB_REPOSITORY_TOP with the real path in above content
b). Change LSF_SERVERDIR to your LSF SERVERDIR direction location.
c).Get a copy of dockerPasswd.sh from github and replace the #JOB_REPOSITORY_TOP# with real value in the following line:
JOBTMPDIR=#JOB_REPOSITORY_TOP#
d). Copy dockerPasswd.sh to JOB_REPOSITORY_TOP/scripts/dockerPasswd.sh and make the script executable
chmod a+x dockerPasswd.sh
e). Restart the LSF mbatchd on the LSF Master by running either badmin mbdrestart or badmin reconfig.
Testing the new container with LSF
$ bsub -app openfoam -I ls /opt
Job <20773> is submitted to default queue <interactive>.
<<Waiting for dispatch ...>>
<<Starting on ac922c>>
OpenFOAM-v1912 ThirdParty-v1912 ibm
$
Testing the new container with MPI Hello World
Make sure MPI is working as expected before attempting to run the OpenFOAM parallel capabilities.
Example of running MPI hello world on a single node with 1 message
$ bsub -app openfoam -I mpirun /usr/local/bin/mpi_hello_world
Job <20774> is submitted to default queue <interactive>.
<<Waiting for dispatch …>>
<<Starting on ac922c>>
Hello world from processor ac922c, rank 0 out of 1 processors
$
Example of running MPI hello world on a single node with 2 job slots or 2 messages.
$ bsub -app openfoam -I -n 2 -R "span[hosts=1]" mpirun /usr/local/bin/mpi_hello_world
Job <20775> is submitted to default queue <interactive>.
<<Waiting for dispatch …>>
<<Starting on ac922c>>
Hello world from processor ac922c, rank 0 out of 2 processors
Hello world from processor ac922c, rank 1 out of 2 processors
$
Example of running MPI hello world on 2 nodes with 1 message per node.
$ bsub -app openfoam -I -n 2 -R "span[ptile=1]" mpirun /usr/local/bin/mpi_hello_world
Job <20776> is submitted to default queue <interactive>.
<<Waiting for dispatch …>>
<<Starting on ac922c>>
Hello world from processor ac922c, rank 0 out of 2 processors
Hello world from processor ac922b, rank 1 out of 2 processors
$
Conclusion
Now, you have a new container image that is ready to run OpenFOAM using Open MPI with LSF. Additionally, a LSF Application Center submission template is available at
OpenFOAM template along with a
ParaView template. Please leave comments or feedback on the above information.
#SpectrumComputingGroup