Mark Twain once said "There is no such thing as a new idea. It is impossible. We simply take a lot of old ideas and put them into a sort of mental kaleidoscope".
Web services are hardly a new idea. Nearly two decades have passed since the Global Grid Forum introduced several innovative concepts, such as OGSA-HPC, SAGA, BES, BES++, and JSDL. These concepts have been incorporated into LSF and the LSF Application Center, notably within the LSF Application Center's RESTful API.
Currently, there is an increased emphasis on integrating LSF with other applications and embedding it within workflows, particularly AI tools. Both the traditional C API and the existing RESTful API present certain limitations, prompting a re-evaluation of potential improvements.
In this initial blog post of a three-part series, I will explore the rationale behind introducing a new API and detail the installation process of the service. The subsequent post will delve into the new client-side tools, and the final entry will examine the OpenAPI specification.
This new web service package is available now as technical preview – I encourage you to give it a go! If you are interested in trying this, please sign up here.
What do we want from this new API?
Overall, we wanted to make it easier to connect to the LSF cluster, from anywhere, while preserving the existing user experience and investment in scripts that use the LSF CLI. Technically speaking you can install the existing LSF Client binaries anywhere – and have them connect to the cluster – if you have the necessary privileges, all the necessary ports are open and appropriate configurations shared. The “necessary ports” are a challenge for Security Professionals who do not want to open custom ports, especially for proprietary protocols that they cannot easily inspect.
With LSF Application Center we do provide a version of the LSF CLI that connects over https to the cluster – which appeases the Security folk, but there are dozens of binaries, and on Windows it requires Administrative privileges to install.
We could just provide 1-to-1 equivalent to the C API – but we want to take this opportunity to address limitations in that API. For example, the default security model assumes an LSF Client is a member of one and only one LSF cluster. If you want the client to submit/query different clusters, you have to jump through a few hoops – which isn’t great usability.
To summarize, we want the:
Server/Server API to be:
• Lightweight and fast: Provides direct access to the LSF cluster (Application Center not required)
• Secure: All communication is encrypted by default (https/TLS) with a choice of authentication methods (User, Single Sign On, API Key).
• Performant: As close as possible to CLI speeds, and can easily be scaled out to support large numbers of connections.
• Easy to use: Self documenting OpenAPI specification.
• Compatibility: Preserves existing integration investment in esub etc
Client/Client API to be:
• Lightweight: We want it to be simple to install, with no elevated privileges required.
• Compatibility: Preserves existing LSF syntax so that it doesn’t break existing scripts.
• Flexible: Supports connecting to multiple LSF Clusters and supports existing CLI and JSON outputs
• Portable: Can be used on a wide range of OS’s.
Installing the new LSF Web Service (LWS)
As root, on an LSF existing LSF host, untar the package: lws10.1.0.14preview_linux-x86_64.tar.Z
root@lsfhost:/tmp# tar xvf lws10.1.0.14preview_linux-x86_64.tar.Z
root@lsfhost:/tmp# cd lws10.1.0.14_linux-x86_64/
root@lsfhost:/tmp/lws10.1.0.14_linux-x86_64# dir
LA ibm-jre-1.8.0-24042206.x86_64.rpm lsf-webservice-10.1.0.14-24042207.x86_64.rpm
LA.tar jre_linux-x86_64.tar lwsinstall.sh
LAPApp.jar linux-x86_64 lwsuninstall.sh
Edit lwsinstall.sh, add a line to source LSF cluster ENV after line “# . $LSF_ENVDIR/profile.lsf”
root@lsfhost:/tmp/lws10.1.0.14_linux-x86_64# vi lwsinstall.sh
…
# . $LSF_ENVDIR/profile.lsf
. /opt/lsf/conf/profile.lsf
…
Run ./lwsinstall.sh , accept IBM license by type “1”. Support for https is enabled by default – enter “N” to keep it enabled, “Y” to disable it. We’ll disable https for now – so we don’t need to worry about installing certificates on the server and client (If we did select “N”, the server side, self signed certificate would be installed automatically). It will install the ibm-jre and lsf-webservice rpm’s into /opt/ibm/. Change directory, source the profile, then start the service:
root@lsfhost:/tmp/lws10.1.0.14_linux-x86_64# cd /opt/ibm/lsfsuite/ext
root@lsfhost:/opt/ibm/lsfsuite/ext# . /opt/ibm/lsfsuite/ext/profile.platform
root@lsfhost:/opt/ibm/lsfsuite/ext# lwsstart.sh
root@lsfhost:/opt/ibm/lsfsuite/ext# lwsstatus.sh
SERVICE STATUS PID PORT HOST_NAME
WEBSERVICE STARTED 3976321 8088 lsfhost
root@lsfhost:/opt/ibm/lsfsuite/ext# netstat -na|grep 8088
tcp6 0 0 :::8088 :::* LISTEN
All good.
At the time of writing, the Service and client have been tested on:
- x86 with RHEL 8.x, RHEL 9.x, SLES 15.5.
- x86 with Ubuntu 20.04 & 22.04 - pam_cap.so must be disabled in /etc/pam.d/common-auth.
We will be adding server support for aarch64 and ppc64le in the near future.
In this blog I’ve covered why we wanted to create a new LSF Web Service, and how to install it. But what can we do with it? How do we interact with it?
In the words of P.T.Barnum “always leave them wanting more” – stay tuned for part two!
#Spectrum-LSF