In this concluding blog of the series, (part 1, part 2) I’ll discuss the Open API Specification for the new LSF Web Service, and how to generate language specific bindings from it, such as Python.
It’s just an API…
LSF is written in C, and no surprise that the public API for LSF is also in C. Over the year we have released Perl and Python bindings, the latter automatically generated from the C API using swig. While you can call cross language API’s, it isn’t straightforward.
“The C language combines all the power of assembly language with all the ease-of-use of assembly language.” - Mark Pearce
Web Services aimed to simplify this, and while the existing LSF Application Center Restful API is documented, and we provided sample Python bindings for it, it was not easy to integrate or extend, and while it was possible to create bindings in other languages, it was a lot of heavy lifting.
An OpenAPI Specification
With the new LSF Web Service (LWS) we set out to create a new API that conformed to the OpenAPI Specification (OAS). Most importantly, this provided a clear separation of the API specification from the language(s) it was implemented in, or consumed in, making it much easier to use from the language of your choice.
The specification can be found in the installation package in “openapi.yaml”, alternatively, once the service is installed and running, it can be generated from the service itself:
billmc@billvm1:/home/billmc$ curl http://lwshostA:8088/openapi
openapi: 3.0.0
info:
title: LSF Web Service
description: LSF Web Service
version: "1.0"
servers:
- url: "{schema}://{host}:{port}/lsf"
variables:
schema:
description: Server schema
enum:
- http
- https
default: http
host:
description: LSF Web Service host
default: localhost
port:
description: LSF Web Service port
default: "8088"
tags:
- name: File Operations
description: "File operations, including upload, download, list file, list repository,\
\ delete"
- name: LSF Cluster
…
Alternatively, you can access the specification UI http://lwshostA:8088/openapi/ui
For those skilled in writing web services, that is likely all you need to use this API. But others may prefer bindings in their language of choice.
Generating Language Specific Bindings
I've been in the HPC game a long time, and if truth be told, my language of choice would be Fortran, but C/C++ will do at a push. But much of today's development is in Python, Go, Ruby or half a dozen other languages. Creating an API for each of these would be impractical, however, there are many tools that can auto generate generate language specific bindings based upon an OpenAPI specification. For this example we will use the “openapitools” package https://github.com/OpenAPITools/openapi-generator which is available in a prebuilt container on dockerhub. This package supports API generation for over two dozen different languages....but apparently not Fortran :-(
Assuming you have docker installed, and that the “openapi.yaml” is in $PWD/local, we can generate Python and Go bindings using:
billmc@billvm1:/home/billmc$ docker run --rm -v "${PWD}:/local" openapitools/openapi-generator-cli generate -i /local/openapi.yaml -g python -o /local/out/py
billmc@billvm1:/home/billmc$ docker run --rm -v "${PWD}:/local" openapitools/openapi-generator-cli generate -i /local/openapi.yaml -g go -o /local/out/go
The generated bindings will be under out/py/ and out/go/ respectively.
Pre-built Python Bindings
If you don’t want to generate it yourself, we have included pre-built python bindings “lws_api_sample.tar.Z”.
Untar this package, and as root:
#python setup.py install
The package includes several test cases that can be found in the “test” subdirectory:
billmc@billvm1:/home/billmc/tmp/lws_api_sample/test$ dir
README job_input.inp test_file_delete.py test_file_tree.py test_list_repos.py
cacert.pem test_cluster.py test_file_download.py test_file_upload.py test_logon.py
These need to be modified to use your credentials and URL. For example:
billmc@billvm1:/home/billmc/tmp/lws_api_sample/test$ more test_logon.py
import openapi_client
from openapi_client.models.user import User
from openapi_client.models.session import Session
from openapi_client.rest import ApiException
from pprint import pprint
json = '{"name":"billmc", "pass":"itsmypassword"}'
user_instance = User.from_json(json)
configuration = openapi_client.Configuration(
host = "https://lwshostA:8448/lsf",
ssl_ca_cert = "cacert.pem"
)
print(configuration.host)
print(user_instance)
with openapi_client.ApiClient(configuration) as api_client:
api_instance = openapi_client.AuthenticationApi(api_client)
try:
api_response = api_instance.logon(user=user_instance)
print("The response of AuthenticationApi->logon:\n")
pprint(api_response)
except Exception as e:
print("Exception when calling AuthenticationApi->logon: %s\n" % e)
We can then connect to the LWS service, and if successful, it will respond with a session key:
billmc@billvm1:/home/billmc/tmp/lws_api_sample/test$ python test_logon.py
Session(token=‘billmc"2024-04-11T01:18:44Z"8UpNKaIaaBDQyxN3/dWafD8WWBpsSsWIZ4honr78mPq3ng3Ok4B4ycjZsUx437tVEW5Xf9BtEOWNdHJM7nJ1Fxlp2SBiEsN881sB7tvtWRSVg4wsVc5r/CWSftZf7NYf"zltCC28BDis+PPYRNRBCQA==', cluster=LSFCluster(jobid_range='999999', cluster_index='10', name='lsf', version='IBM Spectrum LSF Standard 10.1.0.14, Apr 24 2023’))
We can copy this session key to the API_KEY environment variable:
billmc@billvm1:/home/billmc/tmp/lws_api_sample/test$ export API_KEY=‘billmc"2024-04-11T01:18:44Z"8UpNKaIaaBDQyxN3/dWafD8WWBpsSsWIZ4honr78mPq3ng3Ok4B4ycjZsUx437tVEW5Xf9BtEOWNdHJM7nJ1Fxlp2SBiEsN881sB7tvtWRSVg4wsVc5r/CWSftZf7NYf"zltCC28BDis+PPYRNRBCQA==‘
And we can then run other test cases with these credentials:
billmc@billvm1:/home/billmc/tmp/lws_api_sample/test$ test_list_repos.py
The response of FileOperationsApi->list_repos:
Repositories(repos=[RepositoryBean(name='all', path='/home/billmc/')])
Conclusion
By utilizing the OpenAPI specification, we’ve (hopefully!) made the LSF Web Service easier to consume from the language of your choice, and to integrate it into your workflows. If you are interested in trying this preview package, please register here.
Acknowledgements
Credit where credit is due and I'd like to that Jian Jin, Guo Liang Wang and George Gao for creating this new service.