IBM LSF is already widely used to manage training pipelines, simulation, and other batch-oriented AI workloads. The next practical question for many teams is how to operationalize inference?

IBM LSF is designed to schedule workloads with highly-variable runtimes—from seconds to months; treating AI inference as just another long-running LSF job removes operational friction and unlocks immediate business value and provides practical advantages when running both training and inference on the same cluster and hardware.

Of course, for these to be persistent and usable by others we also need to expose and manage endpoints, continuously accept requests, and can be reused by multiple consumers over time. 

This article presents a practical implementation pattern for doing that with IBM LSF and vLLM, using IBM LSF as the backend execution and control layer for persistent inference services.

Implementation goals

This implementation places IBM LSF in a role that is directly relevant to modern AI execution models. IBM LSF already brings together strong resource control, workload lifecycle management, predictable scheduling behavior, and operational observability. Those strengths extend naturally from traditional finite jobs into persistent inference workflows when the runtime is treated as a managed service job.

Technically, the implementation demonstrates that IBM LSF can govern long-running model-serving endpoints in a disciplined way. Strategically, it shows that IBM LSF fits comfortably into an AI operating model that spans both model training and model inference. The same execution environment that organizations trust for training-oriented workloads can also support service-style inference patterns, downstream automation, and endpoint-aware workflows. Specifically it demonstrates:

·       Running a long-lived model-serving endpoint as an IBM LSF-managed job rather than as an unmanaged background process.

·       Exposing a real OpenAI-compatible API surface through vLLM.

·       Making the endpoint discoverable through job-linked metadata so clients do not need to know placement details in advance.

·       Reuse of the same persistent runtime from multiple consumers, including interactive clients (e.g. Jupyter notebooks, Web UI’s) and downstream batch workflows.

·       Be extensible to multi-GPU-backed deployments accept requests continuously and can be reused by multiple consumers over time.

Taken together, these goals move the discussion beyond whether a scheduler can launch a server and toward the more relevant enterprise question: can a scheduler act as a practical control plane for long-running AI inference services and the workflows that depend on them? This implementation shows that the answer is yes.

Environment used

I intentionally kept the test environment small to ensure the pattern remains easy to reproduce and straightforward to explain. The system used for the implementation was a single-node IBM LSF environment with Podman, Python, curl, and a CPU-based vLLM image. The model used was Qwen/Qwen3-0.6B. The purpose was not to maximize model quality or throughput; it was to validate orchestration, lifecycle management, endpoint exposure, and runtime reuse under IBM LSF. Namely:

·       IBM LSF submits and manages a long-running vLLM service job.

·       The service job starts vLLM inside a container, waits for readiness, and writes a registry record containing the resolved host and port.

·       Interactive users can call the service directly through the OpenAI-compatible API using curl, Python, Jupyter notebook, or Open WebUI.

·       Downstream batch workflows can consume the same endpoint through a second IBM LSF job.

·       The same operational pattern extends naturally to GPU-backed nodes for larger models and more demanding latency or throughput targets.

Although the implementation used a CPU-based runtime for simplicity, the pattern is not limited to CPU nodes. The same IBM LSF-managed service model can be extended to GPU-enabled environments where vLLM is deployed against larger models and higher-throughput requirements.

Managing the Service

All implementation artifacts were kept under a dedicated workspace so that service logs, batch logs, registry metadata, model cache, prompt corpus, and output results remained easy to inspect. Keeping those artifacts separate makes the implementation easier to troubleshoot and easier to explain to others.

The service wrapper is the operational heart of the implementation. It converts a standard IBM LSF job into a long-running AI runtime that behaves like a service endpoint. The wrapper selects the model, API key, and container image; starts the vLLM container; waits until the runtime responds successfully on /v1/models; writes a registry file linked to the IBM LSF job ID; and then continues streaming logs so that IBM LSF remains attached to the service lifecycle.

·       The wrapper starts exactly one service instance and is submitted with -n 1.

·       It waits for readiness before registration, ensuring the endpoint record is meaningful.

·       It ties the registry file to the IBM LSF job ID, creating a natural mapping between scheduler state and service discovery.

·       It keeps the service lifecycle visible through IBM LSF and container logs.

Service submission and startup behavior

The service job was submitted through bsub with the wrapper script as the execution command, one slot, explicit memory reservation, and dedicated output and error logs. In the demonstrated run, the service job was assigned IBM LSF job ID 207 and entered RUN state on the target host. That was the first important operational checkpoint: the persistent runtime existed as a scheduler-managed object, not as an unmanaged background process.

JOBID=$(
  bsub -J vllm_service \
       -q normal \
       -n 1 \
       -R 'rusage[mem=4GB]' \
       -oo ~/lsf_vllm_poc/logs/vllm.%J.out \
       -eo ~/lsf_vllm_poc/logs/vllm.%J.err \
       ~/lsf_vllm_poc/start_vllm_lsf.sh \
  | awk '{print $2}' | tr -d '<>'
)

The startup sequence showed the expected progression of a real model-serving runtime: vLLM initialization, model resolution, weight loading, KV-cache setup, warmup, and API server startup. This is important because it demonstrates IBM LSF managing the full lifecycle of a genuine inference service rather than simply holding a placeholder process alive.

As expected, first-run startup on CPU included model initialization and warmup time. That does not change the core result: IBM LSF successfully governed the persistent runtime from submission through readiness and ongoing service availability.

·       Service job submitted as JOBID 207

·       Container name: vllm-job-207

·       vLLM API server started on port 8000 inside the container

·       Host endpoint published as http://centos9-lsf1.fyre.ibm.com:8001/v1

Endpoint registration and discovery

Once readiness succeeded, the wrapper wrote a registry file containing the job ID, service name, model, resolved host, published port, API key, and status. That file served as the discovery record for downstream consumers. Operationally, this is the step that transforms a scheduler-managed execution into a discoverable service runtime.

{
  "jobid": "207",
  "service_name": "qwen-chat",
  "model": "Qwen/Qwen3-0.6B",
  "host": "centos9-lsf1.fyre.ibm.com",
  "port": 8001,
  "api_key": "local-vllm-key",
  "status": "ready"
}

The endpoint URL was then resolved programmatically from the registry file, yielding a clean OpenAI-compatible base URL that clients could consume without hard-coding placement assumptions.

Interactive client validation

A key strength of the implementation is that the same IBM LSF-managed runtime was validated across multiple interactive client patterns. This demonstrates how IBM LSF can serve as the backend execution and control layer while allowing flexibility in the front-end experience.

Direct API validation with curl

The first validation path used the OpenAI-compatible API directly. The /v1/models call confirmed that the model server was reachable, authenticated, and advertising the expected model. A subsequent /v1/chat/completions request returned a valid JSON response, proving that the persistent runtime was alive, discoverable, and callable as a real service.

Jupyter notebook validation

A Jupyter notebook provided a more intuitive interactive client path. This approach maps naturally to how data scientists, developers, and solution architects typically work. In the notebook, the runtime successfully listed the deployed model and executed a chat-completions request with HTTP 200 responses. This confirms that IBM LSF can operate as the backend control plane while familiar user-facing tools interact with the service through a standard API surface.

Jupyter notebook validation against the IBM LSF-managed vLLM service. The notebook successfully listed the deployed model and executed a chat-completions request with HTTP 200 responses.

Open WebUI validation

The same IBM LSF-managed runtime was also validated through a browser-based conversational interface using Open WebUI. This is an important extension of the story because it demonstrates that the persistent inference service can support a familiar user-facing chat experience without changing the underlying IBM LSF execution pattern.

In this validation path, Open WebUI acted as the lightweight front-end client while IBM LSF continued to govern the long-running vLLM service in the backend. The Open WebUI session was configured to use the same Qwen/Qwen3-0.6B runtime that had been launched and registered through the IBM LSF service flow.

·       Persistent IBM LSF-managed vLLM runtime reused through a browser-based chat UI

·       Open WebUI validated as a front-end client against the same registered inference endpoint

·       Consistent service reuse across curl, Jupyter notebook, batch client, and browser-based chat access

Open WebUI connected to the IBM LSF-managed Qwen/Qwen3-0.6B runtime and returning an interactive chat response.

Downstream batch reuse

To demonstrate reuse beyond interactive calls, a small prompt corpus and a simple Python batch client were created. The batch client did not start another model server. Instead, it read the registry file for the service job, resolved the endpoint, submitted prompts to the existing vLLM runtime through /v1/chat/completions, and wrote JSONL output to the results directory.

cat > ~/lsf_vllm_poc/corpus/prompts.txt <<'EOF'
Why do rivers flow towards the sea?
What is the difference between a lion and a tiger?

How do fish survive in water without breathing air like humans?
EOF

The batch client was first validated locally and then submitted as a separate IBM LSF job. This demonstrated that IBM LSF can simultaneously manage a persistent service job and a separate downstream consumer job while the runtime remains reusable under scheduler control.

BATCH_JOBID=$(
  bsub -J vllm_batch \
       -q normal \
       -n 1 \
       -R 'rusage[mem=1GB]' \
       -oo ~/lsf_vllm_poc/logs/batch.%J.out \
       -eo ~/lsf_vllm_poc/logs/batch.%J.err \
       "python3 ~/lsf_vllm_poc/batch_client.py ${JOBID}" \
  | awk '{print $2}' | tr -d '<>'
)

Operational evidence from the run

From an operational perspective, the implementation generated all the right proof points: IBM LSF job state, container identity, endpoint registration, model validation through /v1/models, successful chat-completions requests, notebook success, Open WebUI success, and JSONL batch results. Those artifacts are exactly what an engineer, architect, or operations team would expect to inspect when validating a new runtime pattern.

Production-readiness considerations

·       Keep the service wrapper intentionally small and observable. The more direct the startup path, the easier it is to debug readiness, lifecycle behavior, and service registration.

·       Treat endpoint registration as a first-class concern. The moment a scheduler-managed job becomes a reusable service, discoverability becomes just as important as execution.

·       Separate orchestration validation from model-selection decisions. IBM LSF lifecycle correctness, endpoint exposure, and client reuse can be validated independently of whether a compact demonstration model is the long-term production choice.

·       Prefer a user-facing interactive client such as a Jupyter notebook or lightweight application when presenting the pattern externally. The batch consumer remains valuable as proof of reuse, but interactive front ends make the value easier to understand.

·       Plan early for how the pattern should evolve: resolver APIs, multiple service instances, service pools, GPU-backed deployments, authentication, and monitoring integration all fit naturally on top of the demonstrated foundation.

How this pattern can evolve

The file-based registry used in the implementation was deliberately simple so the pattern would remain transparent and easy to debug. A natural next step is a lightweight resolver service that maps job ID or service name to a resolved endpoint, making client resolution cleaner and closer to production-style service discovery.

Another natural evolution is to run multiple IBM LSF-managed vLLM service instances simultaneously. That would create space for service pools, selection strategies, and richer placement discussions. In GPU-backed environments, the same runtime pattern can be applied to larger models and more demanding latency or throughput objectives while IBM LSF continues to govern allocation and lifecycle behavior.

Conclusion

Framing persistent inference as a natural extension of LSF’s existing long-running job model delivers immediate business and operational advantages: faster deployments, lower costs, consistent governance, and better utilization of shared hardware. By running training and inference on the same cluster under LSF control, organizations gain a single, auditable, and efficient execution model that supports both experimentation and production-grade AI services.

The broader takeaway is clear: IBM LSF can serve as a practical operational control layer for persistent AI inference services while remaining flexible about how those services are consumed. This implementation makes that foundation visible in a concrete, repeatable, and understandable way.

Reference implementation: The code and scripts used in this blog are available in the IBM Spectrum Computing integrations repository:

https://github.com/IBMSpectrumComputing/lsf-integrations/blob/master/LSF-vLLM/README.md