Written By:

Jeremias Werner - Architect, STSM, IBM Cloud Code Engine

Batch Inferencing at Scale with IBM Cloud Code Engine Serverless Fleets

Modern AI workloads increasingly depend on the ability to process massive volumes of data quickly, cost‑efficiently, and without the overhead of managing infrastructure. Whether it’s transforming thousands of documents, running large‑scale simulations, or performing high‑throughput inferencing over large datasets, organizations need compute that elastically scales with demand—yet remains simple for developers and efficient for enterprises.

IBM Cloud Code Engine Serverless Fleets deliver exactly that: a fully managed, massively scalable runtime for compute‑intensive workloads on CPUs or GPUs. In this blog post, we summarize the batch inferencing scenario showcased in a recent technical deep dive, explaining how Code Engine enables developers and client engineering teams to run AI inferencing at scale with minimal operational complexity.

What is IBM Cloud Code Engine?

IBM Cloud Code Engine is a fully managed, serverless compute platform that runs almost any type of code—containers, batch jobs, web apps, functions—without requiring developers to manage servers, clusters, autoscaling, or infrastructure tuning. You bring code; Code Engine handles the rest: deployment, scaling, networking, monitoring, and security.

Key characteristics include:

• Runs any workload: apps, functions, batch jobs, and large‑scale serverless fleets.

• Automatic scaling up and down to zero: no need to size or reserve infrastructure.

• Pay only for what you use: charged per vCPU‑second, GB‑second, and GPU‑second.

• Enterprise‑grade security: SOC2, C5, GDPR, PCI‑DSS, … compliance.

• Runs inside your VPC: required for secure data access in many enterprise scenarios.

Serverless Fleets extend this experience to compute‑heavy, parallelizable workloads—including AI training, doc processing, simulations, and of course batch inferencing.

Understanding Serverless Fleets

Serverless Fleets are designed for massively parallel, run‑to‑completion workloads. Instead of managing clusters or HPC schedulers, developers submit tasks, specify CPU/GPU needs, and Code Engine provisions as many workers as needed—up to thousands or even millions of tasks.

A key advantage is the pull‑based design. Think of airport security rather than supermarket checkout lines:

• Workers pull tasks from a shared task queue.

• Code Engine automatically provisions workers across zones.

• Tasks run concurrently based on CPU/GPU resources you specify.

• When all tasks finish, compute scales to zero and billing stops.

This model delivers high throughput, no idle cost, and zero operations overhead.

Interactive vs. Batch Inferencing

Interactive inferencing

• Single request, expecting milliseconds to seconds latency.

• GPUs must stay pre‑warmed and models must remain loaded.

• Expensive idle time is inevitable.

• Used for chatbots, assistants, and live user queries.

Batch inferencing

• Thousands or millions of independent tasks.

• Throughput matters more than latency.

• Users can wait minutes or hours for results.

• Perfect for serverless GPU launches, since models can be loaded per batch and machines shut down immediately after.

Code Engine Serverless Fleets are optimized for batch inferencing, delivering near‑perfect GPU utilization without paying for idle capacity.

The Batch Inferencing Use Case: Extracting Structure from 8,000 Recipes

Goal

Process 8,000 unstructured cooking recipes and extract temperature and duration for each preparation step—turning messy text into structured, machine‑readable data.

Model & Framework

• Model: IBM Granite model from Hugging Face

• Inference Engine: vLLM (popular for high‑throughput transformer inference)

• Runtime: Python container image running on Code Engine

• Compute: L40s / H100 GPUs, selected per fleet configuration

The Flow

1. Recipes stored in Cloud Object Storage (COS).

2. A fleet is launched with a custom container running vLLM.

3. Code Engine provisions GPU workers (e.g., L40s).

4. Workers are connected to the VPC

5. Each worker downloads the model (cached via COS if desired).

6. Workers run inference for each batch of 1,000 recipes using vLLM

7. Results (augmented JSON) are written back to COS.

8. Workers automatically deprovision—billing stops instantly.

Code Engine handles all infrastructure automation

No user needs to manage GPUs, install CUDA drivers, configure networking, or scale clusters. All GPU enablement, model loading, concurrent task scheduling, and lifecycle management happens automatically.

Why Code Engine is a Strong Fit for Batch Inferencing

1. Cost efficiency

With batch tasks, GPUs run at near 100% utilization, then scale to zero.
No idle GPU pools. No pre‑allocated clusters.

2. High throughput at massive scale

Fleets can run:

• tens of thousands of concurrent tasks

• across hundreds of CPU or GPU workers

• spanning multiple zones

This makes it possible to process millions of documents or simulate thousands of scenarios in parallel.

3. Full control over hardware

Developers select:

• CPU or GPU pools

• GPU family (L4, L40, V100, A100, H100)

• Worker profiles (e.g., number of CPUs per task, amount of memory)

Critical for performance tuning and cost optimization—capabilities traditional inferencing APIs do not provide.

4. Custom code & custom models

Unlike hosted model APIs where you get only “tokens in, tokens out,” fleets let you package any custom logic:

• Your own model weights

• Domain‑specific preprocessing

• vLLM or any other inference runtime

• Additional libraries for analytics or postprocessing

5. Enterprise‑grade isolation

All inferencing runs:

• on single‑tenant virtual machines

• inside your VPC

• with fine‑grained network controls

• and industry certifications (SOC2, C5, GDPR…)

This is essential for regulated industries—banking, insurance, pharma—frequently working with sensitive datasets.

When to Use Serverless Fleets for Inferencing

Fleets are ideal when you need:

✔ High‑throughput batch processing
✔ Custom inferencing code (vLLM, PyTorch, domain logic)
✔ Full control over compute profiles
✔ Secure, single‑tenant execution inside a VPC
✔ Zero idle cost

Try the Tutorial Yourself

The full batch inferencing tutorial—including code, task input files, and instructions—is available here:

👉 https://github.com/IBM/CodeEngine/tree/main/serverless-fleets/tutorials/inferencing

Running it requires only:

• an IBM Cloud account

• a VPC with subnets

• one COS bucket

• the sample fleet definition

Use the One-Time-Setup to deploy a sandbox environment within a few minutes.

From there, you can scale from a single task to thousands of parallel inferencing jobs—simply by changing a few parameters.

Conclusion

Batch inferencing is one of the strongest demonstrations of what IBM Cloud Code Engine Serverless Fleets can deliver:
scalable, serverless, GPU‑powered compute for real enterprise AI workloads.

For developers and client engineering teams, Fleets provide a fast, flexible way to run custom inferencing pipelines. For executives, they deliver predictable cost efficiency, enterprise‑grade security, and almost limitless scale—without infrastructure investment.

If your organization needs to process large data sets, run custom LLM workloads, or execute compute‑intensive pipelines, Code Engine Serverless Fleets offer a modern, cloud‑native way to do it—easily, securely, and at scale.