The Growing Complexity of Modern AI Inference Infrastructure
The focus in enterprise AI has now shifted from training models to inference – delivering answers and outcomes that drive business results. But configuring inferencing systems is a huge challenge, regardless of whether they’re running on premises, in the cloud, or in hybrid configurations. I the optimal hardware and configuration parameters necessary for scalable and robust systems is an engineering problem requiring expert-level knowledge. Though such challenges existed in the past, several aspects of modern inference workloads make them even more difficult today:
-
Hardware diversity: Modern GPUs differ vastly in their compute and storage capabilities and therefore require a careful analysis of which GPU type(s) are optimal for which inference workloads. Beyond GPU memory, inference systems increasingly rely on multi-tier storage architectures to manage runtime state such as KV caches, introducing additional design and operational decisions.
-
Workload diversity: Inference workloads not only consist of user-facing chats, but also enterprise-grade agentic workflows, offline batch inferences, long-document-analysis heavy workloads, etc. These workloads are served and mixed on a common platform, therefore requiring careful attention to ensure reliable service across workloads.
-
Complex Interactions: Performance of an inference request depends not only on the request itself, but also on the requests that came before it, which affect KV cache state, and on the requests currently in the pipeline through batching and scheduling. As a result, compute and I/O demands of a request change dynamically, making performance difficult to predict without detailed operational analysis.
streamline the procurement process, we designed and developed LISA: the LLM Infrastructure Sizing Agent. LISA is an expert LLM infrastructure advisor that enterprise customers can chat with in natural language to explore and quickly find their optimal inference stack. Users simply specify their current or desired environment and receive professional infrastructure and performance analysis within minutes, grounded in state-of-the-art simulation results.
ur solution dramatically lowers the barrier to entry. Users can specify as little as their desired inference model, GPUs, and workload, and the agent translates those high-level goals into detailed simulation configurations automatically. What traditionally required infrastructure expertise and manual parameter tuning becomes a natural language conversation.
Although other LLM simulators exist for similar purposes, they typically require users to understand infrastructure concepts and manipulate configuration files directly. Our solution is the interface that transforms simulation setup, experimentation, and analysis into an interactive conversation. , users can evaluate multiple deployment options and understand the tradeoffs between performance, scalability, and cost. Whether you're investing in your first inference system or optimizing an existing one, LISA helps identify the ideal inference platformfor your needs, giving you confidence in your deployment
LISA is Your Expert Infrastructure Advisor
LISA is designed to support customers of all backgrounds. With over 50 simulation parameters available, users can see how an environment performs under their specific AI use cases. The agent also has presets built in to adjust the conversation to any technical level. This ranges on input from
LISA is an intelligent interface that translates a wide range of user requests into realistic simulations. This translation layer is the key innovation: the agent bridges the gap between user intent and simulator inputs, converting ambiguous business requirements into structured, repeatable experiments. Users can focus on describing the high-level details rather than tweaking nuanced variables like maximum batched tokens or eviction threshold for GPU memory. The work shifts from configuring exact numbers to brainstorming scenarios and solutions.
A Sample Infrastructure Planning Workflow
Let’s walk through a realistic scenario of a user looking to purchase the right infrastructure for their enterprise HR assistant.
A conversation might begin with a prompt such as:
“Compare the NVIDIA/Llama-3.1-70B-Instruct-FP8 model on 4 H100 SXM5 GPUs with 128 GB of CPU memory, with and without 50 TB of Distributed File System storage capacity, for a heavily used HR assistant.”
’s request and generating up to four configurations for simulation. Notice that the user never specifies simulator parameters, cache policies, scheduling settings, or workload distributions. The agent infers and constructs those details using its internal knowledge of the simulator and the user’s request. In this example, the agent understands there are two scenarios: both use the same Hugging Face model, GPU type and count, CPU memory, and workload. The only difference in one scenario is the inclusion of the recommended starting amount of 50 TB of Distributed File System (DFS) storage, defaulted to IBM Storage Scale.
Next, the agent must build a realistic workload for a “heavily used HR assistant”. In this case, the agent understands it is a Retrieval-Augmented Generation (RAG) workload since HR chatbots typically ground their responses by retrievingrelevant internal documents. An HR assistant is one of seven preset RAG workloads the agent offers, allowing users to run popular scenarios right out of the box. If none of the presets align exactly, users can override them or build their own.The RAG workload presets include:
The smart configuration builder was designed to be fast, deterministic, and realistic. More importantly, it allows users to express what they want to learn rather than how a simulation should be configured, shifting the interaction from system setup to infrastructure exploration. We wanted users to interact with the agent in a scientific manner: testing hypotheses and configurations iteratively by modifying one or two variables and observing the impact on performance outcomes, all within a single chat. However, designing an agentic workflow that gives LLMs the freedom to create configurations while including deterministic guardrails for repeatable and realistic simulations posed a challenge. The smart configuration builder satisfies these goals and works seamlessly with the end-to-end system.
After building the configurations, the agent passes them to the simulator to run in parallel. The agent then analyzes the results and generates a comprehensive summary including recommendations and analysis on configuration, performance, capacity, power, and cost. Energyper one million (1M) tokens are two critical metrics added to show users the real business impact. Energy is measured by summing up static baseline hardware energy and runtime energy usage from computation or data Cost per 1M tokens is calculated from estimated energy costs calculations plus amortized hardware across the relevant storage tiers.
Performance results show that the GPU/CPU/DFS configuration s a median time to first token (TTFT) of 375.2 milliseconds, compared with 560.2 milliseconds for GPU/CPU. Furthermore, the GPU/CPU/DFS configuration also demonstrateda 75.5% cache hit rate, meaning more requests could reuse previously stored KV-cache data instead of recomputing it. This reduced GPU computation and contributed to 30% cost savings and higher throughput of 1.32 requests per second, compared with a 43.3% cache hit rate and 0.95 requests per second for GPU/CPU.
The evidence shows that the GPU/CPU/DFS configuration is the best so far, but a user might not be without it further. The conversation might continue with another prompt like,
One of the biggest time savers with the agent is making use of the chat history to easily modify or compare previous simulations. Here, the agent intelligently increases the number of documents from 13 to 17 and doubles the system prompt to 2048 from 1024 tokens, leaving everything else from the GPU/CPU/DFS configuration the same. Results show similar performance metrics with a slightly lower cost and power consumption compared to the previous simulation. Overall, these results suggest that the 50 TB IBM Storage Scale configuration remains robust under the increased workload and can provide meaningful performance and cost benefits for this scenario.
RAG is a popular workload; however, assume a wants to support an agent as well. Fortunately, sers can simulate agentic traces with LISA in JSON or OpenTelemetry formats from different workloads such as SWE-Bench on DeepSeek, or coding harnesses like Claude-code. The tool also has an option to build a completely customized synthetic agentic trace based on any production workload properties. Simulation of some heavy agentic traces like SWE-Bench on DeepSeek may take multiple minutes, but users can specify a time limit for the simulation. That said, expect a potential reduction in validity of the results if not all requests are tried. Thus, a final prompt might ask,
ed an exceptionally fast median TTFT of 55.4 milliseconds and showed the performance benefits robust KV cache offloading, achieving a 98.2% cache hit rate and slashing costs by nearly 50% compared to the RAG simulation results.
Overall, the strength of the agent comes through its ability to turn iterative infrastructure investigation into an interactive, multiturn conversation. The agent can synthesize many simulation results, providing comparative analysis with enterprise-level recommendations weighing performance, cost, and infrastructure tradeoffs. LISA empowers customers to take control of their inference infrastructure by evaluating realistic deployment options and comparing the experimental results for themselves.
1. Prompt agent with desired environment and workload. 2. Divide prompt into parallelsimulations, depending on how many are being compared by the user. 3. Build custom workload for each simulation. 4. Return each simulation result to agent. 5. Agent interprets and explains results, then makes recommendations and comparisons.
Under the hood, LISA’s expert analysis is grounded by an advanced distributed AI and LLM workflow simulator created by IBM Research Zurich. The simulation engine is an infrastructure testing breakthrough, eliminating the need to purchaseexpensive hardware or configure complex infrastructure before choosing an environment. While the simulator provides the analytical foundation, the agent's role is to make those capabilities accessible to users through conversation rather than direct simulator interaction.
The simulator effectively captures the serving an inference request in a distributed setting. This includes simulating complex interactions among:
The simulation is comprehensive and deterministic, providing raw performance results that translate into crucial metrics like TTFT, inter-token latency (ITL), and time per output token (TPOT). Beyond overall performance, the simulator provides detailed measurements of events across the whole distributed infrastructure which are integral to measuring infrastructure at a per-tier level. Because the simulation is deterministic, the same input produces the same output and analysis, thus offering grounded and repeatable comparisons.
Looking Toward Automated Infrastructure Optimization
The agent excels at helping customers identify their optimal inference infrastructure by analyzing simulation results and comparing infrastructure configurations. The next step is to reverse that workflow: given a set of performance targets, business constraints, or workload requirements, what infrastructure configuration should be deployed to meet those targets?
Rather than manually evaluating countless infrastructure options, our vision is to offer users a single-prompt approach to identifying their optimal infrastructure. The agent would loop over deployment configurations, intelligently adjusting parameters until the user’s goals are reached. By combining simulation-driven analysis with intelligent recommendations, the agent could significantly accelerate infrastructure planning, helping customers identify the right inference engine configuration, routing strategy, and hardware architecture for their business.