There's no shortage of enterprise AI reference architectures floating around right now. Every vendor, consultancy, and analyst firm has one. But after 18+ years in enterprise technology - including stints leading cloud architecture at Mastercard, Citibank, and now deploying watsonx on a major programme in the Middle East - I've noticed a pattern: the architectures that look best on slides are rarely the ones that survive contact with production.

I want to share five architecture patterns I've seen succeed at scale, and a few that consistently don't - with the hope of starting a useful conversation here.

Patterns that scale

1. The modular AI platform (not the monolith)

The single biggest architectural decision you'll make is whether to treat AI as a monolithic capability or a modular platform. The watsonx approach - splitting across watsonx.ai (model development), watsonx.data (data governance), and watsonx.governance (responsible AI) - reflects what actually works in the field. Each layer can evolve independently. You can swap foundation models, change your data lakehouse topology, or tighten governance controls without rebuilding everything.

The key insight: AI platforms that scale treat orchestration, data, and governance as separate concerns with clean interfaces between them. The ones that don't try to bundle everything into a single service.

2. Right-sized models over maximum-parameter models

There's a persistent instinct among enterprise architects to reach for the biggest available model. In practice, the organisations getting the best results are doing the opposite. Granite 4.0's approach is instructive here - a hybrid Mamba-2/transformer architecture with Mixture-of-Experts that delivers over 70% lower memory requirements and 2x faster inference than comparably-sized pure transformer models. The Small variant (32B total, 9B active parameters) is outperforming models twice its size on agentic tasks like function calling and instruction following.

The architectural principle: match model size and architecture to the task. A 1B-parameter model running on affordable hardware at the edge can outperform a 70B model stuck behind API rate limits for many enterprise use cases. Design for cost-per-inference, not benchmark bragging rights.

3. The data mesh, not the data dump

I've lost count of how many AI programmes I've seen stall because the team assumed existing data infrastructure was "good enough." Enterprise data architectures built around traditional ETL pipelines and warehouses create friction for agent deployment. The data is structured for humans to query, not for agents to reason over.

The pattern that works: invest early in making your data agent-ready. This means contextualising enterprise data through knowledge graphs and semantic indexing - making information discoverable without requiring the agent to know which database to query. The open data lakehouse architecture in watsonx.data is designed for exactly this, bridging the cost efficiency of object storage with the performance of query engines.

4. Governance as architecture, not afterthought

Research from BCG suggests that nearly 70% of AI failures stem from people and process issues, with only 20% from technology. Yet most architecture diagrams put governance in a dotted-line box off to the side.

The scaling pattern: governance is a first-class architectural component with its own APIs, observability stack, and policy enforcement engine. It's not a review board that meets monthly. watsonx.governance provides the tooling, but the real work is defining your policy model - what can an agent do autonomously, what needs human approval, how do you audit reasoning chains, and how do you detect drift.

5. Hybrid deployment by design

Most enterprise AI architectures assume cloud-first. In the real world - especially in regulated industries and across the Middle East - data residency, latency, and sovereignty requirements mean you need AI workloads running across cloud, on-premises, and edge. Designing for hybrid from day one is far cheaper than retrofitting it later.

This is one area where the watsonx platform's flexibility genuinely differentiates - SaaS on IBM Cloud or AWS, on-premises via Red Hat OpenShift, or hybrid combinations. But the architectural discipline is the same regardless of platform: containerise everything, abstract your model serving layer, and never hardwire assumptions about where compute lives.

Patterns that don't scale

• "One smart agent" architectures. A single monolithic agent handling complex multi-domain workflows collapses under pressure. Intelligence without structure doesn't scale. You need role separation - planners, executors, validators, domain specialists - just as you would in an organisation.
• Bolting AI onto broken processes. AI is an amplifier. If your underlying process is inefficient, AI makes it efficiently inefficient. The organisations that get stuck are the ones trying to automate existing workflows instead of redesigning them for an agentic paradigm.
• Ignoring the orchestration layer. Models are commoditising rapidly. The competitive advantage lives in orchestration - how you design agent interactions, escalation paths, failure handling, and context sharing across systems.

Over to the community:

• What architecture patterns have you seen succeed (or fail) at scale with watsonx?
• How are you approaching the right-sizing question - especially balancing Granite models against third-party options?
• For those working in regulated industries or regions with data sovereignty requirements: what hybrid deployment patterns are working for you?

Keen to hear from others in the trenches.

- Paul Foran Senior Solution Architect, IBM | Abu Dhabi