Overview

In Part 1, we explored why organizations are investing in Enterprise Knowledge Bases and what we learned from customer engagements. In this article, we shift from business outcomes to implementation, walking through the architecture we used to connect enterprise knowledge with watsonx Orchestrate.

The stack:

Although these components can be deployed independently, separating ingestion, retrieval, and orchestration provides greater flexibility. New enterprise content sources can be added without changing the retrieval layer, and improvements to retrieval quality can be made independently of the conversational experience.

Reference Architecture:

Data Ingestion Pipeline

The Challenge

Enterprise data sources each have unique extraction requirements:

Unified Ingestion Design

The ingestion pipeline normalizes all sources to Markdown before downstream processing. This decouples the extraction logic from the chunking and embedding stages.

Core design principles:

1. One adapter per source type — Slack adapter, GitHub adapter, document adapter
2. Normalize to Markdown — common output format across all sources
3. Preserve metadata at extraction time — source, author, timestamp, URL
4. Support incremental updates — only process new or changed content

Example Adapter Flow:

Slack Adapter
├─ Authenticate with Slack API
├─ Fetch messages with pagination (cursor-based)
├─ Resolve user IDs to display names
├─ Fetch thread replies and attach to parent message
└─ Emit: { text: "...", metadata: { channel, author, timestamp, url } }

GitHub Adapter
├─ Fetch issues via REST API (with label/milestone filters)
├─ Include issue comments inline
├─ Format code snippets with language fencing
└─ Emit: { text: "...", metadata: { repo, issue_id, author, labels, url } }

Document Adapter
├─ Detect file type (PDF, DOCX, PPTX, MD)
├─ Extract text and structural elements
├─ Preserve heading hierarchy and lists
└─ Emit: { text: "...", metadata: { filename, modified_date, section_path } }

Intelligent Chunking

Why Chunking Decisions Matter

LLMs operate within context windows (typically 4K–32K tokens). You cannot pass an entire document as context. Chunking breaks content into pieces that are:

• Small enough to fit in a context window
• Large enough to carry self-contained meaning
• Structured so semantically related content stays together

Fixed-size chunking (split every N words) is simple but breaks mid-sentence, mid-argument, or mid-code-block. Heading-based chunking splits at natural document boundaries — section breaks — which preserves semantic coherence.

Heading-Based Chunking Configuration

chunking:
  strategy: heading_based
  max_tokens: 800
  overlap_tokens: 80     # 10% overlap for boundary continuity

Overlap ensures that content near a chunk boundary appears in both the preceding and following chunk. This prevents the retrieval system from missing a relevant passage because it happened to sit at a split point.

Chunking Best Practices

watsonx.ai Embeddings

Embeddings are dense vector representations of text. Semantically similar content produces similar vectors — enabling the similarity search that powers the retrieval step.

Choosing an Embedding Model

The slate-125m-english-rtrvr-v2 model is optimized for retrieval tasks, trained on diverse enterprise content, and delivers sub-100ms latency with batch processing support.

Configuration

Point the client at your watsonx.ai project and select the embedding model:

watsonx_config = {
    "api_key": "your-watsonx-api-key",
    "project_id": "your-project-id",
    "url": "https://us-south.ml.cloud.ibm.com",
    "model_id": "ibm/slate-125m-english-rtrvr-v2"
}

Process chunks in batches of 32 — this keeps requests within API limits and maximises throughput. Each batch returns a list of 768-dimensional vectors that are stored alongside their source chunks.

Building the Vector Retrieval Layer

The retrieval layer is responsible for four things: storing embeddings alongside their source text and metadata, performing fast similarity search against a query vector, supporting metadata filtering to narrow results before or after the vector search, and scaling as the knowledge base grows. Any vector-capable store that satisfies these requirements can serve this role.

Implementation Example: Milvus with watsonx.data

watsonx.data includes a managed Milvus service — enterprise-grade vector storage with built-in security, native integration with watsonx.ai, and scalability to billions of vectors.

Connection Configuration

milvus_config = {
    "host": "your-instance.wxd.lakehouse.appdomain.cloud",
    "port": 443,
    "user": "ibmlhapikey_your-username",
    "password": "your-ibm-cloud-api-key",
    "secure": True,          # TLS required for watsonx.data
    "database": "default",
    "embedding_dim": 768
}

Collection Schema Design

Each collection maps to a knowledge domain (e.g., slack_engineering_kb, github_issues_kb). Each record stores five fields:

Index Configuration

The IVF_FLAT index type provides a good balance of search speed and accuracy for most enterprise deployments:

index_params = {
    "index_type": "IVF_FLAT",
    "metric_type": "L2",       # Use COSINE for normalized embeddings
    "params": {"nlist": 128}   # Number of cluster centroids; tune to collection size
}

L2 vs COSINE: L2 (Euclidean distance) works well when vectors are not normalized. If your embedding model outputs unit-normalized vectors, COSINE similarity is equivalent and often preferred. Verify the normalization behaviour of your chosen model in the watsonx.ai documentation before selecting a metric type.

Multi-Collection Setup

Separate collections by knowledge domain rather than storing everything in one. This enables targeted search, independent update cycles, and cleaner metadata filtering. Typical collection names: slack_general_kb, slack_engineering_kb, github_issues_kb, documentation_kb.

The metadata JSON field travels with every chunk through ingestion and is returned at query time — it is what the agent uses to build citations (channel, author, timestamp, url).

Connecting watsonx Orchestrate to the Retrieval Layer

This is the integration layer where the knowledge pipeline becomes a live agent capability. watsonx Orchestrate uses the Agent Development Kit (ADK) to register custom tools that the agent can call during a conversation.

How the Query Pipeline Works

The key architectural contract here is simple: question → embedding → retrieval backend → context + sources. The agent doesn't need to know how the retrieval backend implements similarity search — that detail is isolated inside the ADK tool.

The query follows a typical retrieval-augmented generation (RAG) flow — from user input through retrieval, grounding, and response generation.

Step 1: Set Up the Vector Store Connection in WxO

Before registering tools, configure both service connections as named application connections in watsonx Orchestrate. This keeps credentials out of tool code and lets the platform manage rotation.

# Vector store (this example uses Milvus via watsonx.data)
orchestrate connections configure --app-id vector_store --env draft --type team --kind basic
orchestrate connections set-credentials --app-id vector_store --env draft \
  --username "ibmlhapikey" --password "your-ibm-cloud-api-key"

# watsonx.ai
orchestrate connections configure --app-id watsonx_ai --env draft --type team --kind api_key
orchestrate connections set-credentials --app-id watsonx_ai --env draft \
  --api-key "your-watsonx-ai-api-key"

Step 2: Implement the ADK Tool

The ADK @tool decorator registers a Python function as a callable tool within watsonx Orchestrate. The structure is straightforward — three steps inside the function body.

The following example uses Milvus as the retrieval backend. The ADK tool interface remains the same if the underlying retrieval technology changes; backend-specific connection and search logic can be isolated within the retrieval implementation.

from ibm_watsonx_orchestrate.agent_builder.tools import tool
from ibm_watsonx_orchestrate.agent_builder.connections import get_application_connection_credentials

@tool(
    name="query_knowledge_base",
    description="Search the enterprise knowledge base for relevant information",
    connection_type="vector_store"
)
def query_knowledge_base(question: str, collection_name: str, top_k: int = 5) -> dict:
    # Pull managed credentials from WxO connection store
    milvus_creds = get_application_connection_credentials("vector_store")
    watsonx_creds = get_application_connection_credentials("watsonx_ai")

    # Embed the question and search the vector store
    query_embedding = WatsonxEmbeddingService(watsonx_creds).embed([question])[0]
    results = connect_to_milvus(milvus_creds).search(
        collection_name=collection_name, data=[query_embedding], limit=top_k,
        search_params={"nprobe": 10}, output_fields=["text", "heading_path", "metadata"]
    )

    # Return context and citations for the LLM
    return {
        "context": [r["text"] for r in results],
        "sources": [r["metadata"] for r in results]
    }

A few design decisions worth calling out:

• Managed credentials — get_application_connection_credentials pulls secrets from WxO's connection store, so no API keys live in tool code.
• output_fields — explicitly request metadata so the agent has everything it needs to build citations.
• nprobe — controls the accuracy/latency tradeoff for IVF_FLAT search. Start at 10 and tune based on observed latency.

Step 3: Register the Tool with watsonx Orchestrate

orchestrate tools import --kind python --file query_knowledge_base.py --app-id vector_store

Step 4: Define the Agent

Wire the tool into an agent with a system prompt that instructs citation behaviour and honest handling of unknowns:

agent:
  name: "Enterprise Knowledge Assistant"
  tools:
    - query_knowledge_base
  system_prompt: |
    You are an enterprise knowledge assistant.
    Use the query_knowledge_base tool to search for relevant information.
    Always cite your sources — include channel, author, and timestamp.
    If the knowledge base does not contain a confident answer, say so clearly.
    Do not speculate beyond what the retrieved content supports.

The system prompt is where you control citation behaviour and tone. Keep it specific — vague prompts produce vague citations.

Step 5: Deploy the Agent

orchestrate agents import --file enterprise_knowledge_rag.yaml
orchestrate agents deploy --name enterprise_knowledge_rag

Retrieval Tuning

Initial deployment is the starting point, not the end state. Retrieval quality is influenced by three things: the number of results retrieved (top_k), metadata filtering (narrowing by source type or date before or after the vector search), and backend-specific search parameters that control the accuracy/latency tradeoff. Start with top_k=5 and no filters, then adjust based on the signals below.

In our Milvus implementation using IVF_FLAT, nprobe is the key backend-specific parameter — it controls how many index clusters are searched during a query. Higher values improve recall at the cost of latency. Start at nprobe=10 and tune from there.

Tuning Guidance

Production Readiness

Security & Governance

• Use IBM Cloud IAM with least-privilege service accounts for each component
• Store all credentials in watsonx Orchestrate's managed connections — never in code or config files
• Use private endpoints for the retrieval backend where available, and require encrypted connections for data in transit. In this implementation, the Milvus service is accessed through watsonx.data with TLS required.
• Rotate API keys on a defined schedule and audit access logs quarterly

Monitoring Metrics

IBM Cloud provides built-in dashboards for infrastructure metrics. Track business-layer metrics through watsonx Orchestrate's built-in conversation logs.

Engineering Lessons

After implementing this architecture across multiple deployments, several technical patterns consistently emerged.

Separate ingestion from retrieval. Keeping the ingestion pipeline independent from the retrieval layer meant we could add new data sources — a new Slack workspace, a GitHub org, a documentation site — without touching the agent or the search logic. Design the boundary early.

Build metadata at ingestion time. Adding source, author, timestamp, and tags during extraction is straightforward. Trying to reconstruct that context from raw text after the fact is expensive and often incomplete. Metadata quality compounds over time — invest in it from day one.

Design for incremental updates from the start. Enterprise knowledge changes continuously. Slack channels generate new messages daily; documentation gets updated; issues are resolved. Treating ingestion as an ongoing pipeline rather than a one-time import changes how you architect the system. Plan for delta processing, not just bulk loads.

Optimize retrieval before changing models. When answer quality fell short, our first instinct was to try a different LLM. In practice, tuning top_k, tightening metadata filters, or improving chunk boundaries consistently had a larger impact than swapping models. Better retrieval feeds the model better context — that matters more than which model processes it.

Final Thoughts

Building an Enterprise Knowledge Base involves much more than connecting an LLM to a vector database. Successful systems depend on high-quality ingestion, thoughtful knowledge organization, well-designed retrieval, and continuous operational improvement. watsonx Orchestrate and watsonx.ai provide the orchestration and AI building blocks, while the retrieval layer can be implemented using the vector-capable backend that best fits the deployment requirements — in this implementation, Milvus through watsonx.data. Long-term success comes from treating enterprise knowledge as a strategic asset rather than simply another AI project.

Additional Resources