Pluggable Analytics for Microservices: Secure, Reusable, and MFE‑Ready

Abstract

Analytics isn’t a dashboard; it’s a platform capability. This article presents a production‑tested pattern to turn microservice exhaust into trusted insights—and ship them as a portable UI when needed. We walk through the end‑to‑end flow—Emit → Ship → Store → Serve → Secure → Present—with a reference stack (Kubernetes/OpenShift, Fluent Bit, Amazon OpenSearch/Elastic, a thin Insights API, OAuth/mTLS/RBAC, and a vendor‑neutral visualization layer such as Grafana). Where teams need product‑embedded views, the same queries can be wrapped as a lightweight embeddable Micro Frontend (MFE). We close by showing how the pipeline naturally extends to API metering for quotas, governance, and billing.

Disclaimer: This article describes a design pattern for building reusable analytics pipelines in microservice platforms. It is not a description of supported APIC Analytics features. None of the technologies, components, or integrations discussed here are officially supported with APIC Analytics.

Who should read this

Platform, SRE, and product engineers building multi‑tenant microservice platforms on Kubernetes/OpenShift who need auditable analytics and a UI they can plug into multiple products.

The core idea

In distributed systems, each service emits logs/metrics, but without a consistent pipeline you get fragmentation, not insight. Treat analytics as a set of contracts—collection, storage, access, security, and presentation—so it’s reusable across products and tenants.

Memorable flow:

Emit → Ship → Store → Serve → Secure → Present

Reference architecture (tool‑agnostic)

• Platform: Kubernetes / Red Hat OpenShift (ROSA/ARO/on‑prem)

• Emit/Ship: App logs + Fluent Bit (DaemonSet or sidecar)

• Store: Amazon OpenSearch (or Elastic) and/or OLAP (ClickHouse/Druid)

• Serve: Thin Insights API (adds caching, RBAC, schema normalization)

• Secure: OAuth/JWT at the edge, mTLS in cluster, secrets in a vault

• Present: Visualization layer that can be Grafana dashboards reading from OpenSearch or an embeddable MFE for product UIs

• Observability glue: Traces/APM (e.g., Instana), synthetic checks

High‑level diagram (text sketch)

+----------------------+        +------------------------+
|  Grafana Dashboards  |<------>|  Insights API (HTTPS)  |
|  (vendor‑neutral)    |  REST  |  cache, RBAC, schema   |
+-----------+----------+        +-----------+------------+
            ^                               |
            |                               | mTLS
            |                               v
            |                       +-------+--------+
            |                       |  OpenSearch    |
            |                       |  (search)      |
            |                       +-------+--------+
            |                               |
            |                               | (optional/and)
            |                               v
            |                       +-------+--------+
            |                       |   OLAP DB      |
            |                       | (aggregates)   |
            |                       +-------+--------+
            |
            | (optional embedding)
+-----------+-----------+   props   
| Host Dashboard(s)     |<----------  MFE (embeddable panel)
| Product A / Product B |
+-----------------------+

Edge: API gateway validates OAuth/JWT, sets tenant headers, applies quotas; forwards with mTLS to Insights API.
Secrets: in a vault; never stored in plaintext ConfigMaps.

The pattern, stage by stage

1) Emit (services)

Emit structured events as JSON for schema evolution and easy parsing.

{
  "ts": "2025-09-02T08:15:30Z",
  "service": "orders",
  "env": "prod",
  "tenant": "acme",
  "event": "http_request",
  "route": "POST /v1/orders",
  "status": 201,
  "latency_ms": 142,
  "user": "u_123",
  "trace_id": "2b7f...",
  "region": "us-east-1"
}

Tips

• Include tenant, env, trace_id, and region.

• Keep payloads small; prefer IDs over blobs.

2) Ship (forwarder/collector)

Start with Fluent Bit as a DaemonSet for zero app changes; use sidecars where you need richer per‑service context.

Minimal Fluent Bit filter (Kubernetes metadata):

[FILTER]
    Name            kubernetes
    Match           kube.*
    Kube_Tag_Prefix kube.var.log.containers.
    Keep_Log        On
    Merge_Log       On
    Annotations     On
    Labels          On

3) Store (datastore)

• Search‑first (OpenSearch/Elastic): ad‑hoc exploration, troubleshooting, dashboards.

• OLAP (ClickHouse/Druid): efficient aggregates for metering and trends.

• Use hot/warm/cold tiering for retention vs. cost.

4) Serve (thin Insights API)

Create a thin API that:

• Normalizes schemas and names

• Adds caching and RBAC

• Provides stable contracts to the UI (hides datastore changes)

OpenAPI excerpt:

openapi: 3.0.3
info: { title: Insights API, version: 1.0.0 }
paths:
  /capabilities:
    get: { responses: { '200': { description: OK } } }
  /insights/{metric}:
    get:
      parameters:
        - { name: metric, in: path, required: true, schema: { type: string } }
        - { name: from, in: query, schema: { type: string, format: date-time } }
        - { name: to, in: query, schema: { type: string, format: date-time } }
        - { name: groupBy, in: query, schema: { type: string } }
      responses: { '200': { description: OK } }
  /health: { get: { responses: { '200': { description: OK } } } }
  /version: { get: { responses: { '200': { description: OK } } } }

5) Secure (edge policy, tenancy, mTLS)

• At the edge: OAuth/JWT validation, tenant extraction, quotas/rate limits.

• In cluster: mTLS between edge and Insights API, service‑to‑service ACLs.

• Secrets: store in a vault; mount via CSI; avoid plaintext in ConfigMaps.

Policy sketch (gateway‑agnostic):

<validate-credentials source="request.headers.Authorization" type="oauth"/>
<set-variable name="var.tenant" value="request.headers.x-tenant"/>
<rate-limit key="{$var.tenant}" threshold="1000/m"/>
<set-header name="x-tenant" value="{$var.tenant}"/>
<forward mTLS="true" />

RBAC pointers

• Shipper: write‑only to indices/streams

• Insights API: read‑only to analytical views

• No wildcard secret access; namespace‑scoped roles

6) Present (Visualization: Grafana or embeddable MFE)

The visualization layer should be vendor‑neutral: start with Grafana dashboards reading from OpenSearch (or your store of choice). For products that need analytics embedded within their UI, wrap the same queries as a lightweight embeddable panel/MFE. This keeps the data path identical while letting each product choose between standalone dashboarding and in‑product views.

Optional MFE host embed (React):

import React from "react";
const AnalyticsMFE = React.lazy(() => import("analytics_mfe/App"));

export default function AnalyticsCard({ orgId, baseUrl }) {
  return (
    <React.Suspense fallback={<div>Loading analytics…</div>}>
      <AnalyticsMFE orgId={orgId} baseUrl={baseUrl} />
    </React.Suspense>
  );
}

Contracted props: orgId, baseUrl, optional capabilities, timeRange, featureFlags.

UX details

• Graceful degraded states (“data may be ~60s old”)

• Empty/permission states

• Link‑back for “open in Grafana” or “open full analytics”

Extension: API Metering on the same pipeline

Once you’re collecting per‑tenant events, metering is a small step:

• Insights API exposes per‑tenant call counts/aggregates

• Metering service pushes counts into billing/entitlement systems

• Edge enforces thresholds: warn at 80%, throttle or block at 100%

• Auditability: queries and decisions are reproducible from stored events

Result: the same analytics spine powers both observability (engineers) and governance (business).

Design choices at a glance

Security & Tenancy checklist

• mTLS between gateway ↔ Insights API; rotate certs

• Per‑tenant isolation at the edge (JWT/headers → policy)

• Least‑privilege RBAC; no wildcard secret access

• Secrets in a vault (not in ConfigMaps); CSI mounts

• Staging ↔ prod parity; synthetic checks hit /health

• Rate limits/quotas by tenant; observability on rejects

“Before/After” mini case (anonymized)

• Context: Multi‑tenant SaaS (~60 services) on Kubernetes across 2 regions

• Before: Per‑team dashboards, no shared metering, fragmented ingestion

• After: Shared pipeline; OpenSearch for search, ClickHouse for aggregates; Grafana for visualization; optional MFE for embedding

• Observed results (typical ranges):

  • MTTR ↓ 30–50% (searchable incident timelines)

  • Time to add analytics to a product ↓ >70% (MFE option)

  • Cost for 90‑day retention ↓ 25–40% (hot/warm/cold)

  • Metering SLA: daily usage exports with <5 min drift

Adoption guide 

1. Standardize event schema (JSON keys for tenant/env/trace).

2. Deploy Fluent Bit as DaemonSet; enable K8s metadata filter.

3. Stand up OpenSearch (hot/warm/cold tiers) and/or ClickHouse.

4. Implement the Insights API (endpoints above); add a cache (e.g., Redis) and RBAC.

5. Harden edge: OAuth/JWT in your gateway; set x-tenant, forward with mTLS.

6. Manage secrets in a vault; mount via CSI; rotate on schedule.

7. Visualize with Grafana; for embedded needs, package an MFE using the same API.

8. Extend to metering: nightly aggregates + real‑time thresholds at the edge.

9. Prove it: Synthetics for /health; latency and error SLOs; drill‑down dashboards.

Validation checklist (pre‑GA)

• 99th percentile latency for /insights under load meets SLO

• Tenancy enforced (negative tests with cross‑tenant tokens)

• Secrets rotation doesn’t interrupt ingestion or reads

• Backfill job validated (recompute aggregates from raw logs)

• Drift tests between OpenSearch vs OLAP aggregates

• Grafana datasource permissions + dashboards tested

• MFE error/isolation tests on host dashboards

• Runbook: on‑call, playbooks, quotas, “what to page on”

Troubleshooting quick wins

• Gaps in data? Check Fluent Bit backpressure, index lifecycle policies

• Slow queries? Add rollups/materialized views; cache hot queries in API

• Tenant bleed‑through? Verify edge policy and query filters; add tests

• MFE blank states? Verify /capabilities and CORS; show actionable empty state

• Grafana auth issues? Align org/team/tenant mapping with Insights API RBAC

Appendix: tiny, real‑world snippets

Fluent Bit output to OpenSearch

[OUTPUT]
    Name            es
    Match           kube.*
    Host            ${OPENSEARCH_HOST}
    Port            443
    TLS             On
    AWS_Auth        On
    AWS_Region      us-east-1
    Logstash_Format On
    Logstash_Prefix logs

Kubernetes NetworkPolicy (only Insights API can read store)

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-insights-to-store, namespace: analytics }
spec:
  podSelector: { matchLabels: { app: opensearch } }
  ingress:
    - from:
        - podSelector: { matchLabels: { app: insights-api } }
      ports: [{ port: 9200, protocol: TCP }]

Embeddable shell for the MFE (optional)

import { Card, CardHeader, CardFooter } from "@carbon/react";
export function AnalyticsShell({ children }) {
  return (
    <Card>
      <CardHeader title="Analytics" subtitle="Live insights"/>
      {children}
      <CardFooter>Data may be up to 60s delayed</CardFooter>
    </Card>
  );
}