Authored by @Yanwei Li, @Ying Mo, @Paras Kampasi

Introduction

Modern observability relies on understanding both what your application is doing and where it's running. When troubleshooting performance issues, you need to connect application behavior to infrastructure context. These two worlds often operate independently.

Imagine investigating a slow API request in your microservices application. Your traces show the payment service took 5 seconds to respond, but you need to understand why. Is it a code issue? Resource exhaustion? A failing pod? Network problems? Without the ability to navigate from your trace to the underlying infrastructure, you're left switching between multiple tools, trying to piece together the story.

Infrastructure correlation solves this by creating a direct link between your application telemetry and the underlying infrastructure. In Instana, this means you can click from a slow trace span directly to the Kubernetes pod running that service, see its resource utilization, check recent events, and understand the complete context. All in one place.

This blog shows you how to set up infrastructure correlation for applications monitored by Instana with OpenTelemetry.

The Easy Path: Instana Distribution for OpenTelemetry (IDOT)

If you're using IDOT (Instana Distribution for OpenTelemetry), infrastructure correlation is automatically configured with optimized default settings. IDOT is designed to work seamlessly with the Instana agent, providing out-of-the-box correlation with minimal configuration.

This is the recommended approach for Instana customers. Learn more about IDOT in the Instana documentation.

The rest of this blog is for users running OpenTelemetry Collectors who may want to customize correlation configuration manually.

How Infrastructure Correlation Works

In Instana, the infrastructure correlation with OpenTelemetry creates a navigable link between your application telemetry and infrastructure entities through OpenTelemetry entities. Here's the key insight: OpenTelemetry entities are created from OpenTelemetry metrics, not traces.

The correlation flow works like this:

1. Your application emits OpenTelemetry metrics with resource attributes (like service.name, k8s.pod.uid)
2. Instana receives these metrics and creates an OpenTelemetry entity
3. Your application emits OpenTelemetry traces with matching resource attributes
4. Instana links the trace spans to the OpenTelemetry entity
5. The entity connects to infrastructure (pods, hosts, containers, processes) using resource attributes
6. Complete chain: Span -> OpenTelemetry Entity -> Pod/Host/Container/Process Entity -> Other Infrastructure

Because traces and metrics share the same resource attributes, they automatically correlate to the same entity, enabling you to navigate from application behavior to infrastructure context.

Required Resource Attributes

For correlation to work, your telemetry must include these resource attributes:

Common Attributes

service.name

• Identifies your service
• Must be unique across your organization
• Examples: payment-service, user-authentication, order-processor

service.instance.id

• Identifies a specific instance of your service
• Must be unique for each running instance
• In Kubernetes: Typically set to k8s.pod.uid
• In VMs: Use hostname or generate a unique ID

Environment-Specific Attributes

k8s.pod.uid (for Kubernetes)

• Critical for linking to Kubernetes pods
• Automatically extracted by the k8sattributes processor in OpenTelemetry Collector

container.id (for Containerd, CRI-O, Docker, OTel Kubernetes Container, or Podman)

• Links to the Containerd, CRI-O, Docker, OTel Kubernetes Container, or Podman entity
• Automatically extracted by the k8sattributes processor in OpenTelemetry Collector

host.id (for VMs and Bare Metal)

• Links to the host entity
• Should match the host ID that Instana agent reports

process.pid (for Process)

• Links to the process entity
• The OpenTelemetry SDK usually sets the process.pid resource attribute automatically.

Configuration Options

You have two options depending on whether your application emits metrics:

Option A: Your Application Emits Both OpenTelemetry Traces and Metrics

This is the most robust approach. If your application already emits both traces and metrics (most modern OpenTelemetry SDKs do this by default), you just need to ensure the required resource attributes are present.

For Non-Kubernetes Environments

Set resource attributes using environment variables:

export OTEL_RESOURCE_ATTRIBUTES="service.name=payment-service,service.instance.id=payment-service-01,host.id=your-host-id"

Or configure them in your OpenTelemetry Collector:

processors:
  batch: {}

  # Add required resource attributes
  resource:
    attributes:
    - key: service.name
      value: payment-service
      action: upsert
    - key: service.instance.id
      value: payment-service-01
      action: upsert
    - key: host.id
      value: your-host-id
      action: upsert

service:
  pipelines:
    traces:
      receivers: [otlp] # Receive traces from application
      processors: [resource, batch] # Add attributes
      exporters: [otlp] # Send to Instana
    
    metrics:
      receivers: [otlp] # Receive metrics from application
      processors: [resource, batch] # Apply same processors as traces
      exporters: [otlp] # Send to Instana

Key points:

• Both traces and metrics pipelines use the same resource processor to ensure consistent attributes.

For Kubernetes Environments

Use the k8sattributes processor to automatically extract Kubernetes metadata:

processors:
  batch: {}

  # Extract Kubernetes metadata
  k8sattributes:
    extract:
      metadata:
      - k8s.pod.uid  # Critical for pod correlation
      - k8s.pod.name
      - container.id # Critical for container correlation
      - ...
    passthrough: false
    pod_association:
    - sources:
      - from: resource_attribute
        name: k8s.pod.ip
    - sources:
      - from: resource_attribute
        name: k8s.pod.uid
    - sources:
      - from: connection
  
  # Set service.instance.id to k8s.pod.uid
  resource:
    attributes:
    - action: insert
      from_attribute: k8s.pod.uid
      key: service.instance.id

service:
  pipelines:
    traces:
      receivers: [otlp] # Receive traces from application
      processors: [k8sattributes, resource, batch] # Enrich with K8s metadata
      exporters: [otlp] # Send to Instana
    
    metrics:
      receivers: [otlp] # Receive metrics from application
      processors: [k8sattributes, resource, batch] # Apply same processores as traces
      exporters: [otlp] # Send to Instana

Key points:

• passthrough: false - Tells k8sattributes to actively query K8s API and add the metadata. With passthrough: true, it would skip the K8s API query and only pass through attributes already in the telemetry, so k8s.pod.uid wouldn't be extracted.
• pod_association - Defines how the collector matches incoming telemetry to specific K8s pods. It tries strategies in order (pod IP → pod UID → connection IP) until one matches the pod.
• action: insert - Only adds service.instance.id if missing, preserving application's service.name
• Processor order matters - k8sattributes must run first to extract k8s.pod.uid, then resource can copy it to service.instance.id. Wrong order means k8s.pod.uid doesn't exist yet.
• All pipelines use same processors - Ensures consistent attributes across traces and metrics

Option B: Your Application Emits Traces Only

If your application only emits traces, you can use the spanmetrics connector in the OpenTelemetry Collector to automatically generate service-level metrics from your traces.

The spanmetrics connector:

1. Receives traces from your application
2. Analyzes spans to extract service-level information
3. Generates metrics (call rates, latencies, error rates)
4. Emits these metrics with the same resource attributes as the traces

This creates the exact same correlation as Option A, the only difference is where the metrics come from.

For Non-Kubernetes Environments

processors:
  batch: {}

  # Add required resource attributes
  resource:
    attributes:
    - key: service.name
      value: payment-service
      action: upsert
    - key: service.instance.id
      value: payment-service-01
      action: upsert
    - key: host.id
      value: your-host-id
      action: upsert

connectors:
  spanmetrics: {}  # Generate metrics from traces

service:
  pipelines:
    traces:
      receivers: [otlp]  # Receive traces from application
      processors: [resource, batch]  # Add attributes
      exporters: [otlp, spanmetrics]  # Send to both Instana and spanmetrics
    metrics:
      receivers: [spanmetrics]  # Receive metrics generated from spans
      processors: [resource, batch]  # Apply same processors as traces
      exporters: [otlp]  # Send to Instana

Key points:

• Both pipelines use the same resource processor, ensuring traces and generated metrics have matching attributes.

For Kubernetes Environments

Combine spanmetrics with the k8sattributes processor:

processors:
  batch: {}

  # Extract Kubernetes metadata
  k8sattributes:
    extract:
      metadata:
      - k8s.pod.uid  # Critical for pod correlation
      - k8s.pod.name
      - container.id # Critical for container correlation
      - ...
    passthrough: false
    pod_association:
    - sources:
      - from: resource_attribute
        name: k8s.pod.ip
    - sources:
      - from: resource_attribute
        name: k8s.pod.uid
    - sources:
      - from: connection
  
  # Set service.instance.id from pod UID
  resource:
    attributes:
    - action: insert
      from_attribute: k8s.pod.uid
      key: service.instance.id

connectors:
  spanmetrics: {}  # Generate metrics from traces

service:
  pipelines:
    traces:
      receivers: [otlp]  # Receive traces from application
      processors: [k8sattributes, resource, batch]  # Enrich with K8s metadata
      exporters: [otlp, spanmetrics]  # Send to both Instana and spanmetrics
    
    metrics:
      receivers: [spanmetrics]  # Receive metrics generated from spans
      processors: [k8sattributes, resource, batch]  # Apply same processors as traces
      exporters: [otlp]  # Send to Instana

Verifying Your Configuration

After configuring infrastructure correlation, verify it's working:

1. Check OpenTelemetry entities - Navigate to Infrastructure → Analyze Infrastructure → OpenTelemetry in Instana
2. View a trace - Open any trace from your application
3. Click on a span - View span details
4. Look for Infrastructure section - You should see a link to the OpenTelemetry entity
5. Navigate to infrastructure - Click through to see pod, host, and other infrastructure details

If you don't see the Infrastructure section in your span details, check that:

• Your metrics are flowing to Instana (creating the OpenTelemetry entity)
• Resource attributes match between traces and metrics
• service.name and service.instance.id are present in both traces and metrics
• For Kubernetes: k8s.pod.uid is present
• For Container: container.id is present
• For VMs: host.id matches what the Instana agent reports
• For Process: process.pid and host.id matches what the Instana agent reports

What's Next

You now have robust infrastructure correlation configured! This provides:

• Automatic entity creation from metrics
• Reliable span-to-entity correlation
• Complete infrastructure visibility from traces

If you have multiple services sharing one collector, read our companion blog on managing service identity to prevent all your services from merging into a single entity. Stay tuned!

#Infrastructure
#OpenTelemetry
#Tracing