Authored by @Yanwei Li, @Ying Mo, @Paras Kampasi

Introduction

The Problem: When Services Lose Their Identity

What Went Wrong

The issue lies in how the collector assigns resource attributes. When multiple services share the same pipeline with the same resource processor configuration, they all receive identical service.name and service.instance.id values.

Remember: Instana creates OpenTelemetry entities based on the combination of service.name and service.instance.id. When all services have the same values, they collapse into a single entity.

Let's look at a problematic configuration:

processors:
  resource:
    attributes:
    - key: service.name
      value: my-service  # Same for all services!
      action: upsert
    - key: service.instance.id
      from_attribute: k8s.pod.uid
      action: insert

service:
  pipelines:
    traces:
      processors: [resource, batch]
    metrics:
      processors: [resource, batch]

What happens:

1. payment-service sends traces with service.name=payment-service
2. Resource processor overwrites it with service.name=my-service
3. inventory-service sends traces with service.name=inventory-service
4. Resource processor overwrites it with service.name=my-service
5. All services now have service.name=my-service
6. Instana creates one entity for all of them

The problem is action: upsert, which overwrites existing values and destroys the service identity that your application's SDK provides.

Let your application's SDK set service.name, and use the collector only to add service.instance.id if it's missing.

For services sharing a collector:

resource:
  attributes:
  - action: insert
    from_attribute: k8s.pod.uid
    key: service.instance.id
  
  # DO NOT set service.name here - let the application provide it

Using action: insert means the collector only adds the attribute if it doesn't already exist. This preserves the service.name that your application's OpenTelemetry SDK provides.

Here's a complete example showing how to handle multiple services with different telemetry sources:

• OTLP services: Multiple microservices sending traces and metrics via OTLP
• Prometheus metrics: Additional metrics scraped from application endpoints

receivers:
  otlp:  # Receive traces and metrics from applications
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  
  prometheus:  # Scrape Prometheus metrics
    config:
      scrape_configs:
      - job_name: 'my-app'
        static_configs:
        - targets: ['localhost:8080']

processors:
  batch: {}
  
  # Extract Kubernetes metadata
  k8sattributes:
    extract:
      metadata:
      - k8s.pod.uid  # Critical for pod correlation
      - k8s.pod.name
      - ...
    passthrough: false  # Actively enrich telemetry
    pod_association:
    - sources:
      - from: resource_attribute
        name: k8s.pod.ip
    - sources:
      - from: resource_attribute
        name: k8s.pod.uid
    - sources:
      - from: connection
  
  # Processor for OTLP services
  # Preserves SDK-provided service.name
  resource/otlp:
    attributes:
    - action: insert  # Only add if missing
      from_attribute: k8s.pod.uid
      key: service.instance.id
  
  # Processor for Prometheus metrics
  # Must set service.name since Prometheus doesn't provide it
  resource/prometheus:
    attributes:
    - action: insert
      from_attribute: k8s.pod.uid
      key: service.instance.id
    - action: insert  # Set service.name for Prometheus
      key: service.name
      value: my-prometheus-app

exporters:
  otlp:  # Send to Instana
    endpoint: instana-endpoint
    headers:
      x-instana-key: your-instana-key

service:
  pipelines:
    traces:
      receivers: [otlp]  # Receive from OTLP services
      processors: [k8sattributes, resource/otlp, batch]  # Preserve service.name
      exporters: [otlp]  # Send to Instana
    
    metrics/otlp:
      receivers: [otlp]  # Receive from OTLP services
      processors: [k8sattributes, resource/otlp, batch]  # Same as traces
      exporters: [otlp]  # Send to Instana
    
    metrics/prometheus:
      receivers: [prometheus]  # Scrape Prometheus metrics
      processors: [k8sattributes, resource/prometheus, batch]  # Set service.name
      exporters: [otlp]  # Send to Instana

Two resource processors:

1. resource/otlp - For OTLP services

   • Uses action: insert for service.instance.id
   • Does NOT set service.name (preserves SDK value)
   • Each service keeps its unique identity

2. resource/prometheus - For Prometheus metrics

   • Uses action: insert for both attributes
   • Must set service.name (Prometheus doesn't provide it)

Pipeline separation:

Traces and metrics from the same service must use the same resource processor to ensure matching attributes.

Configure your application to set its own service.name:

export OTEL_RESOURCE_ATTRIBUTES="service.name=payment-service"

Or in your application code:

# Python example
from opentelemetry.sdk.resources import Resource

resource = Resource.create({
    "service.name": "payment-service"
})

In your collector's resource processor:

# CORRECT - Preserves application's service.name
resource:
  attributes:
  - action: insert
    from_attribute: k8s.pod.uid
    key: service.instance.id

# WRONG - Overwrites application's service.name
resource:
  attributes:
  - action: upsert
    key: service.name
    value: my-service

Traces and metrics from the same service must use the same processor:

traces:
  processors: [k8sattributes, resource/otlp, batch]
metrics:
  processors: [k8sattributes, resource/otlp, batch]  # Identical

When handling different telemetry sources (OTLP, Prometheus, etc.), create separate pipelines with appropriate processors:

metrics/otlp:
  processors: [resource/otlp]  # Preserves service.name
metrics/prometheus:
  processors: [resource/prometheus]  # Sets service.name

After implementing these changes, verify each service has its own entity:

1. Check OpenTelemetry entities - Navigate to Infrastructure → Analyze Infrastructure → OpenTelemetry
2. Count entities - You should see one entity per service instance
3. Verify service names - Each entity should show the correct service name
4. Test correlation - Click through from traces to infrastructure for each service

If services are still merging:

• Check that applications are setting service.name correctly
• Verify you're using action: insert, not action: upsert
• Ensure traces and metrics from the same service use the same processor
• Confirm service.instance.id is unique per instance

Managing multiple services with a shared OpenTelemetry Collector requires careful attention to service identity. The key is to let applications define their own service.name and use the collector only to enrich telemetry with infrastructure-specific attributes like service.instance.id.

By following these patterns:

• Each service maintains its unique identity
• Infrastructure correlation works correctly for all services
• You can scale to dozens of services on a single collector