Observability by Default: Automating Enterprise Observability with Terraform and Instana

Authors: @Jeison Parra Tijaro, Benjamin Lykins, Georgekutty Joseph, @Chinmay Samant, Cesar Araujo

In modern cloud-native environments, observability has become a critical requirement rather than a luxury. As organizations scale their infrastructure across Kubernetes clusters, microservices, and multiple cloud providers, the challenge isn't simply having observability tools, it's ensuring that every resource, every service, and every instance is automatically monitored from the moment it's provisioned.

Manual configuration of observability agents, dashboards, and alerts is time-consuming, error-prone, and doesn't scale. What if observability could be built directly into your infrastructure provisioning process? What if observability was automatically configured for every resource you deploy, following organizational standards without manual intervention?

This is the concept of observability by default a paradigm where observability becomes an inherent characteristic of your infrastructure rather than an afterthought. By leveraging IBM HashiCorp’s Terraform infrastructure-as-code capabilities with IBM Instana's powerful observability platform, organizations can achieve comprehensive, automated observability that scales with their infrastructure (Instana Official Provider). In this article you would find a guide on how to implement observability by default

The Observability Challenge in Enterprise Environments

Traditional approaches to monitoring face several critical challenges at scale:

• Manual overhead: Teams spend countless hours clicking through UIs to configure agents, create dashboards, and set up alerts for each new service.
• Configuration drift: Different environments develop inconsistent observability configurations, making cross-environment debugging difficult and time-consuming.
• Delayed visibility: Observability is often configured after deployment, creating dangerous blind spots during critical launch periods when issues are most likely to occur.
• Lack of standardization: Without enforced standards, different teams implement observability differently, leading to fragmented visibility and inconsistent practices.
• Poor auditability: Manual changes leave no audit trail, making it difficult to track what changed, when, and by whom critical for compliance and troubleshooting.

Who Benefits from Observability by Default?

Platform Engineering Teams can create self-service infrastructure templates with built-in observability, reducing support burden while enabling developer autonomy. Teams can provision infrastructure knowing that observability is automatically configured.

DevOps and SRE Teams gain consistent visibility across all environments without manual configuration. This makes incident response faster and more effective while dramatically reducing operational overhead.

Security and Compliance Teams benefit from complete audit trails provided by Git-managed configurations and can enforce observability standards through policy-as-code, ensuring compliance requirements are met automatically.

Development Teams get immediate visibility into their applications without needing deep expertise in observability tools, allowing them to focus on building features while still maintaining production-grade observability.

Why Instana and Terraform Together?

IBM Instana is purpose-built for modern, dynamic cloud-native applications. Unlike traditional monitoring solutions that require extensive configuration, Instana automatically discovers and monitors your entire application stack. Its key differentiators include:

• Automatic discovery: Instana agents automatically sense and pull in components without explicit configuration, including native integrations with HashiCorp products like Nomad, Vault, and Terraform itself.
• One-second granularity: Provides real-time visibility ideal for microservices and dynamic infrastructure where conditions change rapidly.
• AI-powered insights: Automatically correlates metrics, traces, and logs to identify root causes quickly.
  
  

HashiCorp Terraform is the industry standard for infrastructure-as-code, trusted by enterprises worldwide. Its declarative approach enables:

• Version-controlled infrastructure: All changes are tracked in Git, providing complete audit trails and enabling peer review.
• Consistency across environments: The same configuration can be applied across development, staging, and production with confidence.
• Unified management: Manage infrastructure, applications, and observability configurations from a single platform.

Together, they create a powerful workflow where infrastructure and observability are provisioned, version-controlled, and managed as a unified, declarative configurationenabling true observability by default.

Implementing Observability by Default: A Practical Guide

Let's explore four key patterns that enable observability by default, with real examples from production implementations.

1. Simplify Agent Deployment with Infrastructure-as-Code

The foundation of observability by default is ensuring that every node in your infrastructure automatically gets an observability agent. Using Terraform with orchestration platforms like Nomad, you can ensure comprehensive coverage without manual intervention.

The example below shows how to configure Instana agent deployment using Nomad's system scheduler, which ensures that any new node that joins the cluster automatically receives an Instana agent:

#instana-agent.nomad.hcl
job "instana" {
  namespace    = "terraform-enterprise"
  node_pool    = "control-plane"
  type         = "system"
  group "instana" {
    network {
      mode = "host"
    }    
    task "agent" {
      driver = "docker"
      
      config {
        image = "icr.io/instana/agent:latest"
      }
    }
  }
}

Key benefits of this approach:

• System-level scheduling: The system job type ensures an agent runs on every node automatically, including nodes that join the cluster later.

• Zero-touch observability: New infrastructure automatically receives observability without any manual configuration steps.

• Version-controlled deployment: Agent configuration is stored in Git and deployed consistently across all environments.

Once deployed, the Instana dashboard provides immediate visibility into your infrastructure. The agent automatically senses components and integrates with HashiCorp products like Nomad, Vault, and other platform services without explicit configuration:

2. Enhanced Organization Through Intelligent Tagging

Tags are fundamental to organized, queryable infrastructure. When implemented through Terraform, tags become standardized and automatically applied to all resources. The principle is simple: the more tags, the better.

Why comprehensive tagging matters:

• Better auditing: Track resource ownership, costs, and accountability across teams and projects.
• Simplified alert routing: Automatically route alerts to the correct teams based on tags like team, environment, or service.
• Easier troubleshooting: Filter and query infrastructure by multiple dimensions during incident response, dramatically reducing time to resolution.

Here’s an example of a comprehensive tagging strategy that includes both Terraform-specific and CI/CD metadata:

# Comprehensive resource tagging strategy
resource "aws_instance" "app_server" {
  # ... other configuration ...
  
  tags = {
    # Terraform workspace identification
    terraform.workspace  = "tagging-example"
    
    # CI/CD and Git metadata
    github.actor         = "benjamin-lykins"
    github.repository    = "benjamin-lykins/hashiconf-2025"
    github.run           = "https://github.com/.../runs/17299912220"
    
    # Organization metadata
    team                 = "devops"
    cost_center          = "1234"
  }
}

These tags automatically flow to Instana, where they enable powerful filtering and organization capabilities. You can easily query your entire infrastructure and get a single pane of glass view of specific applications or services:

3. Shift Observability Left: Dashboard Provisioning as Code

Traditional workflows create dashboards after infrastructure is deployed. Observability by default flips this paradigm: dashboards and observability views are provisioned alongside your infrastructure, ensuring comprehensive visibility from the moment resources go live.

Using Terraform’s Instana provider, you can define custom dashboards that automatically reference your infrastructure. As you create resources like databases, caches, and storage, you simultaneously create monitoring dashboards that track their health:

#dashboard.tf - Provisioning infrastructure and Observability together

resource "aws_db_instance" "tfe" {
  # RDS database configuration
}

resource "aws_s3_bucket" "tfe" {
  # S3 bucket configuration
}

resource "aws_elasticache_cluster" "tfe" {
  # Redis cache configuration
}

# Dashboard automatically references the resources above
resource "instana_custom_dashboard" "tfe" {
  title = "Terraform Enterprise"
  
  widgets = templatefile("${path.module}/dashboards/tfe.json", {
    rds    = aws_db_instance.tfe.name
    redis  = aws_elasticache_cluster.tfe.name
    s3     = aws_s3_bucket.tfe.bucket
  })
}

This approach eliminates manual dashboard configuration entirely. When infrastructure is created, the corresponding monitoring dashboards are automatically configured with the correct resource identifiers, providing immediate, comprehensive visibility:

The dashboard updates automatically in real-time, adapting as your infrastructure changes without requiring manual reconfiguration truly dynamic observability.

4. Standardize and Enforce Through Policy-as-Code

The final piece of observability by default is ensuring that standards are consistently followed. Using HashiCorp Sentinel or Open Policy Agent (OPA), you can enforce that all infrastructure includes required tags and observability configurations before deployment.

Enforcement strategies:

• Template IaC pipelines: Provide approved Terraform modules with built-in observability that teams can easily consume.
• Sentinel policy enforcement: Mandate that resources include required tags and observability configurations before Terraform can apply changes.
• Automated tag injection: Use CI/CD pipelines to automatically inject metadata tags like repository, actor, and run information.

Here’s an example GitHub Actions workflow that automatically injects comprehensive tagging metadata:

#github-actions.yml - Automatic tag injection
env:
  TF_VAR_git_tags: '{
    "github.actor": "${{ github.actor }}",
    "github.repository": "${{ github.repository }}",
    "github.run": "https://github.com/${{ github.repository }}/
                    actions/runs/${{ github.run_id }}"
  }'
  
  TF_VAR_team_tags: '{
    "team": "devops",
    "cost_center": "1234"
  }'}

By encoding these requirements into your CI/CD pipelines and using Sentinel policies in Terraform Enterprise, you ensure that observability standards are automatically maintained across all teams and projects no exceptions, no manual enforcement needed.

Real-World Impact and Results

Organizations implementing observability by default with Terraform and Instana report transformative operational improvements:

• Dramatic time savings: What once required hours of manual UI configuration now happens automatically during infrastructure provisioning, reducing observability setup time by 70–80%.
• Zero blind spots: New services go live with full observability from second one, eliminating the dangerous visibility gaps that traditionally exist during launches.
• Consistent Observability everywhere: Development, staging, and production environments have identical observability configurations, dramatically simplifying debugging and reducing environment-specific issues.
• Faster incident resolution: Standardized tagging and dashboard structures enable teams to quickly locate relevant information during incidents, reducing mean time to resolution (MTTR) by up to 45%.
• Improved compliance: Git history provides complete audit trails of all Observability configuration changes, satisfying compliance requirements automatically.
• Better resource optimization: Comprehensive tagging enables accurate cost allocation and identification of optimization opportunities, with some teams reporting 15–20% reductions in Observability-related operational costs.

Getting Started: Your Path to Observability by Default

Implementing observability by default doesn’t require a complete infrastructure overhaul. Start with these practical steps:

• Step 1 — Automate agent deployment: Use Terraform with your orchestration platform (Nomad, Kubernetes, etc.) to ensure Instana agents are deployed automatically to all new resources. This provides immediate baseline visibility.
• Step 2 — Implement comprehensive tagging: Add a standardized tagging module to your Terraform configurations that includes workspace, CI/CD metadata, and organizational information. Start with just a few essential tags and expand over time.
• Step 3 — Create dashboard templates: Build reusable dashboard templates for common infrastructure patterns (web applications, data pipelines, microservices, etc.) that can be instantiated alongside infrastructure.
• Step 4 — Implement policy enforcement: Use Sentinel or OPA to enforce observability standards, starting with simple policies (required tags, monitoring resource creation) and expanding as your practices mature.
• Step 5 — Iterate and improve: Gather feedback from teams, measure the impact, and continuously refine your observability configurations based on real-world usage patterns.

Observability by default represents a fundamental shift in how we approach Observability from a manual, reactive process to an automated, proactive foundation. By treating observability as code and making observability an inherent characteristic of your infrastructure, you’re not just improving operational efficiency you’re building a foundation for reliable, scalable systems that can confidently adapt to whatever challenges the future brings.

Resources and Further Reading

• Instana Terraform Provider Documentation: https://registry.terraform.io/providers/instana/instana/latest/docs
• IBM Instana Observability: https://www.ibm.com/products/instana
• HashiCorp Terraform: https://developer.hashicorp.com/terraform
• HashiCorp Nomad: https://developer.hashicorp.com/nomad
• Terraform Sentinel Policies: https://www.terraform.io/cloud-docs/policy-enforcement
• Instana + HashiCorp Better Together: https://www.instana.com/partners/hashicorp/