Instana

Instana

The community for performance and observability professionals to learn, to share ideas, and to connect with others.

 View Only

Modern Observability Stack: Opentel Meets IBM Instana

By SANJITA GUPTA posted Wed May 07, 2025 02:29 PM

  

In today’s software ecosystem, observability is essential for ensuring modern distributed applications' reliability, performance, and user experience. Two prominent players in this field are IBM Instana and OpenTelemetry. Let’s explore their features, how they are related, and how they together empower developers and operations teams to gain unparalleled insights into their systems.

OpenTelemetry is an open-source observability framework for instrumenting, generating, collecting, and exporting telemetry data such as traces, metrics, and logs. It emerged as the merger of two earlier projects – OpenTracingand OpenCensus –

and is governed by the Cloud Native Computing Foundation (CNCF).

Key Features of OpenTelemetry:

  • Vendor-neutral instrumentation: Allows developers to instrument code once and export data to any backend observability platform.
  • Unified Telemetry: Supports traces, metrics, and logs, creating a standardized observability approach.
  • SDKs and APIs: Offers SDKs in multiple programming languages (e.g., Python, Java, Go, Node.js).
  • Wide Ecosystem Support: Integrates with popular open-source tools and frameworks like Kubernetes, Jaeger, and Prometheus, and with commercial products and is supported by cloud providers

OpenTelemetry primarily focuses on creating and exporting telemetry data. However, it doesn’t store or analyze this data; instead, it forwards it to a backend system like IBM Instana for processing and visualization.

On the other hand, IBM Instana, is an Observability product and Application Performance Monitoring (APM) and that provides automatic and continuous monitoring of applications, infrastructure, and microservices. It isn’t just a monitoring tool; it is a platform to obtain insight and understand a system’s inner workings. Its key differentiator lies in its AI-powered automation and ability to provide full-stack observability with minimal manual configuration.

Key Features of IBM Instana:

  • Automatic Discovery and Instrumentation: IBM Instana discovers services, components, and dependencies automatically, requiring no manual intervention.
  • Real-time Monitoring: Captures and visualizes application performance data in near real-time.
  • AI-Powered Insights: Leverages AI to detect anomalies, analyze root causes, and offer actionable recommendations.
  • End-to-End Tracing: Tracks user transactions across distributed services to identify performance bottlenecks.
  • Infrastructure Monitoring: Includes support for monitoring cloud-native environments like Kubernetes, AWS, and Azure.

IBM Instana excels in collecting and analyzing telemetry data, offering deep insights into system health and performance.

While OpenTelemetry and IBM Instana serve distinct purposes, they complement each other well. IBM Instana can integrate with OpenTelemetry in multiple ways. OpenTelemetry data can be sent to IBM Instana server directly or the data can be locally received by the Instana host agent equipped with a native OTLP endpoint which listens on standard ports 4317 (gRPC) and 4318 (HTTP).

The host agent's key advantage lies in its ability to link incoming telemetry data to the underlying application resources and host infrastructure, which it continuously discovers and monitors. By identifying the specific processes sending OpenTelemetry traces and metrics, the agent establishes correlations across physical and logical dependencies, such as hosts, clusters, containers, and application runtimes, as modeled in IBM Instana's Dynamic Graph.

This infrastructure context is readily accessible through the Context Guide and the Unbounded Analytics workspace, enabling us to analyze individual calls in-depth and gain holistic insights into our system's performance and dependencies.

Here is how they align:

  • Data Collection and Instrumentation: OpenTelemetry acts as a standard framework for instrumenting applications to collect telemetry data. Developers use OpenTelemetry APIs and SDKs to instrument their code, ensuring consistent and vendor-agnostic data collection. IBM Instana integrates with OpenTelemetry, ingesting the telemetry data and enriching it with additional context for visualization, analysis, and alerting.
  • Seamless Integration: OpenTelemetry traces, metrics, and logs can be exported directly to IBM Instana. IBM Instana automatically correlates OpenTelemetry data with its own discoveries, creating a unified observability dashboard.
  • Enhanced Observability with AI: While OpenTelemetry provides raw telemetry data, IBM Instana’s AI capabilities analyze and present this data in meaningful ways, detecting anomalies and providing root cause analysis. Support for Distributed Tracing:Both tools emphasize distributed tracing, which tracks requests as they traverse multiple services. OpenTelemetry generates the trace data, while IBM Instana visualizes the traces and correlates them with other performance metrics.
  • Flexibility and Choice: Developers gain flexibility by using OpenTelemetry to instrument code and IBM Instana as the backend system for analysis. This approach prevents vendor lock-in and ensures scalability.

Here are some of the benefits of Using OpenTelemetry with IBM Instana

  • Standardized Observability: OpenTelemetry ensures a consistent approach to telemetry across teams and technologies, while IBM Instana enhances this data for decision-making.
  • Faster Troubleshooting: IBM Instana’s real-time insights and AI-powered analytics accelerate the identification and resolution of performance issues captured by OpenTelemetry.
  • Scalability and Cloud-Native Support: Both tools are designed for cloud-native environments, making them ideal for Kubernetes, microservices, and serverless architectures.
  • Developer and Ops Collaboration: OpenTelemetry empowers developers to instrument their applications effectively, while IBM Instana provides operations teams with actionable insights, bridging the DevOps gap.

Some Use Cases: OpenTelemetry and IBM Instana in Action

  • Microservices Observability: A large e-commerce platform uses OpenTelemetry to instrument its microservices, generating telemetry data. IBM Instana ingests this data and visualizes service dependencies, identifying bottlenecks during high traffic periods.
  • Kubernetes Monitoring: A SaaS provider runs its workloads on Kubernetes, using OpenTelemetry to capture metrics and traces. IBM Instana integrates this data with its Kubernetes monitoring capabilities, offering a unified view of application and infrastructure health.
  • Hybrid Cloud Monitoring: A financial institution adopts OpenTelemetry to standardize instrumentation across its hybrid cloud environment, while IBM Instana provides AI-driven anomaly detection to ensure seamless operations.

As part of our effort to enhance observability for our product, we successfully implemented IBM Instana to monitor key component Postgres. Here is the detailed procedure, along with tips on how to utilize IBM Instana for meaningful insights. 

Step 1: Initial Setup - Enabling Metrics Collection for Postgres  

We began by enabling metric collection for Postgres: 

  • Modifying the Postgres Cluster CR: The development team updated the Service Operator to include the necessary configurations for enabling metrics collection in the Postgres Cluster Custom Resource (CR). 

  • Creating and Granting Permissions: We created an IBM Instana user in the Postgres database and granted the required permissions to allow access to monitoring data. 

  • Updating Cluster Parameters: Added additional parameters in the Postgres cluster configuration to ensure InstanaIBM Instana could access the necessary data seamlessly. 

Step 2: Setting Up the Centralized InsIBManaIBM Instana Server 

A centralized IBM Instana server was set up by the Cloud Oak team for the Data Performance team, enabling all services across the environment to use it. This central server served as the backbone for collecting and analyzing data from multiple services, including Postgres. 

Step 3: Installing IBM Instana Agents on CP4D Cluster 

With the centralized IBM Instana server in place, the next step was to set up IBM Instana agents in the CP4D cluster where instrumented services, including Postgres, were deployed. 

  • Updating InstanaIBM Instana Agent Configuration: Edited the InstanaIBM Instana agent configuration to include specifications for the Postgres cluster, enabling the agent to connect to Postgres and start collecting metrics. 

  • Verifying Integration: Logged into the InstanaIBM Instana User Interface, navigated to the Kubernetes menu, and located the cluster information under the Infrastructure tab. Validated that the metrics were being collected successfully and associated with the correct workloads and services. 

Step 4: Exploring Metrics in the IBM Instana UI 

Once metrics collection was operational, the IBM Instana UI provided an intuitive way to explore and analyze the data. 

  • Navigating the Cluster: In the Kubernetes menu, selecting the cluster name allowed us to focus on the metrics related to workloads and services specific to Watson Assistant. 

  • Key Metrics and Visualizations: 

  • Graphs and dashboards displayed crucial metrics like: 

  • Total Calls (sum): The total number of calls made by a service. 

  • Erroneous Call Rate (mean): The percentage of calls resulting in errors. 

  • Latency (mean): The average time taken for calls. 

  • Filtering for Specific Data: Used the "Add Filter" option to narrow down the data by fields such as service name, endpoints, or criteria of interest. 

Step 5: Key Metrics to Focus On 

When exploring IBM Instana, it’s easy to get overwhelmed by the sheer amount of data. Start with these foundational checks: 

  • Basic Observations:  

  • Identify all observable services and endpoints. 

  • Check for any missing endpoints or services that should be monitored. 

  • Critical Metrics: Total Calls (sum): Look for an unusually high number of calls that might indicate inefficiencies.Latency (mean): Monitor for long latency calls, which could signal performance bottlenecks.Unexpected Call Flows: Investigate any unanticipated communication between services. 

By following the outlined steps, we successfully established observability for our product, providing valuable insights into its performance and behavior. Building on this success, we are now extending the same observability capabilities to additional components. OpenTelemetry and IBM Instana represent a powerful combination in the observability landscape. OpenTelemetry offers an open, vendor-neutral standard for collecting telemetry data, while IBM Instana excels in analyzing and visualizing this data for actionable insights. Together, they enable organizations to build resilient, high-performing systems in an increasingly complex software ecosystem. 


#Infrastructure
#Kubernetes
#Logs
#OpenTelemetry
#Tracing
#Education

0 comments
272 views

Permalink

https://community.ibm.com/community/user/blogs/sanjita-gupta/2025/05/07/opentel-meets-ibm-instana