Why Observability?

Observability is important for organizations as they rely on complex systems like micro-services, cloud infrastructure, distributed architectures etc. It provides the ability to understand and improve systems by collecting data in real time, allowing organizations to identify issues, optimize performance and ensure reliability.

How OpenTelemetry fits in?

OpenTelemetry is an open-source, vendor agnostic framework designed to collect and manage observability data (logs, metrics, and traces) from applications, services, and infrastructure. It's a set of APIs, libraries, and SDK that allows organizations to gather insights about how their systems behave in real-time.

It helps in creating a single, standardized approach for gathering telemetry data. It provides a consistent way to instrument code, capture telemetry, and export that data to various observability backends. Traces help to visualize how requests flow through your system. Metrics give you performance-related data, such as response times, throughput, and system resource utilization. Logs provide detailed insights into individual events, errors, and system status at a specific moment in time.

Instana and OpenTelemetry

IBM Instana, application performance monitoring (APM) and observability platform that provides real-time insights into the performance and health of your applications, micro-services, and infrastructure, has native support for OpenTelemetry, which allows users to combine Instana’s powerful monitoring and APM capabilities with OpenTelemetry’s flexibility and vendor-neutral approach to observability. This integration allows users to capture traces, metrics, and logs from their applications and then export that data to Instana for deeper analysis.

How Distributed Tracing works in micro-service architecture?

Distributed tracing helps track how transactions move across multiple distributed services. It works by passing context information from one service to another, linking transactions through unique identifiers. However, implementing distributed tracing in a distributed environment comes with challenges. Different vendors often use their own HTTP headers to store trace context, which can break traces when services from different vendors try to communicate. Moreover, since these tracing headers aren’t standardized, they’re not automatically forwarded by middleware like routers, service meshes, or messaging systems. As a result, dropped headers can lead to incomplete or broken traces.

The World Wide Web Consortium (W3C) Trace Context specification solves these issues by defining a standardized format for trace context data exchange.

By standardizing trace context propagation, the W3C Trace Context specification improves visibility into distributed applications, making it easier to analyze performance and resolve issues.

The OpenTelemetry Protocol (OTLP) is the wire protocol used to send telemetry data (such as traces, metrics, and logs) from clients (like application services, infrastructure) to backends (like observability platforms).

In an OpenTelemetry-based setup, OTLP is used to send trace data to observability tools, while the W3C Trace Context specification ensures that trace context (trace ID, span ID) is correctly propagated across service calls, even if different tracing systems or vendors are involved.

Monitoring Infosys Finacle

Infosys Finacle offers advanced digital banking solutions, deployed on the Red Hat OpenShift Container Platform (RHOCP) running on IBM Z and LinuxONE. The solution comprises a wide range of services developed using diverse technology stacks, including C/C++, Java Spring, Node.js, Tomcat, etc.

In collaboration with Infosys Finacle, we have implemented a modern observability architecture to enable end-to-end monitoring and performance visibility. As part of this effort, the OpenTelemetry C SDK has been integrated into Finacle’s application to collect telemetry data—including traces, metrics, and logs. This data is forwarded to the OpenTelemetry Collector, which acts as an intermediary to route it to IBM Instana for real-time visualization and monitoring.

Implementation Architecture Overview

This architecture leverages OpenTelemetry for telemetry collection and IBM Instana for monitoring and visualization. The core components of this architecture include the OpenTelemetry C SDK, the OpenTelemetry Collector, and IBM Instana. Below is a detailed explanation of the architecture:

• The Infosys Finacle application running in an RHOCP cluster has different pods running in each of their services. Since many technology stacks do not support  automatic instrumentation and require applications to explicitly call instrumentation APIs to generate and export telemetry data, OpenTelemetry provides a standardized solution for achieving this. Other services on technology stacks like Tomcat, Java, etc. are automatically instrumented by the Instana agent.
  
• The Finacle C++ application service is instrumented with the help of OpenTelemetry C API that generates the telemetry data. The SDK handles the collection of telemetry data and exports the data to observability backends through the OpenTelemetry Collector. The OpenTelemetry Collector acts as a data processing layer that aggregates and batches the telemetry data before sending it to a backend for analysis. The OpenTelemetry collector is deployed on another pod in our implementation. The Collector then sends the telemetry data to Instana using http or GRPC protocol for visualization and analysis.

Distributed Tracing with OpenTelemetry and W3C Trace Context

The key idea is that traces, which track requests as they move across multiple services, need to be understood by both OpenTelemetry and Instana. This is important because a request may pass through different services, each running on different technology stacks. To represent the entire journey of that request as a single trace, we need to ensure that both OpenTelemetry and Instana can read and follow the trace.

This can be achieved by using the W3C Trace Context standard. When trace data follows this standard, it uses W3C trace headers to pass context (such as trace ID and span ID) across services. By doing so, both OpenTelemetry and Instana can interpret and continue tracking the same trace throughout the lifecycle of the request, even as it moves through different services.

Instana is specifically configured to receive traces that follow the W3C Trace Context standard, ensuring that it can properly interpret and process trace data coming from different sources. OpenTelemetry generates traces in the OTLP (OpenTelemetry Protocol) format and exports them to a central OTEL collector. This collector processes the trace data and sends it to Instana, which can ingest traces in OTLP format.

In addition to the data sent via the OTEL collector, Instana agents deployed across the services also collect and send trace data directly to Instana. This integration ensures that Instana has all the necessary instrumentation data to monitor and manage application performance effectively.

By combining OpenTelemetry for trace collection, W3C Trace Context for trace propagation, and Instana’s ability to receive and process traces, you can achieve full visibility into the distributed system. This approach enables seamless trace tracking across different services and technologies, making it easier to monitor performance and troubleshoot issues.

Metrics sent to Instana can be viewed from the analyze Infrastructure section from the infrastructure tab and the traces can be viewed from the analytics tab on IBM Instana.

Metrics on IBM Instana

Traces on IBM Instana

CONCLUSION

This architecture provides a robust solution to instrument the Infosys Finacle application running on Red Hat OpenShift Container Platform on IBM LinuxONE using OpenTelemetry for collecting and transferring telemetry data to Instana, which can be hosted on IBM Z and LinuxONE.

For more details please contact:

Akshata Kulkarni - Software Engineer, CloudPak on Z - akshatakulkarni@ibm.com
Ashish Agarwal - Senior LinuxONE Solution Architect, IBM Z Ecosystem - ashish.agarwal@in.ibm.com 
Megha R - Team lead, Instana on IBM Z & LinuxONE - megha.r3@ibm.com
Soorej P - Product Manager, Infosys Finacle - soorej_p@edgeverve.com 
Srirama Sharma - STSM, Product owner ICIC, Chief Architect for Cloud Paks on Z - sriramsh@in.ibm.com 
Srinath Kanukolanu - Offering Manager, IBM Z Cloud Paks - srinath.kanukolanu@in.ibm.com 
Vijayalakshmi G - Product Manager, Instana on IBM Z & LinuxONE - Vijayalakshmi.G@ibm.com

#OpenTelemetry
#Self-Hosted
#BusinessObservability