In today’s fast changing and competitive world, the success of an IT enterprise is largely dependent on how fast they are able to detect and resolve customer issues. This is where Observability plays a very crucial role. The three Observability pillars are logs, metrics, and traces. A log is a detailed record of an event that has occurred in an organization’s IT systems or network. Metrics offer numerical data on system performance to analyse trends and monitor the system’s current state. Traces record the flow of requests and transactions in a system to provide a comprehensive view of the system’s components and overall behaviour.
Telemetry is the process of automatic collection and analysis of data from remote IT systems using sensors or agents to gain insights. Open Telemetry (OTel) is an open source observability framework that provides IT teams with standardised protocols and tools for collecting and routing telemetry data.
Logs are one of the biggest legacy telemetry signals supported by OTel instrumentation, and almost all the programming languages are supported with logging libraries and apis. For metrics and traces OTel takes a direct approach by providing a new API with full features support in multiple languages. The approach needed for logs is somewhat different. For OTel to be comprehensive in logging space there is a need to support existing legacy of logs and logging libraries, in addition to offering improvements and better integration with the rest of observability world where possible.
In general many Logging solutions typically have these gaps:
-
Weak integration with the rest of the observability signals.
-
Weak correlation support where tracing and monitoring tools often use limited information such as the time and origin attributes to correlate events.
-
Lack of standardisation for inclusion of information regarding the origin and source of logs such as the application and the infrastructure where the application runs.
This is where Instana comes in. Otel integrated with Instana can provide a comprehensive Observability solution. IBM® Instana® Observability integrates seamlessly with OTel ensuring no service is left un-instrumented supported by complete real-time observability. Instana with it’s powerful dependency map supports upstream and downstream service correlation, and a full-stack visibility helps provide the most comprehensive set of OTel value-added capabilities.
Instana supports the following types of Logs with OTel:
-
System Logs such as logs generated by the operating systems.
-
Infrastructure Logs such as logs generated by various infrastructure components such as Kubernetes events.
-
Third-party Application Logs that applications typically write to standard output, files or other specialized medium. An example is Windows Event Logs for applications.
-
Legacy First-Party Applications Logs like logs from applications that are created in-house.
-
New First-Party Application Logs from greenfield developments. OTel provides recommendations and best practices about how to emit logs (along with traces and metrics) from these applications via File or Standard Output.
How does Instana simplify Open Telemetry(OTel) Logging?
Instana now supports 100% OTel logging. The Instana architecture provides an Otel collector in addition to the normal agent collection framework. There is a dedicated processor for OTel events on the backend server. Customers can send OTel logs on the fly, visualize and monitor them real-time on Instana’s Dashboard. The entire log information from OTel stream is fetched and processed in real-time. This enables Site Reliability Engineers (SREs) to analyze OTel logs on Instana’s user-frendlly UI on the unbounded-analytics Dashboard.
Visualizing OTel logs on Instana’s UnBounded Analytics Dashboard
Instana Dashboard provides a single pane view for all the different log streams. On the Analytics Dashboard for logs, the log stream that needs to be monitored can be selected. The “opentelemetry-stream” can be selected on the left menu to display only OTel logs. The log types and rates over a time period can be seen on the graph. Hovering over a bar in the graph displays the rate of in-flow of the different types(e.g ERROR, DEBUG, WARN and INFO) of logs.
Fig 1. Unbounded Analytics Dashboard
The logs can be filtered based on a time window or log message attributes:
-
The time window can be selected for analysis on the right top of the screen. We can choose preset time windows or specify a custom range. This will filter the logs for the time window for a focused analysis.
-
A more fine grained filter can be selected from the “Filter” option based the OTel log attributes. Here, the logs are filtered based on the “log file path”. All logs from “fileb.log” are displayed on the Dashboard upon filtering. This enables a source based filtering if required.
The logs can also be grouped on the display with the “Group” menu. Here, the “Log Level” has been specified for grouping. A log message can be selected to see details. In the log message details, it is possible to view multi-line stack trace for Error logs.
For further analysis, the infrastructure correlation is also displayed as part of the log details. This is useful for debugging and correlating errors in the system.
A large scale distributed system can have hundreds of micro services interacting with each other. Terabytes of log data are generated. This voluminous data need to be captured, analyzed and correlated to provide support to customer issues, and resolve them on time. Instana equipped with OTel integration provides a highly efficient and comprehensive solution to monitor and analyze such IT environments.
Instana is available as SaaS deployment, and also as a self-hosted deployment. You can quickly deploy Instana SaaS for your IT environments, and analyze tons of logs on the fly!! Observability has never been made this easy!
References:
https://www.ibm.com/products/instana
https://opentelemetry.io/
https://www.datadoghq.com/knowledge-center/opentelemetry/
https://logit.io/blog/post/what-is-telemetry-data/
Author : @Balaji Kadambi , Co-Author : @Ayyappadas G B