OpenTelemetry is a Cloud Native Computing Foundation project that provides an open-source framework for observability. It provides a standardized set of APIs, SDKs, and tools to generate, collect, process, and export telemetry data aligned around three pillars of Metrics (capturing numerical measurements over time), Logs (text events, errors and exceptions) and Tracing (tracking end-to-end transactions across multiple applications). Within the OpenTelemetry standards, ACE has a long track record of promoting the OpenTelemetry approach to tracing, which helps users to diagnose the elapsed time spent by a piece of data within different applications. ACE offers configuration for emitting trace events when interacting with the MQ nodes, HTTP nodes, REST nodes, SOAP nodes, Callable Flow nodes, Kafka nodes, Discovery Connector nodes and, from ACE 13.0.8.0, Compute nodes too. A trace typically consists of many Spans, where a span typically represents the actions that occurred within a particular application and has information to make querying and debugging easier such as a name, a status, a span kind, and time-related data.
Here's a reminder of the recent OpenTelemetry enhancements brought to ACE in version 13 mod releases:
- ACE 13.0.2.0: OpenTelemetry authentication improvements. Security credentials for HTTP and GRPC connections. Basic Auth, API Keys and Bearer Tokens supported.
- ACE 13.0.4.0: Kafka nodes enabled for OpenTelemetry tracing by default. Discovery Connector node OpenTelemetry traces emitted by setting
openTelemetryScope to the value extended in the server.conf.yaml settings.
- ACE 13.0.4.0: Access to telemetry data from Trace nodes and Compute node ESQL (OPENTELEMETRY_TRACE_ID, OPENTELEMETRY_TRACE_PARENT, OPENTELEMETRY_SPAN_ID, OPENTELEMETRY_TRACE_STATE)
- ACE 13.0.8.0: Compute node OpenTelemetry traces emitted by setting
openTelemetryScope to the value extended in the server.conf.yaml settings.
When enabled, every Compute node execution will now emit an OpenTelemetry span. The span carries some basic information about the Compute node processing:
- The attribute compute.mode carries the value of the Compute mode property
- The attribute compute.return specifies whether the ESQL returned TRUE or FALSE
- The attribute compute.propagate.to specifies the propagation target type
- The attribute compute.propagate.terminal specifies the terminal name (when propagating to a terminal)
- The attribute compute.propagate.label specifies the label name (when propagating to a label)
The Compute node will create one span for every PROPAGATE statement in the Compute node ESQL, plus one for the final return from the node. This means if your Compute node doesn't do an explicit propagate then you would get 1 span.
An error span will also be marked in the following circumstances:
- If the Compute node propagates to the failure terminal
- If the Compute node propagates to an invalid label
- If the Compute node catches an exception and rethrows
For more information, take a look at the ACE documentation about OpenTelemetry tracing.