OpenTelemetry

 View Only

Upgraded OpenTelemetry Generic Plugin in Instana

By Zhen Gao posted Wed January 17, 2024 04:59 AM

  

Co-Authors 

Zhen Gao (gaozhenz@cn.ibm.com) - Developer for Instana OTEL 

Zhi Li (bjlizhi@cn.ibm.com) - Developer for Instana OTEL 

Rui Liu (liurui@cn.ibm.com) - Tech lead for Instana OTEL 

OpenTelemetry-based data collectors work with prebuilt generic sensors in Instana to inject the monitoring data 

  • Improved handling and display of complex metrics with histograms: Histograms are used to show complex metrices. Compared to other types of diagrams, histograms has several advantages. Histograms such as bar charts can present highs, lows, concentration, and trends within the data to indicate common value ranges or concentration within specific intervals. Histograms helps to swiftly identify outliers or anomalies within the data, as they might appear as extreme values in the distribution. 

  • Attributes of metrics are retained: All types of data are retained with different dimensions. 

  • Better equipped to manage the dependencies of the apps: The generic sensor and entity plug-in for OpenTelemetry can handle the dependencies of every app that runs in containers.  

Design 

Histograms: 

  • Both DELTA mode and CUMULATIVE mode are supported. 

  • Real-time average value is displayed in the built-in dashboard.  

Note: If data goes through otlp-acceptor at backend, the real-time average value for CUMULATIVEmode is lost as date is not supported at backend. 

  • The _max and _min data points are displayed. 

Attributes: 

  • All OpenTelemetry attributes for data points are retained. 

  • Resource attributes for service.name, pid, container.id, host.name are retained as tags. All other tags for resource attributes are discarded because too many tags might severely decrease the user experience of custom dashboards. 

Dependency with entities: 

  • For in-container apps, the container.id parameter is identified and dependency is established with the corresponding container entity; 

  • The dependency between Generic OpenTelemetry entity and OTelProcess entity is optimized. 

Sample payload of generic OpenTelemetry plug-in  

The following snippet shows a sample payload of Generic OpenTelemetry plug-in: 

{ 

  "pid": "1234", 

  "remote": "28322@d4d7c18968f34007a04b815b27582c62", 

  "resource":{ 

    "service.name":"my-service", 

    "pid": "1234", 

    "hostname": "dk-liunoone.fyre.ibm.com" 

    "container.id": "90202a241f1bc13f96068735ddf4ccbd7b8af7c44c9e2210d835a44c58f39eb4", 

    "sdk.opentelemetry.type": "sensor-sdk:dameng" 

  }, 

  "metrics": { 

    "sums": { 

      "sdk.logs/db.client.connections.idle.min{pool.name=\"HikariPool-1\"}": 10 

    }, 

    "gauges": { 

      "hikaricp/queueSize{logRecordProcessorType=\"BatchLogRecordProcessor\",}": 0 

    }, 

    "histograms": { 

      "hikaricp/db.client.connections.create_time_count{pool.name=\"HikariPool-1\", }": 59, 

      "hikaricp/db.client.connections.create_time_sum{pool.name=\"HikariPool-1\", }": 55.000000, 

      "hikaricp/db.client.connections.create_time_min{pool.name=\"HikariPool-1\", }": 0.000000, 

      "hikaricp/db.client.connections.create_time_max{pool.name=\"HikariPool-1\", }": 8.000000, 

      "hikaricp/db.client.connections.create_time_mean{pool.name=\"HikariPool-1\", }": 3.800000, 

      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"0.0\",}":23.0, 

      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"5.0\",}":35.0, 

      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"10.0\",}":1.0, 

      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"25.0\",}":0.0, 

      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"50.0\",}":0.0, 

      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"75.0\",}":0.0, 

      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"100.0\",}":0.0, 

      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"250.0\",}":0.0, 

      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"500.0\",}":0.0, 

      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"750.0\",}":0.0, 

      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"1000.0\",}":0.0, 

      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"2500.0\",}":0.0, 

      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"5000.0\",}":0.0, 

      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"7500.0\",}":0.0, 

      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"10000.0\",}":0.0, 

      "hikaricp/db.client.connections.create_time_bucket{pool.name=\"HikariPool-1\", le=\"+Inf\",}":0.0 

    }         

  } 

} 

Deploy OpenTelemetry Demo in OCP 

To deploy OpenTelemetry Demo in OCP, complete the following steps: 

  1. Log on to OpenShift CLI (oc). 

  1. Create the namespace otel-demo: 

kubectl create namespace otel-demo 

kubectl project otel-demo 

  1. Install the OpenTelemetry demo application by using Helm: 

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts 

  1. Install helm chart with the release name my-otel-demo:  

helm install my-otel-demo open-telemetry/opentelemetry-demo 

  1. Expose services using kubectl port-forward: 

kubectl port-forward svc/my-otel-demo-frontendproxy 8080:8080 

  1. Create route for services for my-otel-demo: 

my-otel-demo-frontendproxy 

my-otel-demo-grafana 

my-otel-demo-jaeger-query 

 
The following example shows a route that creates my-otel-demo-frontendproxy. 
 
 

Deploying Instana agent in OCP 

Install Instana agent using Helm: 

helm install instana-agent \ 

   --repo https://agents.instana.io/helm \ 

   --namespace instana-agent \ 

   --create-namespace \ 

   --set openshift=true \ 

   --set agent.key=*************** \ 

   --set agent.downloadKey=*********** \ 

   --set agent.endpointHost=ingress-magenta-saas.instana.rocks \ 

   --set agent.endpointPort=443 \ 

   --set cluster.name='cluster101' \ 

   --set zone.name='zone101' \ 

   instana-agent 

Generic OpenTelemetry plug-in for OCP 

Custom dashboard page 

Histogram 
 
Screenshot 2023-08-11 at 12 05 54 

Custom tags implemented for gauge, histogram metrics 

 
Screenshot 2023-08-11 at 12 10 59 

Screenshot 2023-08-11 at 12 09 29 

Generic OpenTelemetry entity plug-in page 

Stack infrastructure of a generic OpenTelemetry entity 
Screenshot 2023-08-11 at 11 36 32 

 

Dependency tree view of a generic OpenTelemetry entity which runs in a CRI-O container.  

A screenshot of a computer

Description automatically generated 

 

Breadcrumb navigation bar of a generic OpenTelemetry entity which runs in a CRI-O container 

A screenshot of a computer

Description automatically generated 

 

Trace of a call from/to OpenTelemetry services page 

Analytics call traces 

A screenshot of a computer

Description automatically generated 

 


#Instana
#IBMInstana
#OpenTelemetry

Permalink