Stan's Corner

 View Only

Monitoring Etcd database using Instana

By Adharsh H posted Mon November 06, 2023 06:46 AM


Co Author: Shivangi Mehra

IBM Instana is an application performance monitoring (APM) tool that offers real-time visibility and insights into the performance of apps and microservices in contemporary IT systems. It is intended to support businesses in managing and monitoring the operation of their infrastructure and applications in complex and dynamic on-premises, hybrid cloud, and cloud-native environments.


Etcd is a distributed, reliable, and highly available key-value store that's frequently used in distributed systems as a configuration and coordination service. It is open source and created by the CoreOS team, which is currently a part of Red Hat and is owned by IBM. The word "etcd" was derived from "etc" and "d"istributed system, the configuration directory for Unix. The purpose of etcd is to offer a dependable method for managing configuration data, service discovery data, and other crucial data in a distributed setting.

Etcd is made to be deployed across several cluster nodes in order to provide high availability and fault tolerance. For the purpose of ensuring data consistency and replication throughout the cluster, a consensus algorithm (often Raft) is used. Different applications and services operating on various nodes within the cluster or even across network borders can remotely access data that is stored in etcd. It is uncommon to use etcd as a local database within an application or on a single machine, although it can be used as a remote database for distributed applications and services. It works well in situations when you need to communicate and coordinate data across several distributed system services or between multiple instances of an application.


The etcd cluster has shared-nothing nodes. The cluster has a leader and followers, with one node acting as the cluster's leader. The Raft algorithm determines the leader node at run-time. The leader node casts votes against followers in response to requests. The leader commits the request and asks followers to commit if the majority of nodes concur. Any node in the cluster may receive a request from an etcd client. If the client sends a request to a follower, the follower relaysthe request to a leader node.

How to interact with etcd?

You can communicate with etcd through the etcd command-line client (etcdctl), the HTTP/gRPC API, or one of the numerous client libraries for various programming languages. To use etcdctl, you can follow the instructions below: 

1. Since etcdctl is frequently included in the etcd distribution, installing etcd is usually required before installing etcdctl. Use the following command to install etcd for Ubuntu/Debian:

sudo apt-get update
sudo apt-get install etcd

2. After the installation is complete, you can verify that etcdctl is installed by running the following command:

etcdctl version

3. To connect to your etcd cluster, etcdctl must be configured. The endpoint of your etcd cluster is typically specified using environment variables like      ETCDCTL_ENDPOINTS.

4. To set the ETCDCTL_API environment variable to specify a particular API version for the etcdctl command-line client, you can use the following syntax:

export ETCDCTL_API=<API version>

Setting the ETCDCTL_API environment variable to the desired API version will instruct etcdctl to use that version when communicating with the etcd cluster. The etcdctl command-line client supports two major API versions: version 2 and version 3. In newer versions of etcd, version 3 is the default API version.

5. Once configured, etcdctl can be used to communicate with your etcd cluster by executing commands like etcdctl put, etcdctl get, etc.

etcdctl set <key> <value>  #Setting a Key-Value Pair for version 2 API
etcdctl put <key> <value>  #Setting a Key-Value Pair for version 3 API
etcdctl get <key>  #Getting a Value by Key
etcdctl del <key>  #Deleting a Key
etcdctl watch <key>  #Watching for Changes



Here is the configuration for monitoring etcd that uses the Instana plug-in. This configuration specifies the paths to the required certificate and key files for establishing a secure connection to the etcd cluster. The CA Certificate is used to verify the authenticity of the etcd server's certificate.

The Client Certificate is used by Instana to authenticate itself to the etcd server. It should be a valid certificate that has been signed by the CA certificate. Client key is the private key used for cryptographic operations to establish a secure connection.

Metrics and dashboards

To ensure the dependability and stability of your distributed systems, it's crucial to monitor etcd performance, especially when it's used as a crucial component in platforms like Kubernetes. A range of performance indicators are offered by etcd to track the functions and overall health of the etcd cluster.

The details provided by the Etcd dashboard includes the number of requests that are sent and received, the amount of incoming and outgoing data in bytes, the number of keys that expire each second, the number of watchers, and the number of operations such as sets, gets, create, update, delete, compare and swap, and compare and delete etc. 

There are currently two dashboards for etcd, one for metrics gathered from /v2 endpoints and the other for metrics gathered from /v3 endpoints.

V2 dashboard

The v2 dashboard provides information like the cluster version, server version, and the number of unsuccessful and successful database operations.

V3 dashboard

The v3 dashboard includes Additional information like the Api version and counts of various database operations as compared to v2 dashboard.


Monitoring etcd using Instana offers a robust and comprehensive solution to ensure the reliability, performance, and health of your etcd clusters. It provides a strong solution for assuring the efficiency and dependability of your distributed etcd clusters. You can proactively manage your etcd infrastructure with its real-time visibility, security, and actionable analytics, and it will provide a solid basis for your distributed applications.