Global Storage Forum

• Fetching current metric states from Storage Insights using its REST API
• Translating those metrics into Prometheus metrics
• Exposing the translated metrics over HTTP

Prometheus instrumentation client libraries help in building exporters by providing interfaces for this use case.

Building Exporter in Go

We will use Prometheus Go client library to build Prometheus exporter in Go.

The Collector Interface

An exporter needs to fetch and translate metrics every time it is scraped. To achieve this in Go, we have to implement the prometheus.Collector interface to create a type that can be registered with a metrics registry. The difference to a normal metric object, e.g., prometheus.Counter, is that the collector does not store the metrics itself, but instead fetches and translates metrics from the third-party system, Storage Insights in our case, for every scrape. The type needs to implement two methods for this:

• Describe(chan<- *Desc) : This method is called upon registration and tells the metric registry which kinds of metric the collector produces.
• Collect(chan<- Metric) : This method is called on every scrape by the registry and produces the actual metrics for that scrape.

The Describe() method

Let's first describe the nature of our metrics to the registry. For the sake of simplicity and also to honor limitations on number of metrics that we can fetch at a time using Storage Insights REST API, we declare the following two metrics only.

var (
	// SI Metrics
	diskTotalDataRate     = prometheus.NewDesc("si_disk_total_data_rate", "Total Backend Data Rate.", []string{}, nil)
	diskTotalResponseTime = prometheus.NewDesc("si_disk_total_response_time", "Total Backend Response Time.", []string{}, nil)
)

Next, get the metric descriptors by calling Collect().

type myCollector struct{}

func (mc myCollector) Describe(ch chan<- *prometheus.Desc) {
	prometheus.DescribeByCollect(mc, ch)
}

The Collect() method

Finally, the Collect() method is in charge of producing current values for each metric during a scrape.

func (mc myCollector) Collect(ch chan<- prometheus.Metric) {
    // 1. Call Storage Insights REST API to get storage systems metrics
    metricsList, err := simetrics.FetchData()
	if err != nil {
		panic(err)
	}
    latest := metricsList[0].(map[string]interface{})
	
   // 2. Assign metric types (gauge, counters, summary, histogram, or unknown) to the returned metrics
    ch <- prometheus.MustNewConstMetric(diskTotalDataRate, prometheus.GaugeValue, float64(latest["disk_total_data_rate"].(float64)))
	ch <- prometheus.MustNewConstMetric(diskTotalResponseTime, prometheus.GaugeValue, float64(latest["disk_total_response_time"].(float64)))
}

Running the Exporter

The code is available on my Github. It consists of three Go modules.

• config module: It makes the application configuration available to the rest of the application. You can get your credentials by following instructions in the docs REST API.
• simetrics module: This module reads the REST API credentials and other details from the config module. It then makes REST API call to fetch token passing the API Key. As long as the token remains valid, it makes calls to metrics API to fetch the desired metrics value. If the token expires, it refreshes the API token.
• main module: The main module exposes the /metrics endpoint for Prometheus to scrape. When the scrape comes, it calls simetrics module to fetch the Storage Insights metrics inside the Collect() method.  

Once you've cloned the repo, provide your REST API authentication details in config.json.

Run the exporter, which causes it to listen on port 8089 (you can adjust the port using the -listen-address flag):

go run main.go --listen-address :8089 --config ../config.json

You can now navigate to the exporter's /metrics endpoint at http://<machine-ip>:8089/metrics.

Besides the default metrics included in the global metrics registry, the /metrics output should now contain the two Storage Insights metrics that we are exporting:

# HELP si_disk_total_data_rate Total Backend Data Rate.
# TYPE si_disk_total_data_rate gauge
si_disk_total_data_rate 889.47
# HELP si_disk_total_response_time Total Backend Response Time.
# TYPE si_disk_total_response_time gauge
si_disk_total_response_time 0.53

Visualizing metrics in Prometheus UI

Download Prometheus for Mac:

wget https://github.com/prometheus/prometheus/releases/download/v3.0.0-beta.0/prometheus-3.0.0-beta.0.darwin-amd64.tar.gz

Extract the tarball and change into the extracted directory:

tar xvzf prometheus-3.0.0-beta.0.darwin-amd64.tar.gz

cd prometheus-3.0.0-beta.0.darwin-amd64

Configure targets and scrape interval in prometheus.yml. Prometheus collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets - which is the exporter that we've build.

# my global config
global:
  scrape_interval: 120s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:8089"]

Start Prometheus with your newly created configuration file:

./prometheus

You should now be able to reach the Prometheus server at http://<machine-ip>:9090. We've prefixed the Storage Insights metrics with si_ as shown below.

Conclusion

We have now built a simple exporter that fetches the desired metrics of storage systems from Storage Insights and translates them into Prometheus metrics so that they can be scraped.