Instana

In today’s complex IT landscape, monitoring and observability are crucial aspects for maintaining high-performing and reliable systems. Tools like Instana and Grafana have emerged as industry leaders in this space and offering powerful solutions for monitoring, visualization, and analysis. One of the key integrations between Instana and Grafana is the Instana Grafana datasource, which allows users to leverage Instana’s rich monitoring data within Grafana’s versatile visualization platform. This blog post covers the Instana Grafana datasource, its benefits, and how to set it up.

Instana

Instana is an automated Application Performance Monitoring (APM) solution designed to provide real-time insights about the performance and health of modern cloud-native applications. With its AI-powered analytics and automatic discovery of application components, Instana offers a comprehensive view of your application stack, from microservices and containers to infrastructure and services.

Grafana

Grafana is an open-source platform for monitoring and observability, specializing in creating beautiful and interactive dashboards for time series data. It supports various data sources and offers a wide range of visualization options. This makes Grafana a popular choice among developers and operators for visualizing metrics, logs, and traces.

Instana Grafana Datasource

The Instana Grafana datasource serves as a connector between Instana’s comprehensive monitoring data and Grafana’s robust visualization platform. It allows users to fetch and display Instana metrics, traces, and other monitoring data within Grafana, and enable the creation of custom dashboards that provide insights into application performance, infrastructure health, and more.

Exploring the features of the Instana Grafana datasource integration

The Instana Grafana datasource integration offers a robust set of features designed to provide users with a seamless and powerful monitoring experience. This integration leverages Instana’s automated Application Performance Monitoring (APM) capabilities and Grafana’s versatile dashboarding tools to deliver comprehensive insights into application and infrastructure performance. In this section, Let’s delve into the key features of the Instana Grafana datasource, highlighting their benefits and practical applications.

Dynamic Focus queries

One of the standout features of the Instana Grafana datasource is its support for Dynamic Focus queries. Dynamic Focus allows users to dynamically filter and focus on specific application components, services, or infrastructure resources within their monitoring data. This feature enables users to drill down into specific areas of interest, facilitating more targeted analysis and troubleshooting.

Benefits

• Focused analysis: Quickly identify and analyze performance issues or anomalies within specific application components or services.
• Customized views: Create custom dashboards and visualizations that focus on the metrics and data relevant to your monitoring objectives.

Applications and websites monitoring

The Instana Grafana datasource extends its monitoring capabilities beyond infrastructure and services to include Applications and Websites. This feature enables users to monitor the performance, availability, and health of their applications and websites directly within Grafana, providing a holistic view of the entire application stack.

Benefits

• Comprehensive monitoring: Gain insights into the end-to-end performance and user experience of applications and websites.
• Unified dashboarding: Consolidate application, website, and infrastructure metrics into a single Grafana dashboard for centralized monitoring and analysis.

Automatic completion for available types and metrics:

To streamline the query-building process and enhance the user experience, the Instana Grafana datasource offers automatic completion for available types and metrics. As users type in the query editor, the datasource provides suggestions and auto-completes available types and metrics, reducing errors and accelerating the query creation process.

Benefits

• Efficient query building: Simplify the process of building queries by leveraging auto-completion and suggestions for available types and metrics.
• Error reduction: Minimize the query errors and syntax issues by utilizing the datasource’s built-in suggestions and validation.

Utilizes Instana REST API

The Instana Grafana datasource utilizes Instana’s REST API to fetch monitoring data, metrics, and insights from the Instana platform. By leveraging the REST API, the integration ensures reliable and efficient data retrieval, enabling real-time monitoring and analysis within Grafana.

Benefits:

• Real-Time Data Retrieval: Fetch and visualize up-to-date monitoring data and metrics from Instana within Grafana.
• Scalability: Benefit from the robustness and scalability of Instana’s REST API, ensuring seamless performance even in dynamic and large-scale environments.

Security with access token

Security is paramount in any monitoring and observability solution, and the Instana Grafana datasource prioritizes this by utilizing access tokens for authentication. User can generate and configure access tokens within Instana and use them to securely authenticate and establish a connection between Grafana and Instana.

Benefits:

• Secure authentication: Protect sensitive monitoring data and ensure secure communication between Grafana and Instana using access tokens.
• After the credentials are saved, the API Token is stored on the server. The user has the option to reset the API Token using the provided reset key Button

Setting up Instana Grafana Datasource

You need to set up Instana Grafana datasource locally.

Prerequisites

Before you set up Instana Grafana Datasource, ensure that the following prerequisites are met:

• Instana 261 or later
• Grafana 10.0.0 or later
• Instana Grafana Datasource 4.0.0 and later
• The Grafana and Instana instances are up and running
• Admin access to Grafana for installing plug-ins and configuring datasources
• Instana API credentials (API endpoint URL, API token) for connecting to the Instana datasource

Installing the plug-in on a local Grafana

For local instances, plug-ins are installed and updated through a simple CLI command. The plug-ins are not updated automatically, however you will be notified when updates are available in your Grafana instance.

1. Install the data source:

You can use any of the following options:

• Use the grafana-cli tool to install Instana plug-in from the command line:

grafana-cli plugins install instana-datasource

The plug-in will be installed into yourGrafana plug-in directory; the default location is /var/lib/grafana/plugins. For more information, see Plugins commands.

• Clone the repository directly from Git, run the following command:

git clone git@github.com:instana/instana-grafana-datasource.git

2. Configure Instana datasource in Grafana:

After installing the Instana plugin, configure the datasource to connect to your Instana instance. 

To configure the datasource:

1. Open Grafana in your web browser.
2. Navigate to Configuration > Data Sources > Add Datasource.
3. Select Instana from the list of available datasources.
4. Enter the following details:

• Name : A name for the datasource (Example: Instana).
• URL : The URL of your Instana installation (Example: https://instana.example.com).
• API Token : API token generated using your Instana instance. Once credentials are saved, the API Token is stored on the server. To reset the API Token, use the reset key button.
• Use-proxy: This option is enabled by default because the only way to use the API token for authentication in Grafana is through Use-Proxy. This option requires Grafana 10.0.1 or later and Instana datasource 4.0.1 or later
• Enable offline snapshots: For enabling querying offline snapshots. Needs Instana release 261 or later and Instana datasource 4.0.1 or later
• Enable Infrastructure Analyze category: Adds a new category that allows usage of Infrastructure Analyze functionality. This feature requires Instana 196 or later and the feature flag must be enabled. 
  If you are interested in this technology, you can submit a request to IBM Support.

5. Click on Save & Test to test the connection and save the datasource configuration.

After you save the datasource, the API Token is configured. To reset the authentication key, click the “Reset API Token” button.

Query Editor

You can submit the query on Instana-Grafana Data Source. The query editor is shown in the following example:

To start, input the Dynamic Focus query. This query must be identical to the query used in the Instana dashboard. You can validate your queries in Instana, and then copy and paste them into Grafana. Saved filters are not currently supported by the Grafana datasource plug-in.

Viewing metrics

You can view metrics on Instana Grafana Datasource.

To view metrics from each category, complete the following steps:

Infrastructure built-in metrics

• Enter your query.
  The “Available Types” list is displayed automatically.
• Select a type from the “Available Types” list.
  The “Available Metrics” dropdown is displayed automatically.
• Pick the metric you’re interested in.
  If your query results in multiple matches, the dataset will display metrics from all those instances. This will generate graphs with multiple plots, similar to the following example.

Infrastructure custom metrics

To select custom metrics that match your query, complete the following steps:

• Choose “Infrastructure custom metrics” from the category dropdown.
  This action will auto-fill the “Available Types” dropdown.
• Select type.
  The “Available Metrics” dropdown will populate automatically.
• Choose your desired metric from this list. Given the potentially large number of custom metrics, you can apply an optional filter to narrow down your selection to a specific subset.

If your Dynamic Focus query fields match, the relevant metrics are displayed as a graph similar to the following example.

Infrastructure Analyze metrics

For self-hosted installations, BeeInstana is required for this endpoint group. For instructions on enabling BeeInstana, see Configuration for the core.

Selecting Infrastructure Analyze Metrics:

• Choose “Infrastructure Analyze” from the category dropdown.
  The “Entity types” list is displayed with a list of all entity types sorted by entity name.
• Select an entity type,
  The “Available Metrics” dropdown will automatically populate.

Grouping and aggregation:

• Provide the appropriate “Group by” tag in the text box.
  Note: You can find the “Group by tag” option within the Instana dashboards.
• Select the appropriate one from the dropdown.
  Most metrics offer multiple aggregation types (for example, SUM, MEAN).

Filtering metrics:

You can the filter in Infrastructure Analyze.

To use this feature:

• Fill the “TagFilterExpression” text box with a tagFilterExpression containing an array of desired filter objects.
• Obtain the tagFilterExpression from the JSON tree under the API query session of the Instana Infrastructure Analytics dashboard.

Displaying metrics:

• If your selection matches the filter criteria, the returned dataset will include metrics, and graphs will be displayed as shown in the following example.

Application metrics

Users can view metrics related to one or multiple applications with this feature. After an application and a metric are selected, the graphs are generated.

Display options based on the application:

• The icon next to the application indicates the type of calls the displayed information is based on:
  a. INBOUND: Metrics based on the calls made by the customers
  b. ALL: Metrics based on all the calls made within the application, including both customer and internal calls.
  Note: These options are only enabled after an application is selected.

Service metrics

With this feature, you can view metrics associated with specific services. Services can be selected either individually or in conjunction with an Application Perspective to provide more detailed metric data.

Selection hierarchy:

• Service: Services can be selected independently or in combination with an Application Perspective.
• Application: Once an application is selected, the dropdown menu for possible services reloads to display only the services used by the selected application.

Display Options Based on Application: When an application is selected, users can specify the type of calls that the displayed information is based on:

• INBOUND: Metrics are based on calls made by consumers of the application.
• ALL: Metrics are based on all calls made within the application, including both customer and internal calls.

Service-Only Selection: If only a service is selected without choosing any application:

• The displayed information will be based on all the calls made within the selected service, including both customer and internal calls.

Endpoint metrics

The Endpoint Metrics Display feature allows users to view metrics associated with specific endpoints. Endpoints can be selected either individually or in conjunction with an Application Perspective and service to present more detailed metric data.

Selection hierarchy:

1. Application: It is advisable to first select an application to ensure the correct endpoint is chosen.
2. Service: After selecting an application, choose a service to further refine the endpoint selection.
3. Endpoint: Finally, select the desired endpoint.

Dynamic Selection: Selectable items like applications, services, and endpoints are reloaded and cached each time a service or application is changed.

Display Options Based on Application: When an application is selected, the user can specify the type of calls that the displayed information is based on:

• INBOUND: Metrics are based on calls made by consumers of the application.
• ALL: Metrics are based on all calls made within the application, including both customer and internal calls.

Endpoint-Only selection: If only an endpoint is selected without choosing any application:

The displayed information will be based on all calls made within the selected application, including both customer and internal calls. It is independent of the selected service.

Analyze application calls

To select the application metrics, follow these steps:

• Choose “Analyze application calls” from the category dropdown.
  This will populate the other dropdown lists.
  The “Application” dropdown will display a list of all applications, sorted by their name.
• Select the desired aggregation type.
  Most metrics offer multiple aggregation types (SUM, MEAN, etc.).
• Additional filters can be added via the “Add Filter” option. Multiple filters are combined using “AND”.

If your selections are correct, the returned dataset will display metrics and provide graphs, as shown in the following example.

Analyze websites

To select EUM website metrics, complete the following steps:

• Choose “Analyze websites” from the category dropdown.
  This will populate the other dropdown lists.
• The “Website” dropdown will display a list of all websites, sorted by their page loads.
• Select the desired aggregation type.
  Most metrics offer multiple aggregation types (SUM, MEAN, etc.).
• Additional filters can be added via the “Add Filter” option. Multiple filters are combined using “AND”.

If your selections are correct, the returned dataset will display metrics and provide graphs, as shown in the following example.

Analyze mobile app

To select EUM mobile app metrics, follow these steps:

• Choose “Analyze mobile app” from the category dropdown.
  This will populate the other dropdown lists.
• The “Mobile-app” dropdown will display a list of all mobile apps, sorted by their session starts.
• Select the desired aggregation type.
  Most metrics offer multiple aggregation types (SUM, MEAN, etc.).
• Additional filters can be added via the “Add Filter” option. Multiple filters are combined using “AND”.

If your selections are correct, the returned dataset will display metrics and provide graphs, as shown in the following example.

SLO information

To display specific SLO/SLI information in Grafana based on Instana:

• Enable the SLO Dashboards option in the Instana Grafana plugin settings.
  After enabling, the SLO/SLI category will appear alongside the others.

SLI visualization:

• Use the Gauge visualization.
• In Threshold settings, reverse the colors (red to green, green to red).
• Enter your SLO to set the threshold.
• Set the unit to “percent” in Field settings.

Remaining error budget:

• Use the Singlestat visualization.
• In Coloring settings, enable the Background toggle and set thresholds to 0,0.
• Invert the colors to turn the background green when there’s a positive remaining minute count.

Time series visualization:

• Choose the graph visualization.
• Select bars instead of lines for the draw mode.
• c. Set the Y-Max value to 1 for the left y-axis.

The dashboard is shown in the following example:

Time Shift

The Time Shift option enables the user to view the outcomes of queries from different points in time.

Usage:

• This feature is useful for comparing two identical queries.
• One query will display the current outcome, while the other will show the outcome from a day earlier.

Custom granularity support

This plug-in supports the ability to select different granularity values, offering an even deeper insight into metrics.

• The output of the query with a granularity of 1 minute is shown in the following example:

• The output of the query with a granularity of 5 minutes is shown in the following example:

Aggregating metrics

You can aggregate graphs at the query level and select one of the following options:

• Display everything, including the individual and aggregated graphs
• Show only the aggregated graph

Contributing to the Instana-Grafana data source

The Instana-Grafana data source is an open-source project.

How to contribute:

• To start contributing, see Instana-Grafana Data Source GitHub repository.
• To submit pull requests and contribute code, see Contribution Guidelines.

Conclusion

The integration of Instana with Grafana as a datasource offers a powerful solution for monitoring and observability, enabling you to gain deeper insights into your applications and infrastructure. By combining the strengths of Instana’s comprehensive monitoring capabilities with Grafana’s flexible visualization and extensibility, you can make data-driven decisions to optimize performance and ensure the reliability of your systems.

For more information on how to integrate and use Instana with Grafana, see Instana Documentation and Grafana Documentation.