Problem space
The Instana Agent requires secret values to communicate with Instana's backend, perform dynamic sensor updates, and connect to external resources at runtime.
In this blog post, I’ll show how to avoid hardcoding credentials and other sensitive information in the Agent’s configuration by integrating with secrets managers.
Disclaimer: The example setup of HashiCorp Vault in this post is for demonstration purposes only and should not be used in production environments.
Main concept
Sensitive information in the Instana Agent’s configuration can be externalized using environment variables or local files.
Secrets managers can export secrets into environment variables or files, depending on the vendor. While not all secrets managers support this natively, open-source tools can help bridge the gap.
Note: Environment variable and file-based configuration is not limited to secrets — it applies to all aspects of agent and sensor configuration. More details can be found in this earlier blog post .
Agent configuration files
The concept of reading sensitive information from agent local environment variables or files applies to nearly every aspect of the agent configuration, but the notation varies and depends on the configuration option and the agent deployment model. All the files that are mentioned in the following paragraph are based in the agent's installation folder under <agent-install-folder>/etc. For example, the file mvn-settings.xml is based in the <agent-install-folder>/etc folder, while instana/configuration.yaml is at <agent-install-folder>/etc/instana/configuration.yaml. The following table lists typical options that require sensitive information and how to externalize the configuration in either an environment variable, a file, or both.
| Configuration option |
configuration file |
env/file |
notation |
| Repository password |
mvn-settings.xml |
env |
<password>${env.INSTANA_AGENT_KEY}</password> |
| Repository proxy password |
mvn-settings.xml |
env |
<password>${env.INSTANA_REPO_PROXY_PWD}</password> |
| Backend agent key |
instana/com.instana.agent.main.sender.Backend.cfg |
env |
key=${env:INSTANA_AGENT_KEY} |
| Backend proxy password |
instana/com.instana.agent.main.sender.Backend.cfg |
env |
proxy.password=${env:INSTANA_PROXY_PWD} |
| Sensor access credentials |
instana/configuration.yaml |
env, file |
configuration_from: { type: agent_env, env_name: MY_ACCESS_CREDENTIALS } |
To summarize the main differences, the file mvn-settings.xml uses the ${env.MY_ENV_VAR} notation, all *.cfg files use the ${env:MY_ENV_VAR} notation (. vs :) and all references to environment variables or files in instana/configuration.yaml[1] require the configuration_from: structure, which allows for powerful agent or workload-specific environment variable expansion or reading contents from a file.
[1]: configuration.yaml can be split into multiple files configuration-XYZ.yaml, with "XYZ" free to choose, and the expansion options apply to each of these.
Different agent deployment models
The integration approach depends on how the Instana Agent is deployed:
- Host-based installations (e.g., via RPM, DEB, or Windows installer): Configuration files must be edited directly, as listed [above](#agent-configuration-files).
- Containerized deployments (e.g., Docker or Kubernetes): Configuration is typically managed via environment variables, which are already exposed for container flexibility. For a list of available environment variables, refer to the Instana Agent Docker repository.
Secrets Managers export to environment or file
Secrets managers store sensitive information securely within your infrastructure. To make this information available to the Instana Agent, a client must retrieve the secrets and expose them as environment variables or files.
The setup and authentication process for the secrets manager client varies by vendor and is beyond the scope of this post. However, the goal remains the same: ensure the client retrieves the required secrets and makes them accessible to the Agent in a supported format.
In the next section, I’ll walk through simplified examples. Note: These examples stripped down on the secrets manager side in order to keep the blog post tidy. The setup is not secure and not intended for production use.
Example for host deployment
The following example is built around three main components:
- Hashicorp Vault running in development mode inside a container
- A simple shell script to access secrets from Vault
- The Instana Agent with configuration files
Setting up Hashicorp Vault (Development Mode)
First, we run Vault in development mode. In this mode, Vault:
- Runs entirely in memory
- Is unsealed by default
- Accepts a root token specified at startup
Warning: This setup is inherently insecure and should only be used for development. Never run Vault in production without following the official documentation to secure your environment.
Start Vault with:
sudo podman run -e 'VAULT_DEV_ROOT_TOKEN_ID=myroot' -e 'VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200' -p 8200:8200 hashicorp/vault
I set the root token to myroot and expose port 8200, so I can access the service from my local host.
The following shell script on my local host will create a new secrets entry for the Instana agent key and a MySQL access password. The agent key is required to authenticate with the Instana backend and for dynamic agent updates from Instana's Maven repository. The MySQL access password is used by the MySQL sensor to pull metrics from a local MySQL database instance.
The following command sends the JSON payload to the Vault service, which is running the [KV secrets engine V2](https://developer.hashicorp.com/vault/api-docs/secret/kv/kv-v2#kv-secrets-engine-version-2-api) on port 8200:
curl \
-H "X-Vault-Token: myroot" \
-H "Content-Type: application/json" \
-X POST \
-d '{"data":{"agent-key":"anpnV6rEQ2az0DFzCQws57", "mysql-password":"very-secret-pwd"}}' \
http://127.0.0.1:8200/v1/secret/data/agent
With the agent key and MySQL password saved to the Vault service, we can move on and integrate it with the Agent startup.
Shell script to access the secrets
In order to access the secrets from vault, a simple HTTP GET request and some parsing with jq is enough. This can be done for any additional secret by adjusting the JSON path jq needs to parse:
# Access the agent-key
curl -s \
-H "X-Vault-Token: myroot" \
http://127.0.0.1:8200/v1/secret/data/agent | jq -r '.data.data."agent-key"'
# Access the MySQL password
curl -s \
-H "X-Vault-Token: myroot" \
http://127.0.0.1:8200/v1/secret/data/agent | jq -r '.data.data."mysql-password"'
Agent configuration
In mvn-settings.xml the following XML snippet is required in the <servers> section:
<servers>
<server>
<id>features</id>
<username>_</username>
<password>${env.INSTANA_KEY}</password>
</server>
<server>
<id>shared</id>
<username>_</username>
<password>${env.INSTANA_KEY}</password>
</server>
Note, how the actual password is referenced from the INSTANA_KEY environment variable.
In the file instana/com.instana.agent.main.sender.Backend.cfg, we reference the agent key for the Instana backend connection by replacing key=<> with the following snippet:
key=${env:INSTANA_KEY}
Finally, to allow the MySQL sensor to connect to the database instance for metrics collection, we modify instana/configuration.yaml end edit the entry for com.instana.plugin.mysql to pick up the value for the password property from the environment variable INSTANA_MY_SQL_PASSWORD:
com.instana.plugin.mysql:
user: 'the-mysql-user'
password:
configuration_from:
type: agent-env
env_name: INSTANA_MY_SQL_PASSWORD
Agent startup
In order to streamline secrets access in this example, I create the following shell script to encapsulate the Vault access:
tee vault.sh <<'EOF'
#!/bin/bash
echo $(curl -s -H 'X-Vault-Token: myroot' 'http://127.0.0.1:8200/v1/secret/data/'"$1" | jq -r ''.data.data.'''"'$2'"')
EOF
chmod +x vault.sh
Now we can prepend the agent startup and export the required secrets into the respective environment variables:
INSTANA_KEY=$(./vault.sh agent agent-key) \
INSTANA_MY_SQL_PASSWORD=$(./vault.sh agent mysql-password) \
/opt/instana/instana-agent/bin/start
Here it becomes clear, the real key and password are only ever exposed to the agent process and security is shifted into the authentication and handling of the vault access token. The same applies for the native Vault client as for any other secrets manager solution.
Example for Kubernetes deployment
The same basics steps apply when running the setup in a Kubernetes or OpenShift environment. In order to skip the lengthy and complex secrets manager setup, I assume reading sensitive information in the cluster is already solved and that secrets can be written into Kubernetes Secret instances.
Leveraging the Kubernetes feature to mount Secrets into environment variables or files, and the ability of Instana agent operator (and Helm chart) to utilize the same when deploying the agent, the exercise is merely to put all pieces together.
Steps:
- Populate Kubernetes Secrets
- Configure the Instana Agent
- Use
agent.keysSecret in the Helm chart or CustomResource to reference the secret.
- Edit
configuration.yaml to reference environment variables or mounted files.
- Mount Secrets into the Agent Pod
- Use the
agent.pod.env property to define environment variables from Secrets. See the agent operator repository for an example.
Assuming the example for host deployment, the adapted agent CustomResource looks like this. Please note, the agent.keysSecret references a Kubernetes Secret which adheres to the agent specification. The MySQL password is stored in the secret mysql-secret under the key password.
apiVersion: instana.io/v1
kind: InstanaAgent
metadata:
name: instana-agent
namespace: instana-agent
spec:
zone:
name: my-zone
cluster:
name: my-cluster
agent:
keysSecret: instana-agent-key-secret
endpointHost: ingress-red-saas.instana.io
endpointPort: "443"
configuration_yaml: |
com.instana.plugin.mysql:
user: 'the-mysql-user'
password:
configuration_from:
type: agent-env
env_name: INSTANA_MY_SQL_PASSWORD
pod:
# New environment variables feature with full Kubernetes EnvVar support
env:
# From secret
- name: INSTANA_MY_SQL_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: password
optional: true
Agent connection setup and configuration.yaml
Properties in the sensor specific `configuration.yaml` file can be edited in place and reference mounted secrets directly. For agent connection settings, like proxies or self-hosted Maven repositories, it's best to use the already exposed environment variables in the agent's docker image. Here is a list of exposed environment variables which can be considered sensitive information and that can be mounted from Kubernetes Secrets directly:
- INSTANA_AGENT_PROXY_USER
- INSTANA_AGENT_PROXY_PASSWORD
- INSTANA_GIT_REMOTE_USERNAME
- INSTANA_GIT_REMOTE_PASSWORD
- INSTANA_REPOSITORY_PROXY_USER
- INSTANA_REPOSITORY_PROXY_PASSWORD
Outlook
Currently, host-based and Kubernetes-based deployments require different approaches to secret management. In the future, we aim to unify these by:
- Offering better external configuration options, independent from the deployment model.
- Supporting direct SecretKeyRef entries for all agent properties.
- Avoiding the need for environment variables by injecting secrets directly into configuration files.
This will enhance security by keeping secrets out of the agent’s environment and leveraging Kubernetes-native mechanisms for secret injection.
#Agent
#Kubernetes
#Integration
#SRE