Applying workload specific observability configuration in Kubernetes or OpenShift clusters
Problem space
Modern IT organizations run infrastructure to serve multiple teams and deployment stages in a single Kubernetes or OpenShift clusters. Sometimes running the same level of observability throughout all deployed workloads is not desirable.
Use cases
1) As a cluster admin, I want to provide different access credentials for individual MySQL processes.
2) As an observability service provider, I want to control the amount of observability data and reduce observability costs for non-critical workloads.
3) As a SRE, I want to (temporarily) exclude certain workloads from monitoring.
All of the above use cases share the same underlying desire: The observability tooling should be able to address different configurations for individual workloads.
In IBM Instana, collecting observability data is mainly tight to our host Agent, which discovers host local processes and carries out monitoring of events, metrics and traces. Most of this is automatic and after installing Agents onto hosts and clusters, data starts to flow in form all workloads, including tracing. This applies to _all_ processes on a host, regardless of meta organization from container schedulers like Kubernetes or OpenShift. And configuration of sensors (for metrics collection) or tracers on Agent level applies to all instances of the sensor or tracer and thus to all workloads on a host and all worker nodes on a cluster.
Provide a default configuration for Agent deployment
Agent configuration is carried out by editing configuration.yaml in etc/instana directory of the Agent install path. For cluster deployments, this is done by providing the contents of configuration.yaml in the Helm chart's values.yaml or the Agent's CustomResource YAML.
The snippet looks the same in both situations. Please note that all other content is omitted for readability.
agent:
configuration_yaml: |
com.instana.plugin.mysql:
user: root
password: very_secure
com.instana.plugin.javatrace:
instrumentation:
enabled: true
sdk:
packages:
- my.custom.web.controller
- my.custom.rest.util
Explanation: The provided configuration.yaml defines settings for the MySQL sensor to connect to every instance using the credentials root/very_secure. Java tracer is configured to pick up Instana trace SDK annotations for two packages.
The previous static configuration will be applied to every MySQL instance and Java application respectively.
Provide dedicated workload configuration
As with the afore mentioned use cases, this might not be desirable in a shared environment.
In this case, the Agent can pull in process specific configuration and apply a different configuration for every single process it discovers.
For this to work, the contents of configuration.yaml need to be adopted to allow for placeholders and a directive to pick up parts of the configuration from the process scope. The general idea is that the workload itself provides the required configuration in either its environment or a process local file:
agent:
configuration_yaml: |
com.instana.plugin.mysql:
user: root
password:
configuration_from:
type: env
env_name: MYSQL_PASSWORD
default_value: very_secure
com.instana.plugin.javatrace:
instrumentation:
enabled:
configuration_from:
type: env
env_name: INSTANA_JAVA_TRACE_ENABLED
default_value: true
sdk:
packages:
configuration_from:
type: file
file_path: /instana-trace-sdk-packages
default_value:
- my.custom.web.controller
- my.custom.rest.util
In order to keep this tutorial crisp, the example above already shows both types of available directives: The blocks indicated by configuration_from will be replaced by the contents of the process local environment variable MYSQL_PASSWORD and for the Java tracer configuration with the contents of the file /instana-trace-sdk-packages. In addition, Java tracing can be disabled all together by setting the environment variable INSTANA_JAVA_TRACE_ENABLED to false.
While type: env for environment variable lookup is applicable to all deployment scenarios, e.g. workloads on a dedicated host or VM, the type: file example is directly targeted towards Kubernetes/OpenShift and containerized workloads, where every process has its own local filesystem.
Kubernetes ConfigMap to environment or file
Of cause, exposing ConfigMap data to container files is also supported by Kubernetes/OpenShift, as documented in the Add ConfigMap data to a Volume section.
Outlook
Of course, there is more to it than addressing individual Deployments or workloads: Large clusters are organized by providing namespaces with dedicated permissions and resource allocations. Observability requirements in the "staging" namespace may be different from the "production" namespace. Different teams or applications running in separated namespaces may have individual requirements as well.
Applying different observability patterns on a namespace level is an important and reoccurring ask from our customers. We have some exciting new features in the pipeline to address those requirements and provide even more flexibility when setting up and deploying IBM Instana for cluster observability. Stay tuned.