Solving Instana DB2 Monitoring: When Local Discovery Fails Silently
Executive Summary
We recently encountered a perplexing issue where Instana's DB2 monitoring sensor was detecting all DB2 instances but creating zero inventories, resulting in no metrics collection. After extensive investigation with TRACE-level logging, we discovered the root cause: Instana is using an outdated Sigar library and sometimes it can not read process ownership information on modern Linux systems, combined with a discovery module bug that silently failed without logging any errors.
The solution: Switch from `local:` to `remote:` configuration, which bypasses process discovery entirely and uses pure JDBC connections.
***************
Update: Instana support has identified a fix and tested which resolves this issue. An updated discovery agent is pending release as of 12/2025. The information in this article is still a viable alternative in situations where remote setup might be needed.
Update: Instana support has created a new a new discovery-db2-1.0.66.jar that involves a fix that resolves this. Working and Resolved 01/2026. A new agent may have an updated version of this in a future release.
***************
---
The Problem: Silent Failure in DB2 Discovery
Initial Symptoms
Our Instana agent was exhibiting the following behavior:
- ✅ DB2 plugin enabled in configuration
- ✅ All 4 DB2 instances detected by process discovery
- ✅ Configuration file parsed successfully
- ✅ Instance names matched correctly (DB2INSTANCE environment variable)
- ❌ **Zero inventories created**
- ❌ **No DB2 sensors activated**
- ❌ **No metrics collected**
- ❌ **No error messages at any log level**
What Made This Challenging
The most frustrating aspect was the **complete lack of error messages**. Even with TRACE-level logging enabled, the discovery module would simply log:
```
DEBUG | DB2 | Detected local process: ... DB2INSTANCE=<instance_name> ...
DEBUG | DB2 | Creating database list to be monitored...
DEBUG | DB2 | Got instance from type:local ... instanceFromConfig:<instance_name>
DEBUG | DB2 | Created TOTAL 0 inventories per instance
```
No exceptions, no warnings, no hints about what went wrong.
---
The Investigation: Deep Dive with TRACE Logging
### Step 1: Enable Comprehensive Logging
We modified `/opt/instana/instana-agent/etc/org.ops4j.pax.logging.cfg` to enable TRACE logging:
# Main Instana logger
log4j2.logger.instana.level = TRACE
# DB2-specific loggers
log4j2.logger.db2discovery.name = com.instana.discovery.db2
log4j2.logger.db2discovery.level = TRACE
log4j2.logger.db2sensor.name = com.instana.sensor.db2
log4j2.logger.db2sensor.level = TRACE
### Step 2: Analyze the Discovery Flow
The TRACE logs revealed the complete discovery process:
1. **Process Detection** ✅ - All 4 DB2 processes found
2. **Environment Variable Parsing** ✅ - DB2INSTANCE correctly identified
3. **Configuration Matching** ✅ - Instance names matched config
4. **Process Owner Detection** ❌ - **userName=\<unknown\>**
5. **Inventory Creation** ❌ - Silently failed, returned 0
### Step 3: The Smoking Gun
The critical clue was in the process information:
```
userName=<unknown>, groupName=<unknown>
```
For comparison, Java processes showed:
```
userName=<lazy>, groupName=<lazy>
```
The `<lazy>` value indicates Sigar's lazy-loading mechanism works, but `<unknown>` means Sigar completely failed to determine the process owner.
Root Cause Analysis
The Outdated Sigar Library
Investigation revealed:
```bash
$ ls -la /opt/instana/instana-agent/lib/libsigar-amd64-linux.so
-rw-r--r--. 1 ID ID 246589 Mar 6 2013
$ ls -la /opt/instana/instana-agent/lib/sigar.jar
-rw-r--r--. 1 ID ID 435546 Oct 29 2024
```
#### Version Analysis
**sigar.jar** (Java library):
- **File date:** October 29, 2024 (recent)
- **Version:** 1.6.5.132-6-INSTANA-13
- **Title:** Instana - Agent - Sigar (patched)
- **Status:** ✅ This is Instana's patched version
**libsigar-amd64-linux.so** (Native library):
- **File date:** March 6, 2013 (11+ years old!)
- **Version:** SIGAR-1.6.5.0
- **Build date:** February 27, 2013 at 9:41 PM
- **Status:** ⚠️ Original outdated version from 2013
```bash
$ strings libsigar-amd64-linux.so | grep "SIGAR-1"
SIGAR-1.6.5.0, SCM revision exported, built 02/27/2013 09:41 PM as libsigar-amd64-linux.so
```
#### The Version Mismatch Problem
There's a critical **mismatch** between the Java library and native library:
- The **JAR file** is Instana's patched version from 2024
- The **native library** is the original unpatched version from 2013
**The ancient native library or the instana discovery code has compatibility issues with:**
- Modern Linux kernels
- Current process security models
- DB2's process initialization methods
- Modern security contexts and namespaces
This explains why the Java library can't properly interface with the native library to read process ownership information on modern systems.
### The Discovery Module Bug
The local discovery module has a flaw:
1. Requires process owner matching - Even when JDBC credentials are provided
2. Can't correctly identify yaml file configuration to match the process 100% of the time.
2. Fails silently - No error logging when userName is unknown
3. No fallback - Doesn't attempt JDBC authentication when process owner can't be determined
### Why This Affects DB2 Specifically
- DB2 processes (`db2sysc`) are started with specific security contexts
- The Sigar library's process enumeration seems to fail for these processes
- Java processes work fine because they use different initialization methods
- The issue is specific to the combination of old Sigar + modern Linux + DB2
---
## The Solution: Remote Configuration
### Understanding Configuration Types
Instana DB2 monitoring supports two configuration types:
1. **Local Configuration** - Uses process discovery + Sigar library
- Detects DB2 processes automatically
- Matches process owner to configured user
- **Fails when process owner cannot be determined**
2. **Remote Configuration** - Uses pure JDBC connections
- Connects directly to host:port
- No process discovery required
- **Works regardless of Sigar library issues**
### Implementation
#### Before (Local Configuration - Broken):
```yaml
com.instana.plugin.db2:
local:
- instance: '<instance_name>'
port: '<port_num>'
user: 'instana'
password: 'your_password'
databases:
- 'DBNAME'
```
#### After (Remote Configuration - Working):
```yaml
com.instana.plugin.db2:
remote:
- host: '127.0.0.1'
port: '<port_num>'
user: 'instana'
password: 'your_password'
poll_rate: 5
custom_polling:
disabled_metrics:
metrics: UNIT_OF_WORK_STATS
sslTrustStorePassword: 'jks_password'
sslTrustStoreLocation: '/opt/instana/instana-agent/keystore/jcekdb.jks'
databases:
- 'DBNAME'
```
### Key Differences
| Aspect | Local Config | Remote Config |
|------------------------|--------------|---------------|
| Process Discovery | Required | Not used |
| Sigar Dependency | Yes | No |
| Process Owner Matching | Required | Not needed |
| Connection Method | Via instance | Via host:port |
| JDBC Authentication | Secondary | Primary |
| Failure Mode | Silent | Logged |
## Results
### Before Fix
```
2025-11-11T09:28:43 | DEBUG | DB2 | Created TOTAL 0 inventories per instance
```
No sensors activated, no metrics collected.
### After Fix
```
2025-11-11T10:46:41 | INFO | DB2 | Activated Sensor: 127.0.0.1:<port>/<db_name>
2025-11-11T10:46:41 | INFO | DB2 | Activated Sensor: 127.0.0.1:<port>/<db_name>
2025-11-11T10:46:41 | INFO | DB2 | Activated Sensor: 127.0.0.1:<port>/<db_name>
2025-11-11T10:46:42 | INFO | DB2 | Activated Sensor: 127.0.0.1:<port>/<db_name>
```
All sensors activated successfully, metrics flowing to Instana backend.
---
## Step-by-Step Resolution Guide
### 1. Backup Your Configuration
```bash
cp /opt/instana/instana-agent/etc/instana/configuration-db2info.yaml \
/opt/instana/instana-agent/etc/instana/configuration-db2info.yaml.backup-$(date +%Y%m%d-%H%M%S)
```
### 2. Convert to Remote Configuration
Edit your DB2 configuration file and change from `local:` to `remote:`:
```yaml
com.instana.plugin.db2:
remote:
- host: '127.0.0.1' # Use localhost for local DB2 instances
port: 'YOUR_DB2_PORT'
user: 'YOUR_MONITORING_USER'
password: 'YOUR_PASSWORD'
poll_rate: 5
sslTrustStorePassword: '<jks_password>'
sslTrustStoreLocation: '/opt/instana/instana-agent/keystore/jcekdb.jks'
databases:
- 'YOUR_DATABASE_NAME'
```
Repeat the configuration block for each DB2 instance you want to monitor.
### 3. Verify Configuration Syntax
Ensure proper YAML indentation (use spaces, not tabs):
- `remote:` should be indented 2 spaces under `com.instana.plugin.db2:`
- Each list item should start with `- ` (dash + space)
- All properties should be properly aligned
### 4. Deploy and Monitor
The Instana agent will automatically reload the configuration within seconds. Monitor the logs:
```bash
tail -f /opt/instana/instana-agent/data/log/agent.log | grep -E "(DB2|Activated|127.0.0.1)"
```
### 5. Verify Success
Look for these messages:
```
INFO | DB2 | Activated Sensor: 127.0.0.1:PORT/DATABASE
```
You should see one activation message for each database configured.
---
## Lessons Learned
### 1. Silent Failures Are the Worst
The discovery module's silent failure made troubleshooting extremely difficult.
### 2. Legacy Dependencies Can Cause Modern Problems
An 11-year-old native library (Sigar) was incompatible with modern systems.
### 3. TRACE Logging Is Essential
Without TRACE-level logging, we would never have discovered the `userName=<unknown>` issue.
### 4. Configuration Flexibility Matters
Having both `local:` and `remote:` configuration options provided a workaround.
### 5. Documentation Gaps
The Instana documentation didn't clearly explain when to use `local:` vs `remote:` configuration, or the Sigar dependency.
---
## Conclusion
What appeared to be a simple configuration issue turned out to be a complex interaction between an outdated native library, modern Linux security models, and insufficient error handling. The solution—switching to remote configuration—is actually more reliable and maintainable than the local discovery approach.
**Key Takeaway:** When monitoring critical infrastructure like databases, prefer explicit configuration (remote) over automatic discovery (local) for reliability and debuggability.
If you're experiencing similar issues with Instana DB2 monitoring, try the remote configuration approach. It bypasses the entire process discovery mechanism and provides a more stable monitoring solution.
---
## Additional Resources
---
## Questions or Comments?
Have you encountered similar issues with Instana or other monitoring tools? Share your experiences in the comments below!
---
**About the Author:** Greg Sorensen is a technical professional specializing in enterprise monitoring and database infrastructure. This investigation was conducted while troubleshooting production monitoring issues, and the detailed analysis and solution have been shared to help others facing similar challenges.
**Last Updated:** November 11, 2025
#Alerting
#Infrastructure
#Database