Introduction
Achieving true observability in enterprise integration landscapes is no longer optional, it’s a necessity for operational excellence, rapid troubleshooting, and proactive optimization. Yet, for many MuleSoft customers, advanced telemetry features to export data to external observability services that would allow for automatic correlation between logs and traces are locked behind expensive licensing tiers like the Anypoint Integration Advanced Package or the Titanium Subscription . This creates a significant gap: organizations need deep visibility, but budgets and compliance requirements often restrict access to premium add-ons.
In this post, I’ll share a practical, standards-based approach to unlocking full-stack observability for MuleSoft Applications, running on CloudHub, in Instana, without the need for premium licensing. Drawing from hands-on experience in real customer environments, I’ll walk you through how to leverage open standards (OpenTelemetry, Fluent Bit) and Instana’s flexible ingestion capabilities to automatically correlate logs and traces. This enables you to:
- Visualize distributed transactions end-to-end across your MuleSoft APIs and microservices
- Accelerate root cause analysis by linking logs and traces in a single pane of glass
- Avoid vendor lock-in and reduce total cost of ownership
If you’re running MuleSoft on CloudHub, this is your guide that provides steps and lessons learned from the field. You’ll see how to instrument your Mule apps, configure log forwarding, and validate that Instana is capturing and correlating your telemetry.
Solution Overview
Our goal: enable end-to-end observability for MuleSoft applications in Instana, leveraging OpenTelemetry (OTel) and Fluent Bit, without relying on premium features.
Architecture Summary:
- MuleSoft APIs instrumented with the Avio OpenTelemetry module and custom log appenders.
- Traces are exported directly form Mule Apps to Instana Backend
- Logs are exported to a Linux VM running both Fluent Bit and the Instana OpenTelemetry Collector.
- This is needed as we cannot export Otel Logs directly from Mule Apps.
- In a Production setup, these two component need to be deployed in HA, so the best way would be on a Kubernetes cluster.
- Data flows into Instana SaaS for correlation and analysis.
Architecture Diagram
Prerequisites
Before you start, ensure you have:
- Access to Anypoint Platform (CloudHub or local deployment).
- Free trial is available to test it
- A Linux VM (for Fluent Bit and Instana OTel Collector).
- Instana API key and Instana Backend available
- Free trial is available at https://www.instana.com/trial/
- Fluent Bit package for your Linux distribution.
- Basic Linux shell access (sudo privileges).
Step-by-Step Implementation
Instrumenting MuleSoft for Traces
Add the Avio OpenTelemetry module to your Mule app, in your pom.xml add the lines in bold:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany</groupId>
<artifactId>hellomule-instana</artifactId>
<version>1.0.0-SNAPSHOT</version>
<packaging>mule-application</packaging>
<name>hellomule-instana-gio</name>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<app.runtime>4.10.0</app.runtime>
<mule.maven.plugin.version>4.5.3</mule.maven.plugin.version>
<avio.otel.version>2.11.1</avio.otel.version>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-clean-plugin</artifactId>
<version>3.2.0</version>
</plugin>
<plugin>
<groupId>org.mule.tools.maven</groupId>
<artifactId>mule-maven-plugin</artifactId>
<version>${mule.maven.plugin.version}</version>
<extensions>true</extensions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.mule.connectors</groupId>
<artifactId>mule-http-connector</artifactId>
<version>1.10.5</version>
<classifier>mule-plugin</classifier>
</dependency>
<dependency>
<groupId>org.mule.connectors</groupId>
<artifactId>mule-sockets-connector</artifactId>
<version>1.2.7</version>
<classifier>mule-plugin</classifier>
</dependency>
<dependency>
<groupId>com.avioconsulting.mule</groupId>
<artifactId>mule-opentelemetry-module</artifactId>
<version>${avio.otel.version}</version>
<classifier>mule-plugin</classifier>
</dependency>
</dependencies>
<repositories>
<repository>
<id>anypoint-exchange-v3</id>
<name>Anypoint Exchange</name>
<url>https://maven.anypoint.mulesoft.com/api/v3/maven</url>
<layout>default</layout>
</repository>
<repository>
<id>mulesoft-releases</id>
<name>MuleSoft Releases Repository</name>
<url>https://repository.mulesoft.org/releases/</url>
<layout>default</layout>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>mulesoft-releases</id>
<name>MuleSoft Releases Repository</name>
<layout>default</layout>
<url>https://repository.mulesoft.org/releases/</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
</project>
Update your Mule configuration XML, add the lines in bold:
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:avio-logger="http://www.mulesoft.org/schema/mule/avio-logger"
xmlns:opentelemetry="http://www.mulesoft.org/schema/mule/opentelemetry"
xmlns:http="http://www.mulesoft.org/schema/mule/http"
xmlns="http://www.mulesoft.org/schema/mule/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/opentelemetry http://www.mulesoft.org/schema/mule/opentelemetry/current/mule-opentelemetry.xsd
http://www.mulesoft.org/schema/mule/avio-logger http://www.mulesoft.org/schema/mule/avio-logger/current/mule-avio-logger.xsd">
<opentelemetry:config name="OTel_Config" serviceName="hellomule-instana-gio" spanAllProcessors="true">
<opentelemetry:exporter>
<opentelemetry:otlp-exporter collectorEndpoint="https://otlp-<region>-saas.instana.io:4318" protocol="HTTP_PROTOBUF">
<opentelemetry:headers>
<opentelemetry:header key="x-instana-key" value="INSTANA_AGENT_KEY" />
</opentelemetry:headers>
</opentelemetry:otlp-exporter>
</opentelemetry:exporter>
</opentelemetry:config>
<http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config" doc:id="87af2094-ad0a-48aa-a766-6c425b69f484" >
<http:listener-connection host="0.0.0.0" port="8081" />
</http:listener-config>
<flow name="hellomule-instanaFlow" doc:id="faf2b496-4b20-45c1-849a-6b9eef783b68" >
<http:listener doc:name="Listener" doc:id="c8286e4c-59a3-402d-8a79-075afd22651b" config-ref="HTTP_Listener_config" path="/hellomule"/>
<set-payload value='"Hello Mule and Instana!"' doc:name="Set Payload" doc:id="38510d83-8bae-4c6e-a83a-7843e9496a91" />
<logger level="INFO" doc:name="Logger" doc:id="475f6a73-48bf-4a9d-9394-6e5a216439da" message="#[attributes.requestPath]"/>
</flow>
</mule>
Exporting Logs
Configure log4j2.xml, add an HTTP appender targeting your Fluent Bit instance, Fluent-bit configuration in bold:
<?xml version="1.0" encoding="utf-8"?>
<Configuration>
<!--These are some of the loggers you can enable.
There are several more you can find in the documentation.
Besides this log4j configuration, you can also use Java VM environment variables
to enable other logs like network (-Djavax.net.debug=ssl or all) and
Garbage Collector (-XX:+PrintGC). These will be append to the console, so you will
see them in the mule_ee.log file. -->
<Appenders>
<RollingFile name="file" fileName="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}hellomule-instana.log"
filePattern="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}hellomule-instana-%i.log">
<PatternLayout pattern="%-5p %d [%t] [trace_id:%X{trace_id}; span_id:%X{span_id}; processor:%X{processorPath}; event:%X{correlationId}] %c: %m%n"/>
<SizeBasedTriggeringPolicy size="10 MB"/>
<DefaultRolloverStrategy max="10"/>
</RollingFile>
<Http name="FluentBitOTLP"
url="http://<fluent-bit-endpoint>:8888/v1/logs"
method="POST"
connectTimeoutMillis="5000"
readTimeoutMillis="5000">
<Property name="Content-Type" value="application/json"/>
<JsonLayout complete="false"
compact="true"
eventEol="true"
properties="true"
includeStacktrace="true"
stacktraceAsString="true">
<!-- static fields -->
<KeyValuePair key="app" value="hellomule-instana-gio"/>
<KeyValuePair key="env" value="dev"/>
<KeyValuePair key="host" value="${hostName}"/>
<!-- Dynamic fields are automatically included thanks to properties="true":
level, loggerName, message, thread, timestamp, mule_correlation_id, mule_processor_path -->
</JsonLayout>
</Http>
</Appenders>
<Loggers>
<!-- Http Logger shows wire traffic on DEBUG -->
<AsyncLogger name="org.mule.service.http.impl.service.HttpMessageLogger" level="DEBUG"/>
<AsyncLogger name="org.mule.service.http" level="WARN"/>
<AsyncLogger name="org.mule.extension.http" level="WARN"/>
<!-- Mule logger -->
<AsyncLogger name="org.mule.runtime.core.internal.processor.LoggerMessageProcessor" level="INFO"/>
<AsyncRoot level="INFO">
<AppenderRef ref="file"/>
<AppenderRef ref="FluentBitOTLP"/>
</AsyncRoot>
</Loggers>
</Configuration>
Linux VM Setup
Fluent Bit Configuration
· Edit /etc/fluent-bit/fluent-bit.conf:
[INPUT]
Name http
Listen 0.0.0.0
Port 8888
Tag mule.logs
[FILTER]
Name nest
Match *
Operation lift
Nested_under contextMap
Add_prefix ctx_
# Remove noise in log
[FILTER]
Name modify
Match *
Remove instant
Remove thread
Remove threadId
Remove threadPriority
Remove loggerFqcn
Remove date
Remove endOfBatch
Remove ctx_processorPath
Remove loggerName
# Add custom tags
[FILTER]
Name modify
Match *
Copy app service_name
[OUTPUT]
Name stdout
Match *
Format json_lines
[OUTPUT]
Name opentelemetry
Match *
Host localhost
Port 24318
Logs_uri /v1/logs
Log_response_payload True
logs_body_key log
logs_body_key_attributes true
logs_span_id_message_key ctx_span_id
logs_trace_id_message_key ctx_trace_id
logs_severity_text_message_key level
· Restart Fluent Bit:
systemctl restart fluent-bit
systemctl status fluent-bit
journalctl -u fluent-bit.service -f
Instana OTel Collector Configuration
Edit /opt/instana/collector/config/config.yaml as needed. Default configuration after installing will work
- If changes were made restart the collector:
/opt/instana/collector/bin/instana_collector_service.sh restart
tail -f /opt/instana/collector/logs/agent.log
ss -tuln
Deployment and Validation
- Deploy your Mule app to Anypoint Platform (via Studio or manual JAR upload). Use the official tutorial in the References (4) to create a mule app.
Testing it with the URL provided in the Anypoint UI
- Check Instana for incoming traces and logs. To generate traces I ran a simple script to call the URL
while true; do
RESPONSE=$(curl -s -w "HTTP_CODE:%{http_code}\n" "$URL")
echo "$(date '+%Y-%m-%d %H:%M:%S') - $RESPONSE"
sleep 1
done
Observability in Action
Application Perspective
Now if I go to my Instana UI, in Applications , under the Service tab, I can look for my application that’s called “hello-mule”
Previously, I created an Application Perspective in Instana. Think of an Application Perspective as a logical grouping that defines what “your application” looks like inside your organization. It’s not just a single service, it’s a way to bring together all the components that make up an app so you can monitor them as one entity.
For this example, I kept it simple: I created an Application Perspective called Hello Mule. To do that, I told Instana to use the service name “hello-mule” as the identifier. This means any traces or logs tagged with that service name will automatically be included in the Hello Mule perspective, giving me a unified view of its performance and health.
Filtered Calls and 100% Trace Collection
Now if I click “Analyze calls” I can dig depper into the traces
Here, we’re looking at the list of calls already filtered by the Application Perspective. This means Instana is only showing the transactions that belong to the “Hello Mule” application we defined earlier, giving us a focused view of its activity.
One key advantage: Instana collects 100% of the traces, no sampling. Unlike other solutions that often sample data and risk missing critical details, Instana captures every trace. This ensures complete visibility into your MuleSoft flows, so you can confidently analyze performance and troubleshoot issues without gaps.
Span Details
When you click one of them you’ll see the trace details. In the next view, we’re looking at a specific trace with full details for each span. This breakdown shows the timing information for every step in the transaction, from the initial HTTP listener to downstream processors. By drilling into these spans, you can quickly identify where latency occurs, which components are performing well, and where potential bottlenecks exist.
This level of granularity is critical for MuleSoft flows because it gives you visibility into the entire execution path, not just the overall response time.
Logs in Context and Smart Alerts
In this view, we’re exploring Instana’s Logs in Context feature. It automatically gathers logs from any source and ties them directly to the related trace spans. This means when you investigate a transaction, you see both the trace timeline and the exact logs that occurred at each step, making troubleshooting faster and more precise.
On top of that, Instana applies smart alerts on logs, detecting anomalies and error patterns proactively. Instead of waiting for issues to escalate, you get real-time notifications when log behavior deviates from normal, helping teams act before problems impact users.
Service Name Filtering
By filtering on the service name hello-mule, we can now see all logs generated by our Mule application. This focused view ensures that every log entry tied to the Hello Mule service is visible and correlated with its respective traces. Combined with Instana’s smart alerts, this makes it easy to detect anomalies or error patterns in real time, so you can quickly act on issues before they impact your integration flows.
Automatic Log-Trace Correlation
When we drill into a specific log entry, we can see it is automatically correlated with the corresponding trace. This means you don’t have to jump between tools or manually match IDs, Instana links logs and traces for you. From the trace page, you can instantly view the related log details, making troubleshooting faster and more accurate whenever issues occur in your MuleSoft flows.
Lessons Learned & Recommendations
- Version Compatibility: We encountered issues with mismatched versions of Fluent Bit and the OTel Collector. Always check compatibility and review release notes before upgrading.
- Configuration Pitfalls: Small typos in config files (especially YAML and XML) can silently break the pipeline. Validate configs and use verbose logging during setup.
- Trace/Log Correlation: Ensuring trace and log IDs are properly propagated and mapped is critical for end-to-end visibility in Instana. Test with sample transactions and verify in the Instana UI.
- Scaling and Performance: For high-throughput environments, monitor resource usage on the Linux VM and consider tuning Fluent Bit’s buffer and batching settings.
- Security: Never expose sensitive credentials in public configs. Use environment variables or secret managers where possible.
Considering Alternatives: IBM WebMethods Hybrid Integration
While this approach unlocks observability for MuleSoft without premium licensing, it’s important to consider the broader integration landscape. If your organization is evaluating alternatives, https://www.ibm.com/products/webmethods-hybrid-integration offers a unified platform for integrating applications, data, and services—on-premises or in the cloud.
Key advantages of IBM WebMethods Hybrid Integration:
- Native observability for every component deployed or managed, regardless of environment
- Centralized monitoring and management across hybrid architectures
- Built-in support for automatic correlation of logs, traces, and metrics
- Flexible deployment options to meet compliance and operational requirements
If you’re looking for a solution that provides comprehensive observability out-of-the-box, with seamless integration and management capabilities, IBM WebMethods Hybrid Integration is worth exploring as a strategic alternative to MuleSoft.
References
- https://avioconsulting.github.io/mule-opentelemetry-module/
- https://www.ibm.com/docs/en/instana-observability/1.0.308?topic=collectors-instana-distribution-opentelemetry-collector
- https://docs.fluentbit.io/manual/3.1/pipeline/outputs/opentelemetry
- https://docs.mulesoft.com/mule-runtime/latest/mule-app-dev-hellomule
- https://github.com/IsReal8a/instana-examples/tree/main/mule-app-hello-world-otel
Contributors
Nuria Alonso Gonzalez - nuria.alonso@ibm.com
Qianqian Ye - Qianqian.Ye1@ibm.com
Mario Gomez Marting - MarioGomez@ibm.com
#Competitive
#OpenTelemetry
#Tracing
#CaseStudy
#Tutorial