Introduction

When you have a slow performing application, it is not trivial to identify the source of the problem, especially if the application is composed of multiple services. Distributed tracing helps you to understand the flow of the requests and capture the time spent on each service. OpenTelemetry is a popular observability framework for capturing telemetry data including traces, metrics and logs. In this article, we will focus on tracing only.

WebSphere Application Server traditional (WAS) does not have OpenTelemetry feature built-in. It is actually very easy to add distributed tracing into WAS with OpenTelemetry automatic instrumentation. In this article, we will describe how to run a demo. We will also show you how to enable OpenTelemetry automatic instrumentation on WAS.

OpenTelemetry automatically instruments a variety of common Java libraries/frameworks, application servers, JVMs and operating systems. The complete list can be found here. OpenTelemetry automatic instrumentation uses a Java agent that can be attached to any Java 8 or later JVM. The agent dynamically injects byte code to your application to capture telemetry data. The data can be exported with OpenTelemetry protocol (OTLP), Jaeger, and Zipkin exporters.

If you are using some other distributed tracing backend that is not listed above, you can export the data with OTLP to an OpenTelemetry collector. OTLP is a vendor agnostic way of exporting telemetry data. The collector has 3 components, a receiver, a processor and an exporter. The receiver can accept a variety of protocols. The processor can handle batching, data filtering and tagging. The exporter can handle retries and encryption. The collector supports exporting data to many others backends. You can get a list of community contributed exporters in this link.

In distributed tracing terms, a trace represents a request in your application. A trace consists of one or many spans. A span is a unit of work with start/end time and some other metadata. We have provided a sample you can run on your own computer so that you can see the traces and spans from a WAS server using the OpenTelemetry agent.

Prerequisite

• Docker desktop installed

• git command installed

You can find the Docker desktop installation instructions here and the git command installation instructions here.

About the demo

The demo consists of 2 containers, WebSphere Application Server traditional and Jaeger. There is a sample JAX-RS application installed on WAS. The automatic instrumentation agent opentelemetry-javaagent.jar is also added to the WAS container and will be attached as the application server starts up.

The source code is located at system/src/main/java/io/openliberty/sample/system folder. A Jython script is located at system/src/main/config/installApp.py. The script installs the application and adds the OpenTelemetry automatic instrumentation agent opentelemetry-javaagent.jar as the Java agent.

The system/Dockerfile uses ibmcom/websphere-traditional:latest-ubi as the base image and it copies the application war file, automatic instrumentation agent jar file and the Jython script into the image. It runs the configuration command at the end.

The docker-compose.yml file at the root folder specifies to use the WAS image built from the Dockerfile and the Jaeger image from jaegertracing/all-in-one. The OpenTelemetry configurations are defined as environment variables under the WAS image.

OTEL_SERVICE_NAME=system

This specifies the name of the service. You will see this name on the Jaeger UI.

OTEL_TRACES_EXPORTER=otlp

This specifies to use the OTLP exporter to export the spans to a Jaeger backend.

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://jaeger:4317

This specifies the host and port of the the Jaeger backend.

OTEL_METRICS_EXPORTER=none

This specifies no metrics exporter.

The Jaeger image requires the environment variable COLLECTOR_OTLP_ENABLED=true to receive OTLP traces.

Getting Started

1. Open the sample GitHub page https://github.com/WASdev/sample.twas.opentelemetry.agent on a browser.

2. Click the Code button and a pop-up dialog appears

3. Click HTTPS and click the copy button beside the https://github.com/…. URL

4. Open a command window and type git clone and paste the URL

5. cd into the project directory

6. To build the sample, run mvn clean install

7. To build the WAS image, run docker compose build

8. To start the demo, run docker compose up -d. It will take a minute or two to start the containers.
   
   

Running the demo

1. Open the Jaeger UI http://localhost:16686/ and you should see “Service (0)”, which indicates there aren’t any traces in Jaeger yet.

2. Open the system application URL https://localhost:9443/system/properties and you should see a list of system properties about your computer.

3. Go back to the Jaeger UI and click the refresh button. Now, you should see “Service (2)”. Select system from the drop down menu and click the Find Traces button. You should see one trace with 3 spans in it.
   
   

   

   
   
   

4. Click the system: /properties to expand it. Click system /properties and system SystemResource.getProperties to expand them. You can see the duration of each span. You can also expand the Tags section to see the tags associated with the span. The first span was the HTTP GET request captured automatically by the agent. The second span was the JAX-RS request, also captured automatically by the agent. Optionally, you can create additional spans with the @WithSpan annotation for code that is not automatically captured. An additional Maven dependency of opentelemetry-instrumentation-annotations is needed. The third span was created using the @WithSpan annotation on the sleep() method in the SystemResource.java class. You can easily identify the third span was the bottleneck of this request. It slept for 358.41 ms.
   
   
   
   
   
   

5. You can access the WAS traditional administrative console at https://localhost:9043/ibm/console. Username is wsadmin and password can be obtained from running this command docker exec system cat /tmp/PASSWORD.

Cleaning Up

You can shut down the containers by running docker compose down and close all browser windows.

Manual Instructions to add the agent

If you want to try the automatic instrumentation on your own environment, you can download the latest version of the Java agent from the OpenTelemetry Java instrumentation Github release page. To add the Java agent manually to WAS,

1. Start WebSphere Application Server

2. Open administrative console in a browser

3. On the left pane, expand Servers

4. Expand Server Types

5. Click WebSphere Application Servers

6. On the right pane, click server1

7. Under the Server Infrastructure section, expand Java and Process Management

8. Click Process definition

9. Under the Additional Properties section, click Java Virtual Machine

10. Add -javaagent:/your-own-path/opentelemetry-javaagent.jar into the Generic JVM arguments text box

11. Click OK and Save to the master configuration

12. Restart the server

#WebSphereApplicationServer #OpenTelemetry #jaeger #docker