opentelemetry-tracing: OpenTelemetry Tracing QuickStart

The opentelemetry-tracing quickstart demonstrates the use of the OpenTelemetry tracing specification in WildFly.

What is it?

OpenTelemetry is a set of APIs, SDKs, tooling and integrations that are designed for the creation and management of telemetry data such as traces, metrics, and logs. OpenTelemetry support in WildFly is limited to traces only. WildFly’s support of OpenTelemetry provides out of the box tracing of Jakarta REST calls, as well as container-managed Jakarta REST Client invocations. Additionally, applications can have injected a Tracer instance in order to create and manage custom `Span`s as a given application may require. These traces are exported to an OpenTelemetry Collector instance listening on the same host.

Architecture

In this quickstart, we have a collection of CDI beans and REST endpoints that expose functionalities of the OpenTelemetry support in WildFly.

Use of the WILDFLY_HOME and QUICKSTART_HOME Variables

In the following instructions, replace WILDFLY_HOME with the actual path to your WildFly installation. The installation path is described in detail here: Use of WILDFLY_HOME and JBOSS_HOME Variables.

When you see the replaceable variable QUICKSTART_HOME, replace it with the path to the root directory of all of the quickstarts.

Prerequisites

To complete this guide, you will need:

less than 15 minutes
JDK 11+ installed with JAVA_HOME configured appropriately
Apache Maven 3.5.3+
Docker Compose, or alternatively Podman Compose

Use of the WILDFLY_HOME and QUICKSTART_HOME Variables

When you see the replaceable variable QUICKSTART_HOME, replace it with the path to the root directory of all of the quickstarts.

Steps

Start the WildFly Standalone Server

Open a terminal and navigate to the root of the WildFly directory.
Start the WildFly server with the default profile by typing the following command.
```
$ WILDFLY_HOME/bin/standalone.sh 
```
Note
For Windows, use the WILDFLY_HOME\bin\standalone.bat script.

Configure the Server

You enable OpenTelemetry by running JBoss CLI commands. For your convenience, this quickstart batches the commands into a configure-opentelemtry.cli script provided in the root directory of this quickstart.

Before you begin, make sure you do the following:
- Back up the WildFly standalone server configuration as described above.
- Start the WildFly server with the standalone default profile as described above.
Review the configure-opentelemtry.cli file in the root of this quickstart directory. This script adds the configuration that enables OpenTelemetry for the quickstart components. Comments in the script describe the purpose of each block of commands.
Open a new terminal, navigate to the root directory of this quickstart, and run the following command, replacing WILDFLY_HOME with the path to your server:
```
$ WILDFLY_HOME/bin/jboss-cli.sh --connect --file=configure-opentelemetry.cli
```
Note
For Windows, use the WILDFLY_HOME\bin\jboss-cli.bat script.

You should see the following result when you run the script:
```
The batch executed successfully
process-state: reload-required
```

You’ll need to reload the configuration after that:

$ WILDFLY_HOME/bin/jboss-cli.sh --connect --commands=reload

Starting the OpenTelemetry Collector

By default, WildFly will publish traces every 10 seconds, so you will soon start seeing errors about a refused connection.

This is because we told WildFly to publish to a server that is not there, so we need to fix that. To make that as simple as possible, you can use Docker Compose to start an instance of the OpenTelemetry Collector.

The Docker Compose configuration file is docker-compose.yaml:

version: "3"
volumes:
  shared-volume:
    # - logs:/var/log
services:
  otel-collector:
    image: otel/opentelemetry-collector
    command: [--config=/etc/otel-collector-config.yaml]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:Z
    ports:
      - 1888:1888 # pprof extension
      - 13133:13133 # health_check extension
      - 4317:4317 # OTLP gRPC receiver
      - 4318:4318 # OTLP http receiver
      - 55679:55679 # zpages extension

The Collector server configuration file is otel-collector-config.yaml:

extensions:
  health_check:
  pprof:
    endpoint: 0.0.0.0:1777
  zpages:
    endpoint: 0.0.0.0:55679

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  logging:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [logging]

  extensions: [health_check, pprof, zpages]

We can now bring up the collector instance:

$ docker-compose up

The service should be available almost immediately, which you can verify by looking for the log entry Everything is ready. Begin running and processing data..

Note	You may use Podman as alternative to Docker if you prefer, in such case the command should be `podman-compose up`.

Note	If your environment does not support Docker or Podman, please refer to Otel Collector documentation for alternatives on how to install and run the OpenTelemetry Collector.

Note	Part of the value of OpenTelemetry is its vendor-agnostic approach to exporting its various supported signals. As such, this demo will only log the incoming traces, leaving the relaying of those signals to a downstream aggregation platform as an exercise for the reader.

Now we can start adding our custom spans from our application.

Creating traces

Implicit tracing of REST resources

The OpenTelemetry support in WildFly provides an implicit tracing of all Jakarta REST resources. That means that for all applications, WildFly will automatically:

extract the Span context from the incoming Jakarta REST request
start a new Span on incoming Jakarta REST request and close it when the request is completed
inject Span context to any outgoing Jakarta REST request
start a Span for any outgoing Jakarta REST request and finish the Span when the request is completed

Explicit tracing

The OpenTelemetry API also supports explicit tracing should your application required it:

package org.wildfly.quickstarts.opentelemetry;

import jakarta.enterprise.context.RequestScoped;
import jakarta.inject.Inject;

import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;

@RequestScoped
public class ExplicitlyTracedBean {

    @Inject
    private Tracer tracer;

    public String getHello() {
        Span prepareHelloSpan = tracer.spanBuilder("prepare-hello").startSpan();
        prepareHelloSpan.makeCurrent();

        String hello = "hello";

        Span processHelloSpan = tracer.spanBuilder("process-hello").startSpan();
        processHelloSpan.makeCurrent();

        hello = hello.toUpperCase();

        processHelloSpan.end();
        prepareHelloSpan.end();

        return hello;
    }
}

Build and Deploy the Quickstart

Make sure WildFly server is started.
Open a terminal and navigate to the root directory of this quickstart.
Type the following command to build the quickstart.
```
$ mvn clean package
```
Type the following command to deploy the quickstart.
```
$ mvn wildfly:deploy
```

This deploys the opentelemetry-tracing/target/opentelemetry-tracing.war to the running instance of the server.

You should see a message in the server log indicating that the archive deployed successfully.

Access the quickstart application

You can either access the application via your browser at http://localhost:8080/opentelemetry-tracing/implicit-trace, or http://localhost:8080/opentelemetry-tracing/explicit-trace. You can also access it from the command line:

$ curl http://localhost:8080/opentelemetry-tracing/implicit-trace
$ curl http://localhost:8080/opentelemetry-tracing/explicit-trace

Either endpoint should return a simple document:

hello

View the traces

You can view the traces by looking at the Collector’s log. You should see something like this:

otel-collector_1  | 2023-12-13T21:05:28.002Z    info    TracesExporter  {"kind": "exporter", "data_type": "traces", "name": "logging", "resource spans": 1, "spans": 1}
otel-collector_1  | 2023-12-13T21:05:28.002Z    info    ResourceSpans #0
otel-collector_1  | Resource SchemaURL: https://opentelemetry.io/schemas/1.20.0
otel-collector_1  | Resource attributes:
otel-collector_1  |      -> service.name: Str(opentelemetry-tracing.war)
otel-collector_1  |      -> telemetry.sdk.language: Str(java)
otel-collector_1  |      -> telemetry.sdk.name: Str(opentelemetry)
otel-collector_1  |      -> telemetry.sdk.version: Str(1.29.0)
otel-collector_1  | ScopeSpans #0
otel-collector_1  | ScopeSpans SchemaURL:
otel-collector_1  | InstrumentationScope io.smallrye.opentelemetry 2.6.0
otel-collector_1  | Span #0
otel-collector_1  |     Trace ID       : c761e8fadec36d222adac36dcff1f4b1
otel-collector_1  |     Parent ID      :
otel-collector_1  |     ID             : 08f93dd25f75b5cd
otel-collector_1  |     Name           : GET /opentelemetry-tracing/implicit-trace
otel-collector_1  |     Kind           : Server
otel-collector_1  |     Start time     : 2023-12-13 21:05:20.560054393 +0000 UTC
otel-collector_1  |     End time       : 2023-12-13 21:05:20.621635685 +0000 UTC
otel-collector_1  |     Status code    : Unset
otel-collector_1  |     Status message :
otel-collector_1  | Attributes:
otel-collector_1  |      -> net.host.port: Int(8080)
otel-collector_1  |      -> http.scheme: Str(http)
otel-collector_1  |      -> http.method: Str(GET)
otel-collector_1  |      -> http.status_code: Int(200)
otel-collector_1  |      -> net.transport: Str(ip_tcp)
otel-collector_1  |      -> user_agent.original: Str(curl/8.2.1)
otel-collector_1  |      -> net.host.name: Str(localhost)
otel-collector_1  |      -> http.route: Str(/opentelemetry-tracing/implicit-trace)
otel-collector_1  |      -> http.target: Str(/opentelemetry-tracing/implicit-trace)
otel-collector_1  |      -> net.sock.host.addr: Str(127.0.0.1)
otel-collector_1  |     {"kind": "exporter", "data_type": "traces", "name": "logging"}

Run the Integration Tests

This quickstart includes integration tests, which are located under the src/test/ directory. The integration tests verify that the quickstart runs correctly when deployed on the server.

Follow these steps to run the integration tests.

Make sure WildFly server is started.
Make sure the quickstart is deployed.
Type the following command to run the verify goal with the integration-testing profile activated.
```
$ mvn verify -Pintegration-testing 
```

Undeploy the Quickstart

When you are finished testing the quickstart, follow these steps to undeploy the archive.

Make sure WildFly server is started.
Open a terminal and navigate to the root directory of this quickstart.
Type this command to undeploy the archive:
```
$ mvn wildfly:undeploy
```

Restore the WildFly Standalone Server Configuration

You can restore the original server configuration using either of the following methods.

You can run the restore-configuration.cli script provided in the root directory of this quickstart.
You can manually restore the configuration using the backup copy of the configuration file.

Restore the WildFly Standalone Server Configuration by Running the JBoss CLI Script

Start the WildFly server as described above.
Open a new terminal, navigate to the root directory of this quickstart, and run the following command, replacing WILDFLY_HOME with the path to your server:
```
$ WILDFLY_HOME/bin/jboss-cli.sh --connect --file=restore-configuration.cli
```
Note
For Windows, use the WILDFLY_HOME\bin\jboss-cli.bat script.

Restore the WildFly Standalone Server Configuration Manually

When you have completed testing the quickstart, you can restore the original server configuration by manually restoring the backup copy the configuration file.

If it is running, stop the WildFly server.
Replace the WILDFLY_HOME/standalone/configuration/standalone.xml file with the backup copy of the file.

Building and running the quickstart application with provisioned WildFly server

Instead of using a standard WildFly server distribution, you can alternatively provision a WildFly server to deploy and run the quickstart, by activating the Maven profile named provisioned-server when building the quickstart:

$ mvn clean package -Pprovisioned-server

The provisioned WildFly server, with the quickstart deployed, can then be found in the target/server directory, and its usage is similar to a standard server distribution, with the simplification that there is never the need to specify the server configuration to be started.

The server provisioning functionality is provided by the WildFly Maven Plugin, and you may find its configuration in the quickstart pom.xml:

        <profile>
            <id>provisioned-server</id>
            <build>
                <plugins>
                    <plugin>
                        <groupId>org.wildfly.plugins</groupId>
                        <artifactId>wildfly-maven-plugin</artifactId>
                        <configuration>
                            <feature-packs>
                                <feature-pack>
                                    <location>org.wildfly:wildfly-galleon-pack:${version.server}</location>
                                </feature-pack>
                            </feature-packs>
                            <layers>...</layers>
                            <!-- deploys the quickstart on root web context -->
                            <name>ROOT.war</name>
                        </configuration>
                        <executions>
                            <execution>
                                <goals>
                                    <goal>package</goal>
                                </goals>
                            </execution>
                        </executions>
                    </plugin>
                    ...
                </plugins>
            </build>
        </profile>

Note	Since the plugin configuration above deploys quickstart on root web context of the provisioned server, the URL to access the application should not have the `/opentelemetry-tracing` path segment after `HOST:PORT`.

Run the Integration Tests with a provisioned server

The integration tests included with this quickstart, which verify that the quickstart runs correctly, may also be run with a provisioned server.

Follow these steps to run the integration tests.

Make sure the server is provisioned.

$ mvn clean package -Pprovisioned-server

Start the WildFly provisioned server, this time using the WildFly Maven Plugin, which is recommended for testing due to simpler automation. The path to the provisioned server should be specified using the jbossHome system property.
```
$ mvn wildfly:start -DjbossHome=target/server 
```
Type the following command to run the verify goal with the integration-testing profile activated, and specifying the quickstart’s URL using the server.host system property, which for a provisioned server by default is http://localhost:8080.
```
$ mvn verify -Pintegration-testing -Dserver.host=http://localhost:8080 
```
Shutdown the WildFly provisioned server, this time using the WildFly Maven Plugin too.
```
$ mvn wildfly:shutdown
```

Building and running the quickstart application in a bootable JAR

You can use the WildFly JAR Maven plug-in to build a WildFly bootable JAR to run this quickstart.

The quickstart pom.xml file contains a Maven profile named bootable-jar which configures the bootable JAR building:

      <profile>
          <id>bootable-jar</id>
          <build>
              <plugins>
                  <plugin>
                      <groupId>org.wildfly.plugins</groupId>
                      <artifactId>wildfly-jar-maven-plugin</artifactId>
                      <configuration>
                          <feature-pack-location>wildfly@maven(org.jboss.universe:community-universe)#${version.server}</feature-pack-location>
                          <layers>
                              <layer>jaxrs-server</layer>
                              <layer>microprofile-config</layer>
                          </layers>
                          <plugin-options>
                              <jboss-fork-embedded>true</jboss-fork-embedded>
                          </plugin-options>
                      </configuration>
                      <executions>
                          <execution>
                              <goals>
                                  <goal>package</goal>
                              </goals>
                          </execution>
                      </executions>
                  </plugin>
              </plugins>
          </build>
      </profile>

Procedure

Build the quickstart bootable JAR with the following command:
```
$ mvn clean package -Pbootable-jar
```
Run the quickstart application contained in the bootable JAR:
```
$ java -jar target/opentelemetry-tracing-bootable.jar
```
You can now interact with the quickstart application.

Note	After the quickstart application is deployed, the bootable JAR includes the application in the root context. Therefore, any URLs related to the application should not have the `/opentelemetry-tracing` path segment after `HOST:PORT`.

Run the Integration Tests with a bootable jar

The integration tests included with this quickstart, which verify that the quickstart runs correctly, may also be run with a bootable jar.

Follow these steps to run the integration tests.

Make sure the bootable jar is provisioned.
```
$ mvn clean package -Pbootable-jar
```
Start the WildFly bootable jar, this time using the WildFly Maven Jar Plugin, which is recommend for testing due to simpler automation.
```
$ mvn wildfly-jar:start -Djar-file-name=target/opentelemetry-tracing-bootable.jar
```
Type the following command to run the verify goal with the integration-testing profile activated, and specifying the quickstart’s URL using the server.host system property, which for a bootable jar by default is http://localhost:8080.
```
$ mvn verify -Pintegration-testing -Dserver.host=http://localhost:8080
```
Shutdown the WildFly bootable jar, this time using the WildFly Maven Jar Plugin too.
```
$ mvn wildfly-jar:shutdown
```

Building and running the quickstart application with OpenShift

Build the WildFly Source-to-Image (S2I) Quickstart to OpenShift with Helm Charts

On OpenShift, the S2I build with Apache Maven uses an openshift Maven profile to provision a WildFly server, deploy and run the quickstart in OpenShift environment.

The server provisioning functionality is provided by the WildFly Maven Plugin, and you may find its configuration in the quickstart pom.xml:

        <profile>
            <id>openshift</id>
            <build>
                <plugins>
                    <plugin>
                        <groupId>org.wildfly.plugins</groupId>
                        <artifactId>wildfly-maven-plugin</artifactId>
                        <configuration>
                            <feature-packs>
                                <feature-pack>
                                    <location>org.wildfly:wildfly-galleon-pack:${version.server}</location>
                                </feature-pack>
                                <feature-pack>
                                    <location>org.wildfly.cloud:wildfly-cloud-galleon-pack:${version.pack.cloud}</location>
                                </feature-pack>
                            </feature-packs>
                            <layers>...</layers>
                            <name>ROOT.war</name>
                        </configuration>
                        <executions>
                            <execution>
                                <goals>
                                    <goal>package</goal>
                                </goals>
                            </execution>
                        </executions>
                    </plugin>
                    ...
                </plugins>
            </build>
        </profile>

You may note that unlike the provisioned-server profile it uses the cloud feature pack which enables a configuration tuned for OpenShift environment.

Getting Started with WildFly for OpenShift and Helm Charts

This section contains the basic instructions to build and deploy this quickstart to WildFly for OpenShift or WildFly for OpenShift Online using Helm Charts.

Prerequisites

You must be logged in OpenShift and have an oc client to connect to OpenShift
Helm must be installed to deploy the backend on OpenShift.

Once you have installed Helm, you need to add the repository that provides Helm Charts for WildFly.

$ helm repo add wildfly https://docs.wildfly.org/wildfly-charts/
"wildfly" has been added to your repositories
$ helm search repo wildfly
NAME                    CHART VERSION   APP VERSION     DESCRIPTION
wildfly/wildfly         ...             ...            Build and Deploy WildFly applications on OpenShift
wildfly/wildfly-common  ...             ...            A library chart for WildFly-based applications

Install OpenTelemetry Collector on OpenShift

The functionality of this quickstart depends on a running instance of the OpenTelemetry Collector.

To deploy and configure the OpenTelemetry Collector, you will need to apply a set of configurations to your OpenShift cluster, to configure the OpenTelemetry Collector as well as any external routes needed:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: collector-config
data:
  collector.yml: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
    exporters:
      logging:
        verbosity: detailed
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [logging]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: opentelemetrycollector
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: opentelemetrycollector
  template:
    metadata:
      labels:
        app.kubernetes.io/name: opentelemetrycollector
    spec:
      containers:
        - name: otelcol
          args:
            - --config=/conf/collector.yml
          image: otel/opentelemetry-collector:0.89.0
          volumeMounts:
            - mountPath: /conf
              name: collector-config
      volumes:
        - configMap:
            items:
              - key: collector.yml
                path: collector.yml
            name: collector-config
          name: collector-config
---
apiVersion: v1
kind: Service
metadata:
  name: opentelemetrycollector
spec:
  ports:
    - name: otlp-grpc
      port: 4317
      protocol: TCP
      targetPort: 4317
    - name: otlp-http
      port: 4318
      protocol: TCP
      targetPort: 4318
  selector:
    app.kubernetes.io/name: opentelemetrycollector
  type: ClusterIP
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: otelcol-otlp-grpc
  labels:
    app.kubernetes.io/name: microprofile
spec:
  port:
    targetPort: otlp-grpc
  to:
    kind: Service
    name: opentelemetrycollector
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect
  wildcardPolicy: None
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: otelcol-otlp-http
  labels:
    app.kubernetes.io/name: microprofile
spec:
  port:
    targetPort: otlp-http
  to:
    kind: Service
    name: opentelemetrycollector
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect
  wildcardPolicy: None

To make things simpler, you can find these commands in charts/opentelemetry-collector.yaml, and to apply them run the following command in your terminal:

$ oc apply -f charts/opentelemetry-collector.yaml

Note	When done with the quickstart, the `oc delete -f charts/opentelemetry-collector.yaml` command may be used to revert the applied changes.

Deploy the WildFly Source-to-Image (S2I) Quickstart to OpenShift with Helm Charts

Log in to your OpenShift instance using the oc login command. The backend will be built and deployed on OpenShift with a Helm Chart for WildFly.

Navigate to the root directory of this quickstart and run the following command:

$ helm install opentelemetry-tracing -f charts/helm.yaml wildfly/wildfly --wait --timeout=10m0s
NAME: opentelemetry-tracing
...
STATUS: deployed
REVISION: 1

This command will return once the application has successfully deployed. In case of a timeout, you can check the status of the application with the following command in another terminal:

oc get deployment opentelemetry-tracing

The Helm Chart for this quickstart contains all the information to build an image from the source code using S2I on Java 17:

build:
  uri: https://github.com/wildfly/quickstart.git
  ref: 32.x
  contextDir: opentelemetry-tracing
deploy:
  replicas: 1
  env:
    - name: OTEL_COLLECTOR_HOST
      value: "opentelemetrycollector"

This will create a new deployment on OpenShift and deploy the application.

If you want to see all the configuration elements to customize your deployment you can use the following command:

$ helm show readme wildfly/wildfly

Get the URL of the route to the deployment.

$ oc get route opentelemetry-tracing -o jsonpath="{.spec.host}"

Access the application in your web browser using the displayed URL.

Note	The Maven profile named `openshift` is used by the Helm chart to provision the server with the quickstart deployed on the root web context, and thus the application should be accessed with the URL without the `/opentelemetry-tracing` path segment after `HOST:PORT`.

Run the Integration Tests with OpenShift

The integration tests included with this quickstart, which verify that the quickstart runs correctly, may also be run with the quickstart running on OpenShift.

Note	The integration tests expect a deployed application, so make sure you have deployed the quickstart on OpenShift before you begin.

Run the integration tests using the following command to run the verify goal with the integration-testing profile activated and the proper URL:

$ mvn verify -Pintegration-testing -Dserver.host=https://$(oc get route opentelemetry-tracing --template='{{ .spec.host }}')

Note	The tests are using SSL to connect to the quickstart running on OpenShift. So you need the certificates to be trusted by the machine the tests are run from.

Undeploy the WildFly Source-to-Image (S2I) Quickstart from OpenShift with Helm Charts

$ helm uninstall opentelemetry-tracing

Conclusion

OpenTelemetry Tracing provides the mechanisms for your application to participate in the distributed tracing with minimal effort on the application side. The Jakarta REST resources are always traced by default, but the specification allows you to create individual spans directly with the CDI injection of the io.opentelemetry.api.trace.Tracer.