Integrate OpenTelemetry with the MP Reactive Messaging Kafka connector

In  microprofile
Tracked by https://github.com/wildfly/wildfly-proposals/issues/636

Overview

SmallRye Reactive Messaging’s Kafka connector allows for integration with OpenTelemetry tracing so that we can track messages sent to and received from Kafka via this connector.

In SmallRye Reactive Messaging, a set of MicroProfile Config properties are used to control whether tracing happens on a reactive messaging channel. These are:

  • mp.messaging.connector.smallrye-kafka.tracing-enabled - This is a connector level property. At present this is set to false in a ConfigSource set up by the integration with an ordinal value set to the maximum value. The result of this is that the user can never enable tracing to happen (since any attempt to set this property to true would be overridden by the mentioned ConfigSource).

  • The above property may in turn be overridden by the following channel level properties.

    • mp.messaging.outgoing.<channel-name>.tracing-enabled - Channel level property, controlling the OpenTelemetry tracing integration for an outgoing channel.

    • mp.messaging.incoming.<channel-name>.tracing-enabled - Channel level property, controlling the OpenTelenetry integration for an incoming channel.

The way the SmallRye Reactive Messaging configuration system works is that it will first check for relevant channel level properties. If those are not present, it will look for the related connector level property, and if that is not available it will use a default.

Internally in SmallRye Reactive Messaging, the default value for the tracing integration is that it is always enabled for a channel if not specified with above properties.

This RFE is about allowing users to enable tracing via OpenTelemetry in WildFly, and also to be able to continue to ban it.

User Stories

Users wish to be able to trace messages sent to/from Kafka via reactive messaging using MicroProfile Telemetry.

Issue Metadata

  • WFLY-19835 - Integrate OpenTelemetry with the MP Reactive Messaging Kafka connector

  • WFLY-19846 - Promote MicroProfile Telemetry 2.0 to WildFly Standard

Affected Projects or Components

Other Interested Projects

None to my knowledge

Relevant Installation Types

  • Traditional standalone server (unzipped or provisioned by Galleon)

  • Managed domain

  • OpenShift Source-to-Image (S2I)

  • Bootable jar

Requirements

It must be possible to turn the tracing on. To do this a resource called subsystem=microprofile-reactive-messaging-smallrye/opentelemetry-tracing=config is introduced. This adds a capability requirement on the MicroProfile Telemetry subsystem, which in turn requires the OpenTelemetry subsystem. The OpenTelemetry subsystem configures how tracing happens.

This resource contains an attribute called kafka-connector, which allows us to control if and how tracing is enabled. It is an enumeration with the following values:

  • NEVER - This is the default value of the attribute, and also the value that will be used if the opentelemetry-tracing=config child resource is not present in the subsystem. This is the same behaviour that has been available today, i.e. the MicroProfile Config properties mentioned in the Overview will not have any effect, and tracing will be disabled.

  • OFF - By default tracing will be disabled, but a user can use the MicroProfile Config properties from the Overview to turn on tracing for a connector or individual channel in an application. We will discuss how the properties are resolved below.

  • ON - Similar to OFF but the default is to enable tracing for all channels, unless overridden for the application with the MicroProfile Config properties from the Overview. We will discuss how the properties are resolved below.

  • ALWAYS - Tracing is enabled for all channels, and can not be turned off with the MicroProfile Config properties from the Overview.

The MicroProfile Config properties in the Overview, are all booleans, and the values have these meanings:

  • false - OpenTelemetry tracing is not enabled

  • true - OpenTelemetry tracing is enabled

When tracing is enabled, and the OpenTelemetry subsystem is configured to connect to a tracing collector, the traces should be available from there.

Calculating whether tracing should happen

If the kafka-connector attribute is NEVER or ALWAYS, tracing will always be totally disabled or enabled, and the MicroProfile Config properties from the Overview will have no effect.

Thus, they only work if the tracing is kafka-connector is OFF or ON. In this case the resolution works as follows for each channel:

  • If the channel is a Kafka channel it will check the channel level mp.messaging.incoming.[channel-name].tracing-enabled or mp.messaging.outgoing.[channel-name].tracing-enabled property, depending on if the channel is incoming or outgoing. If this has a value the value will be used. Otherwise, we go on to the next step.

  • We then check the mp.messaging.connector.smallrye-kafka.tracing-enabled connector level property for the Kafka connector. If this has a value the value will be used. Otherwise, we go on to the next step.

  • Finally, we check the value of the kafka-connector attribute, and use its value. OFF returns false, and ON returns true.

In each of the above steps, if the found value is false, tracing is not enabled. If it is true, tracing is enabled.

Changed requirements

N/A

Non-Requirements

N/A

Future Work

N/A

Backwards Compatibility

If not configured, the default is the same as the existing behaviour, which is to not enable tracing, and to not allow users to enable it.

Default Configuration

No change, and the behaviour is the same as before, which is to not enable tracing.

Importing Existing Configuration

Existing configurations will have the same behaviour as today. i.e. tracing is not enabled, and can not be turned on.

Deployments

No change.

Interoperability

N/A

Implementation Plan

A single pull request to WildFly will be created containing everything.

Admin Clients

It is a simple resource with an attribute, so no work is needed in JBoss CLI/HAL

Security Considerations

None, beyond the consideration that the collector of the traces will receive a lot of the data, and so must be secured as well.

Test Plan

A unit test will be added in WildFly’s wildfly-microprofile-reactive-messaging-config to make sure that the combinations of the new kafka-connector attribute and the MicroProfile Config properties controlling tracing yield the expected final value (e.g. if kafka-connector=NEVER, we check that the resulting value is always false no matter what we specify for the MicroProfile Config properties).

Two main tests will be created in WildFly’s testsuite/integration/microprofile module:

  1. We test all the combinations of the kafka-connector in the new subsystem=microprofile-reactive-messaging-smallrye/opentelemetry-tracing=config resource, and values of mp.messaging.connector.smallrye-kafka.tracing-enabled set via the user’s MicroProfile Config, and make sure that the resulting value of mp.messaging.connector.smallrye-kafka.tracing-enabled in the final MicroProfile Config used for the deployment has the resulting value specified in the Requirements section. We only do this for the connector level attribute since the same code is used to handle the value of kafka-connector at runtime. The mentioned unit test has more coverage.

  2. Testing tracing. The aim here is to test some combinations, and not all.

    1. Ensure that tracing does not happen when tracing is disabled by setting kafka-connector=NEVER

    2. Ensure that tracing happens when kafka-connector=OFF but is turned on for the deployment by specifying mp.messaging.connector.smallrye-kafka.tracing-enabled=true. Both sent and received messages will be traced.

    3. When kafka-connector=ON, and mp.messaging.incoming.<channel-name>.tracing-enabled=false, tracing will onlyt happen on the outgoing channel

Community Documentation

The current MicroProfile Reactive Messaging Document will be enhanced to cover the new resource and its attributes, and how those values combine with the relevant MicroProfile Config Properties to control whether OpenTelemetry tracing happens.

Release Note Content

You can now enable OpenTelemetry tracing for the MicroProfile Reactive Messaging Kafka connector.