Add Micrometer Support to WildFly

In 

Overview

WildFly already supports basic metrics in the base WildFly system, as well as MicroProfile Metrics in the WildFly MicroProfile configuration. In the Java space, however, the library of choice, and the emerging de facto standard for Java developers is Micrometer. This RFE will add support for Micrometer to WildFly, providing an alternative to the WildFly Metrics subsystem, as well as providing a replacement for MicroProfile Metrics, which is slated for removal from WildFly under another JIRA.

Issue Metadata

Issue

Dev Contacts

QE Contacts

Testing By

  • [X] Engineering

  • QE

Affected Projects or Components

  • WildFly

  • Observability

Other Interested Projects

  • MicroProfile

  • Metrics

Relevant Installation Types

  • Traditional standalone server (unzipped or provisioned by Galleon)

  • Managed domain

  • OpenShift s2i

  • Bootable jar

Requirements

Hard Requirements

This feature will integrate Micrometer into the WildFly feature pack. It will provide two modules, one to provide the integration of the library into the application server (org.wildfly.extension.micrometer), and another to expose Micrometer to user deployments via classpath visibility and CDI beans (org.wildfly.micrometer.deployment). These modules will grouped in the new micrometer Galleon layer to allow, for example, for easier Galleon and Bootable Jar provisioning.

The source for this new extension will reside under the observability directory so as to group it with similar extensions, such as those for OpenTelemetry support.

Metrics will be exported by pushing to a collector using the OTLP protocol. The default endpoint is http://localhost:4318/v1/metrics. With this approach, Micrometer will push after a certain number of seconds (configured via the step attribute on the subsystem) to a listening OpenTelemetry collector, which is itself configured to export the metrics data (as well as, for example, tracing and logging data, which OpenTelemetry also supports) to a downstream aggregation system chosen by the administrator. (For a simple example of a collector configuration, see the attached YAML files below)

The WildFly administrator will able to configure both the endpoint and the frequency (or step) with which a push is performed.

This extension will not replace the existing WildFly Metrics solution (exposed at /metrics) in the standard configurations. Administrators wishing to make use of this extension will need to install it manually. Removing WildFly Metrics is completely optional, as the extensions will not conflict with each other (though the duplicated metrics monitoring may cause performance concerns).

Server and JVM metrics will automatically be monitored, as is the case with WildFly Metrics. Server metrics are derived by walking the model tree and registering meters for the appropriate entries.

The Micrometer APIs will be exposed to user deployments, allowing user applications to participate in metrics gathering and exporting. The MeterRegistry created by the application server will be made available to user applications via CDI injection.

Non-Requirements

  • Support for multiple metric backends (e.g., Datadog, New Relic, Prometheus)

  • Automatic timing of REST requests

  • Removal of MicroProfile Metrics (This will be handle via another WildFly JIRA)

  • Removal of application metrics when a user application is undeployed

  • Mechanism for arbitrary subsystems to register meters (e.g., Micrometer’s MeterBinder)

Dependencies

  • CDI/Weld

  • OpenTelemetry (for the OTLP export support)

Backwards Compatibility

There are no backwards compatibility issues with this, as it is a new extension.

Security Considerations

When pushing metrics to the collector, there is no user involved from which to determine credentials/permissions, so the normal RBAC rules cannot be applied directly. The typical Micrometer deployment involves pushing all metrics, irrespective of permissions, to the collector, which then exports the metrics to one or more observability platforms, which are then secured in a manner specific to those systems. The READ_ATTRIBUTE_OPERATION (from the JBoss ModelNode API) to read the metrics values from the WildFly model, though, will be run using the Monitor role to restrict what metrics are pushed to those that should be monitored.

Test Plan

  • Add tests to testsuite/integration/microprofile

    The tests will validate that an application is able to inject a MeterRegistry instance and create/update meters programmatically.

    The test will also spot check a handful of server/JVM metrics to verify that they have been successfully registered.

Community Documentation

The feature will be documented in WildFly Admin Guide in the existing observability section.

Release Note Content

Support for the Micrometer metrics library has been added to the system. Application server and JVM metrics are available, as has been the case in previous releases, but this extension exposes Micrometer’s MeterRegistry via CDI for the inclusion of application metrics as well.

Additional Information

Sample Docker Configuration for Collector

With the following two yaml files, a simple OpenTelemetry collector can started which will accept metrics from WildFly via OTLP, then export them in the Prometheus format at http://localhost:1234 for easy manual verification.

docker-compose.yaml
version: "3"
volumes:
  shared-volume:
    # - logs:/var/log
services:
  otel-collector:
    image: otel/opentelemetry-collector
    command: [--config=/etc/otel-collector-config.yaml]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - 1888:1888 # pprof extension
      - 8888:8888 # Prometheus metrics exposed by the collector
      - 8889:8889 # Prometheus exporter metrics
      - 13133:13133 # health_check extension
      - 4317:4317 # OTLP gRPC receiver
      - 4318:4318 # OTLP http receiver
      - 55679:55679 # zpages extension
      - 1234:1234 # /metrics endpoint
otel-collector-config.yaml
extensions:
  health_check:
  pprof:
    endpoint: 0.0.0.0:1777
  zpages:
    endpoint: 0.0.0.0:55679

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  prometheus:
    endpoint: "0.0.0.0:1234"

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

  extensions: [health_check, pprof, zpages]