High Availability Guide

Configures the mechanism used by clustered caches to communicate with each other. It is only necessary to define a transport if the cache container contains clustered caches.

To create a JGroups transport using the default channel of the server:

To create a JGroups transport using a distinct "alpha" channel, that uses the "tcp" stack:

For a complete list of transport attributes, refer to the WildFly management model documentation

To remove an existing JGroups transport, you can either use the standard remove resource operation:

…​ or by adding the "none" transport (which will auto-remove any existing transport):

Infinispan supports a number of cache types for use in both HA and non-HA server profiles.

The configuration of a cache is divided into several components, each defining a specific cache feature. Because a given cache configuration requires each component relevant to its cache type, cache add operations and cache component add operations are typically batched. Any undefined components are auto-created using their defaults.

e.g. The following cache add operation:

…​ is actually equivalent to the following sequence of operations:

Similarly, you can reset all the attributes of a component by simply removing the component. e.g.

…​ is equivalent to:

The HotRod store uses one dedicated remote cache for each cache created by the server. For Infinispan Server versions supporting protocol version 2.7 and above (Infinispan Server version 9.2) a persistent remote cache will be automatically created based on default configuration. The recommended configuration for the remote cache where session data will be offloaded is transactional distribution mode cache with pessimistic locking. When using Infinispan Server version prior to 9.2, the caches need to be configured manually on the server where cache names correspond to the deployment file names (e.g. test.war).

Once a Remote Cache Container is configured a hotrod store can be configured replacing any existing store. The following CLI script demonstrates a typical use case for offloading sessions using an invalidation-cache with a shared, persistent infinispan-server store referencing an existing remote-cache-container:

To configure a remote-cache-container ensure you have a list of available Infinispan Server nodes. The following example CLI script first adds socket bindings to two known Infinispan Server nodes, followed by configuration of the cluster.

Upon reload, this will register necessary services for the client. A HotRod client can be injected directly into Jakarta EE applications using the @Resource annotation.

The infinispan-session-management resource configures a distributable session manager that uses an embedded Infinispan cache.

e.g. Creating a new session management profile, using ATTRIBUTE granularity with local session affinity:

The hotrod-session-management resource configures a distributable session manager where session data is stored in a remote infinispan-server cluster via the HotRod protocol.

e.g. Creating a new session management profile "foo" using the cache configuration "bar" defined on a remote infinispan server "datagrid" with ATTRIBUTE granularity:

To use an existing distributed session management profile, a web application should include a distributable-web.xml deployment descriptor located within the application’s /WEB-INF directory.

e.g.

Alternatively, the target distributed session management profile can be defined within an existing jboss-all.xml deployment descriptor:

e.g.

If custom session management configuration will only be used by a single web application, you may find it more convenient to define the configuration within the deployment descriptor itself. Ad hoc configuration looks identical to the configuration used by the distributable-web subsystem.

e.g.

Alternatively, session management configuration can be defined within an existing jboss-all.xml deployment descriptor:

e.g.

By default, WildFly’s distributed session manager uses SESSION granularity, meaning that all session attributes are stored within a single cache entry. While this ensures that any object references shared between session attributes are preserved following replication/persistence, it means that a change to a single attribute results in the replication/persistence of all attributes.

If your application does not share any object references between attributes, users are strongly advised to use ATTRIBUTE granularity. Using ATTRIBUTE granularity, each session attribute is stored in a separate cache entry. This means that a given request is only required to replicate/persist those attributes that were added/modified/removed/mutated in a given request. For read-heavy applications, this can dramatically reduce the replication/persistence payload per request.

WildFly’s default distributed session manager behavior is also conservative with respect to concurrent access to a given session. By default, a request acquires exclusive access to its associated session for the duration of a request, and until any async child context is complete. This maximizes the performance of a single request, as each request corresponds to a single cache transaction; allows for repeatable read semantics to the session; and ensures that subsequent requests are not prone to stale reads, even when handled by another cluster member.

However, if multiple requests attempt to access the same session concurrently, their processing will be effectively serialized. This might not be feasible, especially for heavily asynchronous web applications.

Relaxing transaction isolation from REPEATABLE_READ to READ_COMMITTED on the associated cache configuration will allow concurrent requests to perform lock-free (but potentially stale) reads by deferring locking to the first attempt to write to the session. This improves the throughput of requests for the same session for highly asynchronous web applications whose session access is read-heavy.

e.g.

For asynchronous web applications whose session access is write-heavy, merely relaxing transaction isolation is not likely to be sufficient. These web applications will likely benefit from disabling cache transactions altogether. When transactions are disabled, cache entries are locked and released for every write to the session, resulting in last-write-wins semantics. For write-heavy applications, this typically improves the throughput of concurrent requests for the same session, at the cost of longer response times for individual requests.

In WildFly, distributed session attributes are presumed to be mutable objects, unless of a known immutable type, or unless otherwise specified.

Take the following session access pattern:

By default, WildFly replicates/persists the mutable session attributes at the end of the request, ensuring that a subsequent request will read the mutated value, not the original value. However, the replication/persistence of mutable session attributes at the end of the request happens whether or not these objects were actually mutated. To avoid redundant session writes, users are strongly encouraged to store immutable objects in the session whenever possible. This allows the application more control over when session attributes will replicate/persist, since immutable session attributes will only update upon explicit calls to HttpSession.setAttribute(…​).

WildFly can determine whether most JDK types are immutable, but any unrecognized/custom types are presumed to be mutable. To indicate that a given session attribute of a custom type should be treated as immutable by the distributed session manager, annotate the class with one of the following annotations:

e.g.

Alternatively, immutable classes can be enumerated via the distributable-web deployment descriptor.

e.g.

Minimizing the replication/persistence payload for individual session attributes has a direct impact on performance by reducing the number of bytes sent over the network or persisted to storage. A web application can optimize the marshalling of a given session attribute, either through custom JDK serialization logic, or by implementing a custom externalizer. An externalizer is an implementation of the org.wildfly.clustering.marshalling.Externalizer interface, which dictates how a given class should be marshalled. An externalizer reads/writes the state of an object directly from/to an input/output stream, but also:

e.g.

Externalizers are dynamically loaded during deployment via the service loader mechanism. Implementations should be enumerated within a file named: /META-INF/services/org.wildfly.clustering.marshalling.Externalizer Unresolved directive in _high-availability/Subsystem_Support.adoc - include::subsystem-support/Offloading_session_state_to_remote_Infinispan_server_cluster.adoc[]

This element is used to configure the subsystem within a WildFly system profile.

This element is used to configure a JGroups protocol stack.

This element is used to configure the transport layer (required) of the protocol stack.

.

This element is used to configure a (non-transport) protocol layer in the JGroups stack. Protocol layers are ordered within the stack.

This element is used to configure the RELAY protocol for a JGroups stack. RELAY is a protocol which provides cross-site replication between defined sites (data centres). In the RELAY protocol, defined sites specify the names of remote sites (backup sites) to which their data should be backed up. Channels are defined between sites to permit the RELAY protocol to transport the data from the current site to a backup site.

The instance-id or JVMRoute defaults to jboss.node.name property passed on server startup (e.g. via -Djboss.node.name=XYZ).

To configure instance-id statically, configure the corresponding property in Undertow subsystem:

By default, mod_cluster is configured for multicast-based discovery. To specify a static list of proxies, create a remote-socket-binding for each proxy and then reference them in the 'proxies' attribute. See the following example for configuration in the domain mode:

Since WildFly 14 mod_cluster subsystem supports multiple named proxy configurations also allowing for registering non-default Undertow servers with the reverse proxies. Moreover, this allows single application server node to register with different groups of proxy servers.

See the following example which adds another Undertow AJP listener, server and a host and adds a new mod_cluster configuration which registers this host using advertise mechanism.

There are 2 operations that display how Apache httpd sees the node:

Those operations allow to send context related commands to Apache httpd. They are send automatically when deploying or undeploying webapps.

Those operations are like the context operation but they apply to all webapps running on the node and operation that affect the whole node.

Elytron subsystem provides a powerful and flexible model to configure different security aspects for applications and the application server itself. At its core, Elytron subsystem exposes different capabilities to the application server in order centralize security related configuration in a single place and to allow other subsystems to consume these capabilities. One of the security capabilities exposed by Elytron subsystem is a Client ssl-context that can be used to configure mod_cluster subsystem to communicate with a load balancer using SSL/TLS.

When protecting the communication between the application server and the load balancer, you need do define a Client ssl-context in order to:

To define a trust store in Elytron you can execute the following CLI command:

In order to successfully execute the command above you must have a application.truststore file inside your JBOSS_HOME/standalone/configuration directory. Where the trust store is protected by a password with a value password. The trust store must contain the certificates associated with the load balancer or a certificate chain in case the load balancer’s certificate is signed by a CA.

We strongly recommend you to avoid using self-signed certificates with your load balancer. Ideally, certificates should be signed by a CA and your trust store should contain a certificate chain representing your ROOT and Intermediary CAs.

To define a trust manager in Elytron you can execute the following CLI command:

Here we are setting the default-trust-store as the source of the certificates that the application server trusts.

Finally, you can create the Client SSL Context that is going to be used by the mod_cluster subsystem when connecting to the load balancer using SSL/TLS:

Now that the Client ssl-context is defined you can configure mod_cluster subsystem as follows:

Once you execute the last command above, reload the server:

In case you want to validate the load balancer certificate against a Certificate Revocation List (CRL), you can configure the trust-manager in Elytron subsystem as follows:

To use a CRL your trust store must contain the certificate chain in order to check validity of both CRL list and the load balancer`s certificate.

A different way to configure a CRL is using the Distribution Points embedded in your certificates. For that, you need to configure a certificate-revocation-list as follows:

Clustering support is available in the HA profiles of WildFly. In this chapter we’ll be using the standalone server for explaining the details. However, the same applies to servers in a domain mode. Starting the standalone server with HA capabilities enabled, involves starting it with the standalone-ha.xml (or even standalone-full-ha.xml):

This will start a single instance of the server with HA capabilities. Deploying the EJBs to this instance doesn’t involve anything special and is the same as explained in the application deployment chapter.

Obviously, to be able to see the benefits of clustering, you’ll need more than one instance of the server. So let’s start another server with HA capabilities. That another instance of the server can either be on the same machine or on some other machine. If it’s on the same machine, the two things you have to make sure is that you pass the port offset for the second instance and also make sure that each of the server instances have a unique jboss.node.name system property. You can do that by passing the following two system properties to the startup command:

Follow whichever approach you feel comfortable with for deploying the EJB deployment to this instance too.

Now that you have deployed an application with clustered EJBs on both the instances, the EJBs are now capable of making use of the clustering features.

Clustered EJBs have failover capability. The state of the @Stateful @Clustered EJBs is replicated across the cluster nodes so that if one of the nodes in the cluster goes down, some other node will be able to take over the invocations. Let’s see how it’s implemented in WildFly. In the next few sections we’ll see how it works for remote (standalone) clients and for clients in another remote WildFly server instance. Although, there isn’t a difference in how it works in both these cases, we’ll still explain it separately so as to make sure there aren’t any unanswered questions.

A Node encapsulates a member of a group (i.e. a JGroups address). A Node has the following distinct characteristics, which will be unique for each member of the group:

A Membership is an immutable encapsulation of a group membership (i.e. a JGroups view). Membership exposes the following properties:

The Group abstract is effectively a volatile reference to the current membership, and provides a facility for notification of membership changes. It exposes the following properties and operations:

A distributed "Hello world" example that prints joiners and leavers of a group membership:

A command dispatcher is created from a CommandDispatcherFactory, an instance of which is created for every defined channel defined in the JGroups subsystem, as well as a local implementation. e.g.

To ensure that your application functions consistently regardless of server configuration, we recommended that you reference the CommandDispatcherFactory using an alias. Most users should use the "default" alias, which references either:

e.g.

Additionally, WildFly creates a CommandDispatcherFactory alias for every Infinispan cache-container, which references:

This is useful in the case where a CommandDispatcher is used to communicate with members on which a given cache is deployed.

e.g.

A Command encapsulates logic to be executed on a group member. A Command can leverage 2 type of parameters during execution:

The CommandDispatcherFactory creates a CommandDispatcher using a service identifier and a local context. This service identifier is used to segregate commands from multiple command dispatchers. A CommandDispatcher will only receive commands dispatched by a CommandDispatcher with the same service identifier.

Once created, a CommandDispatcher will locally execute any received commands until it is closed. Once closed, a CommandDispatcher is no longer allowed to dispatch commands.

The functionality of a CommandDispatcher boils down to 2 operations:

Both methods return responses as a CompletionStage, allowing for asynchronous processing of responses as they complete.

To demonstrate how to use a CommandDispatcher, let’s create a distributed "hello world" application.

First, let’s create a simple HelloWorld interface which enables the caller to send a specific message to the entire group:

Next, we need to define a local command execution context. This should encapsulate any local information we need to make available to the execution of any command received by our CommandDispatcher. For demonstration purposes, let’s make this a separate interface:

Next we create a "hello world" Command that contains a message from the sender, and responds with a message of its own.

Finally, we create a @Singleton EJB that implements our HelloWorld interface:

Now you can use the HelloWorld.send(…​) operation to say hello to your cluster.

The following is an example of using a ServiceProviderRegistry to publish the availability of a given singleton EJB. The getProviders() method will return the set of nodes on which the ServiceProviderRegistrationBean is deployed.

The following Registry example assigns a UUID to each group member, allowing each member to query the identifier of any other member:

The cache-container and cache attributes of a singleton policy must reference a valid cache from the Infinispan subsystem. If no specific cache is defined, the default cache of the cache container is assumed. This cache is used as a registry of which nodes can provide a given service and will typically use a replicated-cache configuration.

WildFly includes two singleton election policy implementations:

Network partitions are particularly problematic for singleton services, since they can trigger multiple singleton providers for the same service to run at the same time. To defend against this scenario, a singleton policy may define a quorum that requires a minimum number of nodes to be present before a singleton provider election can take place. A typical deployment scenario uses a quorum of N/2 + 1, where N is the anticipated cluster size. This value can be updated at runtime, and will immediately affect any active singleton services.
e.g.

We need to prepare two hosts (or virtual hosts) to do the experiment. We will use these two hosts as following:

Here are some details on what we are going to do:

Please don’t worry if you cannot digest so many details currently. Let’s move on and you will get the points step by step.

In this section we’ll setup both master and slave for them to run in domain mode. And we will configure master to be the domain controller.

First open the host.xml in master for editing:

The default settings for interface in this file is like:

We need to change the address to the management interface so slave could connect to master. The public interface allows the application to be accessed by non-local HTTP, and the unsecured interface allows remote RMI access. My master’s ip address is 10.211.55.7, so I change the config to:

Now we will setup interfaces on slave. Let’s edit host.xml. Similar to the steps on master, open host.xml first:

The configuration we’ll use on slave is a little bit different, because we need to let slave connect to master. First we need to set the hostname. We change the name property from:

to:

Then we need to modify domain-controller section so slave can connect to master’s management port:

As we know, 10.211.55.7 is the ip address of master. You may use discovery options to define multiple mechanisms to connect to the remote domain controller :

Finally, we also need to configure interfaces section and expose the management ports to public address:

10.211.55.2 is the ip address of the slave. Refer to the domain controller configuration above for an explanation of the management, public, and unsecured interfaces.

If you start WildFly on both master and slave now, you will see the slave cannot be started with following error:

Because we haven’t properly set up the authentication between master and slave. Now let’s work on it: