High Availability Guide

The infinispan-session-management resource configures a distributable session manager that uses an embedded Infinispan cache.

e.g. Creating a new session management profile, using ATTRIBUTE granularity with local session affinity:

The hotrod-session-management resource configures a distributable session manager where session data is stored in a remote infinispan-server cluster via the HotRod protocol.

e.g. Creating a new session management profile "foo" using the cache configuration "bar" defined on a remote infinispan server "datagrid" with ATTRIBUTE granularity:

To use an existing distributed session management profile, a web application should include a distributable-web.xml deployment descriptor located within the application’s /WEB-INF directory.

e.g.

Alternatively, the target distributed session management profile can be defined within an existing jboss-all.xml deployment descriptor:

e.g.

If custom session management configuration will only be used by a single web application, you may find it more convenient to define the configuration within the deployment descriptor itself. Ad hoc configuration looks identical to the configuration used by the distributable-web subsystem.

e.g.

Alternatively, session management configuration can be defined within an existing jboss-all.xml deployment descriptor:

e.g.

By default, WildFly’s distributed session manager uses SESSION granularity, meaning that all session attributes are stored within a single cache entry. While this ensures that any object references shared between session attributes are preserved following replication/persistence, it means that a change to a single attribute results in the replication/persistence of all attributes.

If your application does not share any object references between attributes, users are strongly advised to use ATTRIBUTE granularity. Using ATTRIBUTE granularity, each session attribute is stored in a separate cache entry. This means that a given request is only required to replicate/persist those attributes that were added/modified/removed/mutated in a given request. For read-heavy applications, this can dramatically reduce the replication/persistence payload per request.

WildFly’s default distributed session manager behavior is also conservative with respect to concurrent access to a given session. By default, a request acquires exclusive access to its associated session for the duration of a request, and until any async child context is complete. This maximizes the performance of a single request, as each request corresponds to a single cache transaction; allows for repeatable read semantics to the session; and ensures that subsequent requests are not prone to stale reads, even when handled by another cluster member.

However, if multiple requests attempt to access the same session concurrently, their processing will be effectively serialized. This might not be feasible, especially for heavily asynchronous web applications.

Relaxing transaction isolation from REPEATABLE_READ to READ_COMMITTED on the associated cache configuration will allow concurrent requests to perform lock-free (but potentially stale) reads by deferring locking to the first attempt to write to the session. This improves the throughput of requests for the same session for highly asynchronous web applications whose session access is read-heavy.

e.g.

For asynchronous web applications whose session access is write-heavy, merely relaxing transaction isolation is not likely to be sufficient. These web applications will likely benefit from disabling cache transactions altogether. When transactions are disabled, cache entries are locked and released for every write to the session, resulting in last-write-wins semantics. For write-heavy applications, this typically improves the throughput of concurrent requests for the same session, at the cost of longer response times for individual requests.

In WildFly, distributed session attributes are presumed to be mutable objects, unless of a known immutable type, or unless otherwise specified.

Take the following session access pattern:

By default, WildFly replicates/persists the mutable session attributes at the end of the request, ensuring that a subsequent request will read the mutated value, not the original value. However, the replication/persistence of mutable session attributes at the end of the request happens whether or not these objects were actually mutated. To avoid redundant session writes, users are strongly encouraged to store immutable objects in the session whenever possible. This allows the application more control over when session attributes will replicate/persist, since immutable session attributes will only update upon explicit calls to HttpSession.setAttribute(…​).

WildFly can determine whether most JDK types are immutable, but any unrecognized/custom types are presumed to be mutable. To indicate that a given session attribute of a custom type should be treated as immutable by the distributed session manager, annotate the class with one of the following annotations:

e.g.

Alternatively, immutable classes can be enumerated via the distributable-web deployment descriptor.

e.g.

Minimizing the replication/persistence payload for individual session attributes has a direct impact on performance by reducing the number of bytes sent over the network or persisted to storage. See the Marshalling section for more details.

A bean management provider provides access to a given implementation of a bean manager, used by passivation-capable cache factories defined in the ejb3 subsystem to manage passivation and persistence.

Bean management provider elements are named, and represent different implementation and configuration choices for bean management. At least one named bean management provider must be defined in the distributable-ejb subsystem and of those, one instance must be identified as the default bean management provider, using the default-bean-management attribute of the distributable-ejb subsystem.

The available bean management provider is:

A client mappings registry provider provides access to a given implementation of a client mappings registry, used by the EJB client invocation mechanism to store information about client mappings for each node in the cluster. Client mappings are defined in the socket bindings configuration of a server and required to allow an EJB client application to connect to servers which are multi-homed (i.e. clients may access the same server from different networks using a different IP address ad port for each interface on the multi-homed server).

The available client mappings registry providers are:

The distributable-ejb subsystem defines a set of timer management resources that define behavior for persistent or non-persistent EJB timers.

To use distributable timer management for EJB timers, one must first disable the existing in-memory mechanisms in the ejb3 subsystem. See Jakarta Enterprise Beans Distributed Timer documentation for details.

Clustered EJBs have failover capability. The state of the @Stateful @Clustered EJBs is replicated across the cluster nodes so that if one of the nodes in the cluster goes down, some other node will be able to take over the invocations. Let’s see how it’s implemented in WildFly. In the next few sections we’ll see how it works for remote (standalone) clients and for clients in another remote WildFly server instance. Although, there isn’t a difference in how it works in both these cases, we’ll still explain it separately so as to make sure there aren’t any unanswered questions.

In this section we’ll consider a remote standalone client (i.e. a client which runs in a separate JVM and isn’t running within another WildFly 8 instance). Let’s consider that we have 2 servers, server X and server Y which we started earlier. Each of these servers has the clustered EJB deployment. A standalone remote client can use either the JNDI approach or native JBoss EJB client APIs to communicate with the servers. The important thing to note is that when you are invoking clustered EJB deployments, you do not have to list all the servers within the cluster (which obviously wouldn’t have been feasible due the dynamic nature of cluster node additions within a cluster).

The remote client just has to list only one of the servers with the clustering capability. In this case, we can either list server X (in jboss-ejb-client.properties) or server Y. This server will act as the starting point for cluster topology communication between the client and the clustered nodes.

Note that you have to configure the ejb cluster in the jboss-ejb-client.properties configuration file, like so:

When a client connects to a server, the JBoss EJB client implementation (internally) communicates with the server for cluster topology information, if the server had clustering capability. In our example above, let’s assume we listed server X as the initial server to connect to. When the client connects to server X, the server will send back an (asynchronous) cluster topology message to the client. This topology message consists of the cluster name(s) and the information of the nodes that belong to the cluster. The node information includes the node address and port number to connect to (whenever necessary). So in this example, the server X will send back the cluster topology consisting of the other server Y which belongs to the cluster.

In case of stateful (clustered) EJBs, a typical invocation flow involves creating of a session for the stateful bean, which happens when you do a JNDI lookup for that bean, and then invoking on the returned proxy. The lookup for stateful bean, internally, triggers a (synchronous) session creation request from the client to the server. In this case, the session creation request goes to server X since that’s the initial connection that we have configured in our jboss-ejb-client.properties. Since server X is clustered, it will return back a session id and along with send back an "affinity" of that session. In case of clustered servers, the affinity equals to the name of the cluster to which the stateful bean belongs on the server side. For non-clustered beans, the affinity is just the node name on which the session was created. This affinity will later help the EJB client to route the invocations on the proxy, appropriately to either a node within a cluster (for clustered beans) or to a specific node (for non-clustered beans). While this session creation request is going on, the server X will also send back an asynchronous message which contains the cluster topology. The JBoss EJB client implementation will take note of this topology information and will later use it for connection creation to nodes within the cluster and routing invocations to those nodes, whenever necessary.

Now that we know how the cluster topology information is communicated from the server to the client, let see how failover works. Let’s continue with the example of server X being our starting point and a client application looking up a stateful bean and invoking on it. During these invocations, the client side will have collected the cluster topology information from the server. Now let’s assume for some reason, server X goes down and the client application subsequent invokes on the proxy. The JBoss EJB client implementation, at this stage will be aware of the affinity and in this case it’s a cluster affinity. Because of the cluster topology information it has, it knows that the cluster has two nodes server X and server Y. When the invocation now arrives, it sees that the server X is down. So it uses a selector to fetch a suitable node from among the cluster nodes. The selector itself is configurable, but we’ll leave it from discussion for now. When the selector returns a node from among the cluster, the JBoss EJB client implementation creates a connection to that node (if not already created earlier) and creates a EJB receiver out of it. Since in our example, the only other node in the cluster is server Y, the selector will return that node and the JBoss EJB client implementation will use it to create a EJB receiver out of it and use that receiver to pass on the invocation on the proxy. Effectively, the invocation has now failed over to a different node within the cluster.

So far we discussed remote standalone clients which typically use either the EJB client API or the jboss-ejb-client.properties based approach to configure and communicate with the servers where the clustered beans are deployed. Now let’s consider the case where the client is an application deployed another AS7 instance and it wants to invoke on a clustered stateful bean which is deployed on another instance of WildFly. In this example let’s consider a case where we have 3 servers involved. Server X and Server Y both belong to a cluster and have clustered EJB deployed on them. Let’s consider another server instance Server C (which may or may not have clustering capability) which acts as a client on which there’s a deployment which wants to invoke on the clustered beans deployed on server X and Y and achieve failover.

The configurations required to achieve this are explained in this chapter. As you can see the configurations are done in a jboss-ejb-client.xml which points to a remote outbound connection to the other server. This jboss-ejb-client.xml goes in the deployment of server C (since that’s our client). As explained earlier, the client configuration need not point to all clustered nodes. Instead it just has to point to one of them which will act as a start point for communication. So in this case, we can create a remote outbound connection on server C to server X and use server X as our starting point for communication. Just like in the case of remote standalone clients, when the application on server C (client) looks up a stateful bean, a session creation request will be sent to server X which will send back a session id and the cluster affinity for it. Furthermore, server X asynchronously send back a message to server C (client) containing the cluster topology. This topology information will include the node information of server Y (since that belongs to the cluster along with server X). Subsequent invocations on the proxy will be routed appropriately to the nodes in the cluster. If server X goes down, as explained earlier, a different node from the cluster will be selected and the invocation will be forwarded to that node.

As can be seen both remote standalone client and remote clients on another WildFly instance act similar in terms of failover.

The modcluster subsystem supports several operations:

The operations specific to the modcluster subsystem are divided in 3 categories the ones that affects the configuration and require a restart of the subsystem, the one that just modify the behaviour temporarily and the ones that display information from the httpd part.

It is possible to accept a REMOTE_USER already authenticated by the Apache httpd server with Elytron via the AJP protocol. This can be done by setting up Elytron to secure a WildFly deployment and specifying for the External HTTP mechanism to be used. This is done by creating a security domain and specifying the External mechanism as one of the mechanism configurations to be used by the http-authentication-factory:

Elytron will accept the externally authenticated user and use the specified security domain to perform role mapping to complete authorization.

The following is a minimum HAProxy configuration to enable affinity provided by application server to be respected by the given load-balancer.

Notice that the proxy server defined cookie names need to correspond with the application server’s instance-id.

WildFly uses Infinispan to provide high-availability of server-side state. While there are multiple cache modes available, the 'distributed-cache' cache is optimal for cloud environments. This cache mode stores data in a configurable number of cluster members (defaults to 2), and thus allows users to elect the desired balance between availability and capacity. The higher the number of owners, the greater availability (i.e. tolerance to failures), but reduces the effective heap capacity of the cluster. There is a marginal network cost associated with more owners, but this is a secondary concern.

This cache mode relies on consistent hashing to determine the 'primary owner' and the 'backup owners' of given cache entries. In order to achieve optimal performance, the load balancer deployed in front of the application server cluster:

This can be achieved by configuring HAProxy Ingress Controller described below.

A Kubernetes Ingress is an object that manages external access to the services in a cluster, typically via HTTP. It provides load balancing, SSL termination and name-based virtual hosting. In order to support the affinity requirements described above, an Ingress Controller implementation, supporting L7 load balancing and exposes the requisite annotations for configuring session affinity, will need to be configured. The following sections will configure an HAProxy Ingress Controller for optimal routing.

Verifying session affinity has proven to be notoriously challenging for users. This is primarily because, even if the affinity handling is not set up optimally, the tester will not observe any data loss in a distributed HA application. If a request for a given HTTP session is routed to a pod that does not contain the requested session data locally, it will remotely fetch the session data from another pod. While this does not affect functionality per se, this has implications for performance, response time, concurrency, consistency, etc.

The following is a simple guide how to verify correctly functioning session affinity.

First, configure the Undertow web server to attach information about which cluster node actually processed the request. This can be done by creating a filter that adds a header (called JBoss-Node-Name in this example) with a value of the jboss.node.name expression. This value is equal to the Kubernetes pod ID.

Secondly, after deploying any distributable application, query any URL that creates an HTTP session.

If the affinity is working correctly, these two value will match when the cluster topology is stable.

The default subsystem configuration from WildFly’s ha and full-ha profile looks like:

A singleton policy defines:

One can add a new singleton policy via the following management operation:

The singleton subsystem can be used in a non-HA profile, so long as the cache that it references uses a local-cache configuration. In this manner, an application leveraging singleton functionality (via the singleton API or using a singleton deployment descriptor) will continue function as if the server was a sole member of a cluster. For obvious reasons, the use of a quorum does not make sense in such a configuration.

A deployment indicates that it should be deployed as a singleton via a deployment descriptor. This can either be a standalone /META-INF/singleton-deployment.xml file or embedded within an existing jboss-all.xml descriptor. This descriptor may be applied to any deployment type, e.g. JAR, WAR, EAR, etc., with the exception of a subdeployment within an EAR.
e.g.

The singleton deployment descriptor defines which singleton policy should be used to deploy the application. If undefined, the default singleton policy is used, as defined by the singleton subsystem.

Using a standalone descriptor is often preferable, since it may be overlaid onto an existing deployment archive.
e.g.

The singleton subsystem registers a set of runtime metrics for each singleton deployment installed on the server.

e.g.

While singleton MSC services have been around since AS7, WildFly adds the ability to leverage the singleton subsystem to create singleton MSC services from existing singleton policies.

The singleton subsystem exposes capabilities for each singleton policy it defines.

These policies, encapsulated by the org.wildfly.clustering.singleton.service.SingletonPolicy interface, can be referenced via the following capability name: "org.wildfly.clustering.singleton.policy" + policy-name

You can reference the default singleton policy of the server via the name: "org.wildfly.clustering.singleton.default-policy" e.g.

The singleton subsystem registers a set of runtime metrics for each singleton MSC service installed via a given singleton policy.

e.g.

Alternatively, you can configure a singleton policy dynamically, which is particularly useful if you want to use a custom singleton election policy. org.wildfly.clustering.singleton.service.SingletonPolicy is a generalization of the org.wildfly.clustering.singleton.service.SingletonServiceConfiguratorFactory interface, which includes support for specifying an election policy, an election listener, and, optionally, a quorum.

The SingletonElectionPolicy is responsible for electing a member to operate as the primary singleton service provider following any change in the set of singleton service providers. Following the election of a new primary singleton service provider, any registered SingletonElectionListener is triggered on every member of the cluster.

The 'SingletonServiceConfiguratorFactory' capability may be referenced using the following capability name: "org.wildfly.clustering.cache.singleton-service-configurator-factory" + container-name + "." + cache-name

You can reference a 'SingletonServiceConfiguratorFactory' using the default cache of a given cache container via the name: "org.wildfly.clustering.cache.default-singleton-service-configurator-factory" + container-name

e.g.

A Node encapsulates a member of a group (i.e. a JGroups address). A Node has the following distinct characteristics, which will be unique for each member of the group:

A Membership is an immutable encapsulation of a group membership (i.e. a JGroups view). Membership exposes the following properties:

The Group abstract is effectively a volatile reference to the current membership, and provides a facility for notification of membership changes. It exposes the following properties and operations:

A distributed "Hello world" example that prints joiners and leavers of a group membership:

A command dispatcher is created from a CommandDispatcherFactory, an instance of which is created for every defined channel defined in the JGroups subsystem, as well as a local implementation. e.g.

To ensure that your application functions consistently regardless of server configuration, we recommended that you reference the CommandDispatcherFactory using an alias. Most users should use the "default" alias, which references either:

e.g.

Additionally, WildFly creates a CommandDispatcherFactory alias for every Infinispan cache-container, which references:

This is useful in the case where a CommandDispatcher is used to communicate with members on which a given cache is deployed.

e.g.

A Command encapsulates logic to be executed on a group member. A Command can leverage 2 type of parameters during execution:

The CommandDispatcherFactory creates a CommandDispatcher using a service identifier and a local context. This service identifier is used to segregate commands from multiple command dispatchers. A CommandDispatcher will only receive commands dispatched by a CommandDispatcher with the same service identifier.

Once created, a CommandDispatcher will locally execute any received commands until it is closed. Once closed, a CommandDispatcher is no longer allowed to dispatch commands.

The functionality of a CommandDispatcher boils down to 2 operations:

Both methods return responses as a CompletionStage, allowing for asynchronous processing of responses as they complete.

To demonstrate how to use a CommandDispatcher, let’s create a distributed "hello world" application.

First, let’s create a simple HelloWorld interface which enables the caller to send a specific message to the entire group:

Next, we need to define a local command execution context. This should encapsulate any local information we need to make available to the execution of any command received by our CommandDispatcher. For demonstration purposes, let’s make this a separate interface:

Next we create a "hello world" Command that contains a message from the sender, and responds with a message of its own.

Finally, we create a @Singleton Jakarta Enterprise Beans that implements our HelloWorld interface:

Now you can use the HelloWorld.send(…​) operation to say hello to your cluster.

The following is an example of using a ServiceProviderRegistry to publish the availability of a given singleton Jakarta Enterprise Beans. The getProviders() method will return the set of nodes on which the ServiceProviderRegistrationBean is deployed.

The following Registry example assigns a UUID to each group member, allowing each member to query the identifier of any other member:

The JGroups subsystem provides group communication support for HA services in the form of JGroups channels.

Named channel instances permit application peers in a cluster to communicate as a group and in such a way that the communication satisfies defined properties (e.g. reliable, ordered, failure-sensitive). Communication properties are configurable for each channel and are defined by the protocol stack used to create the channel. Protocol stacks consist of a base transport layer (used to transport messages around the cluster) together with a user-defined, ordered stack of protocol layers, where each protocol layer supports a given communication property.

The JGroups subsystem provides the following features:

In the following sections, we describe the JGroups subsystem.

What follows is a sample JGroups subsystem configuration showing all of the possible elements and attributes which may be configured. We shall use this example to explain the meaning of the various elements and attributes.

In many cases, channels will be configured via XML as in the example above, so that the channels will be available upon server startup. However, channels may also be added, removed or have their configurations changed in a running server by making use of the WildFly management API command-line interface (CLI). In this section, we present some key use cases for the JGroups management API.

The key use cases covered are:

KUBE_PING is a discovery protocol for JGroups cluster nodes managed by Kubernetes. Since Kubernetes is in charge of launching nodes, it knows the addresses of all pods it started, and is therefore an ideal place to ask for cluster discovery. Discovery is therefore done by asking Kubernetes for a list of addresses of all cluster nodes. Combined with bind_port / port_range, the protocol will then send a discovery request to all instances and wait for the responses.

To reconfigure an existing server profile with KUBE_PING use the following CLI batch replacing the namespace, labels and stack name (tcp) with the target stack:

For advanced configuration options, please visit protocol’s documentation here.

The org.jgroups.protocols.aws.S3_PING is a discovery protocol using AWS S3 buckets as cluster information store.

To provision using Galleon, use the jgroups-aws layer. Note that the layer does not update configuration, only provisions necessary modules.

The following minimal example updates the existing tcp stack to use this discovery protocol instead of MPING:

The S3 client used by the protocol uses the default credential store provider. Refer to AWS documentation how to generate and supply appropriate credentials. For instance, credentials can be provided by Java system properties (aws.accessKeyId and aws.secretAccessKey), environmental properties (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY), etc.

For advanced configuration options, please visit protocol’s documentation here.

A cache container manages a set of cache configurations that share the same transport and marshalling configuration. Cache containers returned by the Infinispan subsystem are auto-configured with the following customizations:

e.g. To create a new cache container that loads marshallers from the "org.bar" module:

A cache container may designate a specific cache as its default cache, i.e. the cache returned via CacheContainer.getCache():

e.g. To set "bar" as the default cache of the "foo" container:

A cache container can be injected directly into Jakarta EE applications using the @Resource annotation, e.g.

Note that the server will not manage the lifecycle of any caches created from the injected cache manager. The application is responsible for managing the lifecycle of manually created caches.

While Infinispan project is used as a library internally by WildFly to provide data distribution, Infinispan project is also distributed in a standalone server mode. The Infinispan Server cluster operates as a language-independent service accessed remotely over a number of protocols (HotRod, REST, etc).

HotRod is Infinispan’s custom optimized binary protocol which was designed to:

To learn more about the HotRod protocol itself and how to setup and run Infinispan Server, refer to Infinispan documentation for the appropriate version.

JBoss Marshalling is a serialization library for objects implementing java.io.Serializable.

When configured to use JBoss Marshalling, an application can optimize the marshalling of a given object, either through custom JDK serialization logic, or by implementing a custom externalizer. An externalizer is an implementation of the org.wildfly.clustering.marshalling.Externalizer interface, which dictates how a given class should be marshalled. An externalizer reads/writes the state of an object directly from/to an input/output stream, but also:

e.g.

Externalizers are dynamically loaded during deployment via the service loader mechanism. Implementations should be enumerated within a file named: /META-INF/services/org.wildfly.clustering.marshalling.Externalizer

ProtoStream is serialization library based on the Protobuf data format. The nature of the Protobuf data format makes it very easy to evolve classes without breaking serialization compatibility. When compared to JBoss Marshalling, ProtoStream is generally more efficient and generates smaller payloads, especially for objects containing fields that are either optional or have a default value. Since marshallable classes are explicitly enumerated, ProtoStream is not vulnerable to the same arbitrary code execution exploits that affect JDK serialization.

When configured to use ProtoStream, a web application will need to register ProtoStream marshallers/schemas for every application-specific type. WildFly initializes its ProtoStream marshaller using all instances of org.infinispan.protostream.SerializationContextInitializer that are visible to the deployment classpath. Implementations should be enumerated within a file named: /META-INF/services/org.infinispan.protostream.SerializationContextInitializer

Simple objects can leverage ProtoStream annotations and auto-generate marshallers and schemas at build time.

e.g.

Sufficiently complex objects may require a custom org.infinispan.protostream.SerializationContextInitializer implementation to register custom marshaller implementations and schemas. Refer to the Infinispan documentation for details.

We need to prepare two hosts (or virtual hosts) to do the experiment. We will use these two hosts as following:

Here are some details on what we are going to do:

Please don’t worry if you cannot digest so many details currently. Let’s move on and you will get the points step by step.

In this section we’ll setup both the primary and secondary hosts for them to run in domain mode. And we will configure the primary host to be the domain controller.

First open the host.xml in the primary host for editing:

The default settings for interface in this file is like:

We need to change the address to the management interface so the secondary Host Controller could connect to the primary Host Controller. The public interface allows the application to be accessed by non-local HTTP, and the unsecured interface allows remote RMI access. My primary host’s ip address is 10.211.55.7, so I change the config to:

Now we will setup interfaces on the secondary host. Let’s edit host.xml. Similar to the steps on the primary host, open host.xml first:

The configuration we’ll use on the secondary host is a little bit different, because we need to let the secondary Host Controller connect to the primary Host Controller. First we need to set the hostname. We change the name property from:

to:

Then we need to modify domain-controller section so the secondary Host Controller can connect to primary Host Controller management port:

As we know, 10.211.55.7 is the ip address of the primary host. You may use discovery options to define multiple mechanisms to connect to the remote domain controller:

Finally, we also need to configure interfaces section and expose the management ports to public address:

10.211.55.2 is the ip address of the secondary host. Refer to the domain controller configuration above for an explanation of the management, public, and unsecured interfaces.