Transactions propagation and recovery over EJB remoting on OpenShift

Let’s step back to define the example scenario. We have two bare-metal installations of JBoss EAP instances. The first JBoss EAP (naming it as A) calls the second one (naming it as B) with EJB remote call. Let’s depict it as A → B.

In such case, there is a call from one host with an IP address to another host. The server A starts a transaction (we say top-level transaction is started) and make the EJB call to B. Along with this call, the transaction context is passed. Passing context means there is a transaction id passed along the EJB remote call. The receiving side (server B) obtains the transaction id and it starts a child transaction on its side (we call it subordinate transaction). This subordinate transaction have set up the parent transaction being that top-level one started at server A. The subordinate transaction cannot decide independently about the transaction outcome. The subordinate transaction is driven by the top-level.

What happens when a crash of some server happens during the transaction processing? There is a process called recovery manager which periodically tries to finish transactions in error state.

Let' say the server A had crashed somewhere during the work on finishing the transaction. Then the subordinate-B transaction is sitting idle waiting for being instructed from server A. When server A is restarted it loads information about unfinished transactions. The recovery manager at server A makes the remote call to server B to finish transaction subordinate-B.

If there is the server B which crashes then server A periodically tries to connect to server B until the server B is restarted (with the same hostname/IP address it was available before the crash). If B is restarted then it accepts the call and the subordinate-B could be finished.

On top of that, the Narayana transaction manager works with the notion of node identifier. The node identifier is stored as part of the transaction id in the Narayana object store. That means that the transaction id subordinate-B is capable to provide information that node A is the node which started the top level transaction top-level-A which is its parent transaction.

Narayana uses presumed abort strategy. That simply means if there is no information found in the Narayana object store about the transaction then all discovered participants are commanded to roll-back. This for example means that if Narayana has no record about transaction subordinate-B in the object store and after restart of server A it finds that there is such transaction it ask the server B to roll-back it.

The server A finds about existence of the transaction in the server B because the server A continuously asks all the registered remote locations if they have some transactions to finish. Here the server A checks if server B has some in-doubt transaction to finish. Server B responses that there is subordinate-B. Server A does have no record in the objet store thus commands the server B to roll-back it.

It’s important to say that for Narayana JTA implementation the record is saved to object store after successful prepare (as phase of the two-phase commit procedure).

If a user deploys an EAP instance to OpenShift he considers smooth use of horizontal scaling with OpenShift primitives. That’s the JBoss EAP image is deployed under the OpenShift service and a pod is started for it. The user expects to be able to scale up and scale down the number of EAP instances running under the service. By default the offered configuration for EAP is with using the DeploymentConfig which is backed by the ReplicationController.

Use of the ReplicationController means the service provides automatic load balancing with round robin when multiple instances of EAP are started under the scope of one service. The pods started by the ReplicationController are provided with floating hostname which is constructed in a way like eap-app-<arbitrary-hash>. When the pod is rescheduled (for example the node is switched off) the pod is started again but with different hostname. Then the storage of the node is ephemeral by default. But as we said the recovery expects the data to survive restarting the pod.

Let’s show the issues that the OpenShift brings for the transaction propagation.

The solution is about to be constructed on OpenShift primitives to provides a similar environment to bare metal.

The StatefulSet brings storage and hostname stability. The StatefulSet starts pods with stable hostname which is not changed even after pod is restarted/rescheduled. It ensures the same data storage, as it was before pod restart, will be bound to the restarted pod. StatefulSet "deactivates" the service load balancing capabilities and leaves the application to manage the balancing on its own. Here the JBoss EAP clustering abilities will be used to ensures the transaction stickiness. Handling of data from orphaned object store after scale-down will be managed by functionality implemented in WildFly operator. User has to deploy the WildFly operator for the automatic scale-down functionality is available. The WildFly operator is hard requirement for running the transaction recovery fully and with guarantee of data consistency.

If we take the individual issues this setup is about to solve them.

The transaction may be finished with commit or rollback. But there is a third state of the transaction outcome which is [unknown]. That’s named as HEURISTIC and means that there is need a human intervation to decide about the transaction. This is not possible to do in automatic way. In such case the pod is left as active for an administrator may manually check it. The pod is reported as being not-terminated and it won’t be stopped until all heuristics are resolved by administrator (he needs to connect with jboss-cli.sh to the app server in the pod and resolve it).

In the current setup, the transaction propagation with recovery works only when a remote outbound connection is used. Up to that, there are some issues on transaction propagation over EJB which are related to JBEAP-13963.

Up to this the programmatic way for defining the EJB remote call (aka. dynamically call EJB without the use of remote outbound connection configuration for it) should be possible. That’s tracked as issue JBEAP-16149.

For the issues of the subordinate transactions which was fixed recently there is WFTC-52 which was causing OOM on the remote side when EJB remoting with transactions was used.

The other relate WFTC issue is issue WFTC-63 that should bring a way to store WFTC records in JDBC storage.

Then there are few minor WFTC issues about records storage as WFLY-12031 and WFTC-64.

[ ] Engineering

[x] QE