WildFly Testsuite

The testsuite module has a few submodules:

It is expected that test contributions fit into one of these categories.

The pom.xml file located in the testsuite module is inherited by all submodules and is used to do the following:

It should not:

These elements should be defined in logical profiles associated with each logical grouping of tests; e.g., in the pom for the module which contains the tests. The submodule poms contain additional details of their function and purpose as well as expanded information as shown in this document.

You should not activate the abovementioned profiles by -P, because that disables other profiles which are activated by default.

Instead, you should always use activating properties, which are in parentheses in the lists below.

Testsuite profiles are used to group tests into logical groups.

They also prepare WildFly instances and resources for respective testsuite submodules.

Integration testsuite profiles configure surefire executions.

The tests can be run using:

The scripts are wrappers around Maven-based build. Their arguments are passed to Maven (with few exceptions described below). This means you can use:

Pure maven - if you prefer not to use scripts, you may achieve the same result with:

The -rf …​ parameter stands for "resume from" and causes Maven to run the specified module and all successive.

It’s possible to run only a single module (provided the ancestor modules were already run to create the AS copies) :

The -pl …​ parameter stands for "project list" and causes Maven to run the specified module only.

The plugin version needs to be specified in jboss-parent at the <properties> section of the jboss-parent pom.xml file.

See How the JBoss AS instance is built and configured for testsuite modules.

Propagated to tests through arquillian.xml:
<property name="javaVmArguments">${server.jvm.args}</property>
TBD: https://issues.redhat.com/browse/ARQ-647

WildFly instance is copied from ${jboss.dist} to testsuite/target/jbossas. This defaults to WildFly which is built by the project (build/target/wildfly-*).

testsuite/pom.xml

From ${jboss.home} to ${basedir}/target/jbossas phase generate-test-resources, goal resource-plugin:copy-resources

testsuite/integration/pom.xml

phase process-test-resources: antrun-plugin:

Which invokes:

Which invokes:

Please be sure to read Pre-requisites - test quality standards and follow those guidelines.

ManagementClient can be obtained as described below.

@ServerSetup

TBD

TBD

See Arquillian’s Resource Injection docs for more info, https://github.com/arquillian/arquillian-examples for examples.

See also Arquillian Reference.

Note to @ServerSetup annotation: It works as expected only on non-manual containers. In case of manual mode containers it calls setup() method after each server start up which is right (or actually before deployment), but the tearDown() method is called only at AfterClass event, i.e. usually after your manual shutdown of the server. Which limits you on the ability to revert some configuration changes on the server and so on. I cloned the annotation and changed it to fit the manual mode, but it is still in my github branch :)

You need to deploy the same thing twice, so two deployment methods that just return the same thing. And then you have tests that run against each.

Don’t be discouraged, it’s quite easy - a simple use case may even consist of a single short .java file.

Check WildFly test suite test cases for examples.

For more information, see WildFly Testsuite Test Developer Guide. Check the requirements for a test to be included in the testsuite.

Ask for help at WildFly Google user group or Zulip chat.

For information on how to create a GitHub account and push your code therein, see Hacking on WildFly.

Ensure the test validates current WildFly functionality. Do not test the internal functionality of included libraries, as those should be tested in the library’s upstream test suite. However, do test the integration of those libraries with WildFly.

Tests must be clear and verifiably correct. If a test is overly complex or uses questionable logic, refactor it to use clear logic or consult with the original author to clarify intent.

Before adding a test, verify that the functionality is not already covered elsewhere. Use tools like git grep to search for existing test coverage.

Test class and method names should clearly describe what is being tested. Jira IDs alone are not descriptive and require external lookup to understand the test purpose. Jira references may be included in test comments or commit messages, e.g. in Javadoc @see.

All test classes and non-trivial test methods must include Javadoc explaining what functionality is being tested and why.

When adding test coverage for functionality similar to an existing test, consider adding test methods to the existing test class rather than creating a new class. Each new test class adds overhead to execution time. Only create a new class when the functionality is sufficiently different to warrant separation.

Place integration tests in subpackages under the relevant subsystem (e.g., org.jboss.as.test.integration.ejb.async). When a test involves multiple subsystems, place it under the package for the specification that defines the primary behavior being tested.

When testing behavior mandated by specifications but not immediately obvious, include comments explaining the requirement (e.g., "Verifies Jakarta EE X.Y.Z - Description of requirement").

Place integration test resources (deployment descriptors, configuration files, etc.) in the same source directory as the test class. This keeps all test artifacts together and makes tests easier to understand.

Do not hard-code URLs, hostnames, ports, or IP addresses. Use configurable values provided by Arquillian or system properties to ensure tests work with different configurations and IPv6 addresses.

If a necessary configuration property is missing, file a Jira issue with component "Testsuite".

Do not use Thread.sleep() without checking for the actual condition being awaited. Use active waiting with timeouts or timeout mechanisms provided by the API being tested.

Make timeouts configurable with reasonable defaults. For groups of similar tests, use a shared configurable timeout value.

Always include descriptive messages in assert*() and fail() calls. Clear messages make test failures easier to diagnose.

When tests fail due to possible misconfiguration, include the relevant configuration property names and values in exception messages.

Extract configuration values such as timeouts, paths, URLs, and counts into configurable properties defined at the beginning of the test class. This improves test maintainability and flexibility.

Coding standards for production code also apply to test code. Please refer to Hacking on WildFly for the coding standards. Just to mention the most frequent violations, always include the copyright header:

and follow our import order for the project:

Use the testsuite/shared module.

Classes and resources in this module are available in all testsuite modules - i.e. in testsuite/* Only use it if necessary - don’t put things "for future use" in there. Don’t split packages across modules. Make sure the java package is unique in the WildFly project.

Document your util classes* (javadoc) so they can be easily found and reused! A generated list will be put here.

To share component’s test classes with some module in testsuite, you don’t need to split to submodules. You can create a jar with classifier using this:

This creates a jar with classifier "tests", so you can add it as dependency to a testsuite module: