© 2017 The original authors.
1. WildFly Testsuite Overview
This section has not been updated for a long time and some sections are out of date. Contributions are welcome. |
This document will detail the implementation of the testsuite integration submodule as it guides you on adding your own test cases.
The WildFly integration test suite has been designed with the following goals:
-
support execution of all identified test case use cases
-
employ a design/organization which is scalable and maintainable
-
provide support for the automated measurement of test suite quality (generation of feature coverage reports, code coverage reports)
In addition, these requirements were considered:
-
identifying distinct test case runs of the same test case with a different set of client side parameters and server side parameters
-
separately maintaining server side execution results (e.g. logs, the original server configuration) for post-execution debugging
-
running the testsuite in conjunction with a debugger
-
the execution of a single test (for debugging purposes)
-
running test cases against different container modes (managed in the main, but also remote and embedded)
-
configuring client and server JVMs separately (e.g., IPv6 testing)
1.1. Test Suite Organization
The testsuite module has a few submodules:
-
benchmark - holds all benchmark tests intended to assess relative performance of specific feature
-
domain - holds all domain management tests
-
integration - holds all integration tests
-
stress - holds all stress tests
It is expected that test contributions fit into one of these categories.
The pom.xml
file located in the testsuite module is inherited by all
submodules and is used to do the following:
-
set defaults for common testsuite system properties (which can then be overridden on the command line)
-
define dependencies common to all tests (Arquillian, junit or testng, and container type)
-
provide a workaround for
@Resource(lookup=…)
which requires libraries in jbossas/endorsed
It should not:
-
define module-specific server configuration build steps
-
define module-specific surefire executions
These elements should be defined in logical profiles associated with each logical grouping of tests; e.g., in the pom for the module which contains the tests. The submodule poms contain additional details of their function and purpose as well as expanded information as shown in this document.
1.2. Profiles
You should not activate the abovementioned profiles by -P, because that disables other profiles which are activated by default.
Instead, you should always use activating properties, which are in parentheses in the lists below.
Testsuite profiles are used to group tests into logical groups.
-
all-modules.module.profile (all-modules)
-
integration.module.profile (integration.module)
-
compat.module.profile (compat.module)
-
domain.module.profile (domain.module)
-
benchmark.module.profile (benchmark.module)
-
stress.module.profile (stress.module)
They also prepare WildFly instances and resources for respective testsuite submodules.
-
jpda.profile - sets surefire.jpda.args (debug)
-
ds.profile - sets database properties and prepares the datasource (ds=<db id>)
-
Has related database-specific profiles, like mysql51.profile etc.
-
Integration testsuite profiles configure surefire executions.
-
smoke.integration.tests.profile
-
basic.integration.tests.profile
-
clustering.integration.tests.profile
1.3. Integration tests
1.3.1. Smoke -Dts.smoke
Contains smoke tests.
Runs by default; use -Dts.noSmoke
to prevent running.
Tests should execute quickly.
Divided into two Surefire executions:
-
One with full platform
-
Second with web profile (majority of tests).
1.3.2. Basic -Dts.basic
Basic integration tests - those which do not need a special configuration like cluster.
Divided into three Surefire executions:
-
One with full platform,
-
Second with web profile (majority of tests).
-
Third with web profile, but needs to be run after server restart to check whether persistent data are really persisted.
1.3.3. Clustering -Dts.clustering
Contains all tests relating to clustering aspects of the application server, such as:
-
web session clustering,
-
Jakarta Enterprise Beans session clustering,
-
command dispatcher,
-
web session affinity handling,
-
and other areas.
Tests should leverage shared testing logic by extending org.jboss.as.test.clustering.cluster.AbstractClusteringTestCase
.
The test case contract is that before executing the test method, all specified servers are started and all specified
deployments are deployed. This allows Arquillian resource injection into the test case.
There are four WildFly server instances, one load-balancer (Undertow) and one datagrid (Infinispan server) available for the tests.
Maven profiles and Parallelization
There are maven profiles that might come in handy for testing:
-
ts.clustering.common.profile
prepares server configurations used by test execution profiles -
ts.clustering.cluster.ha.profile
runs tests againststandalone-ha.xml
profile -
ts.clustering.cluster.fullha.profile
runs tests which requirestandalone-full-ha.xml
profile; e.g. tests requiring JMS subsystem -
ts.clustering.cluster.ha-infinispan-server.profile
runs tests againststandalone-ha.xml
profile with Infinispan Server provisioned via@ClassRule
-
ts.clustering.single.profile
runs clustering tests that are using a non-HA server profile -
ts.clustering.byteman.profile
runs clustering tests that require installation of byteman rules
For instance, to only run tests that run against full-ha profile, activate clustering tests with -Dts.clustering
and exclude
the other profiles with -P
:
$ ./integration-tests.sh clean install -Dts.noSmoke -Dts.clustering -P \!ts.clustering.cluster.ha.profile,\!ts.clustering.single.profile,\!ts.clustering.byteman.profile
If the testsuite can be run on multiple runners in parallel, the main execution (which takes the majority of the execution time)
can be split by packages using -Dts.surefire.clustering.ha.additionalExcludes
property.
This property feeds a regular expression to exclude sub-packages of the org.jboss.as.test.clustering.cluster
package.
The sub-packages at the time of writing are affinity
, cdi
, dispatcher
, ejb
, ejb2
, group
, jms
, jpa
,
jsf
, provider
, registry
, singleton
, sso
, web
, and xsite
.
For instance, to parallelize testsuite execution on two machines (e.g. when using GitHub actions scripting or alike) the following commands
could be used to split the clustering tests into two executions of similar execution time, the first node can run the first half of the tests in sub-packages, e.g.:
$ ./integration-tests.sh clean install -Dts.noSmoke -Dts.clustering -P="-ts.clustering.cluster.fullha.profile,-ts.clustering.cluster.ha-infinispan-server.profile,-ts.clustering.byteman.profile,-ts.clustering.single.profile" -Dts.surefire.clustering.ha.additionalExcludes=affinity\|cdi\|dispatcher\|ejb\|ejb2\|group\|jms\|jpa
while another node can concurrently run all the other profiles and the other half of sub-packages:
$ ./integration-tests.sh clean install -Dts.noSmoke -Dts.clustering -Dts.surefire.clustering.ha.additionalExcludes=jsf\|provider\|registry\|singleton\|sso\|web\|xsite
If the test packages get out of sync with the excludes this will result in a test running multiple times, rather than tests being omitted.
Running a single test
To run a single test, specifying -Dtest=foo
is the standard way to do this. However, this overrides the includes/excludes
section of the surefire maven plugin execution. So, in case of the clustering testsuite, the profile which this test belongs
to, needs to be specified as well. For instance, to run a single test from the 'single' test execution, exclude the other
test profiles:
$ ./integration-tests.sh clean install -Dts.noSmoke -Dts.clustering -P="-ts.clustering.cluster.ha.profile,-ts.clustering.cluster.fullha.profile,-ts.clustering.cluster.ha-infinispan-server.profile,-ts.clustering.byteman.profile,-ts.clustering.single.profile" -Dtest=org.jboss.as.test.clustering.single.dispatcher.CommandDispatcherTestCase
1.3.4. Running Infinispan Server tests against custom distribution
To run the Infinispan Server-based tests against a custom distribution, a custom location can be specified with -Dinfinispan.server.home.override=/foo/bar
and -Dinfinispan.server.profile.override=infinispan-13.0.xml
to use a corresponding server profile.
The distribution is then copied over to the build directories and patched with user credentials.
$ ./integration-tests.sh clean install -Dts.noSmoke -Dts.clustering -P="-ts.clustering.cluster.ha.profile,-ts.clustering.cluster.fullha.profile,-ts.clustering.single.profile,-ts.clustering.byteman.profile,-ts.clustering.single.profile" -Dinfinispan.server.home.override=/Users/rhusar/Downloads/redhat-datagrid-8.3.0-server
Should it be required, the Infinispan Server driver version can be also overridden by -Dversion.org.infinispan.server.driver=13.0.0.Dev03
.
2. WildFly Integration Testsuite User Guide
See also: WildFly Testsuite Test Developer Guide
Target Audience: Those interested in running the testsuite or a subset thereof, with various configuration options.
2.1. Running the testsuite
The tests can be run using:
-
build.sh
orbuild.bat
, as a part of WildFly build.-
By default, only smoke tests are run. To run all tests, run build.sh install -DallTests.
-
-
integration-tests.sh
orintegration-tests.bat
, a convenience script which uses bundled Maven (currently 3.0.3), and runs all parent testsuite modules (which configure the AS server). -
pure maven run, using
mvn install
.
The scripts are wrappers around Maven-based build. Their arguments are passed to Maven (with few exceptions described below). This means you can use:
-
build.sh (defaults to install)
-
build.sh install
-
build.sh clean install
-
integration-tests.sh install
-
…etc.
2.1.1. Supported Maven phases
Testsuite actions are bounds to various Maven phases up to verify
.
Running the build with earlier phases may fail in the submodules due to
missed configuration steps. Therefore, the only Maven phases you may
safely run, are:
-
clean
-
install
-
site
2.1.2. Testsuite structure
testsuite
integration
smoke
basic
clust
iiop
multinode
xts
compat
domain
mixed-domain
stress
benchmark
2.1.3. Test groups
To define groups of tests to be run, these properties are available:
-
-DallTests
- Runs all subgroups. -
-DallInteg
- Runs all integration tests. Same ascd testsuite/integration; mvn clean install -DallTests
-
-Dts.integ
- Basic integration + clustering tests. -
-Dts.clustering
- Clustering tests. -
-Dts.iiop
- IIOP tests. -
`-Dts.multinode `- Tests with many nodes.
-
-Dts.manualmode
- Tests with manual mode Arquillian containers. -
-Dts.bench
- Benchmark tests. -
-Dts.stress
- Stress tests. -
-Dts.domain
- Domain mode tests. -
-Dts.compat
- Compatibility tests.
2.2. Examples
-
integration-tests.sh [install] ` `
-- Runs smoke tests.
-
integration-tests.sh clean install — Cleans the target directory, then runs smoke tests.
-
integration-tests.sh install -Dts.smoke ` `
-- Same as above.
-
integration-tests.sh install -DallTests ` `
-- Runs all testsuite tests.
-
integration-tests.sh install -Dts.stress — Runs smoke tests and stress tests.
-
integration-tests.sh install -Dts.stress -Dts.noSmoke — Runs stress tests only.
Pure maven - if you prefer not to use scripts, you may achieve the same result with:
-
mvn … -rf testsuite
The -rf …
parameter stands for "resume from" and causes Maven to run
the specified module and all successive.
It’s possible to run only a single module (provided the ancestor modules were already run to create the AS copies) :
-
mvn … -pl testsuite/integration/cluster
The -pl …
parameter stands for "project list" and causes Maven to
run the specified module only.
2.2.2. Other options
-DnoWebProfile
- Run all tests with the full profile (
standalone-full.xml
). By default, most tests are run under web
profile ( standalone.xml
).
-Dts.skipTests
- Skip testsuite’s tests. Defaults to the value of
-DskipTests
, which defaults to false
. To build AS, skip unit tests
and run testsuite, use -DskipTests -Dts.skipTests=false
.
2.2.3. Timeouts
Surefire execution timeout
Unfortunatelly, no math can be done in Maven, so instead of applying a timeout ratio, you need to specify timeout manually for Surefire.
-Dsurefire.forked.process.timeout=900
In-test timeout ratios
Ratio in prercent - 100 = default, 200 = two times longer timeouts for given category.
Currently we have five different ratios. Later, it could be replaced with just one generic, one for database and one for deployment operations.
-Dtimeout.ratio.fsio=100
-Dtimeout.ratio.netio=100
-Dtimeout.ratio.memio=100
-Dtimeout.ratio.proc=100
-Dtimeout.ratio.db=100
2.2.4. Running a single test (or specified tests)
Single test is run using -Dtest=… . Examples:
-
./integration-tests.sh install
-Dtest='*Clustered*'
-Dintegration.module
-Dts.clustering
-
./integration-tests.sh clean install -Dtest=org
/
jboss
/
as
/
test
/
integration
/
ejb/async/*TestCase.java
-Dintegration.module
-Dts.basic
-
cd testsuite; mvn install
-Dtest='*Clustered*'
-Dts.basic
# No need for -Dintegration.module - integration module is active by default.
The same shortcuts listed in "Test groups" may be used to activate the module and group profile.
Note that -Dtest=
overrides <includes>
and <exludes>
defined in
pom.xml, so do not rely on them when using wildcards - all compiled test
classes matching the wildcard will be run.
Which Surefire execution is used?
Due to Surefire’s design flaw, tests run multiple times if there are
multiple surefire executions.
To prevent this, if -Dtest=
… is specified, non-default executions
are disabled, and standalone-full is used for all tests.
If you need it other way, you can overcome that need:
-
basic-integration-web.surefire with standalone.xml - Configure standalone.xml to be used as server config.
-
basic-integration-non-web.surefire - For tests included here, technically nothing changes.
-
basic-integration-2nd.surefire - Simply run the second test in another invocation of Maven.
2.2.5. Running against existing AS copy (not the one from
build/target/jboss-as-*)
-Djboss.dist=<path/to/jboss-as> will tell the testsuite to copy that AS into submodules to run the tests against.
For example, you might want to run the testsuite against AS located in
/opt/wildfly-8
:
./integration-tests.sh -DallTests -Djboss.dist=/opt/wildfly-8
The difference between jboss.dist and jboss.home:
jboss.dist is the location of the tested binaries. It gets copied to testsuite submodules.
jboss.home is internally used and points to those copied AS instances (for multinode tests, may be even different for each AS started by Arquillian).
Running against a running JBoss AS instance
Arquillian’s WildFly 32 container adapter allows specifying
allowConnectingToRunningServer
in arquillian.xml
, which makes it
check whether AS is listening at managementAddress:managementPort
, and
if so, it uses that server instead of launching a new one, and doesn’t
shut it down at the end.
All arquillian.xml’s in the testsuite specify this parameter. Thus, if you have a server already running, it will be re-used.
Running against JBoss Enterprise Application Platform (EAP) 6.0
To run the testsuite against AS included JBoss Enterprise Application Platform 6.x (EAP), special steps are needed.
Assuming you already have the sources available, and the distributed EAP
maven repository unzipped in e.g. /opt/jboss/eap6-maven-repo/
:
1) Configure maven in settings.xml to use only the EAP repository. This
repo contains all artifacts necessary for building EAP, including maven
plugins.
The build (unlike running testsuite) may be done offline.
The recommended way of configuring is to use special settings.xml, not
your local one (typically in .m2/settings.xml).
<mirror>
<id>eap6-mirror-setting</id>
<mirrorOf>
*,!central-eap6,!central-eap6-plugins,!jboss-public-eap6,!jboss-public-eap6-plugins
</mirrorOf>
<name>Mirror Settings for EAP 6 build</name>
<url>file:///opt/jboss/eap6-maven-repo</url>
</mirror>
</mirrors>
\2) Build EAP. You won’t use the resulting EAP build, though. The purpose is to get the artifacts which the testsuite depends on.
mvn clean install -s settings.xml -Dmaven.repo.local=local-repo-eap
\3) Run the testsuite. Assuming that EAP is located in /opt/eap6
, you
would run:
./integration-tests.sh -DallTests -Djboss.dist=/opt/eap6
2.2.6. Running with a debugger
Argument | What will start with debugger | Default port | Port change arg. |
---|---|---|---|
-Ddebug |
AS instances run by Arquillian |
8787 |
-Das.debug.port=… |
-Djpda |
alias for -Ddebug |
|
|
-DdebugClient |
Test JVMs (currently Surefire) |
5050 |
-Ddebug.port.surefire=… |
-DdebugCLI |
AS CLI |
5051 |
-Ddebug.port.cli=… |
Examples
./integration-tests.sh install -DdebugClient -Ddebug.port.surefire=4040
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Listening for transport dt_socket at address: 4040
./integration-tests.sh install -DdebugClient -Ddebug.port.surefire
...
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Listening for transport dt_socket at address: 5050
./integration-tests.sh install -Ddebug
./integration-tests.sh install -Ddebug -Das.debug.port=5005
JBoss AS is started by Arquillian, when the first test which requires given instance is run. Unless you pass -DtestLogToFile=false, there’s (currently) no challenge text in the console; it will look like the first test is stuck. This is being solved in http://jira.codehaus.org/browse/SUREFIRE-781. |
Depending on which test group(s) you run, multiple AS instances may be started. In that case, you need to attach the debugger multiple times. |
2.2.7. Running tests with custom database
To run with different database, specify the -Dds
and use these
properties (with the following defaults):
-Dds.jdbc.driver=
-Dds.jdbc.driver.version=
-Dds.jdbc.url=
-Dds.jdbc.user=test
-Dds.jdbc.pass=test
-Dds.jdbc.driver.jar=${ds.db}-jdbc-driver.jar
driver
is JDBC driver class. JDBC url
, user
and pass
is as
expected.
driver.version
is used for automated JDBC driver downloading. Users
can set up internal Maven repository hosting JDBC drivers, with
artifacts with
GAV = jdbcdrivers:${ds.db}:${ds.jdbc.driver.version
}
Internally, JBoss has such repo at http://nexus.qa.jboss.com:8081/nexus/content/repositories/thirdparty/jdbcdrivers/ .
The ds.db
value is set depending on ds. E.g. -Dds=mssql2005
sets
ds.db=mssql
(since they have the same driver). -Dds.db
may be
overriden to use different driver.
In case you don’t want to use such driver, set just
-Dds.db= (empty) and provide the driver to the AS manually.
Not supported; work in progress on parameter to provide JDBC Driver
jar.
2.2.8. Running tests with IPv6
-Dipv6
- Runs AS with
-Djava.net.preferIPv4Stack=false -Djava.net.preferIPv6Addresses=true
and the following defaults, overridable by respective parameter:
Parameter | IPv4 default | IPv6 default | |
---|---|---|---|
-Dnode0 |
127.0.0.1 |
::1 |
Single-node tests. |
-Dnode1 |
127.0.0.1 |
::1 |
Two-node tests (e.g. cluster) use this for the 2nd node. |
-Dmcast |
230.0.0.4 |
ff01::1 |
ff01::1 is IPv6 Node-Local scope mcast addr. |
-Dmcast.jgroupsDiag |
224.0.75.75 |
ff01::2 |
JGroups diagnostics multicast address. |
-Dmcast.modcluster |
224.0.1.105 |
ff01::3 |
mod_cluster multicast address. |
Values are set in AS configuration XML, replaced in resources (like ejb-jar.xml) and used in tests.
2.2.9. Running tests with security manager / custom security policy
-Dsecurity.manager
- Run with default policy.
-Dsecurity.policy=<path>
- Run with the given policy.
-Dsecurity.manager.other=<set of Java properties>
- Run with the given
properties. Whole set is included in all server startup parameters.
Example:
./integration-tests.sh clean install -Dintegration.module -DallTests \
\"-Dsecurity.manager.other=-Djava.security.manager \
-Djava.security.policy==$(pwd)/testsuite/shared/src/main/resources/secman/permitt_all.policy \
-Djava.security.debug=access:failure \"
Notice the \" quotes delimiting the whole -Dsecurity.manager.other
property.
2.2.10. Creating test reports
Test reports are created in the form known from EAP 5. To create them, simply run the testsuite, which will create Surefire XML files.
Creation of the reports is bound to the site
Maven phase, so it must
be run separatedly afterwards. Use one of these:
./integration-tests.sh site
cd testsuite; mvn site
mvn -pl testsuite site
Note that it will take all test results under testsuite/integration/
-
the pattern is **/*TestCase.xml
, without need to specify -DallTests
.
2.2.11. Creating coverage reports
Coverage reports are created by JaCoCo.
During the integration tests, Arquillian is passed a JVM argument which
makes it run with JaCoCo agent, which records the executions into
${basedir}/target/jacoco
.
In the site
phase, a HTML, XML and CSV reports are generated. That is
done using jacoco:report
Ant task in maven-ant-plugin
since JaCoCo’s
maven report goal doesn’t support getting classes outside
target/classes.
Usage
./build.sh clean install -DskipTests
./integration-tests.sh clean install -DallTests -Dcoverage
./integration-tests.sh site -DallTests -Dcoverage ## Must run in separatedly.
Alternative:
mvn clean install -DskipTests
mvn -rf testsuite clean install -DallTests -Dcoverage
mvn -rf testsuite site -DallTests -Dcoverage
2.2.12. Cleaning the project
To have most stable build process, it should start with:
-
clean target directories
-
only central Maven repo configured
-
clean local repository or at least:
-
free of artefacts to be built
-
free of dependencies to be used (especially snapshots)
-
To use , you may use these commands:
mvn clean install -DskipTests -DallTests ## ...to clean all testsuite modules.
mvn dependency:purge-local-repository build-helper:remove-project-artifact -Dbuildhelper.removeAll
In case the build happens in a shared environment (e.g. network disk), it’s recommended to use local repository:
cp /home/hudson/.m2/settings.xml .
sed "s|<settings>|<settings><localRepository>/home/ozizka/hudson-repos/$JOBNAME</localRepository>|" -i settings.xml
Or:
mvn clean install ... -Dmaven.repo.local=localrepo
3. WildFly Testsuite Harness Developer Guide
Audience: Whoever wants to change the testsuite harness
JIRA: WFLY-576
3.1. Testsuite requirements
http://community.jboss.org/wiki/ASTestsuiteRequirements will probably be merged here later.
3.2. Adding a new maven plugin
The plugin version needs to be specified in jboss-parent at the <properties>
section of the jboss-parent pom.xml file.
3.5. Properties and their propagation
Propagated to tests through arquillian.xml:
<property name="javaVmArguments">${server.jvm.args}</property>
TBD: https://issues.redhat.com/browse/ARQ-647
3.5.1. JBoss AS instance dir
integration/pom.xml
(currently nothing)
*-arquillian.xml
<container qualifier="jboss" default="true">
<configuration>
<property name="jbossHome">${basedir}/target/jbossas</property>
3.5.2. Server JVM arguments
<surefire.memory.args>-Xmx512m -XX:MaxPermSize=256m</surefire.memory.args>
<surefire.jpda.args></surefire.jpda.args>
<surefire.system.args>${surefire.memory.args} ${surefire.jpda.args}</surefire.system.args>
3.6. Debug parameters propagation
<surefire.jpda.args></surefire.jpda.args> - default
<surefire.jpda.args>-Xrunjdwp:transport=dt_socket,address=${as.debug.port},server=y,suspend=y</surefire.jpda.args> - activated by -Ddebug or -Djpda
testsuite/pom.xml: <surefire.system.args>... ${surefire.jpda.args} ...</surefire.system.args>
testsuite/pom.xml: <jboss.options>${surefire.system.args}</jboss.options>
testsuite/integration/pom.xml: <server.jvm.args>${surefire.system.args} ${jvm.args.ip.server} ${jvm.args.security} ${jvm.args.timeouts} -Dnode0=${node0} -Dnode1=
integration/pom.xml:
<server.jvm.args>${surefire.system.args} ${jvm.args.ip.server} ${jvm.args.security} ${jvm.args.timeouts} -Dnode0=${node0} -Dnode1=${node1} -DudpGroup=${udpGroup} ${jvm.args.dirs}</server.jvm.args>
arquillian.xml:
<property name="javaVmArguments">${server.jvm.args} -Djboss.inst=${basedir}/target/jbossas</property>
4. How the WildFly is built and configured for testsuite modules.
Refer to Shortened Maven Run Overview to see the mentioned build steps.
\1) AS instance is copied from ${jboss.dist
} to
testsuite/target/jbossas
.
Defaults to AS which is built by the project (
build/target/jboss-as-*
).
2)
testsuite/pom.xml:
from $\{jboss.home} to ${basedir}/target/jbossas
phase generate-test-resources: resource-plugin, goal copy-resources
testsuite/integration/pom.xml:
phase process-test-resources: antrun-plugin:
<ant antfile="$\{basedir}/src/test/scripts/basic-integration-build.xml">
<target name="build-basic-integration"/>
<target name="build-basic-integration-jts"/>
</ant>
Which invokes
<target name="build-basic-integration" description="Builds server configuration for basic-integration tests">
<build-server-config name="jbossas"/>
Which invokes
<!-- Copy the base distribution. -->
<!-- We exclude modules and bundles as they are read-only and we locate the via sys props. -->
<copy todir="@{output.dir}/@{name}">
<fileset dir="@{jboss.dist}">
<exclude name="**/modules/**"/>
<exclude name="**/bundles/**"/>
</fileset>
</copy>
<!-- overwrite with configs from test-configs and apply property filtering -->
<copy todir="@{output.dir}/@{name}" overwrite="true" failonerror="false">
<fileset dir="@{test.configs.dir}/@{name}"/>
<filterset begintoken="${" endtoken="}">
<filter token="node0" value="${node0}"/>
<filter token="node1" value="${node1}"/>
<filter token="udpGroup" value="${udpGroup}"/>
<filter-elements/>
</filterset>
</copy>
5. Plugin executions matrix
x - runs in this module
xx - runs in this and all successive modules
x! - runs but should not.
TS | integ | smoke | basic | clust | iiop | comp | domain | bench | stress | |
---|---|---|---|---|---|---|---|---|---|---|
initialize |
||||||||||
maven-help-plugin |
xx |
x |
x |
x |
x |
x |
x |
x |
x |
x |
properties-maven-plugin:write-project-properties |
x |
|
|
|
|
|
|
|
|
|
maven-antrun-plugin:1.6:run (banner) |
|
|
|
|
|
|
|
|
|
|
process-resources |
||||||||||
maven-resources-plugin:2.5:resources (default-resources) |
xx |
|
|
|
|
|
|
|
|
|
maven-dependency-plugin:2.3:copy (copy-annotations-endorsed) |
xx! |
|
|
|
|
|
|
|
|
|
compile |
||||||||||
maven-compiler-plugin:2.3.2:compile (default-compile) |
xx |
|
|
|
|
|
|
|
|
|
generate-test-resources |
||||||||||
maven-resources-plugin:2.5:copy-resources (build-jbossas.server) |
xx! |
|
|
|
|
|
|
|
|
|
Should be: |
x |
|
|
|
|
|
|
|
|
|
maven-resources-plugin:2.5:copy-resources (ts.copy-jbossas) |
|
x |
|
|
|
|
|
|
|
|
maven-resources-plugin:2.5:copy-resources (ts.copy-jbossas.groups) |
|
x |
x |
x |
? |
? |
? |
! |
|
|
Should be: |
|
xx |
x |
x |
x |
x |
x |
x |
x |
x |
process-test-resources |
||||||||||
maven-resources-plugin:2.5:testResources (default-testResources) |
xx |
|
|
|
|
|
|
|
|
|
maven-antrun-plugin:1.6:run (build-smoke.server) |
|
|
x |
|
|
|
|
|
|
|
maven-antrun-plugin:1.6:run (prepare-jars-basic-integration.server) |
|
|
|
x |
|
|
|
|
|
|
maven-antrun-plugin:1.6:run (build-clustering.server) |
|
|
|
|
x |
x!? |
|
|
|
|
test-compile |
||||||||||
maven-compiler-plugin:2.3.2:testCompile (default-testCompile) |
xx |
|
|
|
|
|
|
|
|
|
xml-maven-plugin:1.0:transform (update-ip-addresses-jbossas.server) |
x |
|
|
|
|
|
|
|
|
|
maven-antrun-plugin:1.6:run (build-jars) |
|
|
|
|
|
|
x |
|
|
|
test |
||||||||||
maven-surefire-plugin:2.10:test (smoke-full.surefire) |
|
|
|
|
|
|
|
|
|
|
maven-surefire-plugin:2.10:test (smoke-web.surefire) |
|
|
|
|
|
|
|
|
|
|
maven-surefire-plugin:2.10:test (default-test) |
|
|
|
|
x |
x |
x |
x |
|
|
maven-surefire-plugin:2.10:test (basic-integration-default-full.surefire) |
|
|
|
x |
|
|
|
|
|
|
maven-surefire-plugin:2.10:test (basic-integration-default-web.surefire) |
|
|
|
x |
|
|
|
|
|
|
maven-surefire-plugin:2.10:test (basic-integration-2nd.surefire) |
|
|
|
x |
|
|
|
|
|
|
maven-surefire-plugin:2.10:test (tests-clust-multi-node-unm…surefire) |
|
|
|
|
x |
|
|
|
|
|
maven-surefire-plugin:2.10:test (tests-clustering-single-node.surefire) |
|
|
|
|
x |
|
|
|
|
|
maven-surefire-plugin:2.10:test (tests-clustering-multi-node.surefire) |
|
|
|
|
x |
|
|
|
|
|
maven-surefire-plugin:2.10:test (tests-iiop-multi-node.surefire) |
|
|
|
|
|
x |
|
|
|
|
package |
||||||||||
maven-jar-plugin:2.3.1:jar (default-jar) |
xx! |
|
|
|
|
|
|
|
|
|
maven-source-plugin:2.1.2:jar-no-fork (attach-sources) |
x |
|
|
|
|
|
|
|
|
|
install |
||||||||||
maven-install-plugin:2.3.1:install (default-install) |
xx! |
|
|
|
|
|
|
|
|
|
|
TS |
integ |
smoke |
basic |
clust |
iiop |
comp |
domain |
bench |
stress |
6. Shortened Maven Run Overview
6.1. How to get it
./integration-tests.sh clean install -DallTests | tee TS.txt | testsuite/tools/runSummary.sh
6.2. How it’s done
Run this script on the output of the AS7 testsuite run:
## Cat the file or stdin if no args,
## filter only interesting lines - plugin executions and modules separators,
## plus Test runs summaries,
## and remove the boring plugins like enforcer etc.
cat $1 \
| egrep ' --- |Building| ---------|Tests run: | T E S T S' \
| grep -v 'Time elapsed'
| sed 's|Tests run:| Tests run:|' \
| grep -v maven-clean-plugin \
| grep -v maven-enforcer-plugin \
| grep -v buildnumber-maven-plugin \
| grep -v maven-help-plugin \
| grep -v properties-maven-plugin:.*:write-project-properties \
;
You’ll get an overview of the run.
6.3. Example output with comments.
ondra@ondra-redhat: ~/work/AS7/ozizka-as7 $ ./integration-tests.sh clean install -DallTests | tee TS.txt | testsuite/tools/runSummary.sh
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Building JBoss Application Server Test Suite: Aggregator 7.1.0.CR1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] --- maven-dependency-plugin:2.3:copy (copy-annotations-endorsed) @ jboss-as-testsuite ---
Copies org.jboss.spec.javax.annotation:jboss-annotations-api_1.1_spec to ${project.build.directory}/endorsed .
Inherited - needed for compilation of all submodules.
[INFO] --- maven-resources-plugin:2.5:copy-resources (build-jbossas.server) @ jboss-as-testsuite ---
Copies ${jboss.home} to target/jbossas . TODO: Should be jboss.dist.
[INFO] --- xml-maven-plugin:1.0:transform (update-ip-addresses-jbossas.server) @ jboss-as-testsuite ---
Changes IP addresses used in server config files -
applies ${xslt.scripts.dir}/changeIPAddresses.xsl on ${basedir}/target/jbossas/standalone/configuration/standalone-*.xml
Currently inherited, IMO should not be.
[INFO] --- maven-source-plugin:2.1.2:jar-no-fork (attach-sources) @ jboss-as-testsuite ---
TODO: Remove
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ jboss-as-testsuite ---
[INFO] ------------------------------------------------------------------------
[INFO] Building JBoss Application Server Test Suite: Integration Aggregator 7.1.0.CR1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] --- maven-dependency-plugin:2.3:copy (copy-annotations-endorsed) @ jboss-as-testsuite-integration-agg ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (build-jbossas.server) @ jboss-as-testsuite-integration-agg ---
TODO: Remove
[INFO] --- maven-resources-plugin:2.5:copy-resources (ts.copy-jbossas) @ jboss-as-testsuite-integration-agg ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (ts.copy-jbossas.groups) @ jboss-as-testsuite-integration-agg ---
[INFO] --- xml-maven-plugin:1.0:transform (update-ip-addresses-jbossas.server) @ jboss-as-testsuite-integration-agg ---
TODO: Remove
[INFO] --- maven-source-plugin:2.1.2:jar-no-fork (attach-sources) @ jboss-as-testsuite-integration-agg ---
TODO: Remove
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ jboss-as-testsuite-integration-agg ---
[INFO] ------------------------------------------------------------------------
[INFO] Building JBoss AS Test Suite: Integration - Smoke 7.1.0.CR1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] --- maven-dependency-plugin:2.3:copy (copy-annotations-endorsed) @ jboss-as-testsuite-integration-smoke ---
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ jboss-as-testsuite-integration-smoke ---
TODO: Remove
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ jboss-as-testsuite-integration-smoke ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (build-jbossas.server) @ jboss-as-testsuite-integration-smoke ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (ts.copy-jbossas.groups) @ jboss-as-testsuite-integration-smoke ---
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ jboss-as-testsuite-integration-smoke ---
[INFO] --- xml-maven-plugin:1.0:transform (update-ip-addresses-jbossas.server) @ jboss-as-testsuite-integration-smoke ---
TODO: Remove
[INFO] --- maven-antrun-plugin:1.6:run (build-smoke.server) @ jboss-as-testsuite-integration-smoke ---
[echo] Building AS instance "smoke" from /home/ondra/work/EAP/EAP6-DR9 to /home/ondra/work/AS7/ozizka-as7/testsuite/integration/smoke/target
TODO: Should be running one level above!
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ jboss-as-testsuite-integration-smoke ---
[INFO] --- maven-surefire-plugin:2.10:test (smoke-full.surefire) @ jboss-as-testsuite-integration-smoke ---
T E S T S
Tests run: 4, Failures: 0, Errors: 4, Skipped: 0
6.4. Example output, unchanged
ondra@lenovo:~/work/AS7/ozizka-git$ ./integration-tests.sh clean install -DallTests | tee TS.txt | testsuite/tools/runSummary.sh
SSCmeetingWestfordJan [copy] Warning: /home/ondra/work/AS7/ozizka-git/testsuite/integration/src/test/resources/test-configs/smoke does not exist.
[copy] Warning: /home/ondra/work/AS7/ozizka-git/testsuite/integration/src/test/resources/test-configs/clustering-udp-0 does not exist.
[copy] Warning: /home/ondra/work/AS7/ozizka-git/testsuite/integration/src/test/resources/test-configs/clustering-udp-1 does not exist.
[copy] Warning: /home/ondra/work/AS7/ozizka-git/testsuite/integration/src/test/resources/test-configs/iiop-client does not exist.
[copy] Warning: /home/ondra/work/AS7/ozizka-git/testsuite/integration/src/test/resources/test-configs/iiop-server does not exist.
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Building JBoss Application Server Test Suite: Aggregator 7.1.0.Final-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] --- maven-antrun-plugin:1.6:run (banner) @ jboss-as-testsuite ---
[INFO] --- maven-dependency-plugin:2.3:copy (copy-annotations-endorsed) @ jboss-as-testsuite ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (build-jbossas.server) @ jboss-as-testsuite ---
[INFO] --- xml-maven-plugin:1.0:transform (update-ip-addresses-jbossas.server) @ jboss-as-testsuite ---
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ jboss-as-testsuite ---
[INFO] ------------------------------------------------------------------------
[INFO] Building JBoss Application Server Test Suite: Integration 7.1.0.Final-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] --- maven-dependency-plugin:2.3:copy (copy-annotations-endorsed) @ jboss-as-testsuite-integration-agg ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (build-jbossas.server) @ jboss-as-testsuite-integration-agg ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (ts.copy-jbossas) @ jboss-as-testsuite-integration-agg ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (ts.copy-jbossas.groups) @ jboss-as-testsuite-integration-agg ---
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ jboss-as-testsuite-integration-agg ---
[INFO] ------------------------------------------------------------------------
[INFO] Building JBoss Application Server Test Suite: Integration - Smoke 7.1.0.Final-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] --- maven-dependency-plugin:2.3:copy (copy-annotations-endorsed) @ jboss-as-testsuite-integration-smoke ---
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ jboss-as-testsuite-integration-smoke ---
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ jboss-as-testsuite-integration-smoke ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (build-jbossas.server) @ jboss-as-testsuite-integration-smoke ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (ts.copy-jbossas.groups) @ jboss-as-testsuite-integration-smoke ---
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ jboss-as-testsuite-integration-smoke ---
[INFO] --- maven-antrun-plugin:1.6:run (build-smoke.server) @ jboss-as-testsuite-integration-smoke ---
[echo] Building AS instance "smoke" from /home/ondra/work/AS7/ozizka-git/testsuite/integration/smoke/../../../build/target/jboss-as-7.1.0.Final-SNAPSHOT to /home/ondra/work/AS7/ozizka-git/testsuite/integration/smoke/target
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ jboss-as-testsuite-integration-smoke ---
[INFO] --- maven-surefire-plugin:2.10:test (smoke-full.surefire) @ jboss-as-testsuite-integration-smoke ---
T E S T S
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
[INFO] --- maven-surefire-plugin:2.10:test (smoke-web.surefire) @ jboss-as-testsuite-integration-smoke ---
T E S T S
Tests run: 116, Failures: 0, Errors: 0, Skipped: 6
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ jboss-as-testsuite-integration-smoke ---
[INFO] Building jar: /home/ondra/work/AS7/ozizka-git/testsuite/integration/smoke/target/jboss-as-testsuite-integration-smoke-7.1.0.Final-SNAPSHOT.jar
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ jboss-as-testsuite-integration-smoke ---
[INFO] ------------------------------------------------------------------------
[INFO] Building JBoss Application Server Test Suite: Integration - Basic 7.1.0.Final-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] --- maven-dependency-plugin:2.3:copy (copy-annotations-endorsed) @ jboss-as-testsuite-integration-basic ---
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ jboss-as-testsuite-integration-basic ---
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ jboss-as-testsuite-integration-basic ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (build-jbossas.server) @ jboss-as-testsuite-integration-basic ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (ts.copy-jbossas.groups) @ jboss-as-testsuite-integration-basic ---
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ jboss-as-testsuite-integration-basic ---
[INFO] --- maven-antrun-plugin:1.6:run (prepare-jars-basic-integration.server) @ jboss-as-testsuite-integration-basic ---
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ jboss-as-testsuite-integration-basic ---
[INFO] --- maven-surefire-plugin:2.10:test (basic-integration-default-full.surefire) @ jboss-as-testsuite-integration-basic ---
T E S T S
Tests run: 323, Failures: 0, Errors: 4, Skipped: 30
[INFO] ------------------------------------------------------------------------
[INFO] Building JBoss Application Server Test Suite: Integration - Clustering 7.1.0.Final-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] --- maven-dependency-plugin:2.3:copy (copy-annotations-endorsed) @ jboss-as-testsuite-integration-clust ---
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ jboss-as-testsuite-integration-clust ---
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ jboss-as-testsuite-integration-clust ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (build-jbossas.server) @ jboss-as-testsuite-integration-clust ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (ts.copy-jbossas.groups) @ jboss-as-testsuite-integration-clust ---
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ jboss-as-testsuite-integration-clust ---
[INFO] --- maven-antrun-plugin:1.6:run (build-clustering.server) @ jboss-as-testsuite-integration-clust ---
[echo] Building config clustering-udp-0
[echo] Building AS instance "clustering-udp-0" from /home/ondra/work/AS7/ozizka-git/testsuite/integration/clust/../../../build/target/jboss-as-7.1.0.Final-SNAPSHOT to /home/ondra/work/AS7/ozizka-git/testsuite/integration/clust/target
[echo] Building config clustering-udp-1
[echo] Building AS instance "clustering-udp-1" from /home/ondra/work/AS7/ozizka-git/testsuite/integration/clust/../../../build/target/jboss-as-7.1.0.Final-SNAPSHOT to /home/ondra/work/AS7/ozizka-git/testsuite/integration/clust/target
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ jboss-as-testsuite-integration-clust ---
[INFO] --- maven-surefire-plugin:2.10:test (tests-clustering-multi-node-unmanaged.surefire) @ jboss-as-testsuite-integration-clust ---
T E S T S
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0
[INFO] --- maven-surefire-plugin:2.10:test (tests-clustering-single-node.surefire) @ jboss-as-testsuite-integration-clust ---
T E S T S
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] --- maven-surefire-plugin:2.10:test (tests-clustering-multi-node.surefire) @ jboss-as-testsuite-integration-clust ---
T E S T S
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ jboss-as-testsuite-integration-clust ---
[INFO] Building jar: /home/ondra/work/AS7/ozizka-git/testsuite/integration/clust/target/jboss-as-testsuite-integration-clust-7.1.0.Final-SNAPSHOT.jar
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ jboss-as-testsuite-integration-clust ---
[INFO] ------------------------------------------------------------------------
[INFO] Building JBoss Application Server Test Suite: Integration - IIOP 7.1.0.Final-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] --- maven-dependency-plugin:2.3:copy (copy-annotations-endorsed) @ jboss-as-testsuite-integration-iiop ---
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ jboss-as-testsuite-integration-iiop ---
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ jboss-as-testsuite-integration-iiop ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (build-jbossas.server) @ jboss-as-testsuite-integration-iiop ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (ts.copy-jbossas.groups) @ jboss-as-testsuite-integration-iiop ---
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ jboss-as-testsuite-integration-iiop ---
[INFO] --- maven-antrun-plugin:1.6:run (build-clustering.server) @ jboss-as-testsuite-integration-iiop ---
[echo] Building config iiop-client
[echo] Building AS instance "iiop-client" from /home/ondra/work/AS7/ozizka-git/testsuite/integration/iiop/../../../build/target/jboss-as-7.1.0.Final-SNAPSHOT to /home/ondra/work/AS7/ozizka-git/testsuite/integration/iiop/target
[echo] Building config iiop-server
[echo] Building AS instance "iiop-server" from /home/ondra/work/AS7/ozizka-git/testsuite/integration/iiop/../../../build/target/jboss-as-7.1.0.Final-SNAPSHOT to /home/ondra/work/AS7/ozizka-git/testsuite/integration/iiop/target
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ jboss-as-testsuite-integration-iiop ---
[INFO] --- maven-surefire-plugin:2.10:test (tests-iiop-multi-node.surefire) @ jboss-as-testsuite-integration-iiop ---
T E S T S
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ jboss-as-testsuite-integration-iiop ---
[INFO] Building jar: /home/ondra/work/AS7/ozizka-git/testsuite/integration/iiop/target/jboss-as-testsuite-integration-iiop-7.1.0.Final-SNAPSHOT.jar
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ jboss-as-testsuite-integration-iiop ---
[INFO] ------------------------------------------------------------------------
[INFO] Building JBoss Application Server Test Suite: Compatibility Tests 7.1.0.Final-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] --- maven-dependency-plugin:2.3:copy (copy-annotations-endorsed) @ jboss-as-testsuite-integration-compat ---
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ jboss-as-testsuite-integration-compat ---
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ jboss-as-testsuite-integration-compat ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (build-jbossas.server) @ jboss-as-testsuite-integration-compat ---
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ jboss-as-testsuite-integration-compat ---
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ jboss-as-testsuite-integration-compat ---
[INFO] --- maven-antrun-plugin:1.6:run (build-jars) @ jboss-as-testsuite-integration-compat ---
[INFO] --- maven-surefire-plugin:2.10:test (default-test) @ jboss-as-testsuite-integration-compat ---
T E S T S
Tests run: 7, Failures: 0, Errors: 4, Skipped: 3
[INFO] ------------------------------------------------------------------------
[INFO] Building JBoss Application Server Test Suite: Domain Mode Integration Tests 7.1.0.Final-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] --- maven-dependency-plugin:2.3:copy (copy-annotations-endorsed) @ jboss-as-testsuite-integration-domain ---
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ jboss-as-testsuite-integration-domain ---
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ jboss-as-testsuite-integration-domain ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (build-jbossas.server) @ jboss-as-testsuite-integration-domain ---
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ jboss-as-testsuite-integration-domain ---
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ jboss-as-testsuite-integration-domain ---
[INFO] --- maven-surefire-plugin:2.10:test (default-test) @ jboss-as-testsuite-integration-domain ---
T E S T S
Tests run: 89, Failures: 0, Errors: 0, Skipped: 4
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ jboss-as-testsuite-integration-domain ---
[INFO] Building jar: /home/ondra/work/AS7/ozizka-git/testsuite/domain/target/jboss-as-testsuite-integration-domain-7.1.0.Final-SNAPSHOT.jar
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ jboss-as-testsuite-integration-domain ---
[INFO] ------------------------------------------------------------------------
[INFO] Building JBoss Application Server Test Suite: Benchmark Tests 7.1.0.Final-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] --- maven-dependency-plugin:2.3:copy (copy-annotations-endorsed) @ jboss-as-testsuite-benchmark ---
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ jboss-as-testsuite-benchmark ---
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ jboss-as-testsuite-benchmark ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (build-jbossas.server) @ jboss-as-testsuite-benchmark ---
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ jboss-as-testsuite-benchmark ---
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ jboss-as-testsuite-benchmark ---
[INFO] --- maven-surefire-plugin:2.10:test (default-test) @ jboss-as-testsuite-benchmark ---
T E S T S
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ jboss-as-testsuite-benchmark ---
[INFO] Building jar: /home/ondra/work/AS7/ozizka-git/testsuite/benchmark/target/jboss-as-testsuite-benchmark-7.1.0.Final-SNAPSHOT.jar
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ jboss-as-testsuite-benchmark ---
[INFO] ------------------------------------------------------------------------
[INFO] Building JBoss Application Server Test Suite: Stress Tests 7.1.0.Final-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] --- maven-dependency-plugin:2.3:copy (copy-annotations-endorsed) @ jboss-as-testsuite-stress ---
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ jboss-as-testsuite-stress ---
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ jboss-as-testsuite-stress ---
[INFO] --- maven-resources-plugin:2.5:copy-resources (build-jbossas.server) @ jboss-as-testsuite-stress ---
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ jboss-as-testsuite-stress ---
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ jboss-as-testsuite-stress ---
[INFO] --- maven-surefire-plugin:2.10:test (default-test) @ jboss-as-testsuite-stress ---
T E S T S
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ jboss-as-testsuite-stress ---
[INFO] Building jar: /home/ondra/work/AS7/ozizka-git/testsuite/stress/target/jboss-as-testsuite-stress-7.1.0.Final-SNAPSHOT.jar
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ jboss-as-testsuite-stress ---
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
7. WildFly Testsuite Test Developer Guide
See also: WildFly Integration Testsuite User Guide
7.1. Pre-requisites
Please be sure to read Pre-requisites - test quality standards and follow those guidelines.
7.3. ManagementClient and ModelNode usage example
final ModelNode operation = new ModelNode();
operation.get(ModelDescriptionConstants.OP).set(ModelDescriptionConstants.READ_RESOURCE_OPERATION);
operation.get(ModelDescriptionConstants.OP_ADDR).set(address);
operation.get(ModelDescriptionConstants.RECURSIVE).set(true);
final ModelNode result = managementClient.getControllerClient().execute(operation);
Assert.assertEquals(ModelDescriptionConstants.SUCCESS, result.get(ModelDescriptionConstants.OUTCOME).asString());
ManagementClient can be obtained as described below.
7.4. Arquillian features available in tests
@ServerSetup
TBD
@ContainerResource private ManagementClient managementClient;
final ModelNode result = managementClient.getControllerClient().execute(operation);
TBD
@ArquillianResource private ManagementClient managementClient;
ModelControllerClient client = managementClient.getControllerClient();
@ArquillianResource ContainerController cc;
@Test
public void test() {
cc.setup("test", ...properties..)
cc.start("test")
}
<arquillian>
<container qualifier="test" mode="manual" />
</arquillian>
// Targeted containers HTTP context.
@ArquillianResource URL url;
// Targeted containers HTTP context where servlet is located.
@ArquillianResource(SomeServlet.class) URL url;
// Targeted containers initial context.
@ArquillianResource InitialContext|Context context;
// The manual deployer.
@ArquillianResource Deployer deployer;
See Arquillian’s Resource Injection docs for more info, https://github.com/arquillian/arquillian-examples for examples.
See also Arquillian Reference.
Note to @ServerSetup annotation: It works as expected only on non-manual containers. In case of manual mode containers it calls setup() method after each server start up which is right (or actually before deployment), but the tearDown() method is called only at AfterClass event, i.e. usually after your manual shutdown of the server. Which limits you on the ability to revert some configuration changes on the server and so on. I cloned the annotation and changed it to fit the manual mode, but it is still in my github branch :)
7.5. Properties available in tests
7.5.1. Directories
-
jbosssa.project.dir - Project’s root dir (where ./build.sh is).
-
jbossas.ts.dir - Testsuite dir.
-
jbossas.ts.integ.dir - Testsuite’s integration module dir.
-
jboss.dist - Path to AS distribution, either built (build/target/jboss-as-…) or user-provided via -Djboss.dist
-
jboss.inst - (Arquillian in-container only) Path to the AS instance in which the test is running (until ARQ-650 is possibly done)
-
jboss.home - Deprecated as it’s name is unclear and confusing. Use jboss.dist or jboss.inst.
7.5.3. Time-related coefficients (ratios)
In case some of the following causes timeouts, you may prolong the timeouts by setting value >= 100:
100 = leave as is,
150 = 50 % longer, etc.
-
timeout.ratio.gen - General ratio - can be used to adjust all timeouts.When this and specific are defined, both apply.
-
timeout.ratio.fs- Filesystem IO
-
timeout.ratio.net - Network IO
-
timeout.ratio.mem - Memory IO
-
timeout.ratio.cpu - Processor
-
timeout.ratio.db - Database
Time ratios will soon be provided by
org.jboss.as.test.shared.time.TimeRatio.for*()
methods.
7.6. Negative tests
To test invalid deployment handling: @ShouldThrowException
Currently doesn’t work due to WFLY-673.
optionally you might be able to catch it using the manual deployer
@Deployment(name = "X", managed = false) ...
@Test
public void shouldFail(@ArquillianResource Deployer deployer) throws Exception {
try {
deployer.deploy("X")
}
catch(Exception e) {
// do something
}
}
7.7. Clustering tests (WFLY-616)
You need to deploy the same thing twice, so two deployment methods that
just return the same thing.
And then you have tests that run against each.
@Deployment(name = "deplA", testable = false)
@TargetsContainer("serverB")
public static Archive<?> deployment()
@Deployment(name = "deplB", testable = false)
@TargetsContainer("serverA")
public static Archive<?> deployment(){ ... }
@Test
@OperateOnDeployment("deplA")
public void testA(){ ... }
@Test
@OperateOnDeployment("deplA")
public void testA() {...}
7.8. How to get the tests to main branch
-
First of all, be sure to read the "Before you add a test" section.
-
Fetch the newest main:
git fetch upstream # Provided you have the wildfly/wildfly GitHub repo
as a remote called 'upstream'.
-
Rebase your branch: git checkout WFLY-1234-your-branch; git rebase upstream/main
-
Run whole testsuite (integration-tests -DallTests). You may use https://ci.wildfly.org/viewType.html?buildTypeId=WF_Nightly.
-
If any tests fail and they do not fail in main, fix it and go back to the "Fetch" step.
-
-
Push to a new branch in your GitHub repo:
git push origin WFLY-1234-new-XY-tests
-
Create a pull-request on GitHub. Go to your branch and click on "Pull Request".
-
If you have a jira, start the title with it, like - WFLY-1234 New tests for XYZ.
-
If you don’t, write some apposite title. In the description, describe in detail what was done and why should it be merged. Keep in mind that the diff will be visible under your description.
-
-
Keep the branch rebased daily until it’s merged (see the Fetch step). If you don’t, you’re dramatically decreasing chance to get it merged.
-
You might have someone with merge privileges to cooperate with you, so they know what you’re doing, and expect your pull request.
-
When your pull request is reviewed and merged, you’ll be notified by mail from GitHub.
-
You may also check if it was merged by the following:
git fetch upstream; git cherry
<branch> ## Or
git branch --contains\{\{<branch> - see}}here
-
Your commits will appear in main. They will have the same hash as in your branch.
-
You are now safe to delete both your local and remote branches:
git branch -D WFLY-1234-your-branch; git push origin :WFLY-1234-your-branch
-
8. How to Add a Test Case
(Please don’t (re)move - this is a landing page from a Jira link.)
Thank you for finding time to contribute to WildFly 32 quality.
Covering corner cases found by community users with tests is very
important to increase stability.
If you’re providing a test case to support your bug report, it’s very
likely that your bug will be fixed much sooner.
8.1. 1) Create a test case.
It’s quite easy - a simple use case may even consist of one short .java file.
Check WildFly 32 test suite test cases for examples.
For more information, see WildFly Testsuite Test Developer Guide. Check the requirements for a test to be included in the testsuite.
Ask for help at WildFly 32 forum or at IRC - #wildfly @ FreeNode.
8.2. 2) Push your test case to GitHub and create a pull request.
For information on how to create a GitHub account and push your code therein, see Hacking on WildFly.
If you’re not into Git, send a diff file to JBoss forums, someone might pick it up.
9. Before you add a test
Every added test, whether ported or new should follow the same guidelines:
9.1. Verify the test belongs in WildFly
AS6 has a lot of tests for things that are discontinued. For example
the
legacy JBoss Transaction Manager which was replaced by Arjuna. Also we
had tests for SPIs that no longer exist. None of these things should
be
migrated.
9.2. Only add CORRECT and UNDERSTANDABLE tests
If you don’t understand what a test is doing (perhaps too complex), or
it’s going about things in a strange way that might not be correct,
THEN
DO NOT PORT IT. Instead we should have a simpler, understandable, and
correct test. Write a new one, ping the author, or skip it altogether.
9.3. Do not add duplicate tests
Always check that the test you are adding doesn’t have coverage
elsewhere (try using "git grep"). As mentioned above we have some
overlap between 6 and 7. The 7 test version will likely be better.
9.4. Don’t name tests after JIRAs
A JIRA number is useless without an internet connection, and they are
hard to read. If I get a test failure thats XDGR-843534578 I have to
dig
to figure out the context. It’s perfectly fine though to link to a
JIRA
number in the comments of the test. Also the commit log is always
available.
9.5. Tests should contain javadoc that explains what is being tested
This is especially critical if the test is non-trivial
9.6. Prefer expanding an EXISTING test over a new test class
If you are looking at migrating or creating a test with similar
functionality to an exiting test, it is better to
expand upon the existing one by adding more test methods, rather than
creating a whole new test. In general each
new test class adds at least 300ms to execution time, so as long as it
makes sense it is better to add it to an
existing test case.
9.7. Organize tests by subsystem
Integration tests should be packaged in subpackages under the relevant
subsystem (e.g org.jboss.as.test.integration.ejb.async). When a test
impacts multiple subsystems this is a bit of a judgement call, but in
general the tests should go into the package of
the spec that defines the functionality (e.g. Jakarta Contexts and Dependency Injection based constructor
injection into an Jakarta Enterprise Beans, even though this involves Jakarta Contexts and Dependency Injection and Jakarta Enterprise Beans,
the Jakarta Contexts and Dependency Injection spec defines this behaviour)
9.8. Explain non-obvious spec behavior in comments
The EE spec is full of odd requirements. If the test is covering
behavior that is not obvious then please add something like "Verifies
EE
X.T.Z - The widget can’t have a foobar if it is declared like blah"
9.9. Put integration test resources in the source directory of the test
At the moment there is not real organization of these files. It makes
sense for most apps to have this separation, however the testsuite is
different. e.g. most apps will have a single deployment descriptor of
a
given type, for the testsuite will have hundreds, and maintaining
mirroring
package structures is error prone.
This also makes the tests easier to understand, as all the artifacts
in
the deployment are in one place, and that place tends to be small
(only
a handful of files).
9.10. Do not hard-code values likely to need configuration (URLs, ports, …)
URLs hardcoded to certain address (localhost) or port (like the default
8080 for web) prevent running the test against different address or with
IPv6 adress.
Always use the configurable values provided by Arquillian or as a system
property.
If you come across a value which is not configurable but you think it
should be, file an WildFly 32 jira issue with component "Test suite".
See
@ArquillianResourrce
usage example.
9.11. Follow best committing practices
-
Only do changes related to the topic of the jira/pull request.
-
Do not clutter your pull request with e.g. reformatting, fixing typos spotted along the way - do another pull request for such.
-
Prefer smaller changes in more pull request over one big pull request which are difficult to merge.
-
Keep the code consistent across commits - e.g. when renaming something, be sure to update all references to it.
-
Describe your commits properly as they will appear in main’s linear history.
-
If you’re working on a jira issue, include it’s ID in the commit message(s).
9.12. Do not use blind timeouts
Do not use Thread.sleep() without checking for the actual condition you
need to be fulfilled.
You may use active waiting with a timeout, but prefer using timeouts of
the API/SPI you test where available.
Make the timeouts configurable: For a group of similar test, use a configurable timeout value with a default if not set.
9.13. Provide messages in assert*() and fail() calls
Definitely, it’s better to see "File x/y/z.xml not found" instead of:
junit.framework.AssertionFailedError
at junit.framework.Assert.fail(Assert.java:48) [arquillian-service:]
at junit.framework.Assert.assertTrue(Assert.java:20) [arquillian-service:]
at junit.framework.Assert.assertTrue(Assert.java:27) [arquillian-service:]
at org.jboss.as.test.smoke.embedded.parse.ParseAndMarshalModelsTestCase.getOriginalStandaloneXml(ParseAndMarshalModelsTestCase.java:554) [bogus.jar:]
9.14. Provide configuration properties hints in exceptions
If your test uses some configuration property and it fails possibly due to misconfiguration, note the property and it’s value in the exception:
File jdbcJar = new File( System.getProperty("jbossas.ts.dir", "."),
"integration/src/test/resources/mysql-connector-java-5.1.15.jar");
if( !jdbcJar.exists() )
throw new IllegalStateException("Can't find " + jdbcJar + " using $\{jbossas.ts.dir} == " + System.getProperty("jbossas.ts.dir") );
10. Shared Test Classes and Resources
10.1. Among Testsuite Modules
Use the testsuite/shared module.
Classes and resources in this module are available in all testsuite modules - i.e. in testsuite/* .
Only use it if necessary - don’t put things "for future use" in there.
Don’t split packages across modules. Make sure the java package is unique in the WildFly project.
Document your util classes (javadoc) so they can be easily found and reused! A generated list will be put here.
10.2. Between Components and Testsuite Modules
To share component’s test classes with some module in testsuite, you don’t need to split to submodules. You can create a jar with classifier using this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>test-jar</goal>
</goals>
</execution>
</executions>
</plugin>
This creates a jar with classifier "tests", so you can add it as dependency to a testsuite module:
<dependency>
<groupId>org.wildfly</groupId>
<artifactId>wildfly-clustering-common</artifactId>
<classifier>tests</classifier>
<version>${project.version}</version>
<scope>test</scope>
</dependency>