Cucumber JVM + JUnit: Re-run failed tests

Automated tests should run reliably and provide predictable results. At the same time there are some spontaneous temporary errors which distort the entire picture while reviewing test results. And it's really annoying when tests fail on some temporary problem and mainly pass on next run. Of course, one thing is when we forgot to add some waiting time out to wait for element to appear before interacting with it. But there are cases when the reason of such temporary problem lays beyond automated tests implementation but mainly related to environment which may cause some delays of downtime for short time. So, normal reaction on that is to re-run failed tests and confirm functionality is fine. But this is too routine task and it doesn't really require some intellectual work to perform. It's simply additional logic which handles test result state and triggers repetitive run in case of error. If test fails permanently we'll still see the error but if test passes after that then the problem doesn't require too much attention.

And generally, if you simply stepped out it's not the reason to fail.

This problem is not new and it's been resolved already for many particular cases. E.g. here is the JUnit solution example. In this post I'll show you how to perform the same re-run for Cucumber-JVM in combination with JUnit as I'm actively using this combination of engines and it is quite popular. The solution shown in previous link doesn't really fit the Cucumber as each specific JUnit test in Cucumber-JVM corresponds to some specific step rather than entire scenario. Thus, the re-run functionality for this combination of engines looks a bit different. So, let's see how we can re-run our Cucumber tests in JUnit.

Ploblem Solved: Cucumber-JVM running actions before and after tests execution with JUnit

Background

It is frequent case when we need to do some actions before and/or after entire test suite execution. Mainly, such actions are needed for global initialization/cleanup or some additional reporting or any other kind of pre/post-processing. There may be many different reasons for that and some test engines provide such ability, e.g. TestNG has BeforeSuite and AfterSuite annotations, the JUnit has test fixtures which may run before/after test class (it's not really the same but when we use Cucumber-JVM it's very close to what we need).

Problem

The problem appears when you want to add some actions at the very latest or very earlies stages of tests execution and you use Cucumber-JVM with JUnit. In my case I wanted to add some reports post-processing to make an advanced Cucumber report generation. In this case JUnit fixtures didn't help as AfterClass-annotated method runs before Cucumber generates final reports.

At the same time adding @BeforeAll and @AfterAll hooks question raised on Cucumber side as well. And there was even some solution proposed. Unfortunately, authors decided to revert those changes as there were some cases when it does not work.

So, the problem is that I need something to run after entire Cucumber-JVM suite is done but neither Cucumber nor JUnit gives me built-in capability for doing this.

Solution

Aerial: Create New Project Using Archetype

Since 0.0.5 version Aerial has an ability to generate Java projects from archetypes to get sample working project quickly. In this post I'll describe the steps for doing this.

Problem Solved: run TestNG test from JUnit using Maven

Background

Recently I've been developing some applicaiton component which was supposed to run with TestNG. Actually, it was another extension of TestNG test. So, in order to test that functionality I had to emulate TestNG suite run within the test body. Well, performing TestNG run programmatically wasn't a problem. It can be done using code like:

import org.testng.TestListenerAdapter;
import org.testng.TestNG;

.......
        TestListenerAdapter tla = new TestListenerAdapter();
        TestNG testng = new TestNG();
        testng.setTestClasses(new Class[] {SomeTestClass.class});
        testng.addListener(tla);
        testng.run();
.......

where SomeTestClass is some existing TestNG class. This can be even used with JUnit (which was my case as I mainly used JUnit for the test suite). So, technically TestNG can be executed from JUnit test and vice versa.

Problem

The problem appeared when I tried to run JUnit test performing TestNG run via Maven. Normally tests are picked up using surefire plugin which can be included into Maven pom.xml file with the entry like this:

 <plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.18.1</version>
 </plugin>

If you use JUnit only it picks up JUnit tests. In my case I also used JUnit for running test suites but one test required TestNG test class (actually the class with TestNG @Test annotation) as well as I had to add TestNG Maven dependency. In this case only this TestNG class was executed during test run. So, how to let Maven know that I want to run exactly JUnit tests but not TestNG ones while both of them are present within the same Maven project?