Cucumber JVM: Advanced Reporting

Advanced Cucumber Reporting Introduction

The Cucumber JVM contains some set of predefined reports available as the plugin option. By default we have some raw reports. Some of them are ready to be provided for end users (e.g. HTML report) while others still require some post-processing like JSON reports (both for usage and results reports). Also, available standard HTML report isn't really good enough. It's good for analysis (partially) but if we want to provide some kind of overview information we don't need so much details as well as we may need some more visualized statistics rather than just one simple table.

Well, there are some already existing solutions for such advanced reporting, e.g. Cucumber Reports by masterthought.net. That's really nice results reporting solution for Cucumber JVM. But when I tried to apply this solution I encountered several problems with Maven dependencies resolution and report processing. That became a reason for me to look at something custom as I couldn't smoothly integrate this reporting to my testing solution. Additionally, it covers only results reporting while I'm also interested in steps usage statistic information to make sure that we use our Cucumber steps effectively. And apparently I didn't find reporting solution covering this area in appropriate way for Cucumber JVM.

Being honest, I'm not really big fan of writing custom reporting solution at all as for test automation engineers existing information is more than enough in most of the cases. But if you need to provide something more specific and preferably in e-mailable form to send to other members of project team we need something else. That's why I created some components which generate some Cucumber reports like results and usage reports (see sample screen shots below).

The above samples show e-mailable reports which mainly provide results overview information which can be sent via e-mail as well as additional HTML report summarizing usage statistics. In this post I'll describe my reporting solution with some examples and some detailed explanations.

Cucumber JVM Advanced Reporting Setup

Cucumber reporting solution is provided as a Java library which can be included as any dependency via Jar, Maven, Gradle or any other solution. Since library is included it provides the API which converts basic Cucumber JVM reports (mainly JSON-based) into HTML reports. Now let's bring more details on that.

Add Java Dependency

As it was mentioned before my custom reporting solution is provided as the library which can be included via Maven:

<dependency>
 <groupId>com.github.mkolisnyk</groupId>
 <artifactId>cucumber-reports</artifactId>
 <version>0.0.2</version>
</dependency>

or the same thing for Gradle:

'com.github.mkolisnyk:cucumber-reports:0.0.2'

or simply download Jar some here (depending on version the URL can be different). After that the API is accessible within the project and we can write the code which generates reports.

Report Generation Samples

Report generation can be triggered somewhere within Java code. It is assumed that base JSON reports have already been generated by Cucumber JVM, so we just have to point to existing files. In real life this API should be called somewhere as the part of hooks or some other similar code blocks which are executed after entire test suite is completed.

The following code sample generates results overview report based on Cucumber JSON report stored at ./src/test/resources/cucumber.json location. The output directory is target and the file prefix is cucumber-results. The code is:

CucumberResultsOverview results = new CucumberResultsOverview();
results.setOutputDirectory("target");
results.setOutputName("cucumber-results");
results.setSourceFile("./src/test/resources/cucumber.json");
results.executeFeaturesOverviewReport();

As the result of this code there would be target/cucumber-results-feature-overview.html file generated.

Cucumber usage report works in the same fashion except it uses report produced as the part of usage Cucumber JVM plugin. The following example generates usage report into the target folder based on JSON usage report generated by Cucumber and located at ./src/test/resources/cucumber-usage.json path:

CucumberUsageReporting report = new CucumberUsageReporting();
report.setOutputDirectory("target");
report.setJsonUsageFile("./src/test/resources/cucumber-usage.json");
report.executeReport();

The output would be the HTML file located at target/cucumber-usage-report.html.

This is how those reports are generated. Now let's take a look at what exactly is generated.

Cucumber HTML Results Report

Cucumber results report contains 3 major sections:

• Overview Chart - contains pie charts showing the ratio of passed/failed features and scenarios. Scenario is considered as failed when it has failed steps. Otherwise if scenario has undefined steps (without any failed steps) the scenario status is undefined. In all other cases scenario is treated as passed. Features status uses similar logic but in this case the elementary part is scenario. In other words if feature contains at least one failed scenario it is treated as failed. If no fails occurred but there are some undefined scenarios the feature is undefined. Otherwise it is treated as passed. Eventually, the overview chart looks like this:
  
• Features Status - it is the table containing the list of features by their names and scenario run statistics. It shows the number of passed, failed and undefined scenarios for each specific features. Here is the sample of feature overview table:

  Feature Name	Status	Passed	Failed	Undefined
  Live Departure Board	passed	6	0	0
  Payments	failed	6	1	0
  Application start up	passed	2	0	0
  Journey options feature	passed	3	0	0
  RTJP - Seats left	failed	1	1	0
  Rename 2 singles to cheapest tickets	passed	2	0	0
  Journey Search	failed	27	4	0

• Scenario Status - it is more detailed breakdown where features are also split into scenarios. The table contain the number of passed, failed and undefined steps for each specific scenarios. Sample table looks like (sample fragment):

  Feature Name	Scenario	Status	Passed	Failed	Undefined
  Live Departure Board	Make a search through Live Departure Board	passed	9	0	0
  Live Departure Board	Make a search through Live Departure Board	passed	9	0	0
  Live Departure Board	Make a search through Live Departure Board	passed	9	0	0
  Live Departure Board	Make a search through Live Departure Board	passed	9	0	0
  Live Departure Board	No direct route	passed	8	0	0
  Live Departure Board	Recent searches LDB list	passed	7	0	0

Since this format is adapted to be an e-mailable report there is no need to add steps breakdown as thus the report becomes too big as well as the actual steps breakdown can be taken from standard Cucumber JVM HTML output.

Cucumber HTML Usage Report

This report is more specific and it is mainly targeted to analyze how effectively we use our Cucumber keywords as well as provides major information about use of those keywords. This report contains 2 major sections:

• Usage Overview - contains graph and table summarizing general keywords usage statistics
• Usage Detailed Information - contains the set of tables showing where and how each specific keyword is used

Now let's describe both sections in more details.

Usage Overview

Usage overview contains 2 parts:

• Cucumber Keywords Usage Graph - graphical representation of general usage for all the keywords. Sample usage graph looks like:
  
  This is major item of the entire report and mainly it is enough to view it in order to get an idea about how good we are at steps re-use. What exactly does it contain? The X-axis goes through the number of steps re-use. The Y-axis shows how many steps are re-used X times. So, if we see on this graph the point with dimensions like X = 28 and Y = 2 it indicates that we have 2 keywords which are re-used 28 times within the test suite.

  Another major item in this graph is average and median numbers (highlighted with red and yellow lines correspondingly). Average number is mathematical average. This value indicates the average steps re-use count across the entire set of steps. Since it's mathematical average number it doesn't reflect statistical distribution of all steps. It means that high average number may be reached even having most of the steps used just once but 1 or 2 steps are re-used big number of times. In this case mainly we don't re-use steps but our numbers still look good. Thus our information is a bit distorted and we can get some wrong conclusions during the analysis.

  In order to give more precise picture of average re-use count which also takes into account the actual distribution of the steps re-use data the graph also shows the median value. It is also average number but it indicates that at least 50% of steps are re-used less than this median value times. Thus, in case of huge number of single time used steps our median will still be 1 even despite we have some steps which are re-used hundreds of times while simple mathematical average shows better number.

  When we analyze the above values we should make sure that both of them are at least greater than 2. This means that in average we use each step at least once without necessity of spending time on writing this step implementation. This already makes Gherkin use effective. If the above numbers are bigger it's even better.

  Eventually, we have Re-use ratio which is displayed as a big number on the graph. The ratio of %N indicates that in current test suite %N of steps were written without implementing actual glue code. In order words this number means that %N of all our steps in current test suite are automated even during test design stage. So, the bigger this ratio is the more effective we use our steps.

• Cucumber Keywords Usage Table - keywords usage table shows the same data as the above graph but this data with precise numbers is grouped and represented in tabular form. It is needed just because graph has some restrictions in displaying all numbers so in some cases we cannot get precise information about actual re-use count. So, table provides this information.

Usage Detailed Information

Detailed information part shows each keyword usage with all variations of all parameters the keyword is used with. E.g:

^(?:I |)click on the "([^"]*)" button$

Step Name	Duration	Location
I click on the "OK" button	-	-
	2.858510898	LiveTrainTimes.feature:48
I click on the "Skip" button	-	-
	1.862363906	Startup.feature:7
I click on the "Login" button	-	-
	2.242663886	Startup.feature:7
I click on the "Buy Tickets" button	-	-
	19.839095216	booking/JourneyOptions.feature:12
	21.44855246	booking/JourneyOptions.feature:30
	20.263704459	booking/JourneyOptions.feature:87
	22.396167947	journeysearch/BON-80_RTJP_SeatsLeft.feature:14
	20.97751731	journeysearch/BON-88_SingleOptionsForJourney.feature:15
	21.321668302	journeysearch/BON-88_SingleOptionsForJourney.feature:79
	21.294036775	journeysearch/BasicJourneySearch.feature:7
	19.723224101	journeysearch/BasicJourneySearch.feature:22
	21.087017738	journeysearch/BasicJourneySearch.feature:221
I click on the "Earlier Trains" button	-	-
	28.285814568	booking/JourneyOptions.feature:18
I click on the "Later Trains" button	-	-
	18.404438358	booking/JourneyOptions.feature:36
I click on the "First Class" button	-	-
	0.815324731	booking/JourneyOptions.feature:94
click on the "Buy Tickets" button	-	-
	29.302681774	journeysearch/BasicJourneySearch.feature:231

Here we can identify which parts of the keyword are really varying and how many different variations are used. Mainly, based on that information we can identify which regular expressions can be optimized or joint in order to make more effective use of keywords.

Summary

The above reports aren't complete set of available reports which can be made for Cucumber JVM output processing. But these are major ones I have to use at the moment. And again, this project is still in development so we may have some more reports produced based on standard output formats.