Advanced Cucumber Reporting Introduction
The Cucumber JVM contains some set of predefined reports available as the plugin option. By default we have some raw reports. Some of them are ready to be provided for end users (e.g. HTML report) while others still require some post-processing like JSON reports (both for usage and results reports). Also, available standard HTML report isn't really good enough. It's good for analysis (partially) but if we want to provide some kind of overview information we don't need so much details as well as we may need some more visualized statistics rather than just one simple table.
Well, there are some already existing solutions for such advanced reporting, e.g. Cucumber Reports by masterthought.net. That's really nice results reporting solution for Cucumber JVM. But when I tried to apply this solution I encountered several problems with Maven dependencies resolution and report processing. That became a reason for me to look at something custom as I couldn't smoothly integrate this reporting to my testing solution. Additionally, it covers only results reporting while I'm also interested in steps usage statistic information to make sure that we use our Cucumber steps effectively. And apparently I didn't find reporting solution covering this area in appropriate way for Cucumber JVM.
Being honest, I'm not really big fan of writing custom reporting solution at all as for test automation engineers existing information is more than enough in most of the cases. But if you need to provide something more specific and preferably in e-mailable form to send to other members of project team we need something else. That's why I created some components which generate some Cucumber reports like results and usage reports (see sample screen shots below).
Cucumber JVM Advanced Reporting Setup
Cucumber reporting solution is provided as a Java library which can be included as any dependency via Jar, Maven, Gradle or any other solution. Since library is included it provides the API which converts basic Cucumber JVM reports (mainly JSON-based) into HTML reports. Now let's bring more details on that.
Add Java Dependency
As it was mentioned before my custom reporting solution is provided as the library which can be included via Maven:
<dependency> <groupId>com.github.mkolisnyk</groupId> <artifactId>cucumber-reports</artifactId> <version>0.0.2</version> </dependency>or the same thing for Gradle:
'com.github.mkolisnyk:cucumber-reports:0.0.2'or simply download Jar some here (depending on version the URL can be different). After that the API is accessible within the project and we can write the code which generates reports.
Report Generation Samples
Report generation can be triggered somewhere within Java code. It is assumed that base JSON reports have already been generated by Cucumber JVM, so we just have to point to existing files. In real life this API should be called somewhere as the part of hooks or some other similar code blocks which are executed after entire test suite is completed.
The following code sample generates results overview report based on Cucumber JSON report stored at ./src/test/resources/cucumber.json location. The output directory is target and the file prefix is cucumber-results. The code is:
CucumberResultsOverview results = new CucumberResultsOverview(); results.setOutputDirectory("target"); results.setOutputName("cucumber-results"); results.setSourceFile("./src/test/resources/cucumber.json"); results.executeFeaturesOverviewReport();As the result of this code there would be target/cucumber-results-feature-overview.html file generated.
Cucumber usage report works in the same fashion except it uses report produced as the part of usage Cucumber JVM plugin. The following example generates usage report into the target folder based on JSON usage report generated by Cucumber and located at ./src/test/resources/cucumber-usage.json path:
CucumberUsageReporting report = new CucumberUsageReporting(); report.setOutputDirectory("target"); report.setJsonUsageFile("./src/test/resources/cucumber-usage.json"); report.executeReport();The output would be the HTML file located at target/cucumber-usage-report.html.
This is how those reports are generated. Now let's take a look at what exactly is generated.
Cucumber HTML Results Report
Cucumber results report contains 3 major sections:
- Overview Chart - contains pie charts showing the ratio of passed/failed features and scenarios. Scenario is considered as failed when it has failed steps. Otherwise if scenario has undefined steps (without any failed steps) the scenario status is undefined. In all other cases scenario is treated as passed. Features status uses similar logic but in this case the elementary part is scenario. In other words if feature contains at least one failed scenario it is treated as failed. If no fails occurred but there are some undefined scenarios the feature is undefined. Otherwise it is treated as passed. Eventually, the overview chart looks like this:
- Features Status - it is the table containing the list of features by their names and scenario run statistics. It shows the number of passed, failed and undefined scenarios for each specific features. Here is the sample of feature overview table:
Feature Name Status Passed Failed Undefined Live Departure Board passed 6 0 0 Payments failed 6 1 0 Application start up passed 2 0 0 Journey options feature passed 3 0 0 RTJP - Seats left failed 1 1 0 Rename 2 singles to cheapest tickets passed 2 0 0 Journey Search failed 27 4 0
- Scenario Status - it is more detailed breakdown where features are also split into scenarios. The table contain the number of passed, failed and undefined steps for each specific scenarios. Sample table looks like (sample fragment):
Feature Name Scenario Status Passed Failed Undefined Live Departure Board Make a search through Live Departure Board passed 9 0 0 Live Departure Board Make a search through Live Departure Board passed 9 0 0 Live Departure Board Make a search through Live Departure Board passed 9 0 0 Live Departure Board Make a search through Live Departure Board passed 9 0 0 Live Departure Board No direct route passed 8 0 0 Live Departure Board Recent searches LDB list passed 7 0 0
Cucumber HTML Usage Report
This report is more specific and it is mainly targeted to analyze how effectively we use our Cucumber keywords as well as provides major information about use of those keywords. This report contains 2 major sections:
- Usage Overview - contains graph and table summarizing general keywords usage statistics
- Usage Detailed Information - contains the set of tables showing where and how each specific keyword is used
Usage overview contains 2 parts:
- Cucumber Keywords Usage Graph - graphical representation of general usage for all the keywords. Sample usage graph looks like:
Another major item in this graph is average and median numbers (highlighted with red and yellow lines correspondingly). Average number is mathematical average. This value indicates the average steps re-use count across the entire set of steps. Since it's mathematical average number it doesn't reflect statistical distribution of all steps. It means that high average number may be reached even having most of the steps used just once but 1 or 2 steps are re-used big number of times. In this case mainly we don't re-use steps but our numbers still look good. Thus our information is a bit distorted and we can get some wrong conclusions during the analysis.
In order to give more precise picture of average re-use count which also takes into account the actual distribution of the steps re-use data the graph also shows the median value. It is also average number but it indicates that at least 50% of steps are re-used less than this median value times. Thus, in case of huge number of single time used steps our median will still be 1 even despite we have some steps which are re-used hundreds of times while simple mathematical average shows better number.
When we analyze the above values we should make sure that both of them are at least greater than 2. This means that in average we use each step at least once without necessity of spending time on writing this step implementation. This already makes Gherkin use effective. If the above numbers are bigger it's even better.
Eventually, we have Re-use ratio which is displayed as a big number on the graph. The ratio of %N indicates that in current test suite %N of steps were written without implementing actual glue code. In order words this number means that %N of all our steps in current test suite are automated even during test design stage. So, the bigger this ratio is the more effective we use our steps.
- Cucumber Keywords Usage Table - keywords usage table shows the same data as the above graph but this data with precise numbers is grouped and represented in tabular form. It is needed just because graph has some restrictions in displaying all numbers so in some cases we cannot get precise information about actual re-use count. So, table provides this information.
Usage Detailed Information
Detailed information part shows each keyword usage with all variations of all parameters the keyword is used with. E.g:
^(?:I |)click on the "([^"]*)" button$
|I click on the "OK" button||-||-|
|I click on the "Skip" button||-||-|
|I click on the "Login" button||-||-|
|I click on the "Buy Tickets" button||-||-|
|I click on the "Earlier Trains" button||-||-|
|I click on the "Later Trains" button||-||-|
|I click on the "First Class" button||-||-|
|click on the "Buy Tickets" button||-||-|
The above reports aren't complete set of available reports which can be made for Cucumber JVM output processing. But these are major ones I have to use at the moment. And again, this project is still in development so we may have some more reports produced based on standard output formats.