Cucumber JVM: Advanced Reporting 2. Detailed Report (HTML, PDF)

Introduction

Previously I've described basic report samples based on Cucumber-JVM. Since I'm using this reporting solution on practice quite frequently the number of requirements and enhancements grows so since last time I had to add some more additional features which can be in use.

Some of new features are:

• Cucumber extension to support failed tests re-run
• Detailed results report which supports the following:
  • Detailed results report generation in HTML format
  • Detailed results report generation in PDF format
  • Screen shot are included into both the above reports

All those new features are included into 0.0.5 version, so we can add Maven dependency:

<dependency>
 <groupId>com.github.mkolisnyk</groupId>
 <artifactId>cucumber-reports</artifactId>
 <version>0.0.5</version>
</dependency>

or the same thing for Gradle:

'com.github.mkolisnyk:cucumber-reports:0.0.5'

Since Cucumber failed tests re-run functionality was described before in this post I'll concentrate more on detailed results report features.

Why do we need the detailed results report?

First question which may appear here is: why do we need such report? Most of the details can be retrieved from standard Cucumber HTML report. Well, main reason is that existing HTML results report isn't enough and on practice we need some additional features. Some of them are:

• Report itself should have minimal external dependencies (e.g. styles, scripts) so that it is easier to publish it as build artifact
• It is very useful to have report showing screen shots on every error encountered
• The entire report should also have some portable and self-contained format so that it can be sent via e-mail. Thus, PDF format can be quite useful for that

Also, standard HTML report looks pretty "standard" and empty. So, I've added some features to make new detailed report more convenient to read and navigate through the list of test results.

Detailed Report Structure

The image below shows how detailed report looks like:

It contains 3 major sections:

• Overview
• Table of Contents
• Detailed Results Report

Overview section

Overview section contains aggregated data on entire run results. In particular it shows the number of passed/failed/skipped features/scenarios and steps with the % of passed items in comparison to entire number of items. Here is how typical overview section looks like:

	Passed	Failed	Undefined	%Passed
Features	4	3	0	57.14
Scenarios	47	6	0	88.68
Steps	406	6	28	92.27

Table of Contents section

Table of contents is the hyper-linked list of features and scenarios highlighting the execution status. It is convenient addition to the report which provides possibility to navigate to each specific scenario result. Thus we have fast way to get to the test we're interested in.

Detailed Results Report section

Detailed results report looks similar to standard HTML report and simply contains the description of steps with the status and detailed stack trace is case of error. Also, every scenario result contains hyperlink to the table of contents. Thus we can easily navigate to scenario we need and then switch to any other scenarios we're interested in.

How to Generate Detailed Report

The report generation is based on standard Cucumber JSON results report post-processing. So, before generating detailed results report we should specify which results report file we should use for report generation. So, typical Java code for report producing looks like:

        CucumberDetailedResults results = new CucumberDetailedResults();
        results.setOutputDirectory("target/");
        results.setOutputName("cucumber-results");
        results.setSourceFile("./src/test/resources/cucumber.json");
        results.executeDetailedResultsReport(false);

This code produces detailed report stored at the following path: target/cucumber-results-test-results.html.

Adding Screen Shots

Often it's very useful to have the screen shot on error to see what was the real problem which caused error. It's not the silver bullet but it's definitely helpful for analysis. Since currently described report is produced as the result of post-processing but not as the part of hooks the way screen shots are generated are out of report generator scope. We simply should assume that we already have some folder containing screen shot files which fit the following rules:

• Image files are of PNG format
• Each failed test contains only one screen shot as we normally have only one error on failed test. Maybe in the future I'll add some additional options to handle multiple files but currently it is the way we do things now.
• Screen shot file names are taken from the names of corresponding scenario IDs where special characters like ; or spaces are replaced with underscore.

Once we meet such requirements all we need is to define such screen shot locations. Thus, the Java code producing detailed report now looks like:

        CucumberDetailedResults results = new CucumberDetailedResults();
        results.setOutputDirectory("target/");
        results.setOutputName("cucumber-results");
        results.setSourceFile("./src/test/resources/cucumber.json");
        results.setScreenShotLocation("../src/test/resources/");
        results.executeDetailedResultsReport(false);

The highlighted part defines where to look for screen shots.

PDF Generation

And the last feature is the ability to generate file in PDF format. Actually, when we generate HTML report with all the above features it is still hard to transport as it may contain references to image files. In order to make the report more portable (e.g. if we need to send it then via e-mail). In order to reach this we simply need to convert generated detailed report HTML to PDF file. This flow is handled by the parameter of the executeDetailedResultsReport method. If it is set to true the PDF report is generated in addition to HTML one. If it is false only HTML report is generated. So, in order to produce PDF output we should modify previous code sample this way:

        CucumberDetailedResults results = new CucumberDetailedResults();
        results.setOutputDirectory("target/");
        results.setOutputName("cucumber-results");
        results.setSourceFile("./src/test/resources/cucumber.json");
        results.setScreenShotLocation("../src/test/resources/");
        results.executeDetailedResultsReport(true);

The above example will produce PDF file located by the path: target/cucumber-results-test-results.pdf.

Summary

That were additional Cucumber reporting features introduced recently. So, currently we have reports which can be set as e-mail message body + some additional reports which can be provided as attachments. This is good foundation for quite convenient notification system where we don't need to keep all our reports as build artifacts as well as now we have one major report where we can get both overview and detailed information. The list of enhancements will grow as long as more additional requirements appears. But the stuff we have right now is already good.