Search

Sunday 3 May 2015

Cucumber JVM: Advanced Reporting

Cucumber JVM: Advanced Reporting

Advanced Cucumber Reporting Introduction

The Cucumber JVM contains some set of predefined reports available as the plugin option. By default we have some raw reports. Some of them are ready to be provided for end users (e.g. HTML report) while others still require some post-processing like JSON reports (both for usage and results reports). Also, available standard HTML report isn't really good enough. It's good for analysis (partially) but if we want to provide some kind of overview information we don't need so much details as well as we may need some more visualized statistics rather than just one simple table.

Well, there are some already existing solutions for such advanced reporting, e.g. Cucumber Reports by masterthought.net. That's really nice results reporting solution for Cucumber JVM. But when I tried to apply this solution I encountered several problems with Maven dependencies resolution and report processing. That became a reason for me to look at something custom as I couldn't smoothly integrate this reporting to my testing solution. Additionally, it covers only results reporting while I'm also interested in steps usage statistic information to make sure that we use our Cucumber steps effectively. And apparently I didn't find reporting solution covering this area in appropriate way for Cucumber JVM.

Being honest, I'm not really big fan of writing custom reporting solution at all as for test automation engineers existing information is more than enough in most of the cases. But if you need to provide something more specific and preferably in e-mailable form to send to other members of project team we need something else. That's why I created some components which generate some Cucumber reports like results and usage reports (see sample screen shots below).

The above samples show e-mailable reports which mainly provide results overview information which can be sent via e-mail as well as additional HTML report summarizing usage statistics. In this post I'll describe my reporting solution with some examples and some detailed explanations.

Cucumber JVM Advanced Reporting Setup

Cucumber reporting solution is provided as a Java library which can be included as any dependency via Jar, Maven, Gradle or any other solution. Since library is included it provides the API which converts basic Cucumber JVM reports (mainly JSON-based) into HTML reports. Now let's bring more details on that.

Add Java Dependency

As it was mentioned before my custom reporting solution is provided as the library which can be included via Maven:

<dependency>
 <groupId>com.github.mkolisnyk</groupId>
 <artifactId>cucumber-reports</artifactId>
 <version>0.0.2</version>
</dependency>
or the same thing for Gradle:
'com.github.mkolisnyk:cucumber-reports:0.0.2'
or simply download Jar some here (depending on version the URL can be different). After that the API is accessible within the project and we can write the code which generates reports.

Report Generation Samples

Report generation can be triggered somewhere within Java code. It is assumed that base JSON reports have already been generated by Cucumber JVM, so we just have to point to existing files. In real life this API should be called somewhere as the part of hooks or some other similar code blocks which are executed after entire test suite is completed.

The following code sample generates results overview report based on Cucumber JSON report stored at ./src/test/resources/cucumber.json location. The output directory is target and the file prefix is cucumber-results. The code is:

CucumberResultsOverview results = new CucumberResultsOverview();
results.setOutputDirectory("target");
results.setOutputName("cucumber-results");
results.setSourceFile("./src/test/resources/cucumber.json");
results.executeFeaturesOverviewReport();
As the result of this code there would be target/cucumber-results-feature-overview.html file generated.

Cucumber usage report works in the same fashion except it uses report produced as the part of usage Cucumber JVM plugin. The following example generates usage report into the target folder based on JSON usage report generated by Cucumber and located at ./src/test/resources/cucumber-usage.json path:

CucumberUsageReporting report = new CucumberUsageReporting();
report.setOutputDirectory("target");
report.setJsonUsageFile("./src/test/resources/cucumber-usage.json");
report.executeReport();
The output would be the HTML file located at target/cucumber-usage-report.html.

This is how those reports are generated. Now let's take a look at what exactly is generated.

Cucumber HTML Results Report

Cucumber results report contains 3 major sections:

  • Overview Chart - contains pie charts showing the ratio of passed/failed features and scenarios. Scenario is considered as failed when it has failed steps. Otherwise if scenario has undefined steps (without any failed steps) the scenario status is undefined. In all other cases scenario is treated as passed. Features status uses similar logic but in this case the elementary part is scenario. In other words if feature contains at least one failed scenario it is treated as failed. If no fails occurred but there are some undefined scenarios the feature is undefined. Otherwise it is treated as passed. Eventually, the overview chart looks like this:
  • Features Status - it is the table containing the list of features by their names and scenario run statistics. It shows the number of passed, failed and undefined scenarios for each specific features. Here is the sample of feature overview table:
    Feature NameStatusPassedFailedUndefined
    Live Departure Boardpassed600
    Paymentsfailed610
    Application start uppassed200
    Journey options featurepassed300
    RTJP - Seats leftfailed110
    Rename 2 singles to cheapest ticketspassed200
    Journey Searchfailed2740
  • Scenario Status - it is more detailed breakdown where features are also split into scenarios. The table contain the number of passed, failed and undefined steps for each specific scenarios. Sample table looks like (sample fragment):
    Feature NameScenarioStatusPassedFailedUndefined
    Live Departure BoardMake a search through Live Departure Boardpassed900
    Live Departure BoardMake a search through Live Departure Boardpassed900
    Live Departure BoardMake a search through Live Departure Boardpassed900
    Live Departure BoardMake a search through Live Departure Boardpassed900
    Live Departure BoardNo direct routepassed800
    Live Departure BoardRecent searches LDB listpassed700
Since this format is adapted to be an e-mailable report there is no need to add steps breakdown as thus the report becomes too big as well as the actual steps breakdown can be taken from standard Cucumber JVM HTML output.

Cucumber HTML Usage Report

This report is more specific and it is mainly targeted to analyze how effectively we use our Cucumber keywords as well as provides major information about use of those keywords. This report contains 2 major sections:

  • Usage Overview - contains graph and table summarizing general keywords usage statistics
  • Usage Detailed Information - contains the set of tables showing where and how each specific keyword is used
Now let's describe both sections in more details.

Usage Overview

Usage overview contains 2 parts:

  • Cucumber Keywords Usage Graph - graphical representation of general usage for all the keywords. Sample usage graph looks like:
    This is major item of the entire report and mainly it is enough to view it in order to get an idea about how good we are at steps re-use. What exactly does it contain? The X-axis goes through the number of steps re-use. The Y-axis shows how many steps are re-used X times. So, if we see on this graph the point with dimensions like X = 28 and Y = 2 it indicates that we have 2 keywords which are re-used 28 times within the test suite.

    Another major item in this graph is average and median numbers (highlighted with red and yellow lines correspondingly). Average number is mathematical average. This value indicates the average steps re-use count across the entire set of steps. Since it's mathematical average number it doesn't reflect statistical distribution of all steps. It means that high average number may be reached even having most of the steps used just once but 1 or 2 steps are re-used big number of times. In this case mainly we don't re-use steps but our numbers still look good. Thus our information is a bit distorted and we can get some wrong conclusions during the analysis.

    In order to give more precise picture of average re-use count which also takes into account the actual distribution of the steps re-use data the graph also shows the median value. It is also average number but it indicates that at least 50% of steps are re-used less than this median value times. Thus, in case of huge number of single time used steps our median will still be 1 even despite we have some steps which are re-used hundreds of times while simple mathematical average shows better number.

    When we analyze the above values we should make sure that both of them are at least greater than 2. This means that in average we use each step at least once without necessity of spending time on writing this step implementation. This already makes Gherkin use effective. If the above numbers are bigger it's even better.

    Eventually, we have Re-use ratio which is displayed as a big number on the graph. The ratio of %N indicates that in current test suite %N of steps were written without implementing actual glue code. In order words this number means that %N of all our steps in current test suite are automated even during test design stage. So, the bigger this ratio is the more effective we use our steps.

  • Cucumber Keywords Usage Table - keywords usage table shows the same data as the above graph but this data with precise numbers is grouped and represented in tabular form. It is needed just because graph has some restrictions in displaying all numbers so in some cases we cannot get precise information about actual re-use count. So, table provides this information.

Usage Detailed Information

Detailed information part shows each keyword usage with all variations of all parameters the keyword is used with. E.g:

^(?:I |)click on the "([^"]*)" button$

Step NameDurationLocation
I click on the "OK" button - -
2.858510898LiveTrainTimes.feature:48
I click on the "Skip" button - -
1.862363906Startup.feature:7
I click on the "Login" button - -
2.242663886Startup.feature:7
I click on the "Buy Tickets" button - -
19.839095216booking/JourneyOptions.feature:12
21.44855246booking/JourneyOptions.feature:30
20.263704459booking/JourneyOptions.feature:87
22.396167947journeysearch/BON-80_RTJP_SeatsLeft.feature:14
20.97751731journeysearch/BON-88_SingleOptionsForJourney.feature:15
21.321668302journeysearch/BON-88_SingleOptionsForJourney.feature:79
21.294036775journeysearch/BasicJourneySearch.feature:7
19.723224101journeysearch/BasicJourneySearch.feature:22
21.087017738journeysearch/BasicJourneySearch.feature:221
I click on the "Earlier Trains" button - -
28.285814568booking/JourneyOptions.feature:18
I click on the "Later Trains" button - -
18.404438358booking/JourneyOptions.feature:36
I click on the "First Class" button - -
0.815324731booking/JourneyOptions.feature:94
click on the "Buy Tickets" button - -
29.302681774journeysearch/BasicJourneySearch.feature:231
Here we can identify which parts of the keyword are really varying and how many different variations are used. Mainly, based on that information we can identify which regular expressions can be optimized or joint in order to make more effective use of keywords.

Summary

The above reports aren't complete set of available reports which can be made for Cucumber JVM output processing. But these are major ones I have to use at the moment. And again, this project is still in development so we may have some more reports produced based on standard output formats.

215 comments:

  1. Hi
    Am getting below error message. Please help me to resolve this issue.

    java.lang.NoClassDefFoundError: com/cedarsoftware/util/io/JsonReader
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.readFileContent(CucumberResultsOverview.java:75)
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeFeaturesOverviewReport(CucumberResultsOverview.java:153)
    at com.cuckeadv.testcases.stepdefinitions.CucumberReportStepdefs.runReport(CucumberReportStepdefs.java:26)

    ReplyDelete
    Replies
    1. Hi Naren,

      This error looks like dependency problem. So, you can check the following:
      1) You use the latest version of cucumber-reports. At the moment it's 0.0.3
      2) All dependencies are extracted
      3) Your dependencies are set for proper stage. E.g. if you use that reporting during test execution that should be test dependencies, not just compilation ones. E.g.:

      testCompile 'com.github.mkolisnyk:cucumber-reports:0.0.3'

      or in Maven:

      <dependency>
      <groupId>com.github.mkolisnyk</groupId>
      <artifactId>cucumber-reports</artifactId>
      <version>0.0.3</version>
      <scope>test</scope>
      </dependency>

      Actually, you can try both.
      It's mainly about dependencies as com.cedarsoftware.util.io package is defined as the part of 'com.cedarsoftware:json-io:2.2.31' dependency. If any you can also try to add the com.cedarsoftware > json-io dependency of version 2.2.31.

      Delete
    2. Hi Kolesnik Nickolay,
      Its working now. I am searching for this for a long time. Much thanks for the post and your support.

      Delete
  2. How to integrate Advance cucumber report setup in ruby code. I can generate JSON report.

    ReplyDelete
    Replies
    1. Since it is Java library you can create Java-based command line utility which will pick up your report and generate output.

      Delete
  3. HI iam getting this error please help

    org.apache.maven.reporting.MavenReportException: Error occured while generating Cucumber usage report
    at com.github.mkolisnyk.cucumber.reporting.CucumberUsageReporting.executeReport(CucumberUsageReporting.java:385)

    ReplyDelete
    Replies



    1. CucumberUsageReporting report = new CucumberUsageReporting();
      report.setOutputDirectory("target");
      report.setJsonUsageFile("path/to/cucumber.Json");
      try {
      report.executeReport();
      } catch (MavenReportException e1) {
      // TODO Auto-generated catch block
      e1.printStackTrace();
      }

      Delete
    2. 1) Firstly, make sure you point to proper JSON file. Usage report uses JSON generated as the usage output. It's not the results report

      2) Is that complete stack trace?

      3) What is the version of library?

      Delete
    3. Just reproduced the problem. For me such error appeared when I tried to generate usage report for empty JSON file. This usage JSON is generated empty when we do dry run. The behaviour is as planned. So, when you use the report generator make sure you point to usage JSON file and this file is not empty. Otherwise, the format is violated and exception is thrown. I can add some additional handling but at the moment it already works as expected

      Delete
  4. This comment has been removed by the author.

    ReplyDelete
  5. Hi Kolesnik,

    I tried implementing cucumber-report, however I get below error while i execute code.

    com.asite.adoddle.testrunner.RunCukesTest Time elapsed: 0.03 sec <<< ERROR!
    java.io.IOException: Input is invalid JSON; does not start with '{' or '[', c=-1
    at com.cedarsoftware.util.io.JsonReader.readJsonObject(JsonReader.java:1494)
    at com.cedarsoftware.util.io.JsonReader.readObject(JsonReader.java:707)
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.readFileContent(CucumberResultsOverview.java:81)


    Below is my Runner class,

    public class RunCukesTest{

    @AfterClass
    public static void generateReport() throws Throwable
    {
    CucumberResultsOverview results = new CucumberResultsOverview();
    results.setOutputDirectory(".//target");
    results.setOutputName(".//target//cucumber-results");
    results.setSourceFile(".//target//cucumber.json");
    results.executeFeaturesOverviewReport();
    }
    }

    I verified that my JSON is not empty and validated as well on http://jsonlint.com/.

    ReplyDelete
  6. PLS CAN SOMEONE HELP ME ON HOW TO INTEGRATE THIS WITH MY FRAMEWORK.
    package com.chroma.aodb.stepsdefs;

    import cucumber.api.CucumberOptions;

    import cucumber.api.testng.AbstractTestNGCucumberTests;


    @CucumberOptions(

    features = {"src/test/resources/features/regression"},
    glue = "com.chroma.aodb.stepsdefs",
    tags = {"@arrival"},
    monochrome = false,
    plugin = {"html:target/html/cucumber-html-report", "pretty"
    , "json:target/Json/cucumber.json"
    ,"junit:target/Junit/cucumber.xml"})

    public class TestRunner extends AbstractTestNGCucumberTests{

    }

    THIS IS MY TESTRUNNER CLASS. I AM USING TESTNG. WHERE DO I HAVE TO PUT THE CUCUMBERRESULTS CODE AND CUCUMBERREPORT CODE IN THE FRAMEWORK. THANKS

    ReplyDelete
    Replies
    1. I'm not sure if your example works but if it does then you could try to override tear down method. So, you need to add something line this:

      @Override
      @AfterClass
      public void tearDownClass() throws Exception {
      super.tearDownClass();
      // TODO: Add your reporting here
      }

      The TODO comment contains area where you can paste your report generation code.

      Delete
  7. i have tried with @After annotations from cucumber API.
    @After
    public void afterScenario() throws Exception{
    {
    CucumberResultsOverview results = new CucumberResultsOverview();
    results.setOutputDirectory(".//targets");
    results.setOutputName(".//targets//cucumber-results");
    results.setSourceFile(".//targets//cucumberresult.json");
    results.executeFeaturesOverviewReport();
    CucumberUsageReporting report = new CucumberUsageReporting();
    report.setOutputDirectory("target");
    report.setJsonUsageFile("./src/test/resources/cucumber-usage.json");
    report.executeReport();
    }
    }
    I got an error.
    java.io.FileNotFoundException
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.readFileContent(CucumberResultsOverview.java:76)
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeFeaturesOverviewReport(CucumberResultsOverview.java:189)
    at com.chroma.aodb.stepsdefs.Hooks.afterScenario(Hooks.java:23)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at cucumber.runtime.Utils$1.call(Utils.java:37)
    at cucumber.runtime.Timeout.timeout(Timeout.java:13)
    at cucumber.runtime.Utils.invoke(Utils.java:31)
    at cucumber.runtime.java.JavaHookDefinition.execute(JavaHookDefinition.java:60)
    at cucumber.runtime.Runtime.runHookIfTagsMatch(Runtime.java:223)
    at cucumber.runtime.Runtime.runHooks(Runtime.java:211)
    at cucumber.runtime.Runtime.runAfterHooks(Runtime.java:205)
    at cucumber.runtime.model.CucumberScenario.run(CucumberScenario.java:46)
    at cucumber.runtime.model.CucumberFeature.run(CucumberFeature.java:165)
    at cucumber.api.testng.TestNGCucumberRunner.runCucumber(TestNGCucumberRunner.java:63)
    at cucumber.api.testng.AbstractTestNGCucumberTests.feature(AbstractTestNGCucumberTests.java:21)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:639)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:816)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1124)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:124)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
    at org.testng.TestRunner.privateRun(TestRunner.java:773)
    at org.testng.TestRunner.run(TestRunner.java:623)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:359)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:354)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:312)
    at org.testng.SuiteRunner.run(SuiteRunner.java:261)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1185)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1110)
    at org.testng.TestNG.run(TestNG.java:1018)
    at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:112)
    at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:205)
    at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:176)

    ReplyDelete
    Replies
    1. Hi Purnima,

      I've already explained the problem in parallel topic here

      It also contains some possible solution. The idea is that you should call report generation after everything related to test run is done. Let me know if you need more details

      Delete
  8. Hi,
    I have tried to include the report generation methods in After Hook by cucumber api, but i am getting this error

    [31mFailure in after hook: [0m [31mHooks.afterScenario() [0m
    [31mMessage: [0m [31mjava.io.FileNotFoundException
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.readFileContent(CucumberResultsOverview.java:76)
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeFeaturesOverviewReport(CucumberResultsOverview.java:189)
    at com.chroma.aodb.stepsdefs.Hooks.afterScenario(Hooks.java:25)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at cucumber.runtime.Utils$1.call(Utils.java:37)
    at cucumber.runtime.Timeout.timeout(Timeout.java:13)
    at cucumber.runtime.Utils.invoke(Utils.java:31)
    at cucumber.runtime.java.JavaHookDefinition.execute(JavaHookDefinition.java:60)
    at cucumber.runtime.Runtime.runHookIfTagsMatch(Runtime.java:223)
    at cucumber.runtime.Runtime.runHooks(Runtime.java:211)
    at cucumber.runtime.Runtime.runAfterHooks(Runtime.java:205)
    at cucumber.runtime.model.CucumberScenario.run(CucumberScenario.java:46)
    at cucumber.runtime.junit.ExecutionUnitRunner.run(ExecutionUnitRunner.java:91)

    My After hook is like this,

    @After
    public static void afterScenario() throws Exception {
    CucumberResultsOverview results = new CucumberResultsOverview();
    results.setOutputDirectory(".//targets");
    results.setOutputName(".//targets//cucumber-results");
    results.setSourceFile(".//targets//cucumberresult.json");
    results.executeFeaturesOverviewReport();
    CucumberUsageReporting report = new CucumberUsageReporting();
    report.setOutputDirectory("target");
    report.setJsonUsageFile("./src/test/resources/cucumber-usage.json");
    report.executeReport();
    }

    ReplyDelete
    Replies
    1. Just view the previous replies in this topic. Details also can be explained here

      Delete
  9. pls can you help to resolve this. I have included these dependency in my pom.xml file

    com.github.mkolisnyk
    cucumber-reports
    0.0.3
    test



    net.masterthought
    cucumber-reporting
    0.4.0


    com.cedarsoftware
    json-io
    2.2.31

    ReplyDelete
    Replies
    1. Please, update cucumber-reports version to the latest one. Currently it is 0.0.11

      Delete
    2. Yes i did update thee cucumber-reports.

      com.github.mkolisnyk
      cucumber-reports
      0.0.11
      test


      But I still get the error.
      [31mFailure in after hook: [0m [31mHooks.afterScenario() [0m
      [31mMessage: [0m [31mjava.io.FileNotFoundException
      at com.github.mkolisnyk.cucumber.reporting.CucumberResultsCommon.readFileContent(CucumberResultsCommon.java:43)
      at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeOverviewReport(CucumberResultsOverview.java:154)
      at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeFeaturesOverviewReport(CucumberResultsOverview.java:168)
      at com.chroma.aodb.stepsdefs.Hooks.afterScenario(Hooks.java:25)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:497)
      at cucumber.runtime.Utils$1.call(Utils.java:37)
      at cucumber.runtime.Timeout.timeout(Timeout.java:13)
      at cucumber.runtime.Utils.invoke(Utils.java:31)
      at cucumber.runtime.java.JavaHookDefinition.execute(JavaHookDefinition.java:60)
      at cucumber.runtime.Runtime.runHookIfTagsMatch(Runtime.java:223)
      at cucumber.runtime.Runtime.runHooks(Runtime.java:211)
      at cucumber.runtime.Runtime.runAfterHooks(Runtime.java:205)
      at cucumber.runtime.model.CucumberScenario.run(CucumberScenario.java:46)
      at cucumber.runtime.junit.ExecutionUnitRunner.run(ExecutionUnitRunner.java:91)
      at cucumber.runtime.junit.FeatureRunner.runChild(FeatureRunner.java:63)
      at cucumber.runtime.junit.FeatureRunner.runChild(FeatureRunner.java:18)
      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
      at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      at cucumber.runtime.junit.FeatureRunner.run(FeatureRunner.java:70)
      at cucumber.api.junit.Cucumber.runChild(Cucumber.java:93)


      Thanks.

      Delete
    3. This error appears because at the moment this code was invoked the actual JSON file wasn't populated yet as all data is filled after all tests and all after-actions are done. To overcome this problem you can use Cucumber extension from the same library. The test code looks like:

      import org.junit.runner.RunWith;

      import cucumber.api.CucumberOptions;

      import com.github.mkolisnyk.cucumber.runner.ExtendedCucumberOptions;
      import com.github.mkolisnyk.cucumber.runner.ExtendedCucumber;

      @RunWith(ExtendedCucumber.class)
      @ExtendedCucumberOptions(
      jsonReport = "build/cucumber.json",
      jsonUsageReport = "build/cucumber-usage.json",
      outputFolder = "build/",
      detailedReport = true,
      detailedAggregatedReport = true,
      overviewReport = true,
      usageReport = true,
      coverageReport = false,
      retryCount = 0,
      screenShotLocation = "screenshots/",
      screenShotSize = "300px",
      toPDF = true
      )
      @CucumberOptions(
      plugin = {"html:build/cucumber-html-report",
      "junit:build/cucumber-junit.xml",
      "json:build/cucumber.json",
      "pretty:build/cucumber-pretty.txt",
      "usage:build/cucumber-usage.json"
      },
      features = {"src/test/java/com/mkolisnyk/common/features"},
      glue = {"com/mkolisnyk/common/steps"},
      tags = {"@test"}
      )
      public class AcceptanceTest {
      }

      Of course, you should set proper references to paths and files. Nota that I'm using mix of CucumberOptions and ExtendedCucumberOptions

      Delete
  10. thanks a lot. I will try this. Thanks a lot for the immediate response.

    ReplyDelete
  11. I have tried as you suggested, and I have set the references path and files. My scenarios are running but I got this error. Sorry for being a trouble.

    2 Scenarios ( [31m2 failed [0m)
    6 Steps ( [32m6 passed [0m)
    0m15.939s

    java.io.FileNotFoundException
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsCommon.readFileContent(CucumberResultsCommon.java:43)
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeOverviewReport(CucumberResultsOverview.java:154)
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeFeaturesOverviewReport(CucumberResultsOverview.java:168)
    at com.chroma.aodb.stepsdefs.Hooks.afterScenario(Hooks.java:25)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at cucumber.runtime.Utils$1.call(Utils.java:37)
    at cucumber.runtime.Timeout.timeout(Timeout.java:13)
    at cucumber.runtime.Utils.invoke(Utils.java:31)
    at cucumber.runtime.java.JavaHookDefinition.execute(JavaHookDefinition.java:60)
    at cucumber.runtime.Runtime.runHookIfTagsMatch(Runtime.java:223)
    at cucumber.runtime.Runtime.runHooks(Runtime.java:211)
    at cucumber.runtime.Runtime.runAfterHooks(Runtime.java:205)
    at cucumber.runtime.model.CucumberScenario.run(CucumberScenario.java:46)
    at com.github.mkolisnyk.cucumber.runner.ExtendedExecutionUnitRunner.run(ExtendedExecutionUnitRunner.java:35)
    at com.github.mkolisnyk.cucumber.runner.ExtendedFeatureRunner.runChild(ExtendedFeatureRunner.java:76)
    at com.github.mkolisnyk.cucumber.runner.ExtendedFeatureRunner.runChild(ExtendedFeatureRunner.java:22)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at cucumber.runtime.junit.FeatureRunner.run(FeatureRunner.java:70)
    at com.github.mkolisnyk.cucumber.runner.ExtendedCucumber.runChild(ExtendedCucumber.java:72)
    at com.github.mkolisnyk.cucumber.runner.ExtendedCucumber.runChild(ExtendedCucumber.java:28)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at com.github.mkolisnyk.cucumber.runner.ExtendedCucumber.run(ExtendedCucumber.java:141)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

    java.io.FileNotFoundException
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsCommon.readFileContent(CucumberResultsCommon.java:43)
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeOverviewReport(CucumberResultsOverview.java:154)


    ReplyDelete
  12. java.lang.IllegalArgumentException: Cannot locate declared field class org.apache.http.impl.client.HttpClientBuilder.sslcontext
    at org.apache.commons.lang3.Validate.isTrue(Validate.java:155)
    at org.apache.commons.lang3.reflect.FieldUtils.readDeclaredField(FieldUtils.java:512)
    at com.gargoylesoftware.htmlunit.HttpWebConnection.createConnectionManager(HttpWebConnection.java:890)
    at com.gargoylesoftware.htmlunit.HttpWebConnection.getResponse(HttpWebConnection.java:166)
    at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1316)
    at com.gargoylesoftware.htmlunit.WebClient.loadWebResponse(WebClient.java:1233)
    at com.gargoylesoftware.htmlunit.html.HtmlPage.loadJavaScriptFromUrl(HtmlPage.java:1130)
    at com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:1073)
    at com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:395)
    at com.gargoylesoftware.htmlunit.html.HtmlScript$3.execute(HtmlScript.java:276)
    at com.gargoylesoftware.htmlunit.html.HtmlScript.onAllChildrenAddedToPage(HtmlScript.java:290)
    at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:

    ReplyDelete
    Replies
    1. If you still have method annotated with @After, you should remove it. Reports are generated as the part of ExtendedCucumber runner

      Delete
  13. This comment has been removed by the author.

    ReplyDelete
  14. @RunWith(ExtendedCucumber.class)
    @ExtendedCucumberOptions(
    jsonReport = "build/cucumber.json",
    jsonUsageReport = "build/cucumber-usage.json",
    outputFolder = "build/",
    detailedReport = true,
    detailedAggregatedReport = true,
    overviewReport = true,
    usageReport = true,
    coverageReport = false,
    retryCount = 0,
    screenShotLocation = "screenshots/",
    screenShotSize = "300px",
    toPDF = true
    )
    @CucumberOptions(
    plugin = {"html:build/cucumber-html-report",
    "junit:build/cucumber-junit.xml",
    "json:build/cucumber.json",
    "pretty:build/cucumber-pretty.txt",
    "usage:build/cucumber-usage.json"
    },
    features = {"src/test/resources/features/regression"},
    glue = {"com.chroma.aodb.stepsdefs"},
    tags = {"@sampleTest"}
    )
    public class AcceptanceTest {
    }

    ReplyDelete
  15. now I get a different error,
    java.lang.Exception: Error occured while generating Cucumber usage report
    at com.github.mkolisnyk.cucumber.reporting.CucumberUsageReporting.executeReport(CucumberUsageReporting.java:678)
    at com.github.mkolisnyk.cucumber.runner.ExtendedCucumber.runUsageReport(ExtendedCucumber.java:80)
    at com.github.mkolisnyk.cucumber.runner.ExtendedCucumber.run(ExtendedCucumber.java:146)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
    Caused by: java.io.FileNotFoundException
    at com.github.mkolisnyk.cucumber.reporting.CucumberUsageReporting.getStepSources(CucumberUsageReporting.java:627)
    at com.github.mkolisnyk.cucumber.reporting.CucumberUsageReporting.executeReport(CucumberUsageReporting.java:659)
    ... 8 more
    java.io.FileNotFoundException
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsCommon.readFileContent(CucumberResultsCommon.java:43)
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeOverviewReport(CucumberResultsOverview.java:154)
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeFeaturesOverviewReport(CucumberResultsOverview.java:168)
    at com.github.mkolisnyk.cucumber.runner.ExtendedCucumber.runOverviewReport(ExtendedCucumber.java:92)
    at com.github.mkolisnyk.cucumber.runner.ExtendedCucumber.run(ExtendedCucumber.java:149)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

    ReplyDelete
  16. Update ExtendedCucumberOptions. In particular make sure you have attribute like:

    usageReport = false,

    ReplyDelete
  17. I got this error,
    Oct 16, 2015 3:49:08 PM com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify
    WARNING: Obsolete content type encountered: 'text/javascript'.
    Oct 16, 2015 3:49:09 PM com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify
    WARNING: Obsolete content type encountered: 'text/javascript'.
    Oct 16, 2015 3:49:09 PM com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify
    WARNING: Obsolete content type encountered: 'application/x-javascript'.
    Oct 16, 2015 3:49:11 PM com.gargoylesoftware.htmlunit.DefaultCssErrorHandler error
    WARNING: CSS error: 'https://ajax.googleapis.com/ajax/static/modules/gviz/1.1/core/tooltip.css' [3:38] Error in pseudo class or element. (Invalid token " ". Was expecting one of: , ":", , , .)
    Oct 16, 2015 3:49:11 PM com.gargoylesoftware.htmlunit.DefaultCssErrorHandler warning
    WARNING: CSS warning: 'https://ajax.googleapis.com/ajax/static/modules/gviz/1.1/core/tooltip.css' [3:38] Ignoring the whole rule.
    Oct 16, 2015 3:49:12 PM com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify
    WARNING: Obsolete content type encountered: 'text/javascript'.
    Oct 16, 2015 3:49:12 PM com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify
    WARNING: Obsolete content type encountered: 'text/javascript'.
    Oct 16, 2015 3:49:13 PM com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify
    WARNING: Obsolete content type encountered: 'application/x-javascript'.
    Oct 16, 2015 3:49:14 PM com.gargoylesoftware.htmlunit.DefaultCssErrorHandler error
    WARNING: CSS error: 'https://ajax.googleapis.com/ajax/static/modules/gviz/1.1/core/tooltip.css' [3:38] Error in pseudo class or element. (Invalid token " ". Was expecting one of: , ":", , , .)
    Oct 16, 2015 3:49:14 PM com.gargoylesoftware.htmlunit.DefaultCssErrorHandler warning
    WARNING: CSS warning: 'https://ajax.googleapis.com/ajax/static/modules/gviz/1.1/core/tooltip.css' [3:38] Ignoring the whole rule.

    ReplyDelete
    Replies
    1. It's not the error. It's just HtmlUnit output. We have dedicated issue: https://github.com/mkolisnyk/cucumber-reports/issues/33
      Major thing is that you should see files generated. If you see reports, your solution works OK.

      Delete
  18. Hi,
    Thanks a lot. It's working. It has generated the report but I couldn't see that in Eclipse, but when I go to the folder, it was there. Can I send the report to my email address?.

    Thanks a lot again.

    ReplyDelete
    Replies
    1. Sending e-mail notification is out of the scope of current library but you can create separate utility which does it. You can find some e-mail sending samples, e.g. Java - Sending Email.

      I can think about adding such functionality (e.g. I can implement extra annotation for that) but it's not something which is available at the moment.

      Delete
  19. Ok no problem. I can take a look at it.
    What is this screenShotLocation = "screenshots/", screenShotSize = "300px" in the ExtendedCucumberOptions?.
    Is it for failed scenarios?

    ReplyDelete
    Replies
    1. Yes, in case of failing tests reporting solution expects some image files in specified locations with specific names. The way they are dropped there is also out of library scope as it is even out of Cucumber scope (mainly screen shot generation is up to engine which interacts with UI).

      So, if image is available it will be included into the detailed report. Otherwise, you'll simply get results with error and stack trace information as it is normally reported by Cucumber JVM

      Delete
  20. Hi,
    I have a question. I executed 39 scenarios and this 39 scenarios is shown in my console as passed but when i take a look at the pdf report, it is mentioned that 62 scenarios passed. I am not sure where this no is from.

    ReplyDelete
    Replies
    1. looks like it is also counting the background steps as one scenario.

      Delete
    2. Yes, background is calculated as scenario. This is the way backgrounds are represented in the JSON report which is the source for other reports. In this report background is still another form of scenario.

      Delete
  21. But it shouldn't actually count Background as a scenario because it is a part of a scenario.
    So if i have 2 scenarios with a Background then it will be total of 4 scenarios where in reality it is just two scenarios, isn't it?.

    ReplyDelete
    Replies
    1. It is reported as the scenario. All the library does is it takes JSON content, converts it into data structure and generates the HTML content in generic way. So, that's why results are represented in the way we see.

      But I agree that we can work on that to make results more precise. I've created dedicated issue for that: https://github.com/mkolisnyk/cucumber-reports/issues/34

      Delete
  22. alright, since it has already been reported as a bug. Thanks a lot for your help.

    ReplyDelete
    Replies
    1. Just to clarify, this bug was just created by me after your feedback. Disregard the actual implementation specifics report should not produce misleading information. So, I'll see what I can do with that.

      Delete
  23. Replies
    1. The retryCount field is used for automated failed tests re-run. The number specified by this option is the number of attempts to re-run failed test until it goes green or the number of retries is exceeded. A lot of times I had a situation when some tests fail just due to some temporary problem and it is normally passes on next run. So, with the retryCount option we can define how many times the engine will automatically re-run tests.

      Delete
  24. Hi, looks like you have fixed the bug with Background. How do I get the fix in my project?. Thanks.

    ReplyDelete
    Replies
    1. It's not yet fixed. I fixed just one part of it. Once I complete with this and release new version (0.0.12) then you'll be able to use the new functionality.

      Delete
    2. Thanks for your awesome work. I am using 1.0.0 version and still the Background bug is there. When will the fix be released for this issue? Thanks

      https://github.com/mkolisnyk/cucumber-reports/issues/34

      Delete
    3. It was fixed as the part of 0.0.12. So, now when we have 1.0.0 it's also there. Very likely there are some situations when background isn't handled properly. E.g. if test fails on background or at the first step it may still be reported separately as for reporting the background and the scenario are completely independent entities.

      Delete
    4. Thanks for the reply Nickolay Kolesnik. Are you planning to fix for these situations where background isn't handled properly with the examples you provided.

      Delete
  25. Hi Nickolay,
    I saw another issue with the report. I dont know if you are aware of it.
    When I run a test suite which has more than 45 scenarios, and if any of the scenario has failed, the aggregate result, result coverage, cucumber test result-result.html and the pdf doesn't show the latest report. It will always show the previous tests run with all passed scenarios.

    The feature overview result is fine. It shows the latest report with failed scenarios in it.

    ReplyDelete
    Replies
    1. The only case when it potentially can happen is when results report isn't overwritten. All those reports are post-processors and there can be 2 major cases when they produce previous output:

      1) The initial source JSON report file is still the same
      2) Reports themselves were failed to generate, so you actually see old reports.

      If you see the situation like you described the 2nd case is very likely. You should check whether console output contains some errors. Also, check both HTML and PDF reports.

      Delete
  26. the console is showing the failed scenarios. I have checked the pdf report and it still shows the previous tests which are passed. It never really update the report when there is any failed scenarios.

    ReplyDelete
    Replies
    1. Try to remove old output (all files) before running scenario again. Very likely you simply see old reports or at least some resources are still old.

      Delete
  27. i ran into some issues. cant seem to execute the tests.
    when i right click on the RunCukesTest.java file and click on Run as Junit, I am getting error.
    Class not found com.chroma.userInterface.executor.RunCukesTest
    java.lang.ClassNotFoundException: com.chroma.userInterface.executor.RunCukesTest
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

    do u know how to resolve this?.

    ReplyDelete
    Replies
    1. It means that there's no class files in the output directory. You should rebuild your Java project so that classes files appear. In Eclipse it's done via menu Project > Clean

      If you use Maven or Gradle you can run command which does all actions before running tests. E.g.

      mvn clean compile

      of for Gradle:

      griddle clean compile

      Delete
  28. I did like u hv suggested me, but when i run maven build, it's giving errors. It doens't seem to find any of my classes and packages in the project.

    ReplyDelete
    Replies
    1. Perform Project > Clean operation from your IDE. Maybe you can change some file a bit (just add some space) and save to trigger IDE-level compilation. That's probably the cause of your errors. The binary classes were not updated so when you run your tests you are actually running old code.

      Delete
  29. i just undo all the changes from the project i had in source control. so i got the previous build from source control now. and it seems fine, but i dont know if this issue will appear again if I try to refresh the project.

    ReplyDelete
    Replies
    1. It is something with IDE settings when changes are not built automatically. This way if you start running individual test you can get the same results as before just because actual binaries are still old. Normally, running Maven build from command line and settings refresh helps. But, this topic is a bit irrelevant to subject

      Delete
  30. Hi,
    can we use Junit annotations with cucumber jvm?

    ReplyDelete
    Replies
    1. You can but they would hardly work. At least JUnit @Before and @After annotated methods will not work for sure. That's related to the fact that Cucumber-JVM JUnit runner doesn't handle JUnit standard events. That's why it has hooks mechanisms where you can define custom @Before and @After methods.

      Delete
  31. ok. But @Before and @After accepts only Scenario scenario argument, isn't it?. when I have tried with more than one argument, it's giving me error. thanks a lot.

    ReplyDelete
    Replies
    1. Do not confuse JUnit native annotations and Cucumber-JVM annotations (which are mainly from the cucumber.api.java). If you use Cucumber-JVM and its' runner all non-Cucumber annotations will be skipped.

      Delete
  32. Hi Nickolay,

    I have a question about the RetryCount. I have set it to 1 in RunCukesTest class. When I run my cucumber tests, I got 5 failed scenarios, but it doesn't seem to re run all the failed scenarios. Are you aware of this?.

    The Junit Result shown in the console, it;s showing the step name and the scenario name. for ex:
    Given I am in the login page(Scenario: User login)
    When I enter username and password(Scenario: User login)
    Then I should be able to login(Scenario: User login)

    I don't think it's suppose to show them in this way. Thanks.

    ReplyDelete
    Replies
    1. Regarding retry count. It worked well. Are you sure it really doesn't re-run some test? Is it possible to localise which test wasn't re-run and where it failed initially. Potentially, if test fails on some pre-conditions like Before-step there can be some problems (at least I didn't check those edge cases). But when test fails during steps execution re-run definitely happens. So, if possible, please, check if that is the case. If not, maybe you can figure out some other feature which makes re-run not working. For now existing information is not enough to make any specific conclusions.

      As for console output. Per my understanding this is the output we observe using pretty formatter plugin. Honestly, I didn't check the behaviour for this formatter. I was mainly interested in consistent JUnit and JSON formatters output. I can take a look at this at least to see if pretty format output changes on re-run. But you also can check that. You can create some tests and run them using only standard Cucumber runner. If the output is the same, there's nothing I can do with that because all works fine. Otherwise, I will need to take a look where I can make changes.

      Delete
  33. Hi Nickolay,
    Can I run test parallely using ExtendedCucumber Class?.

    ReplyDelete
    Replies
    1. Currently this feature is just planned. I even have dedicated issue for that (#31). But I cannot give you any precise timelines when it is going to be done. Standard Cucumber JUnit runner doesn't seem to be designed to support parallel run. So, this task will take a while.

      Delete
  34. public class AcceptanceITCaseTest {

    @AfterClass
    public static void afterClass() {
    getDriver().quit();


    CucumberResultsOverview results = new CucumberResultsOverview();
    results.setOutputDirectory("target");
    results.setOutputName("cucumber-results");
    results.setSourceFile("./target/cucumber-report.json");
    try {
    results.executeFeaturesOverviewReport();
    }catch (Exception e){
    e.printStackTrace();
    }
    }
    }
    above is my code and when the code is trying to generate report, it finds cucumber-report.json empty and as a result, it throws an exception saying json is invalid. We can not specify anything to execute after "@AfterClass" Or can we?

    Also, I renamed already generated report as "cucumber-report.json" to "cucumber-report1.json" made change in above snippet as well, and rerun the code just to see if report is getting generated or not against the existing report. However, here i got null pointer exception.

    Can you please tell me how can I make use of this library?

    ReplyDelete
    Replies
    1. For the first problem there is dedicated post describing problem and the solution.

      As for second problem there may be thousands of reasons. But major checkpoints are:

      1) Make sure you use the latest library version (currently it's 1.0.0)
      2) Make sure that you refreshed all your dependencies so that no old library versions are available
      3) Make sure you don't use different versions of libraries which cucumber-reports library uses as the dependency (here is the reference to the pom.xml which is used by library). If possible, try to avoid conflicting dependencies
      4) Make sure the source file really exists at the specified location

      If problem still exists, feel free to create new issue in the GitHub project. While describing issue, please, drop the source JSON file content and the stack trace which is generated.

      Delete
  35. hi, can we download the new version of the report?.

    ReplyDelete
    Replies
    1. Here is the link to the Maven repository viewer: http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22cucumber-reports%22.
      Here you can view which version is now the latest. Also, this resource can provide the link to download binaries and view how to include dependencies. That can be the source where you can see whether new version is available.

      Also, the GitHub project shows the list of releases: https://github.com/mkolisnyk/cucumber-reports/releases

      Delete
  36. Hi, when I try to update the version to 0.0.12, I get an error Missing artifact com.github.mkolisnyk:cucumber-reports:jar:0.0.12. How do I resolve this?. Thanks.

    ReplyDelete
    Replies
    1. Hi,

      There can be multiple reasons. Just go through the following check-points:

      1) Make sure you type artefact name and version properly. The artefact you're referring to definitely exists in the repository and can be viewed here

      2) Make sure your Maven settings refer to central Maven repository where those artifacts are taken from (it shouldn't be a case as you already used some other versions but worth trying)

      3) Check proxy settings

      Missing artefact error generally means that Maven wasn't able to retrieve artefact from the repository.

      BTW, the most recent version of the library is 1.0.1 at the moment, so you can try to update dependency to this version

      Delete
  37. Hi Kolesnik,

    Could you please share some info on passing test data to stepdefination files. I want to run one feature file for different test data (reuse). how can I generalize stepdef file so that it will run for different test data. Please help me.

    Thanks a lot.

    ReplyDelete
    Replies
    1. Take a look at the keyword called Scenario outline: and corresponding Examples:. It is actually the way to make data-driven tests. Also, examples may have dedicated tags. At least they can be used in exclusion patterns.

      Delete
  38. Hi Kolesnik,

    Is there any code which would take screenshots and place in a specific folder for cucumber ??

    also is there a way to give a convention for the screenshots naming for each feature

    ReplyDelete
    Replies
    1. You can view the conversation here. All the answers to your questions can be found here.

      Delete
  39. can we use apache poi to read data external to step definitions class in cucumber files ?? if yes can you share an example of the same

    ReplyDelete
    Replies
    1. Of course you can. Actually, POI and Cucumber are independent on each other, so you can use the POI the same way as any other Java code. I don't know what specifics do you expect.

      Delete
  40. @RunWith(ExtendedCucumber.class)
    @ExtendedCucumberOptions(jsonReport = "target/cucumber.json",
    retryCount = 0,
    detailedReport = true,
    detailedAggregatedReport = true,
    overviewReport = true,
    coverageReport = true,
    jsonUsageReport = "target/cucumber-usage.json",
    usageReport = true,
    toPDF = true,
    //excludeCoverageTags = {"@flaky" },
    //includeCoverageTags = {"@passed" },
    outputFolder = "target")
    @CucumberOptions(
    features = { "./src/test/resource" },
    plugin = { "html:target/cucumber-html-report", "json:target/cucumber.json", "pretty:target/cucumber-pretty.txt", "usage:target/cucumber-usage.json", "junit:target/cucumber-results.xml" },
    tags = {"@ETV"})
    public class RunnerTest {

    @AfterSuite
    public static void reporting() throws Exception {
    CucumberUsageReporting report = new CucumberUsageReporting();
    report.setOutputDirectory("target");
    report.setJsonUsageFile("target/cucumber-usage.json");
    report.executeReport();

    }
    public static void generateReport() throws Throwable
    {
    CucumberResultsOverview results = new CucumberResultsOverview();
    results.setOutputDirectory(".//target");
    results.setOutputName(".//target//cucumber-results");
    results.setSourceFile(".//target//cucumber.json");
    results.executeFeaturesOverviewReport();
    }


    My runner class is defined as above but in debug mode i see that the after class
    report.executeReport(); and results.executeFeaturesOverviewReport();
    are not executing due to which I am not getting any reports ...
    Need help to understand where the reports are failing

    Dependency used

    com.github.mkolisnyk
    cucumber-reports
    1.0.1

    ReplyDelete
    Replies
    1. Look at related post for this. Briefly speaking, you don't need to add report generation instructions into AfterSuite method. You use the ExtendedCucumber, so you can use its' options for this

      Delete
    2. can you please explain this in more in detail please

      Advanced reporting is mainly about Cucumber standard reports post-processing so we should make consistent reference between values in CucumberOptions and ExtendedCucumberOptions annotations. Mainly it's about JSON file paths as for CucumberOptions it is output file while ExtendedCucumberOptions annotation utilizes this file as an input file.

      Delete
    3. Right, the library uses the JSON reports. But it also contains the runner which is responsible for proper report generation as soon as the proper input is available. So, the runner provided by the library contains all necessary processing. But when you do the AfterSuite method it becomes just the part of that processing which is normally happens before the final report generation. So, you can use it for various post-condition actions but not the report generation as the report is not available at the time such method is called

      Delete
    4. so does it mean that output for json is mentioned in cucumber options and the same needs to be given as input for the extended cucumber options method??

      Delete
    5. Yes, correct. ExtendedCucumberOptions are processed after entire run completes.

      Delete
  41. Hi,
    Getting the below after i integrated,
    java.io.FileNotFoundException
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsCommon.readFileContent(CucumberResultsCommon.java:43)
    at com.github.mkolisnyk.cucumber.reporting.CucumberDetailedResults.executeDetailedResultsReport(CucumberDetailedResults.java:338)
    at com.github.mkolisnyk.cucumber.runner.ExtendedCucumber.runDetailedAggregatedReport(ExtendedCucumber.java:120)
    at com.github.mkolisnyk.cucumber.runner.ExtendedCucumber.run(ExtendedCucumber.java:155)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
    Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.65 sec <<< FAILURE! - in com.dwp.runCarer.RunnerCarerTest
    com.dwp.runCarer.RunnerCarerTest Time elapsed: 0.47 sec <<< ERROR!
    java.lang.NoSuchMethodError: cucumber.runtime.model.CucumberFeature.getUri()Ljava/lang/String;
    at cucumber.runtime.junit.FeatureRunner.run(FeatureRunner.java:68)
    at com.github.mkolisnyk.cucumber.runner.ExtendedCucumber.runChild(ExtendedCucumber.java:72)
    at com.github.mkolisnyk.cucumber.runner.ExtendedCucumber.runChild(ExtendedCucumber.java:28)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at com.github.mkolisnyk.cucumber.runner.ExtendedCucumber.run(ExtendedCucumber.java:141)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

    ReplyDelete
  42. Nickoley,
    My runner class is below,
    @RunWith(ExtendedCucumber.class)
    @ExtendedCucumberOptions(
    jsonReport = "json:target/cucumber.json",
    retryCount = 3,
    detailedReport = false,
    detailedAggregatedReport = true,
    overviewReport = false,
    toPDF = false,
    outputFolder = "target/")
    @CucumberOptions(
    plugin = {"html:target/cucumber-html-report",
    "json:target/cucumber.json",
    "pretty:target/cucumber-pretty.txt",
    "usage:target/cucumber-usage.json",
    "junit:target/cucumber-junit.xml"
    },
    features="src/test/resources/com/test/testScriptCarers",
    glue={"com.dwp.testScriptCarers"},
    tags = {"@TestEndToEnd"}
    )

    Can you please help me where it is going wrong

    ReplyDelete
    Replies
    1. I see a few options to check:
      1) Was the target/cucumber.json actually generated?
      2) What is the version of the cucumber-reports library you are using (the latest version for now is 1.0.3)?
      3) Are you adding separate dependency for Cucumber-JVM? If so, which version it is?

      I see actually 2 errors:
      1) Something wrong happened at the Cucumber level which prevented report from being generated
      2) Since no report is generated the cucumber-reports library didn't find the file.

      So, I suspect something wrong happened before my library started generating any reports.

      Delete
  43. Finally made it was because of the dependency issue and certain classes where not getting identified added the below dependency for both cucumber and your in pom

    com.github.mkolisnyk
    cucumber-reports
    1.0.3


    and cucumber version as
    1.2.4

    and cucumber-jvm-deps version is not 1.2.4 but 1.0.5


    but my Scenario Outline is going into infinite loop
    will try to analyse is this because of the code or other one

    ReplyDelete
    Replies
    1. You can try dry run to filter out scenario implementation problems. For this you need to add dryRun = true flag into CucumberOptions annotation

      Delete
    2. I Nickoley,
      Thanks for the swift response.I added dryRun=true but excpetion is thrown but if i added dryRun=false i am able to execute the script but it is going into infinite loop.....

      java.io.FileNotFoundException
      at com.github.mkolisnyk.cucumber.reporting.CucumberResultsCommon.readFileContent(CucumberResultsCommon.java:107)
      at com.github.mkolisnyk.cucumber.reporting.CucumberResultsCommon.readFileContent(CucumberResultsCommon.java:127)
      at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeOverviewReport(CucumberResultsOverview.java:121)
      at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeFeaturesOverviewReport(CucumberResultsOverview.java:143)
      at com.github.mkolisnyk.cucumber.runner.ReportRunner.runOverviewReport(ReportRunner.java:43)
      at com.github.mkolisnyk.cucumber.runner.ReportRunner.run(ReportRunner.java:183)
      at com.github.mkolisnyk.cucumber.runner.ExtendedCucumber.run(ExtendedCucumber.java:111)
      at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
      at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
      java.io.FileNotFoundException
      at com.github.mkolisnyk.cucumber.reporting.CucumberResultsCommon.readFileContent(CucumberResultsCommon.java:107)
      at com.github.mkolisnyk.cucumber.reporting.CucumberResultsCommon.readFileContent(CucumberResultsCommon.java:127)
      at com.github.mkolisnyk.cucumber.reporting.CucumberDetailedResults.executeDetailedResultsReport(CucumberDetailedResults.java:472)
      at com.github.mkolisnyk.cucumber.runner.ReportRunner.runDetailedReport(ReportRunner.java:77)
      at com.github.mkolisnyk.cucumber.runner.ReportRunner.run(ReportRunner.java:185)
      at com.github.mkolisnyk.cucumber.runner.ExtendedCucumber.run(ExtendedCucumber.java:111)
      at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
      at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
      java.io.FileNotFoundException
      at com.github.mkolisnyk.cucumber.reporting.CucumberResultsCommon.readFileContent(CucumberResultsCommon.java:107)
      at com.github.mkolisnyk.cucumber.reporting.CucumberResultsCommon.readFileContent(CucumberResultsCommon.java:127)
      at com.github.mkolisnyk.cucumber.reporting.CucumberDetailedResults.executeDetailedResultsReport(CucumberDetailedResults.java:472)
      at com.github.mkolisnyk.cucumber.runner.ReportRunner.runDetailedAggregatedReport(ReportRunner.java:95)
      at com.github.mkolisnyk.cucumber.runner.ReportRunner.run(ReportRunner.java:186)
      at com.github.mkolisnyk.cucumber.runner.ExtendedCucumber.run(ExtendedCucumber.java:111)
      at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
      at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

      Delete
    3. How many tests are actually executed when you do a dry run? If there's no matching test, the report won't be generated as there's nothing to generate.

      Also, make sure you define parameters properly. In the test class example I see that you define the feature option as string. But it is an array. So, try to make sure that it looks something like:
      features= {"src/test/resources/com/test/testScript"}

      Delete
    4. Appreciate your valuable time and feedback, My Feature file has a scenario similar to the below one , if i give dryRun=false it is going into infinite loop,but if i give dryRun=true i am getting the above exception,will invest some time myself and try to find out what the issue is, because it is working for others ....
      in the mean while can you please share the git repository url where your example is ..
      One more question can we show Pie chart and bar charts after including your plugin??
      Scenario Outline: Only some roles have access to the address book
      When I sign in as a
      And I go to the address book
      Then I should be access
      Examples:
      | role | access |
      | admin | granted |
      | sales | granted |
      | typist | denied |
      | staff | denied |

      Delete
    5. OK. In your main test class you use tag attribute. Does this feature contain tag as well? If not, nothing is to be generated. Also, try to run using standard Cucumber runner (just use @RunWith(ExtendedCucumber.class) annotation) to see if infinite loop still persists. We definitely need to filter out errors irrelevant to reporting library before I can help you.

      As for other information you ask for:
      1) The repository is located by the following URL: https://github.com/mkolisnyk/cucumber-reports. You can find some examples in the tests location: here. Also, there is official documentation site for the library, however the documentation is still in process and I still have to document a lot of things.
      2) As for the ability to draw pie or bar charts. Well, some of supported reports include pie charts, some of them include bar charts. But each report is of specific purpose and you have quite limited abilities to configure it. In particular, the layout is now pretty much fixed.

      Delete
  44. @RunWith(ExtendedCucumber.class)
    @ExtendedCucumberOptions(
    jsonReport = "json:target/cucumber.json",
    retryCount = 10,
    detailedReport = true,
    detailedAggregatedReport = true,
    overviewReport = true,
    toPDF = true,
    outputFolder = "target/"

    )
    @CucumberOptions(
    dryRun = true,
    plugin = {"html:target/cucumber-html-report",
    "json:target/cucumber.json",
    "pretty:target/cucumber-pretty.txt",
    "usage:target/cucumber-usage.json",
    "junit:target/cucumber-junit.xml"
    },
    features="src/test/resources/com/test/testScript",
    glue={"com.test.testScript"},
    tags = {"@TestEndToEnd"}

    )

    ReplyDelete
  45. Hi Nickolay,

    Gone through the entire page, seems promising and thanks much for this amazing thing.

    However I am not able to generate the report. Even,"RunTest.java" is not being created.

    Below is my runner class:
    @CucumberOptions(
    plugin={"pretty", "html:target/cucumber-html-report",
    "json:target/cucumber-report.json",
    "pretty:target/cucumber-pretty.txt",
    "usage:target/cucumber-usage.json",
    "junit:target/cucumber-junit.xml"},
    dryRun=false,
    features ="src/test/resources/features/TestFeature.feature",
    format = {"json:target/cucumber-report.json", "html:target/cucumber"}
    // glue ={"scr/test/java/stepDefinations/",}
    // tags="@bmw"
    )

    public class RunTest extends AbstractTestNGCucumberTests{

    @AfterClass
    public static void generateReport() throws Throwable
    {
    CucumberResultsOverview results = new CucumberResultsOverview();
    results.setOutputDirectory(".//target");
    results.setOutputName(".//target//cucumber-results");
    results.setSourceFile(".//target//cucumber.json");
    results.executeFeaturesOverviewReport();
    }
    }

    ********************
    Few more information:
    I am using testng and eclipse.

    Please help.

    ReplyDelete
    Replies
    1. I've even created dedicated post for that. AfterClass is still invoked before the JSON report is generated. You should try ExtendedCucumberOptions instead. The latest version of library (which is now 1.0.3) supports TestNG.

      Here is sample TestNG class initialised for the advanced reporting and cucumber:


      @ExtendedCucumberOptions(jsonReport = "target/cucumber.json",
      retryCount = 0,
      featureOverviewChart = true,
      detailedReport = true,
      detailedAggregatedReport = true,
      overviewReport = true,
      coverageReport = true,
      jsonUsageReport = "target/cucumber-usage.json",
      usageReport = true,
      toPDF = true,
      breakdownReport = true,
      breakdownConfig = "src/test/resources/breakdown-source/simple.json",
      knownErrorsReport = true,
      knownErrorsConfig = "src/test/resources/known-errors-source/sample_model.json",
      consolidatedReport = true,
      reportPrefix = "testng-result",
      consolidatedReportConfig = "src/test/resources/consolidated-source/s_ample_batch.json",
      outputFolder = "target")
      @CucumberOptions(
      plugin = {"html:target/cucumber-html-report",
      "json:target/cucumber.json",
      "pretty:target/cucumber-pretty.txt",
      "usage:target/cucumber-usage.json"
      },
      features = {"src/test/java/com/github/mkolisnyk/cucumber/features/" },
      glue = {"com/github/mkolisnyk/cucumber/steps" },
      tags = {"@passed"}
      )
      public class TestSubClass extends ExtendedTestNGRunner {
      @BeforeSuite
      public void setUp() {
      System.out.println("Setup");
      }
      @AfterSuite
      public void tearDown() {
      System.out.println("Teardown");
      }
      }

      Delete
  46. Hi Nickolay,

    Gone through the page and tried the same with my test runner, however I am not able to generate reports. I am getting below error:
    java.lang.UnsupportedClassVersionError: com/github/mkolisnyk/cucumber/runner/ExtendedCucumberOptions : Unsupported major.minor version 52.0

    My test runner is as below:
    @RunWith(Cucumber.class)
    @ExtendedCucumberOptions(
    jsonReport = "build/cucumber.json",
    jsonUsageReport = "build/cucumber-usage.json",
    outputFolder = "build/",
    detailedReport = true,
    detailedAggregatedReport = true,
    overviewReport = true,
    usageReport = true,
    coverageReport = false,
    retryCount = 0,
    screenShotLocation = "screenshots/",
    screenShotSize = "300px",
    toPDF = true
    )
    @CucumberOptions(
    plugin={"pretty", "html:target/cucumber-html-report",
    "json:target/cucumber-report.json",
    "pretty:target/cucumber-pretty.txt",
    "usage:target/cucumber-usage.json",
    "junit:target/cucumber-junit.xml"},
    dryRun=false,
    features ="src/test/resources/features/TestFeature.feature",
    format = {"json:target/cucumber-report.json", "html:target/cucumber"}
    // glue ={"scr/test/java/stepDefinations/",}
    // tags="@bmw"
    )

    public class RunTest extends AbstractTestNGCucumberTests{
    }

    Note: I am using TestNg.

    ReplyDelete
    Replies
    1. This indicates that you are using java version less than 1.8. The latest library version is mainly targeted to 1.8 java as I'm using some of that features. However, TestNG support was added quite while ago and if you downgrade cucumber-reports library to 1.0.1 version you should be able to see results.

      Delete
    2. Thanks much Nickolay for the prompt response .

      Yes, I am using Java 1.7. I downgraded the cucumber report to 1.0.1 version. However I am still not getting the report. Can you please tell me which folder to look in. I am seeing in Target folder which contains two subfolders who hold reports but they are basic reports which gets generated without any customization.

      Delete
    3. The output folder is defined by the outputFolder field in the @ExtendedCucumberOptions annotation. Also, make sure that jsonReport is referred to the same location as standard Cucumber JSON report. E.g. taking your sample code from the first comment this field should be set into something like:
      jsonReport = "target/cucumber-report.json"

      Delete
    4. I changed the said thing in @ExtendedCucumberOptions as below:

      jsonReport = "target/cucumber-report.json",
      jsonUsageReport = "target/cucumber-usage.json",
      outputFolder = "target/",

      Still no success, No reports are being generated. However I am not getting any error.

      Please pardon me for bugging you for silly.

      Delete
    5. Please, drop the entire test class with all annotations. Also, make sure you refresh the content of the output folder as for Eclipse it takes a while to reflect current folder state.

      Delete
    6. Yes, I am refreshing it. Even I am checking it in actual folders on the machine. Just a clarification, which name this report would get generated with?

      Please find below the entire test runner class:

      package stepDefinations;

      import org.junit.runner.RunWith;
      import org.testng.annotations.AfterClass;
      import java.io.IOException;

      import org.testng.annotations.AfterClass;
      import org.testng.annotations.Test;

      import com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview;
      import com.github.mkolisnyk.cucumber.runner.ExtendedCucumberOptions;

      import cucumber.api.CucumberOptions;
      import cucumber.api.junit.Cucumber;
      import cucumber.api.testng.AbstractTestNGCucumberTests;
      import cucumber.api.testng.TestNGCucumberRunner;
      @RunWith(Cucumber.class)
      @ExtendedCucumberOptions(
      jsonReport = "target/cucumber-report.json",
      jsonUsageReport = "target/cucumber-usage.json",
      outputFolder = "target/",
      detailedReport = true,
      detailedAggregatedReport = true,
      overviewReport = true,
      usageReport = true,
      coverageReport = false,
      retryCount = 0,
      screenShotLocation = "screenshots/",
      screenShotSize = "300px",
      toPDF = true
      )
      @CucumberOptions(
      plugin = {"html:target/cucumber-html-report",
      "json:target/cucumber.json",
      "pretty:target/cucumber-pretty.txt",
      "usage:target/cucumber-usage.json"
      },
      dryRun=false,
      features ="src/test/resources/features/TestFeature.feature",
      format = {"json:target/cucumber-report.json", "html:target/cucumber"}
      // glue ={"scr/test/java/stepDefinations/",}
      // tags="@bmw"
      )

      public class RunTest extends AbstractTestNGCucumberTests{

      }

      Delete
    7. Major root cause found. Please, extend your test class from ExtendedTestNGRunner class as this is major runner which is responsible for running tests and reports generation.

      And just some additional comments:

      1. remove format field in @CucumberOptions annotation. It is deprecated and in your example it is duplicated by plugin field which looks OK.
      2. uncomment glue field.
      3. If you run tests via TestNG you don't need the @RunWith(Cucumber.class) annotation

      Delete
  47. Hi Nickolay,

    Is there any way to send the reports generated to a mail directly ??

    also can I get PDF for Overview Report generated ??

    ReplyDelete
    Replies
    1. Hi Uma,

      Regarding e-mail sending. This is generally not the responsibility of the library. You can add some utility which sends reports after test execution is done. You can find some code samples for that here. Maybe in the future such thing would be added into the library as well (maybe as an additional annotation) but I wouldn't expect it to be done in the nearest future.

      As for PDF generation. The @ExtendedCucumberOptions annotation contains toPDF field. If it is set to true every report would have PDF version (except usage report). This functionality is available since 1.0.3 library version. If you use earlier versions this flag is working for detailed reports only. And yes, the latest library versions already require 1.8 Java.

      Delete
  48. Thank you Nickolay would try it out :)

    ReplyDelete
  49. Hi Nickolay,

    I still didn't get success. I uncommented the glue then it is giving below error.

    java.lang.NullPointerException
    at com.github.mkolisnyk.cucumber.runner.ExtendedRuntimeOptions.(ExtendedRuntimeOptions.java:25)
    at com.github.mkolisnyk.cucumber.runner.ExtendedTestNGRunner.runCukes(ExtendedTestNGRunner.java:37)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:86)
    at org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:197)
    at cucumber.api.testng.AbstractTestNGCucumberTests.run(AbstractTestNGCucumberTests.java:19)
    at org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:209)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:639)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:820)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1128)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
    at org.testng.TestRunner.privateRun(TestRunner.java:782)
    at org.testng.TestRunner.run(TestRunner.java:632)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:366)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319)
    at org.testng.SuiteRunner.run(SuiteRunner.java:268)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1246)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1171)
    at org.testng.TestNG.run(TestNG.java:1066)
    at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:113)
    at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:206)
    at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:177)


    However having glue commented I was able to get few more files getting generated under Target folder, but still these are the basic reports which are getting generated under "cucumber" and "cucumber-html-report" folders.

    ReplyDelete
    Replies
    1. Looks like test class didn't recognise annotation for extended options. I've raised an issue for further investigation.

      Delete
    2. Just an update on this. I've released new 1.0.4 version where TestNG problem was fixed. Please, try and let me know if there are some problems

      Delete
  50. Nikolay, thank you for this plugin, it's briliant, works like a magic!
    Thank you very much, this is exactly what I was looking for!
    Nina

    ReplyDelete
    Replies
    1. Thank you! I'm always glad to know when my solution is useful.

      Delete
  51. Hi NIckolay, great library! I'm excited to use it. Is there the ability to set the extended cucumber options from the command line? For example, with Cucumber options you can have a Maven command that sets them like this:

    mvn -Dcucumber.options="--tags @smoke" verify

    ReplyDelete
    Replies
    1. Hi. Currently there's no such possibility as it wasn't really needed having ExtendedCucumber runner in place (which was supposed to do all magic). But if it's needed I can create separate Maven plugin for the reporting. I've created the task for this here

      Delete
    2. Thanks for the quick response! Here is some context to why I would like the command line options.

      I run my integration tests with Maven through Jenkins. The main purpose I would use the command line options for is to configure the retryCount per Jenkins job. I have multiple Jenkins jobs that may require a different rerun count but use the same Cucumber runner. So it would be ideal if I could run the following command:

      mvn -Dcucumber.options="--retryCount=1" verify

      Let me know if I can help in any way. Much appreciated!

      Delete
    3. I see. So, you need to override ExtendedCucumber runner options which is a bit different. OK, I've created another issue (#76) for that. I doubt it would be included in the most recent release which is to be done in the nearest future but the release after that would definitely contain the change.

      Meanwhile you can use copies of Cucumber class with different retryCount parameters (maybe you'll need to vary something else as well). This would be some kind of interim solution

      Delete
    4. Thanks, I hope I clarified my use case better. I wasn't sure what you meant exactly by the Maven plugin. =]

      I am working on implementing the separate Cucumber runners now, but so far the only variation is the retryCount.

      Delete
  52. Question: If we are currently on Java 7, do you know if it is possible create a version that only requires junit 4.11 and Java 7 or any workarounds?

    ReplyDelete
  53. For JUnit version it's trivial. You have to have appropriate dependency version and exclude any other dependencies. As for Java version itself it's a bit more complicated as the Java has backwards compatibility but if something is done for later version there's high probability it wouldn't work for earlier versions.

    ReplyDelete
  54. hi all,
    i got a problem any one help me .i am using cucumber jar is :cucumber-reports-1.0.5.jar. what's the issue
    java.lang.UnsupportedClassVersionError: com/github/mkolisnyk/cucumber/reporting/CucumberResultsOverview : Unsupported major.minor version 52.0
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(Unknown Source)
    at java.security.SecureClassLoader.defineClass(Unknown Source)
    at java.net.URLClassLoader.defineClass(Unknown Source)
    at java.net.URLClassLoader.access$100(Unknown Source)
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at reports.CucumberRepo.repo(CucumberRepo.java:11)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

    ReplyDelete
    Replies
    1. Hi,

      Probably you use Java 1.7 or lower. The reporting library requires 1.8

      Delete
  55. Dear Nikolay,
    i tried your style of reporting using your dependency and code. i get the report generated however, the result is undefined. both the circles are in grey colour.
    I wonder what i am doing wrong here. any help much appreciated please.

    many thanks,
    Raj

    ReplyDelete
    Replies
    1. You should check that:
      1) Your input JSON contains any results (not dry run)
      2) Path to the JSON file is proper

      For more details you can refer to the documentation page

      Delete
    2. Hi Niko!
      thanks for the prompt reply. i tried transferring the dependency and code to an other project but unfortunately its throwing a file not found exception. it looks like its not able to create target folder for some reason.

      Delete
    3. The target folder is something to be created by Maven during build. Also, the reporting library is just post-processor for standard Cucumber output. So, if you have file not found it likely means that this source file wasn't generated. So, the problem may happen even before you start running anything related to reporting library

      Delete
    4. This comment has been removed by the author.

      Delete
    5. This comment has been removed by the author.

      Delete
  56. Replies
    1. the problem was the fact that I did not create the cucumber.jason file with data in it. once i have done that, it works fine now. but now i have one more problem. i have put the generate report method under @AfterClass. now, as long as all the tests pass i get the report generated. if there is a failure such as element not found excretion etc, the tests fail, exception is thrown and the generate report method is not read. i have put all my code in to a try catch block and run the generate reports in the finally block. unfortunately the report shows green only. its not showing any failures. i am not sure how to generate a report which can differentiate pass with failures.

      Delete
    2. This problem is related to the topic described in neighbouring post. The idea is that explicit report generation is not really applicable for After-methods as the JSON report is generated at very late stage. But cucumber-reports library provides extension for standard Cucumber runner. Thus, you can declare your test class like this and any supported report will be enabled just by specifying appropriate flag in ExtendedCucumberOption annotation. Also, notice that it uses @RunWith(ExtendedCucumber.class).

      Delete
  57. When I am running my below Runner class

    @RunWith(ExtendedCucumber.class)
    @ExtendedCucumberOptions(jsonReport = "target/cucumber.json",
    overviewReport = true,
    outputFolder = "target")
    @CucumberOptions(plugin = { "html:target/cucumber-html-report",
    "json:target/cucumber.json", "pretty:target/cucumber-pretty.txt",
    "usage:target/cucumber-usage.json", "junit:target/cucumber-results.xml" },
    features = { "src/main/resources/Login.feature" },
    glue = { "com/github/mkolisnyk/cucumber/steps" })
    public class RunnerClass {

    }


    I am getting the below exception, could you please help me on this


    java.lang.IllegalArgumentException: Cannot locate declared field class org.apache.http.impl.client.HttpClientBuilder.sslcontext
    at org.apache.commons.lang3.Validate.isTrue(Validate.java:155)
    at org.apache.commons.lang3.reflect.FieldUtils.readDeclaredField(FieldUtils.java:512)

    ReplyDelete
    Replies
    1. This error is related to HtmlUnit and the cucumber-reports library doesn't use it for ages. If this error is (anyhow) related to the cucumber-reports library you should update it to the latest version. Currently it is 1.0.6

      Delete
  58. Hi,

    The screen shots are not getting attached in my pdf report. CucumberOptions and ExtendedCucumberOptions in my runner class are given below

    @RunWith(ExtendedCucumber.class)
    @ExtendedCucumberOptions(jsonReport = "Reports/cucumber.json", jsonUsageReport = "Reports/cucumber-usage.json", screenShotLocation = "/Screenshots/",screenShotSize = "300px",detailedReport = true, detailedAggregatedReport = true, overviewReport = true, overviewChartsReport = true, featureOverviewChart = true, usageReport = true, coverageReport = true, toPDF = true, pdfPageSize = "auto", outputFolder = "Reports")
    @CucumberOptions(plugin = { "html:Reports/cucumber-html-report", "json:Reports/cucumber.json",
    "pretty:Reports/cucumber-pretty.txt", "usage:Reports/cucumber-usage.json",
    "junit:Reports/cucumber-results.xml" }, monochrome = true)

    I have tried the following 2 After methods

    @After
    public void takeScreenshot(Scenario scenario) {
    if (scenario.isFailed()) {
    WebDriver driver1 = new Augmenter().augment(driver);
    File file = ((TakesScreenshot) driver1).getScreenshotAs(OutputType.FILE);
    try {
    FileUtils.copyFile(file, new File(
    System.getProperty("user.dir") + "/Reports/Screenshots/" + scenario.getName() + ".png"));
    } catch (IOException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
    }
    }
    }

    @After
    public void takeScreenshot(Scenario scenario) {
    if (scenario.isFailed()) {
    final byte[] data = ((TakesScreenshot) driver).getScreenshotAs(OutputType.BYTES);
    scenario.embed(data, "image/png");
    }
    }

    None of the above scripts worked. When I checked the Screenshots folder I can see the screenshots in both cases. But its not attached in the pdf report.

    ReplyDelete
    Replies
    1. Screenshots location right now doesn't work properly. But embedded screenshots should apply. If you use the latest library version (now it's 1.0.7) you can use something like this:

      @After
      public void takeScreenshot(Scenario scenario) {
      if (scenario.getStatus().equalsIgnoreCase("failed")) {
      try {
      WebDriver augmentedDriver = new Augmenter().augment(Driver.current());
      File scrFile = ((TakesScreenshot) augmentedDriver).getScreenshotAs(OutputType.FILE);
      byte[] data = FileUtils.readFileToByteArray(scrFile);
      scenario.embed(data, "image/png");
      } catch (Exception e) {
      e.printStackTrace();
      }
      }
      }

      This code is taken from real project. So, it should work.

      Delete
    2. I tried the same using the latest library version (1.0.7). But still the screen shots are not getting attached in my pdf report. I have used the same After method that you have specified. Once I migrated to the latest version I got the following error when the script is run using Apache Ant

      java.lang.NoClassDefFoundError: org/joda/time/ReadableInstant
      at com.github.mkolisnyk.cucumber.reporting.CucumberUsageReporting.getOutputDirectory(CucumberUsageReporting.java:54)
      at com.github.mkolisnyk.cucumber.reporting.CucumberUsageReporting.executeReport(CucumberUsageReporting.java:700)
      at com.github.mkolisnyk.cucumber.runner.ReportRunner.runUsageReport(ReportRunner.java:29)
      at com.github.mkolisnyk.cucumber.runner.ReportRunner.run(ReportRunner.java:173)
      at com.github.mkolisnyk.cucumber.runner.ExtendedCucumber.run(ExtendedCucumber.java:164)
      at org.junit.runners.Suite.runChild(Suite.java:128)
      at org.junit.runners.Suite.runChild(Suite.java:27)
      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
      at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
      at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
      at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
      at org.junit.runner.JUnitCore.run(JUnitCore.java:105)
      at org.junit.runner.JUnitCore.run(JUnitCore.java:94)
      at com.settings.RunFeatures.main(RunFeatures.java:40)
      Caused by: java.lang.ClassNotFoundException: org.joda.time.ReadableInstant
      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      ... 18 more

      In order to fix the error I tried to associate the following libraries one by one
      org.apache.servicemix.bundles.joda-time-2.3_1.jar
      org.apache.servicemix.bundles.jruby-1.5.6_1.jar
      jruby-complete-1.2.0.jar
      joda-time-2.9.4.jar

      In all cases the error got fixed but I am not getting the screenshots attached in the pdf. Do I need to add any specific libraries(jar files) to attach the screenshots in pdf? Another thing I noted is that when I check the Screenshots folder there are 2 screenshots for a single failure. Do I need to remove the attribute Screenshot location from the ExtendedCucumberOptions? Could you please have a look at the CucumberOptions and ExtendedCucumberOptions in my runner class that I have posted earlier. Am I missing any thing?

      Delete
    3. 1) If you run via Apache Ant you'll definitely have some dependencies lost. Use Maven instead. It will carry out all dependencies. There are about 60 of them (including transitive dependencies). It's hard to handle them all manually.

      2) You may get 2 identical screen shots during PDF processing. It means that embedded image was recognised

      3) I see the screenShotLocation has value of "/Screenshots/". This may be treated as absolute path. Try to set as relative path, e.g. try this: screenShotLocation = "Screenshots/" (I've removed first slash in the path)

      Delete
    4. Thanks a lot. That fix worked for me. Removed the first slash from Screenshots and added a slash at the end of Reports in outputFolder

      Delete
  59. Hi,

    I have read your entire page including all questions and answers but I am still confusing of where I can add your below codes to have report generated.

    /* your code */
    CucumberResultsOverview results = new CucumberResultsOverview();
    results.setOutputDirectory("target");
    results.setOutputName("cucumber-results");
    results.setSourceFile("./src/test/resources/cucumber.json");
    results.executeFeaturesOverviewReport();

    Within some questions and answers, you seem to recommend to use included in the runner class. Am I confusing here?

    I have my hook as below. Should I add a method in it and modify your method (pass parameters) in it?

    /********************************************************
    * Setup and Get driver before start a test
    ********************************************************/
    @Before
    public void testSetup() {
    WebDriverFactory wdf = new WebDriverFactory();
    driver = wdf.getDriver();
    }

    /**
    * Take screenshot and quit
    * @param
    * @throws IOException
    */
    @After
    public void shutdownDriver() throws IOException {
    try {
    String screenshotFile = env.getScreenshotLocation() + new Object(){}.getClass().getEnclosingClass().getSimpleName();
    takeScreenShot(driver, screenshotFile);
    logger.info("***Screenshot is taken.");
    } catch (final WebDriverException somePlatformsDontSupportScreenshots) {
    System.err.println(somePlatformsDontSupportScreenshots.getMessage());
    }

    driver.quit();
    logger.info("***Close all browsers and stop driver.");
    }
    }

    ReplyDelete
    Replies
    1. I suggest using extended cucumber runner which is the part of this library. I wrote relevant post for that.

      Also, you can see some samples on the official documentation.

      Notice that current post was written more than 1 year ago, so some things have changed that time. So, right now the major idea is that you don't have to generate the report explicitly as Cucumber and JUnit after-methods are not proper place for that (those methods are normally triggered before source son report is generated). If you use ExtendedCucumber runner and ExtendedCucumberOptions annotation those reports would be generated. Please, refer the above links for examples. This is how the library is supposed to be used now.

      Delete
  60. Thank you for quick response.

    I attempted to add to my existing (below) runner class. However, the "@RunWith(ExtendedCucumber.class)" gives me error such as "Class cannot be resolved to a type".

    I could not figure out the cause since I import "import com.github.mkolisnyk.cucumber.runner.ExtendedCucumberOptions;"


    import org.junit.runner.RunWith;

    import com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview;
    import com.github.mkolisnyk.cucumber.runner.ExtendedCucumberOptions;

    import cucumber.api.CucumberOptions;
    import cucumber.api.junit.Cucumber;


    //@RunWith(ExtendedCucumber.class)
    @ExtendedCucumberOptions(jsonReport = "target/cucumber.json",
    retryCount = 0,
    detailedReport = true,
    detailedAggregatedReport = true,
    overviewReport = true,
    coverageReport = true,
    jsonUsageReport = "target/cucumber/Jason-Home/addFlight/addFlightTests.jason",
    usageReport = true,
    toPDF = true,
    outputFolder = "target/cucumber/reports/"
    )


    //@RunWith(Cucumber.class)
    @CucumberOptions(
    glue = { "classpath:cbp/test/stepdefs", "classpath:cbp/inttest/framework/scenariohook" },
    features = { "src/test/resources/cucumber/test.addFlight/AddFlightTests.feature" },
    format = { "pretty", "html:target/cucumber/addFlight/addFlightTests",
    "json:target/cucumber/Jason-Home/addFlight/addFlightTests.jason" },

    tags = {"~@Skip"}
    )

    public class AddFlightTests_IT {

    }


    Also, where I am suppose to add the below codes for report generating?

    public static void generateReport() throws Throwable {
    CucumberResultsOverview results = new CucumberResultsOverview();
    results.setOutputDirectory(".//target");
    results.setOutputName(".//target//cucumber-results");
    results.setSourceFile(".//target//cucumber//Jason-Home//addFlight//addFlightTests.jason");
    //results.execute();
    results.executeFeaturesOverviewReport();
    }

    ReplyDelete
    Replies
    1. I'll reply to the second question first:

      where I am suppose to add the below codes for report generating?

      Nowhere. ExtendedCucumber does it. Once you add @RunWith(ExtendedCucumber.class) the runner itself will generate the report.

      Now let's take a look at your ClassNotFoundException. I see you generate JSON report to the file at the location: target/cucumber/Jason-Home/addFlight/addFlightTests.jason. So, you need to update jsonReport field in ExtendedCucumberOptions annotation with this path. Otherwise, you look for non-existing report.

      Also, since you are not generating usage report you should remove usageReport and jsonUsageReport fields.

      Delete
    2. This comment has been removed by the author.

      Delete
  61. The reports look great. Thank so much for sharing.

    One of the report (pdf) is chopped off it legend on the right of the page. It is "cucumber-results-feature-overview.pdf". The html version look fine.

    How do I get the pdf report not to chop off on the right of the page?


    I have the below:
    @ExtendedCucumberOptions(
    jsonReport = "target/test-reports/addFlight/addFlightTests.jason",
    retryCount = 0,
    detailedReport = true,
    detailedAggregatedReport = true,
    overviewReport = true,
    coverageReport = true,
    toPDF = true,
    excludeCoverageTags = {"@flaky" },
    includeCoverageTags = {"@passed" },
    outputFolder = "target/test-reports/addFlight/reports/"
    )

    ReplyDelete
    Replies
    1. There is pdfPageSize field which defines the page size of generated PDF reports. Possible values can be found in corresponding CSS Specification page

      Delete
  62. Thank you for the respond. I used pdfPagesize = "landscape" and it works for me.

    I have another question. My test contains 5 scenarios and each scenario shares (few steps) background. When run, the report shows 10 tests because of it treats the background as a test.

    Is there a way I can fix this matter?

    Example.

    Background:
    Given I am at login site
    When I am authenticated for the site
    Then the system should display ABC page
    When I navigate to CDE page
    Then the system should display CDE page

    Scenario: test1
    do something

    Scenario: test1
    do something

    ReplyDelete
    Replies
    1. It is supposed to be like that for Overview reports only. For detailed reports background had to be added to scenario. At least I remember doing this enhancement.

      Anyway, right now it is how it should be. The problem is that this is the way results are reported in standard JSON file. All those backgrounds are reported as separate scenarios. So, it's definitely not that much I can do.

      Delete
  63. Hi,

    I have another question.

    How the "Cucumber-result-coverage" works?

    When run a test:
    1) Test results show in "cucumber-result-feature-overview" that there are three tests fail
    2) BUT ... "Cucumber-result-coverage" show all features pass.

    ReplyDelete
    Replies
    1. Coverage report is targeted to show if the all the expressions defined in feature files have implementations. It's more about static analysis. For coverage report tests would fail if there are some expressions which don't have Java implementation. So. tests may fail due to errors but coverage report shows if they are runnable in principle.

      Delete
  64. Thanks.

    I have another question.

    Is there a way I can give a name to the test result? eg. abc-test-result instead of "Cucumber-result-coverage"

    ReplyDelete
    Replies
    1. ExtendedCucumberOptions annotation has the field named reportPrefix. You can specify your own prefix and it will generate report with the name you need. The only thing is that you can change just prefixes. Suffixes are hard-coded as ExtendedCucumber generates multiple reports, so there should be guarantee that different reports will not overwrite themselves within the same run.

      Delete
  65. Hello,

    I have a little complex structure as I am using Cucumber, Testng, Maven and selenium grid to build my testing framework on Eclipse. It is also deploys a parallel running to perform tests on different browsers, as IE/chrome/FF at the same time.
    I wanna generate a separate report for every browser, and aggregated graphical report for all browsers/features.

    The issue is, all files regarding the graphical report are generated successfully, but the cucumber-html-report!
    This is one of the browsers specific run class..
    package cucumber.examples.java.testNG.runners;

    import org.junit.runner.RunWith;

    import com.github.mkolisnyk.cucumber.runner.ExtendedCucumber;
    import com.github.mkolisnyk.cucumber.runner.ExtendedCucumberOptions;

    import cucumber.api.CucumberOptions;
    import cucumber.api.testng.AbstractTestNGCucumberTests;

    @RunWith(ExtendedCucumber.class)
    @ExtendedCucumberOptions(jsonReport = "target/cucumber.json",
    detailedReport=true,
    detailedAggregatedReport = true,
    overviewReport = false,
    toPDF = false,
    outputFolder = "target")
    @CucumberOptions(
    features = "src/test/resources/features",
    glue = {"cucumber.examples.java.testNG.stepDefinitions"},
    tags = "@Demo",
    format = {"pretty",
    "html:target/cucumber-report/FF",
    "json:target/cucumber-report/FF/cucumber.json",
    "junit:target/cucumber-report/FF/cucumber.xml"},
    plugin={"html:target/cucumber-html-report",
    "json:target/cucumber.json", "pretty:target/cucumber-pretty.txt",
    "usage:target/cucumber-usage.json", "junit:target/cucumber-results.xml"})
    public class RunCukesTestInFirefox extends AbstractTestNGCucumberTests {

    }

    and this is my after suite method

    @AfterSuite
    public void reports() throws Exception{

    results.setOutputDirectory("target");
    results.setOutputName("cucumber-results");
    results.setSourceFile("./target/cucumber.json");
    results.execute();

    /* report.setOutputDirectory(".//target");
    report.setJsonUsageFile(".//target/cucumber-usage.json");
    report.executeReport();*/
    }

    Now, I have all files generated successfully, as cucumber.json and cucumber-results.xml, but I don't have cucumber-html-report

    Any help?

    ReplyDelete
    Replies
    1. The cucumber-html-report is standard Cucumber-JVM output file. Extended Cucumber runner isn't responsible for generating it.

      Delete
    2. Thanks for your quick reply.
      So what do you think could be the reason for not generating the cucumber-html-report?

      Delete
    3. It looks like Cucumber thing but one thing I'd suggest you to try is to avoid multiple sub-folders in the output path. E.g. you can define HTML output as "html:target/cucumber-reportFF". I suspect that if interim folder doesn't exist it wouldn't be created. But defining output folder directly inside existing folder should make things working.

      Delete
    4. Unfortunately, it didn't work!
      I just wonder which files the cucumber-html-report depends on to be created, and what could be the reason that made all files being generated except the graphical html file. I already followed the documentation step by step but no solution.

      Note: there is no errors at all in my console!

      Delete
    5. All I can advice is to cleanup you CucumberOptions annotation. I see that it has both format and plugin fields. Leave just plugin as the format field is deprecated. Also, make sure that the output path isn't overwritten by any other tests.

      Delete
  66. Is there a way that we can add custom messages to the report. For example a test step has passed and we can also print some custom message on the report for the passed test step? I know this can be done on the out of the box cucumber report using Scenario.write(String message) method but not sure if we can achieve this somehow on these reports as well.

    ReplyDelete
    Replies
    1. Since library version 1.0.7 there is handling of the Scenario.write output messages in detailed report.

      Delete
    2. That worked perfectly well. Thanks a lot. Really appreciate @Nickolay.

      Delete
  67. Hi ,
    am getting below error
    java.lang.NoClassDefFoundError: com/cedarsoftware/util/io/JsonReader
    at com.github.mkolisnyk.cucumber.reporting.interfaces.CucumberResultsCommon.readFileContent(CucumberResultsCommon.java:136)
    at com.github.mkolisnyk.cucumber.reporting.interfaces.AggragatedReport.readFileContent(AggragatedReport.java:35)
    at com.github.mkolisnyk.cucumber.reporting.interfaces.AggragatedReport.readFileContent(AggragatedReport.java:48)
    at com.github.mkolisnyk.cucumber.reporting.interfaces.AggragatedReport.readFileContent(AggragatedReport.java:42)
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeOverviewReport(CucumberResultsOverview.java:141)
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.execute(CucumberResultsOverview.java:202)
    at com.github.mkolisnyk.cucumber.reporting.interfaces.AggragatedReport.execute(AggragatedReport.java:23)
    at StepDefinition.Feature_StepDefinition.ReportGeneration(Feature_StepDefinition.java:34)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at

    Please help me with this issue.
    I'm using 1.0.6 cucumber version

    ReplyDelete
    Replies
    1. There can be one of the following reasons:
      1) You include jar directly without dependencies. Then you need to use either Maven or Gradle
      2) You use dependency management but you have your own dependency on com.cedarsoftware:json-io library. Remove it from your pom.xml or build.gradle (whatever you use).

      Delete
  68. Hi Nickolay, after execution i got a wonderful html report on my mail.But Graphs was missing.Just showing "100% (10) Passed0% (0) Failed0% (0) Undefined0% (0) Known". Any idea on this

    Thanks,
    Vipin

    ReplyDelete
    Replies
    1. Hi Vipin.

      Very likely you use MS Outlook (because on Mac everything is rendered properly). It doesn't support rendering of SVG which is used to display charts.
      SVG was selected because I needed fully standalone report file without any external dependencies.
      At the moment there's no alternative for that.

      Delete
    2. Hi Nickolay,

      Can you please provide overview report in some image format (*.png or *.jpeg), so that we can easily insert in mails.


      Regards,
      vipin

      Delete
  69. can we add a total number of feature and scenario couloumn execution in the overview of the report section.

    ReplyDelete
    Replies
    1. in the report the overalall duration is not displaying the time how can i fix it and when i get the total number of coloumn and can you provide me the source code of the report generating

      Delete
    2. Source code can be found here: https://github.com/mkolisnyk/cucumber-reports

      Delete
  70. can u add totalnumber of coloumn in the report

    ReplyDelete
  71. Hi Nickolay Kolensnik, My all scenarios showing as Passed in Report while some of them are failed.What I am missing ?

    ReplyDelete
    Replies
    1. I see several potential reasons for that:

      1) You are looking at wrong report. Either you take file which wasn't generated by the test or you might need to refresh it if you overwrote existing report. Also, you will see such results if you look at the coverage report which shows a bit different information and, yes, all tests may look like passed there, however this report shows a bit different information. So, make sure you look at results report not coverage.

      2) Another reason can be related to the way you indicate test as failed. It should be the same way as the Cucumber itself indicates the failed test. E.g. if you use Unit-Based runner you should use JUnit assertions to perform verifications.

      These are my guesses.

      Delete
  72. I am using below code foe generating Usage report but nothing is generated.I can generate overview report.

    CucumberUsageReporting report = new CucumberUsageReporting();
    report.setOutputDirectory("target");
    report.setJsonUsageFile("./src/test/resources/cucumber-usage.json");
    report.executeReport();

    ReplyDelete
  73. I am not able to generate Usage report but overview report is generating for me.Below is the code for Usage report.

    CucumberUsageReporting report = new CucumberUsageReporting();
    report.setOutputDirectory("target");
    report.setJsonUsageFile("./src/test/resources/cucumber-usage.json");
    report.executeReport();

    ReplyDelete
    Replies
    1. How your inability to generate usage report is reflected? Do you receive any errors? Can you see the output file?

      Apart from that make sure the JSON file, you specified, actually exists and it is exactly usage report (not just simple JSON output)

      Delete
  74. I am now getting following exception when try to generate Overview Report..


    java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.mbfs.devops.fabrics.runner.ExtendedCucumberRunner.runPredefinedMethods(ExtendedCucumberRunner.java:40)
    at com.mbfs.devops.fabrics.runner.ExtendedCucumberRunner.run(ExtendedCucumberRunner.java:56)
    at org.junit.runners.Suite.runChild(Suite.java:128)
    at org.junit.runners.Suite.runChild(Suite.java:27)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
    at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
    at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
    at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
    at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
    at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
    Caused by: java.lang.NoSuchFieldError: INTERNET_EXPLORER_11
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeOverviewReport(CucumberResultsOverview.java:159)
    at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeFeaturesOverviewReport(CucumberResultsOverview.java:168)
    at com.mbfs.devops.fabrics.reporting.html.HTMLReportingEngine.generateReport(HTMLReportingEngine.java:57)
    at com.mbfs.devops.automation.connector.CucumberRunnerTest.tearDown(CucumberRunnerTest.java:92)
    ... 23 more

    ReplyDelete
    Replies
    1. 1) You probably need to update the reporting library version. The latest one is 1.0.8 however 1.0.7 also should work.

      2) Also, I see that you are trying to invoke report generation from some After-method. That is not recommended. You can use annotations to define which report to generate. Similar problem and solution was described in the dedicated post

      Delete
  75. Hi Nickolay,

    I am using below code in stepdef file
    @Before
    public void setUp(Scenario scenario) throws Exception {
    this.scenario = scenario;
    }
    and i need to write input value into my html file, and i used
    scenario.write("some value") but it is writing into joson file but not able to get in html report can you please advise how to use this

    ReplyDelete
    Replies
    1. Which HTML report you refer to? Standard HTML output should have this feature by default and scenario.write will publish all necessary output. If you talk about Cucumber reporting library then such functionality is supported for detailed reports starting from 1.0.7 version.

      Delete
    2. am now getting following exception when try to generate Overview Report..


      java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at com.mbfs.devops.fabrics.runner.ExtendedCucumberRunner.runPredefinedMethods(ExtendedCucumberRunner.java:40)
      at com.mbfs.devops.fabrics.runner.ExtendedCucumberRunner.run(ExtendedCucumberRunner.java:56)
      at org.junit.runners.Suite.runChild(Suite.java:128)
      at org.junit.runners.Suite.runChild(Suite.java:27)
      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
      at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
      at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
      at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
      at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
      at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
      at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
      at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157)
      at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
      at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
      at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
      Caused by: java.lang.NoSuchFieldError: INTERNET_EXPLORER_11
      at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeOverviewReport(CucumberResultsOverview.java:159)
      at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeFeaturesOverviewReport(CucumberResultsOverview.java:168)
      at com.mbfs.devops.fabrics.reporting.html.HTMLReportingEngine.generateReport(HTMLReportingEngine.java:57)
      at com.mbfs.devops.automation.connector.CucumberRunnerTest.tearDown(CucumberRunnerTest.java:92)
      ... 23 more

      Delete
    3. Hi Nickolay Kolesnik,

      i am using your cucumber jars, so you mean to say i have to use 1.0.7 version of your jars or need to use cucumber jars?

      Delete
    4. You definitely need to update dependencies. It's not just my library jar but also subsequent dependencies.

      Delete
    5. Hi Nickolay,
      DO you have any update to this error.I am only getting this error while generating overview report but usage report & detail report I can generate.Also program is exiting at the overview report generation step and not going forward with other execution.Please advice.


      am now getting following exception when try to generate Overview Report..


      java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at com.mbfs.devops.fabrics.runner.ExtendedCucumberRunner.runPredefinedMethods(ExtendedCucumberRunner.java:40)
      at com.mbfs.devops.fabrics.runner.ExtendedCucumberRunner.run(ExtendedCucumberRunner.java:56)
      at org.junit.runners.Suite.runChild(Suite.java:128)
      at org.junit.runners.Suite.runChild(Suite.java:27)
      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
      at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
      at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
      at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
      at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
      at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
      at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
      at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:157)
      at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
      at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
      at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
      Caused by: java.lang.NoSuchFieldError: INTERNET_EXPLORER_11
      at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeOverviewReport(CucumberResultsOverview.java:159)
      at com.github.mkolisnyk.cucumber.reporting.CucumberResultsOverview.executeFeaturesOverviewReport(CucumberResultsOverview.java:168)
      at com.mbfs.devops.fabrics.reporting.html.HTMLReportingEngine.generateReport(HTMLReportingEngine.java:57)
      at com.mbfs.devops.automation.connector.CucumberRunnerTest.tearDown(CucumberRunnerTest.java:92)
      ... 23 more

      Delete
    6. Update library dependency. Use the latest versions of the library. It should start from 1.0.7

      Delete
    7. I am using library 1.0.8.Also how can change the css of the reporting

      Delete
    8. 1) According to the stack trace you sent, you're definitely not using the latest version. You can see the source code and see that, for instance, the executeFeaturesOverviewReport which threw an exception simply doesn't exist. But the similar class for version 0.0.10 contains problematic field and it fits the stack trace numbers. So, it means you still use old version of the library. You need to update the dependency.

      2) You cannot change CSS at the moment.

      Delete
    9. Thanks for your response.I have another requirement.Can we add date of execution & application name in the report.

      Delete
    10. Such functionality isn't available yet. I can create some task to add parameters definition section.
      However, at the moment you can parameterise report file names to reflect necessary data. You can find some samples at the documentation.

      Delete
    11. If you add those parameters in the report it will be very helpful.How soon can you add that.

      Delete
    12. There's no specific timelines as there are some other pending tasks. Also, this functionality can be replicated with existing Consolidated report. It simply merges several HTML reports into one.

      So, you can add some code which generates file with application information and then you can generate consolidated report which merges application information with the detailed report. Thus, you'll have everything in one file.

      Delete
    13. Is dates are already added in Consolidated report.I am plaaning on generating consolidated report,just wondering if it contains date field

      Delete
    14. There's no need of date in the consolidated report as it simply merges multiple files into one. If you need date you can generate some small HTML with date and whatever info you need and include it first.

      Delete
  76. Is it possible that we can hide the rerun scenarios which are failed and only show the last one if that was a passed? Currently if the rerun count is set to 1 and if first execution of test is failed where as rerun was a passed, in the report it'll show 2 runs (1st as failed and 2nd one as passed). Is it possible that in this case we'll only show the 2nd one which is a pass. Also if a test is repeatedly failing -- maybe just display the last run failed in the report only. Please let me know if this is possible or would need some code change. Thanks.

    ReplyDelete
    Replies
    1. Such functionality is implemented using aggregations. Please, refer to the Configuration Options section for re-run functionality. Mainly, you need to add detailedAggregatedReport = true into the @ExtendedCucumberOptions annotation.

      Delete
  77. Thanks for your response.I am trying to generate consolidated report but getting following error.
    Caused by: java.lang.AssertionError: Consolidated Report: Configuration file specified is of improper format.
    This is my JSon file.What mistake I am doing.

    {
    "@type": "com.github.mkolisnyk.cucumber.reporting.types.consolidated.ConsolidatedReportBatch",
    "models": [
    {
    "items": [
    {
    "path": "C:\Users\samukhe\workspace\ECM\target/cucumber-results-feature-overview-chart.html",
    "title": "Overview Chart"
    },

    {
    "path": "C:\Users\samukhe\workspace\ECM\target/cucumber-results-agg-test-results.html",
    "title": "Detailed Results"
    },

    ],
    "reportSuffix": "batch-config1",
    "title": "Overall Results Batch 1",
    "useTableOfContents": true
    },
    ]
    }

    ReplyDelete
    Replies
    1. It indicates that "Configuration file specified is of improper format". It means that library couldn't parse the JSON. Make sure it is in valid format. E.g. in you sample I see extra commas after last items of the list. That may cause the error. Generally, I'd recommend you to run some JSON validators. There are some available online.

      Delete