JBehave vs Cucumber JVM comparison

The interest in BDD approach is growing and the more and more different engines appear in that space. However, there're some well known players in that area. We can compare their features however the final choice is still based on the technology used. E.g. if we compare Behat and SpecFlow we definitely can find some gaps and advantages but if we're going to use e.g. C# the Behat is not a subject of interest here. And vice versa.

The more interesting case is when 2 or more engine appear in the same technological area. And it's twice more interesting when these are one of the most popular engines on one of the most popular technology. In this post we'll compare JBehave vs Cucumber against their applicability for Java test automation. We compare them using the same scale as for BDD Engines Comparison post with some additional detalization on features. Additionally, I'd like to mention that we should compare apples to apples so there's no need to dig the Ruby implementation of Cucumber. It's interesting to check the Cucumber Java capabilities against JBehave to make comparison more or less valuable.

All right, we have JBehave, we have Cucumber, so ...

LET THE MORTAL COMBAT COMPARISON BEGIN!!!

What would be compared

Firstly, I'd like to mention that we should compare apples to apples so there's no need to dig the Ruby implementation of Cucumber. It's interesting to check the Cucumber Java capabilities against JBehave to make comparison more or less valuable.

During comparison we'll cover the following areas/features:

• Documentation
• Flexibility in passing parameters
• Auto-complete
• Scoping
• Composite steps
• Backgrounds and Hooks
• Binding to code
• Formatting flexibility
• Built-in reports
• Conformity to Gherkin standard
• Input Data Sources

And again we'll use 4 grade scale for evaluating each of the features. The grades are classified in the following way:

Grade	Criteria
0	Feature is not represented
1	Feature is partially available with some restrictions
2	Mainly feature exists but without any extra options
3	Full-featured support

Documentation

The documentation is the first entry point while trying to use any library/tool/other software whatever. So, we'll start with it here as well.While evaluating this feature we'll take a look at the following items:

• Official site - every more or less formed solution should have an official site and an entry point for new people. Otherwise it would be hard to find it within miriads pages in the Internet space
• Getting started guide - when you're new to any engine there should be some dedicated page where to start from. Basically it contains some step-by-step instruction for creating first sample using the engine/tool whatever. Having such page is the big plus to the documentation
• General Usage guide - indeed for many different reasons we should refer to manual for some specific feature or some complex action. So, it would be great if we have consolidated set of pages for reference
• API documentation - since both engines are used via API it would be nice to have them documented on the API level as well
• Code examples - that would be another reference point demonstrating some specific feature capabilities. In most cases the only live sample is much better than tons of documentation pages
• Specialized forums - every well-grown solution consolidates people around. They make communities, discussion forums. Such resources are very useful for discussing some specific cases or non-standard tricks which are quite frequent but problem-specific almost all the time

All those items are evaluated using boolean logic like "exists" or "doesn't exist". And again, our comparison will be more targeted to Java platform, so if something exists for Cucumber but without similar stuff for Java it's treated the same way as it doesn't exists. So, here is the comparison:

Feature	Available in JBehave	Evidence	Available in Cucumber	Evidence
Official site		JBehave Official site		Cucumber Official site NOTE There's no dedicated site for Cucumber-JVM however it's quite stupid to say that there's no official site at all. So, for now this option counts for Cucumber JVM but for any next split decision point I'll tend to say rather no than yes for feature availability
Getting started guide		Getting started page		The information about that is distributed between various samples and blog posts but there's no dedicated page showing all steps in Java
General Usage guide		Introduction Core Reference		The reason is the same as before. There's no dedicated reference to Java usage as there're too few Java-specific features
API documentation		JBehave Javadoc modules		Javadoc main page
Code examples		JBehave examples: A dime a dozen modules		Cucumber examples
Specialized forums		JBehave - BDD LinkedIn Group		Cucumber group on LinkedIn - quite populated place with big number of active discussions Cukes Google group

Based on above features I can evaluate JBehave and Cucumber with the following grades:

Engine	Grade
JBehave	3
Cucumber-JVM	2

Well, I wouldn't say that Cucumber has weaker documentation. The key thing is that Cucumber is spread across multiple platforms and it's the core part which is mostly documented. At the same time JBehave is more specialized. So, the documentation is more targeted and technology specific.

Flexibility in passing parameters

In this section we'll compare JBehave and Cucumber capabilities against their flexibility in building instructions and varying them in different fashion to cover more specific cases which correspond to the same action. Here we'll check the following options:

• Parameter variants - in some cases we have fixed scope of possible values to pass as the parameter. Instead of using generic regular expression we should be able to enumerate acceptable options. That's what's done using parameter variants
• Parameters injection - in some cases the sequence of the input parameters to method is not the same as we see convenient for our phrases. In such cases we should be able to define explicitly which part of the text goes to some specific parameter
• Tabular parameters - when we pass some complex structures or arrays of complex structures there should be some mechanism to group them in granular way so they are still readable and at the same time they're convenient to parse. Tabular parameters is one of the ways for that
• Examples - that looks similar to tabular parameters except this feature is used for different purpose when we want to run the same scenario against multiple different input parameters
• Multi-line input - relatively small feature however it's quite frequently used and it's nice when such feature is supported
• Extra features - any other features. They're not so essential as previous features however they're nice to have

So, the following table compares JBehave and Cucumber against the above features:

Feature	Available in JBehave	Evidence	Available in Cucumber	Evidence
Parameter variants		Pattern Variances		Sample code
Parameters injection		Reference page		Something similar
Tabular parameters		Tabular parameters in JBehave		DataTable samples
Examples		Parametrized scenarios		Scenario Outlines
Multi-line input				Docstrings sample
Formatted input		Parameter converters - nice feature for passing complex types within one instruction + the ability to customize conversion		Sample code containing an example. In particular: @Given("^today is (.+)$") public void today_is(@Format("yyyy-MM-dd") Date date) { calculator = new DateCalculator(date); }

There may be some other small features but in most cases they're rather some small syntax sugar or missing just because there's no need of them in some specific engine.

After the above features comparison we have the following grades for our engines:

Engine	Grade
JBehave	2
Cucumber-JVM	3

Looks like it's getting hot with such battle. No wonder, as we compare quite well-grown engines which have a bunch of features.

Auto-complete

This is very useful feature as when the test automation solution grows the number of keywords is growing as well. So, after some time it's hard to keep all phrases in memory. Indeed, some standards and agreements may help with that but only partially. The most effective helper for writing BDD instructions is an auto-complete ability displaying the most applicable options for the typed text.

Since the feature is useful it's represented with various plugins for IDE:

• JBehave update site for Eclipse with autocomplete support in feature files
• Cucumber update site for Eclipse containing various features including text highlight and autocomplete

NOTE

Since both plugins are applied to the same file types they may be in conflict with each other. So, in order to omit such problem it's recommended using only one of those plugins

So, both engines have such support. That's reflected in the evaluation scores:

Engine	Grade
JBehave	3
Cucumber-JVM	3

Scoping

Scoping is needed to narrow down the set of classes which would contain necessary bindings of simply get the sub-set of tests. So, there're 2 ways of stope restriction:

• Tests scope - identifies which sub-set of tests to run. Additionally it can be sub-divided into:
  • Feature/story scope - only some specific stories are in scope
  • Scenario scope - only some specific scenarios are in scope
• Step scope - identifies which sub-set of steps to use while looking for step binding

That makes 3 criteria which fits the grade scale. So, we can evaluate both engines against those features:

Feature	Available in JBehave	Evidence	Available in Cucumber	Evidence
Feature/Story scope		Meta-filtering		Tags
Scenario scope		Meta-filtering		Tags
Step scope		The scope of classes is defined during at the configuration so steps are initially scoped by design		For Cucumber it's treated as rather antipattern than some built-in solution

So, based on the above features we're getting the following grades:

Engine	Grade
JBehave	3
Cucumber-JVM	2

Well, the difference here is not so essential and mostly based on ideological aspects however things which are objectively useful are available in both engines.

Composite steps

This feature is useful when we'd like to re-use steps so that we can call some test instructions by code bound. However, since all the steps are actually annotated methods this feature is not essential. The Composite steps in JBehave is built-in feature. It is also available in Cucumber but:

• These's no trace of it in JVM part
• There's no actual need of it in Java implementation
• If there's a real need of it it can be done using reflection

So, JBehave supports composite steps as full-scale feature. Cucumber-JVM can do the same but it's rather workaround than something fully supported. So, it makes the following grades:

Engine	Grade
JBehave	3
Cucumber-JVM	1

Backgrounds and Hooks

This part of features covers the ability to perform some additional actions prior/after main events like feature/scenario/step execution. I've distributed this group into Backgrounds and Hooks as these features are similar by idea but fundamentally different by the purpose. When backgrounds are the part of the scenarios the hooks are rather part of the engine and their usage is mostly intended to expand the default engine behavior. That's why we'll evaluate engines accross those features separately.

Backgrounds

Backgrounds are supported by Cucumber however that's the weakest place of JBehave. There's no backgrounds there yet. That's one of the major things which don't conform the common Gherkin standards which are used accross many other BDD engines. Hope, that would be added in the nearest future but for now we have the following grades:

Engine	Grade
JBehave	0
Cucumber-JVM	3

Hooks

For hooks the situation is a bit more tricky. Hooks are available in JBehave as well as they're defined for Cucumber though the Java implementation has 1 major restriction: there's no hook before/after step. That's valid for both engine. All other hooks are defined and if necessary we can include hooks for test engines in use (JUnit or TestNG). But it's a matter of fact the hooks cover only some part of possible cases. Here is the comparison table:

Hook	Available in JBehave	Available in Cucumber
Before/After step
Before/After scenario
Before/After feature
Before/After run

So, both engines are equal here and both of them have the same grade:

Engine	Grade
JBehave	2
Cucumber-JVM	2

Binding to code

Well, this check-point is redundant here as JBehave and Cucumber use the same approach for binding text instructions to the code using annotations. This is the most compact way for that. So, we'll evaluate both engines with maximal grade:

Engine	Grade
JBehave	3
Cucumber-JVM	3

Formatting flexibility

Before I compared multiple engines against this characteristics and JBehave got minimal grade just because by default it doesn't ignores heading spaces before Given\When\Then statements. That's frustrating gap. However, if we dig deeper we can find that JBehave has StoryLoader interface and we can implement loadStoryAsText method to handle heading spaces and use it. Thus, we won't have such problems with formatting. Despite such flexibility isn't enabled by default it can be included using standard libraries. So, in this area JBehave isn't far behind and both engines can be evaluated with the following grades:

Engine	Grade
JBehave	2
Cucumber-JVM	3

Built-in reports

Indeed there's nothing we can do with the testing unless we have some representative results. So, reporting and results representation in general is the vital part of any engine. Also, there can be some other reporting types showing rather usage statistics than test execution status. It's useful information as well (e.g. it can be used to analyze how frequently we use the steps). Also, different formats are applicable for different areas. E.g. the HTML report format is nice to have as an output artifact while it's not convenient to see it in the console output. So, there're different report types and the more of them are available the more information we can get from run results. So, let's compare JBehave and Cucumber against supported report types:

Report type	Available in JBehave	Available in Cucumber
Console output
Pretty console output
Structured file (e.g. XML)
Well-formatted readable file (e.g. HTML)
Usage report
Extra report types

One thing to mention here is the availability of extra report types. The JBehave has PrintStreamOutput which allows customizing the output and write results in your own format. At the same time the Cucumber has various extra pre-defined formats. Again, these're likely bells and whistles but yet that's nice additional feature. So, that makes the following grades:

Engine	Grade
JBehave	2
Cucumber-JVM	3

Conformity to Gherkin standards

Both JBehave and Cucumber provide extra layer for test instruction definitions. It's the level of scenarios in natural language which are represented as external source for the core engine. It gives an ability to switch the core code or the entire programming language when necessary. Or it's also good when you write the API for multi-lingual use. In this case we should test it against the same scenarios but using different implementation basically depending to the programming language. For this purpose it is important to have all libraries in use to conform the common standards. For BDD engines it's the Gherkin. Many engines support it so if you create tests using one of those engines and then you decide to switch to some other platform you don't need to change at least test scenarios. That's why the compliance to some common standards is important. In this section we'll take one of the most frequently used keywords/attributes and check how do they conform Gherkin.

Keyword/attribute	Available in JBehave	Available in Cucumber	Comment
Feature			In JBehave the analog is Story. However, there's support of Gherkin style keywords which includes support of the Feature
Scenario
Given\When\Then
And\But
Scenario Outline
Examples
Background			No backgrounds no support of it
Tag			Tags have the same format but JBehave uses Meta keyword for that and that brings differences
Tables

Given the above results I can conclude that JBehave conforms Gherkin by near 2/3 while Cucumber is 100% compliant. That's reflected in grades:

Engine	Grade
JBehave	2
Cucumber-JVM	3

Input Data Sources

In some cases it's useful to store scenarios somewhere at the external location e.g. some tracking system. I'd say even more. If we want to involve many different people in tests automation we should use some shared resource accessible to everyone where those scenarios are defined. Also, some scenario may be an input for other scenarios or we simply need to reduce the copy/paste for repetitive set of instructions the same way we do while writing the program code. So, that's what all those features are about. So, we'll check them separately.

External Data

It's needed when we want to customize the input for our test instructions. For this purpose JBehave has StoryLoader interface and we can implement loadStoryAsText method which can be defined the way we need and read data from whatever we can reach. I won't even dig too deep to get some example. Recently, I made an integration between JBehave and JIRA. What about Cucumber? Nothing at the moment.

Nuff said. That's the score:

Engine	Grade
JBehave	3
Cucumber-JVM	0

Inclusions

The inclusions are needed to minimize code duplications. That's why they're are used in code, that's the purpose for scenarios as well. JBehave provides something similar to that named Given Stories it runs the stories specified in the referenced file prior to main story. It's not pure inclusion and mostly it can be omitted by groupping stories into one file. However, there're some cases where something like that is needed. Unfortunately Cucumber doesn't support this feature. Well, in some sense it's even good as such distributed scenarios are harder to trace and harder to maintain as it leads to external dependencies resolution. The funniest thing here is that such feature could be really needed for backgrounds as a lot of cases have some common pre-condition accross the features (usually it's login/logout operations). But Cucumber doesn't support inclusions and JBehave doesn't support backgrounds.

For this feature we'll make the following evaluation scores:

Engine	Grade
JBehave	2
Cucumber-JVM	0

Well, generally speaking the external data support is the weakest place in Cucumber. It was as good as JBehave in many other features but here it fails by all points.

Summary

It's time to summarize all the results and join them under total score table. Here it is:

Engine\Feature	Documentation	Flexibility in passing parameters	Auto-complete	Scoping	Composite steps	Backgrounds and Hooks		Binding to code	Formatting flexibility	Built-in reports	Conformity to standards	Input Data Sources		Total
						Backgrounds	Hooks					External Data	Inclusions
JBehave	3	2	3	3	3	0	2	3	2	2	2	3	2	30
Cucumber-JVM	2	3	3	2	1	3	2	3	3	3	3	0	0	28

Well, that was a good battle but JBehave appeared to be stronger at the end. However, Cucumber appeared to be quite strong opponent despite Java is not the main area of it's applicability. It simply lost some score points just due to some features were not migrated to Java yet. Nevertheless, I can conclude that both JBehave and Cucumber are equally applicable for Java-based automation and the choice between them is mostly based on what you need. If we think about choice between those engine I'd propose the following: if you look for some sort of "golder standard" the Cucumber is for you. But if you want something more you should look at JBehave