Cucumber: How to avoid major mistakes

Recently I came across several posts criticizing Cucumber and approach itself. Some of the links:

• BEHAVIOR-DRIVEN DEVELOPMENT VS. TESTING by James Bach
• Testing like the TSA (here is the criticizm of TDD approach but there is one recomendation here which fits the topic. I mean the recomendation not to use Cucumber)

and many others. Generally speaking the idea is that we should put off our pink sun glasses and look at the real world. Cucumber (and all it's analogs designed for other programming languages) is quite attractive solution providing the ability to write descriptive automated tests. But this attractiveness hides a myriad ways to "shoot your leg". And the more people use the approach of executable specifications the more people have bad experience. Why this happens? Key thing here is that each approach requires specific set of agreements as well as it always has some restrictions. BDD aproach itself is not an exception. So, before using it we should understand what the potential problems can be and where it is applicable at all. Most of failures related to BDD and Cucumber in particular are related to the fact that it was used in umproper way and at improper plase. At the same time there are some quite simple rules which can help to avoid essential part of the problems. Some of them are:

1. Do not use it for white box unit testing - Main reason is that BDD makes an overhead in a form of extra resources (like feature files) and the code binding text instructions to the actual executable code. Another key factor here is that white box unit testing is done by developers for developers. It means that the ability to represent readable tests isn't advantage here as developers understand the code pretty well. All the above facts are multiplied by the fact that test development shouldn't take more time than development as the value the developer brings is reflected in the code under test rether then in tests covering that code. All those things make Cucumber not applicable for developers
2. Make conventions for building test instructions - In many cases I encountered the situation when the same action was represented in many different forms with minimal differences. E.g. there were phrases like:

   When I click on Submit button
   When I click on Submit
   When I click on the Submit button
   When I click on the "Submit" button
   

   As you can see the meaning of the above strings is the same though for Cucumber they are different. Use common convention to follow in order to prevent such chaos. E.g. you can state that all control names must be quoted and preceeded with the article. Also you should identify what the control type is. In this case only last line is valid. Of course, initially it will require a lot of reviews and corrections but after some time it becomes a habit. It's actually the same as coding style. It good practice when all people write the code in the same fashion so any developer can read and understand the code of others. The same thing is here.
3. Use proper level of detail - There are tests which verify some specific work flow checking all steps in detail. But also there are tests which just use that work flow but they are intereted in further checks affected by some other functionality. In this case we don't need to specify all details. Some general action description should be enough. E.g. we have test like:

   Given I'm on home page
   When I click on the "Orders" link
   Then I should see the "Orders" page is open
   When I click on the "Create" link
   Then I should see the "Create Order" page is open
   When I enter "Test" value into the "Order Name" field
   And click on the "Submit" button
   Then I should see the "Success" message appears
   

   and we have another test which requires several orders for further processing. The second test can use the following:

   When I create the "Test" order
   And I create the "Test1" order
   

   In this example we put instructions from the first test and wrapped them with the single instruction. Actually if we do the same with the code we just make a layer of re-usable functions which represent some domain layer. So, that would be done anyway. We just have to group that functionality properly. But the main idea is that if we don't care about how we do some actions but we just need the result then we can use more high-level instructions.
4. Use parameters - key flexibility of Cucumber and all other similar engines is in ability to vary some part of instructions. Actually the code is bound to some text in feature files using regular expressions with back-linked variables. E.g. if we have instructions like:

   When I click on the "Create" button
   And I click on the "Submit" button
   

   we don't need to bind separately both instructions. We can create method for clicking any button by their name and bind it to the following regular expression:

   /I click on the "(.*)" button/
   

   In the cucumber the text in quotes will be passed as parameter. If we need to pass complex parameter (e.g. we need to fill some set of fields in some specific form) we can use tables and test instructions will look like:

   When I fill in the "Order" form with the following options:
     | Order Name | Date       | Client  | Price |
     | Test1      | 2012-02-12 | John B. | 200$  |
   

   Of course, we should implement appropriate method performing those actions.
5. Try to re-use text instructions as much as possible - BDD with it's extra layer of text instructions requires extra eforts to support them and their bindings to the actual code. In order to optimize that expences we should re-use existing text instructions. The more time each specific text instruction is in use the less effort is required to maintain the code in case we need to do some changes which should be reflected in all affected tests.
6. Use examples when you check multiple input options for the same work flow - It's more about test design and it can be the biggest pain. Actually, here is an example of such situation. Let's take an example of poorly designed test:

   Scenario 1: Account is in credit+
   Given the account is in credit
   And the card is valid
   And the dispenser contains cash
   When the customer requests cash
   Then check that the account is debited
   And check that cash is dispensed
   And check that the card is returned
   And check that nothing happens that shouldn’t happen and everything else happens that should happen for all variations of this scenario and all possible states of the ATM and all possible states of the customer’s account and all possible states of the rest of the database and all possible states of the system as a whole, and anything happening in the cloud that should not matter but might matter.
   

   The problem here is that there's necessity of tests verifying different combinations of inputs (different situation) but it was designed in plain manner where we perform basic check and then specify in high-level that we should check various combinations. Actually, when we want to check the system behavior under different input but with the same work flow we should use data-driven approach when we define common workflow, identify varying parameters and bind that work flow to the table of input data and expected results. In this example the common work flow skeleton can be represented in the following way:

   Scenario Outline: Account is in credit+
   Given the account is <Account State>
   And the card is <Card State>
   And the dispenser <Dispenser State>
   When the customer requests cash
   Then check that the account <Account Result>
   And check that cash <Cash Result>
   And check that the card <Card Result>
   

   Note that the keyword Scenario was replaced with Scenario Outline and varying parameters were replaced with some keywords between < and > characters. The only thing left here is to add the data table with inputs and expected results. So, the above test should be appended with the following lines:

   Examples:
     | Account State | Card State | Dispenser State | Account Result | Cash Result     | Card Result |
     | in credit     | valid      | contains cash   | is debited     | is dispensed    | is returned |
     | in credit     | invalid    | contains cash   | isn't debited  | isn't dispensed | is returned |
     | in credit     | expired    | contains cash   | isn't debited  | isn't dispensed | is returned |
     ....
   

   Here is example how we extend our test coverage varying Card State option. The more rows we add the more cases will be checked. The scenario outline will be executed as much as the number of rows and each run uses data from appropriate row.

This is the major list of good practices. Of course, it may be updated with many other practices but, anyway, if you follow the above rules you can seriously minimize the problems you may encounter which using BDD approach and Cucumber (or any similar engine) in your test automation.