Cucumber JVM + JUnit: Re-run failed tests

Automated tests should run reliably and provide predictable results. At the same time there are some spontaneous temporary errors which distort the entire picture while reviewing test results. And it's really annoying when tests fail on some temporary problem and mainly pass on next run. Of course, one thing is when we forgot to add some waiting time out to wait for element to appear before interacting with it. But there are cases when the reason of such temporary problem lays beyond automated tests implementation but mainly related to environment which may cause some delays of downtime for short time. So, normal reaction on that is to re-run failed tests and confirm functionality is fine. But this is too routine task and it doesn't really require some intellectual work to perform. It's simply additional logic which handles test result state and triggers repetitive run in case of error. If test fails permanently we'll still see the error but if test passes after that then the problem doesn't require too much attention.

And generally, if you simply stepped out it's not the reason to fail.

This problem is not new and it's been resolved already for many particular cases. E.g. here is the JUnit solution example. In this post I'll show you how to perform the same re-run for Cucumber-JVM in combination with JUnit as I'm actively using this combination of engines and it is quite popular. The solution shown in previous link doesn't really fit the Cucumber as each specific JUnit test in Cucumber-JVM corresponds to some specific step rather than entire scenario. Thus, the re-run functionality for this combination of engines looks a bit different. So, let's see how we can re-run our Cucumber tests in JUnit.

Cucumber JVM: Advanced Reporting

Advanced Cucumber Reporting Introduction

The Cucumber JVM contains some set of predefined reports available as the plugin option. By default we have some raw reports. Some of them are ready to be provided for end users (e.g. HTML report) while others still require some post-processing like JSON reports (both for usage and results reports). Also, available standard HTML report isn't really good enough. It's good for analysis (partially) but if we want to provide some kind of overview information we don't need so much details as well as we may need some more visualized statistics rather than just one simple table.

Well, there are some already existing solutions for such advanced reporting, e.g. Cucumber Reports by masterthought.net. That's really nice results reporting solution for Cucumber JVM. But when I tried to apply this solution I encountered several problems with Maven dependencies resolution and report processing. That became a reason for me to look at something custom as I couldn't smoothly integrate this reporting to my testing solution. Additionally, it covers only results reporting while I'm also interested in steps usage statistic information to make sure that we use our Cucumber steps effectively. And apparently I didn't find reporting solution covering this area in appropriate way for Cucumber JVM.

Being honest, I'm not really big fan of writing custom reporting solution at all as for test automation engineers existing information is more than enough in most of the cases. But if you need to provide something more specific and preferably in e-mailable form to send to other members of project team we need something else. That's why I created some components which generate some Cucumber reports like results and usage reports (see sample screen shots below).

The above samples show e-mailable reports which mainly provide results overview information which can be sent via e-mail as well as additional HTML report summarizing usage statistics. In this post I'll describe my reporting solution with some examples and some detailed explanations.

GitHub: test tracking via Maven plugin

Recently we've organized integration with GitHub by retrieving issues content into the files. All this was done in a form of command line utility. In this post I'll extend the functionality to use this solution as Maven plugin.

Sirius C#: adding UI Automation library

Recently we've created code interacting with Win32 and Web. Now it's time to expand the coverage to the .NET area. For this purpose we have dedicated library called UI Automation. This library is provided with .NET framework and contains basic API interacting with window objects. At the same time this library can be used for interaction with the standard Win32 controls, however it can be used as an auxiliary modules to expand the coverage of existing library I created before for interacting with Win32 elements. In this article I will create sample control classes and make some demo showing how it works. Also, in this example I'll take the tab control and create simple tests interacting with it. At this point that would be just stand-alone library but in the future it will be integrated into entire Sirius project.

Sirius: adding Web client API

We've done some work for Web core functionality. To make the work complete the client API should be added in the same fashion as for Win32 library. At the end of current development there should be an example how to interact with web using Web client.

The following topics will be covered here:

• General Web client architecture
• Web client proxy updates
• Web client class library implementation

Sirius: adding Web core functionality

A lot of work were done for Win32 stuff starting from initial core functionality preparation and up to basic operations implementations as well as handling parent/child windows and user input. So, now we can work with Win32 (not in full extand but at least for base operations). One of the further expansion direction is the coverage of the Web functionality. That's the completely new area for Sirius Test Automation Platform and in this post I'll describe the core part creation which mainly affect server side.

Sirius: Win32 - add Dialog handling and basic input

Recently I've added functionality working with main windows. Next step is to apply functionality to interact with dialog boxes and child controls. It's almost the same except these are child windows and we should carry out additional relationship to main window. Additionally, we should add basic ability to send mouse and keyboard input. Thus, we'll complete basic functionality for Win32. All other code to be written should cover some control specifics rather than common functionality.

Sirius: modularize the solution

In the previous post I've added another module to support Win32 operations. Actually, there should be a lot of modules to be added. But I put it directly into the Server module. The more components we cover the more modules I have to include. Thus the server code as well as clients become heavy-weight and entire solution becomes huge. But firstly, it breaks one of the ideas of the Sirius platform. It should have an extensible engine which can plug in the modules on fly. Also, it should be adopted to different operating systems (that's why I chose Java as the main language) but what should the Win32 module do, for instance, on Unix? Nothing. So, in such case I don't need to include it. Or even more, I want to make different complectations for Server side depending on what I actually want to use. For this purpose I have to split our components into functional modules and provide the ability to configure what modules I should include. Also, I would be useful to provide the ability to deliver either entire solution or some parts of it. In this post I'll describe the steps for doing this.

Sirius: core Win32 functionality support

Once we've done with infrastructure it's time to work on development itself. Now I'll start working on the functionality performing interaction with Win32 objects. In this article I'll describe basic preparations for Win32 interactions as well as I'll describe the core functionality to capture the window on screen and perform some manipulations with it. And of course this activity is done in the scope of Sirius development. So, it would be bound to the entire architecture of the platform.

Sirius: Testing setup

Since Sirius is going to be used for testing purposes it's not polite to leave it without testing. Major specifics of this solution is that different parts of entire system are written using different programming languages. At the same time all those parts reflect the same functionality. It means that the tests would be the same for all clients. Well, they would be the same for server part as well (at least most of the core tests). It's not very nice to copy/paste them and then migrate to different languages. So, obviously there's going to be a lot of duplicating work. So, I have to do something to minimize that duplication as well as minimize the maintenance effort in case I want to change some tests. It's valid especially for client side as core part doesn't have any local specific logic. All immediate client methods are actually callers for the same server side methods.

So, there's going to be multiple languages but the same interface for tests. It's not trivial solution but I found that.

Sirius: solution improvements

Overview

The more code is prepared the more items require various corrections and enhancements. Also, some things weren't done initially but they were planned. So, in this article I'll describe various changes applied to the entire project. Here I'll cover the following topics:

1. Adding functionality to stop service endpoints
2. Server side updates for more informative generated code
3. Generate Ruby client from Java server code
4. Migrating solution from Ant to Maven

Sirius: First steps

In the previous article we've created basic skeleton for entire solution. Now it's time to add the content. In this article I'll take sample method, write it's server part and create clients for all programming languages we already prepared infrastructure for. The example would be simple but it's enough to find out how it's better to design entire solution and what is the base flow for adding new functionality. Also it would be clearly seem how it is better to use such solution. So, in this article I'll describe the following:

1. Sample server side method creation (actually that would be the sample of SOAP web-service creation on Java)
2. Java client creation
3. Ruby client creation
4. C# client creation

OK. We're good to go.

Sirius: Dev environment setup

Introduction

All right, I've decided to start development but it's not enough just to write the code. I also should be able to build the packages, test them somehow, provide some documentation and do many other activities. But firstly, I don't want to make a lot of routine work by myself. Let the machine do that for me. Also, I want to be sure that the way the system works for me locally is the same as for any other machine. And eventually, I don't want to keep the source code only locally. It's unsafe as my local machine can be damaged and all information would be lost in this case. So, we need some additional software which makes the development infrastructure and which should be configured in the way convenient to use.

This article is about to describe the infrastructure setup as well as how to configure all this software to point to each other.

Sirius: ice is broken

Pre-phrase

This is the new series of articles dedicated to the Sirius - the new test automation platform development. This is the work I've been doing for several years accumulating different pieces of code with different languages, for different technologies, for different testing types.

The goal of my work is to combine existing solutions under common architecture and represent some unified and extensible design which can be a basis for another brand-new test automation solution or even common platform for delivering test automation types. This is the not the first time I am getting back to this but the more results I get the more different solutions I can integrate under the same framework. I don't expect it to be another widely used platform (I am not even talking about replacement of any vendor tool) but that can be a good prototype. I doubt there were a lot of people who believed in Selenium eight years ago when only some draft version appeared. But take a look at the situation on test automation market now. Selenium became one of the most popular test automation tools and the popularity is growing. Even QTP loses its positions and now it's quite easy to find test automation specialists who are familiar with some popular programming language like Java, C#, Ruby, Python, PHP and many others which were traditionally used for development.