Video Course: Data-Driven Testing in Selenium

This is the Data-driven testing in Selenium video course created by me.

Video Course: Automated UI Testing in Android

This is the Automated UI Testing in Android video course created by me. It is the continuation of series of courses dedicated to step-by-step UI automation framework development. Just click on the image below for more details.

Video Course: Automated UI Testing in Java

OK. This is the Automated UI Testing in Java video course created by me. Just click on the image below for more details.

Cucumber JVM: Advanced Reporting 2. Detailed Report (HTML, PDF)

Introduction

Previously I've described basic report samples based on Cucumber-JVM. Since I'm using this reporting solution on practice quite frequently the number of requirements and enhancements grows so since last time I had to add some more additional features which can be in use.

Some of new features are:

• Cucumber extension to support failed tests re-run
• Detailed results report which supports the following:
  • Detailed results report generation in HTML format
  • Detailed results report generation in PDF format
  • Screen shot are included into both the above reports

All those new features are included into 0.0.5 version, so we can add Maven dependency:

<dependency>
 <groupId>com.github.mkolisnyk</groupId>
 <artifactId>cucumber-reports</artifactId>
 <version>0.0.5</version>
</dependency>

or the same thing for Gradle:

'com.github.mkolisnyk:cucumber-reports:0.0.5'

Since Cucumber failed tests re-run functionality was described before in this post I'll concentrate more on detailed results report features.

Cucumber JVM + JUnit: Re-run failed tests

Automated tests should run reliably and provide predictable results. At the same time there are some spontaneous temporary errors which distort the entire picture while reviewing test results. And it's really annoying when tests fail on some temporary problem and mainly pass on next run. Of course, one thing is when we forgot to add some waiting time out to wait for element to appear before interacting with it. But there are cases when the reason of such temporary problem lays beyond automated tests implementation but mainly related to environment which may cause some delays of downtime for short time. So, normal reaction on that is to re-run failed tests and confirm functionality is fine. But this is too routine task and it doesn't really require some intellectual work to perform. It's simply additional logic which handles test result state and triggers repetitive run in case of error. If test fails permanently we'll still see the error but if test passes after that then the problem doesn't require too much attention.

And generally, if you simply stepped out it's not the reason to fail.

This problem is not new and it's been resolved already for many particular cases. E.g. here is the JUnit solution example. In this post I'll show you how to perform the same re-run for Cucumber-JVM in combination with JUnit as I'm actively using this combination of engines and it is quite popular. The solution shown in previous link doesn't really fit the Cucumber as each specific JUnit test in Cucumber-JVM corresponds to some specific step rather than entire scenario. Thus, the re-run functionality for this combination of engines looks a bit different. So, let's see how we can re-run our Cucumber tests in JUnit.

Ploblem Solved: Cucumber-JVM running actions before and after tests execution with JUnit

Background

It is frequent case when we need to do some actions before and/or after entire test suite execution. Mainly, such actions are needed for global initialization/cleanup or some additional reporting or any other kind of pre/post-processing. There may be many different reasons for that and some test engines provide such ability, e.g. TestNG has BeforeSuite and AfterSuite annotations, the JUnit has test fixtures which may run before/after test class (it's not really the same but when we use Cucumber-JVM it's very close to what we need).

Problem

The problem appears when you want to add some actions at the very latest or very earlies stages of tests execution and you use Cucumber-JVM with JUnit. In my case I wanted to add some reports post-processing to make an advanced Cucumber report generation. In this case JUnit fixtures didn't help as AfterClass-annotated method runs before Cucumber generates final reports.

At the same time adding @BeforeAll and @AfterAll hooks question raised on Cucumber side as well. And there was even some solution proposed. Unfortunately, authors decided to revert those changes as there were some cases when it does not work.

So, the problem is that I need something to run after entire Cucumber-JVM suite is done but neither Cucumber nor JUnit gives me built-in capability for doing this.

Solution

Cucumber JVM: Advanced Reporting

Advanced Cucumber Reporting Introduction

The Cucumber JVM contains some set of predefined reports available as the plugin option. By default we have some raw reports. Some of them are ready to be provided for end users (e.g. HTML report) while others still require some post-processing like JSON reports (both for usage and results reports). Also, available standard HTML report isn't really good enough. It's good for analysis (partially) but if we want to provide some kind of overview information we don't need so much details as well as we may need some more visualized statistics rather than just one simple table.

Well, there are some already existing solutions for such advanced reporting, e.g. Cucumber Reports by masterthought.net. That's really nice results reporting solution for Cucumber JVM. But when I tried to apply this solution I encountered several problems with Maven dependencies resolution and report processing. That became a reason for me to look at something custom as I couldn't smoothly integrate this reporting to my testing solution. Additionally, it covers only results reporting while I'm also interested in steps usage statistic information to make sure that we use our Cucumber steps effectively. And apparently I didn't find reporting solution covering this area in appropriate way for Cucumber JVM.

Being honest, I'm not really big fan of writing custom reporting solution at all as for test automation engineers existing information is more than enough in most of the cases. But if you need to provide something more specific and preferably in e-mailable form to send to other members of project team we need something else. That's why I created some components which generate some Cucumber reports like results and usage reports (see sample screen shots below).

The above samples show e-mailable reports which mainly provide results overview information which can be sent via e-mail as well as additional HTML report summarizing usage statistics. In this post I'll describe my reporting solution with some examples and some detailed explanations.

Aerial: Create New Project Using Archetype

Since 0.0.5 version Aerial has an ability to generate Java projects from archetypes to get sample working project quickly. In this post I'll describe the steps for doing this.

Aerial: Introduction to Test Scenarios Generation Based on Requirements

In one of the previous articles dedicated to moving to executable requirements I've described some high-level ideas about how such requirements should look like. Main idea is that document containing requirements/specifications can be the source for generating test cases. But in the previous article we just made brief overview on how it should work in theory. Now it's time to turn theory into practice. In this article I'll make brief introduction to new engine I've started working with. So, let me introduce the Aerial the magic box which makes our requirements executable.

In this article I'll describe main principles of this engine and provide some base examples to bring an idea of what it is and where it is used.

GitHub: integrating project into entire GitHub infrastructure

GitHub provides quite good set of toolsets which can be used to support full development cycle and prepare infrastructure to all necessary output artifacts. Recently, I've described samples how to provide integration with GitHub issue tracker and to wrap it with Maven plugin. So, actually, I have dedicated Maven plugin project which I can use as a separate project. In order to make it as much self-contained as it's needed we should:

1. Set up new project in the GitHub which includes:
   1. New repository setup
   2. Build script preparation
   3. Release settings
2. Prepare project documentation generation
3. Integration with CI

So, at the end, all we have to do is to work with our IDE on adding new code or making some changes to existing modules. Everything else is supported by GitHub infrastructure as soon as we push our changes into remote repository.

Not bad at the beginning. So, let's get started.

GitHub: test tracking via Maven plugin

Recently we've organized integration with GitHub by retrieving issues content into the files. All this was done in a form of command line utility. In this post I'll extend the functionality to use this solution as Maven plugin.

GitHub: organizing and automating test tracking with Java

GitHub provides some tracker where we can keep our tasks. The GitHub API provides the ability to access that resources programmatically. Thus, with the help of API I could integrate JBehave and GitHub so that I can run JBehave tests stored in GitHub tracker. But it's not the only case.

With the help of API I can set the correspondence between tests and features. This gives the ability to make the traceability of tests and features they're applied for. Also, what if we use something different from JBehave, e.g. Cucumber, Sprecflow which use flat files for input? In this article I'll describe how to:

• Organize tests with GitHub tracker
• Automatically generate traceability matrix
• Automatically generate feature files based on the content of the test issues

Java: JBehave integration with GitHub

During BDD engines comparison and comparing with Cucumber I've found out really serious advantage of JBehave in comparison to other similar engines. It's the ability to read data from external sources different from text files in specific locations as well as the ability to customize that. And I made some sample JBehave integration with JIRA. I did that because I had a plans for Sirius platform to store all tests in JIRA and read them from that location. Well, that link is still useful for projects which use JIRA as tracking system.

But for my project it still doesn't work as I need a shared JIRA instance so I can read tests from there. I don't have it yet... and then I found I don't need that, actually. Since I store Sirius project on GitHub it would be reasonable to share test related resources on GitHub as well. But it's not just about sharing but reading. Actually, GitHub contains it's own tracker. So, in this article I'll describe how to read JBehave scenarios stored as the tickets in the GitHub issue tracker.

Sirius: running as windows service

Sirius server part is delivered as a set of Java archives and this part is initiated via command line interface. It brings some inconveniences while running the tests. Thus, before start using client API we should make sure that the server is up and running. Of course, we can do that manually but:

1. we should be able to do that automatically
2. when we test client side we shouldn't touch server side at all

So, it's quite convenient to deliver Sirius Server start/stop process to the system. For Windows it can be done by setting server component as the windows service. Since, Sirius Server is just another Java application it requires some additional efforts for making such task however such task appeared multiple times. Here are the related topics on StackOverflow:

• Install java program as a windows service: Alternative to JavaService?
• How to create a windows service from java app

So, I'm looking for something similar for Sirius Server. In this post I'll describe the settings needed for making Sirius Java application running as the windows service.

Sirius: Adding code style checks

We already set up the build process for Sirius as well as we reserved the placeholders for testing. So, further improvement in this area is to extend build with different tests and checkpoints which make more detail verification. This time we'll add more modules which perform static testing for our components and they should fail the build if static checks didn't pass (especially taking into account that it was initially planned).

What would we look for? Basically that's:

• Conformity to coding standards - all code standards are made to provide unified approach of writing code as well as reading and maintaining it.
• Various cases of improper code usage - a lot of actual application errors are results of improper code use or use of potentially dangerous code constructions. So, the earlier we find them the less number of silly mistakes we take during further testing stages

Actually, we've made first approximation for that while we checked the compilation ability. If system isn't compilable it's not working at all. So, we're just making more detailed checks.

Since Sirius modules use different programming languages the tool set will be varying based on that. We'll use the following tool set for that:

Language	Engine
Java	Check Style PMD
C#	StyleCop
Ruby	rubocop

OK. Let's see how it would be added.

JBehave vs Cucumber JVM comparison

The interest in BDD approach is growing and the more and more different engines appear in that space. However, there're some well known players in that area. We can compare their features however the final choice is still based on the technology used. E.g. if we compare Behat and SpecFlow we definitely can find some gaps and advantages but if we're going to use e.g. C# the Behat is not a subject of interest here. And vice versa.

The more interesting case is when 2 or more engine appear in the same technological area. And it's twice more interesting when these are one of the most popular engines on one of the most popular technology. In this post we'll compare JBehave vs Cucumber against their applicability for Java test automation. We compare them using the same scale as for BDD Engines Comparison post with some additional detalization on features. Additionally, I'd like to mention that we should compare apples to apples so there's no need to dig the Ruby implementation of Cucumber. It's interesting to check the Cucumber Java capabilities against JBehave to make comparison more or less valuable.

All right, we have JBehave, we have Cucumber, so ...

LET THE MORTAL COMBAT COMPARISON BEGIN!!!

Sirius: adding Web client API

We've done some work for Web core functionality. To make the work complete the client API should be added in the same fashion as for Win32 library. At the end of current development there should be an example how to interact with web using Web client.

The following topics will be covered here:

• General Web client architecture
• Web client proxy updates
• Web client class library implementation

Sirius: adding Web core functionality

A lot of work were done for Win32 stuff starting from initial core functionality preparation and up to basic operations implementations as well as handling parent/child windows and user input. So, now we can work with Win32 (not in full extand but at least for base operations). One of the further expansion direction is the coverage of the Web functionality. That's the completely new area for Sirius Test Automation Platform and in this post I'll describe the core part creation which mainly affect server side.

Sirius: Win32 - add Dialog handling and basic input

Recently I've added functionality working with main windows. Next step is to apply functionality to interact with dialog boxes and child controls. It's almost the same except these are child windows and we should carry out additional relationship to main window. Additionally, we should add basic ability to send mouse and keyboard input. Thus, we'll complete basic functionality for Win32. All other code to be written should cover some control specifics rather than common functionality.

Java: JBehave integration with JIRA

During BDD engines comparison I noticed that JBehave supports various data sources for the stories but not just text files. This is quite valuable feature as we can share the stories accross the teams. But what if we share that using existing infrastructure? I mean all the tests are typically stored under some tracking system where they can be easily queried from. Such tracking systems should be the place where the tests are written to and updated at. So, why shouldn't we integrate that with the engine so that tests are designed, stored and modified in the tracking system and the engine like JBehave picks them up from there? That would be the tight integration between tests and their automated implementation. In this post I'll show how to make such integration between JIRA and JBehave.