WebDriving Test Automation

Introduction

WebDriver shows growing popularity for many years and at least for past 4 years it is number 1 player on the market of web test automation solutions. The growth hasn't been stopped yet. The WebDriver exposes open interface and it's server side API is documented in W3C Standard. Thus, a lot of people can implement their own back-end API and expand technology support. So, WebDriver can become more than another option among web test automation tools list. It may become (if not yet) the most wide spread test automation platform so that the term of UI test automation may be replaced with another term like Web-Driving. In this post I'll describe where the WebDriver solution "web-drives" us to and where are the areas of further growth.

NBehave vs SpecFlow Comparison

It's always good when you use some technology and you have a choice between various tools/engines. In some cases it makes a problem like it happens sometimes with BDD Engines especially when we have to choose between similar engines which are widely used and at first glance they seem to be identical. Some time ago I've made JBehave vs Cucumber-JVM comparison to spot some differences and comparative characteristics of the most evolved engines in Java world. And as I can see from the pge views statistics it's quite interesting topic. At the same there is .NET technology which has another set of BDD engines. And they are quite popular as well. So, in this world we may encounter question like: What is better, NBehave or SpecFlow ?

This answer isn't so trivial. When I did cross-platform BDD Engines comparison almost 3 years ago some of the engines weren't well enough or at least their documentation was on low level. At that time NBehave didn't seem to look well. But since that time a lot of things changed and now both NBehave and SpecFlow are turned into full-featured Gherkin interpreter engines. So, the choice of better tool among then isn't so trivial anymore. It means we'll start new comparison between NBehave or SpecFlow

So, let's find out who's winner in this battle!!!

Measure Automated Tests Quality

Introduction

There's one logical recursion I encounter with test automation. Test automation is about developing software targeted to test some software. So, the output of test automation is another software. This is one of the reason for treating the test automation as the development process (which is one of the best practices for test automation). But how are we going to make sure that the software we create for testing is good enough? Indeed, when we develop the software we use testing (and test automation) as one of the tools for checking and measure the quality of the software under test.

So, what about software we create for test automation?

On the other hand we use testing to make sure that the software under test is of acceptable quality. In case of test automation we use another software for this. And in some cases this software becomes complicated as well. So, how can we rely on non-tested software for making any conclusions about the target product we develop? Of course, we can make test automation simple but it's not the common solution. So, we should find some compromise where we use reliable software to check the target software (the system under test). Also, we should find the way to find out how deep testing can be and how we can measure that.

So, main questions which appear here are:

• How can we identify that the automated tests we have are enough to measure quality of end product?
• How can we identify that our tests are really good?
• How can we keep quality control on our automated tests?
• How can we identify if our tests are of acceptable complexity?

In this article I'll try to find out answers to many of those questions.

Sirius: 1 year retrospective

1 year ago I've started working on the Sirius project. 6 month ago there was the first retrospective where I summarized all achievements and future goals. Generally last 6 years were not the most productive. Nevertheless, there were some achievements even there. So, in this topic I'll summarize again where we are with the Sirius now.

JBehave vs Cucumber JVM comparison

The interest in BDD approach is growing and the more and more different engines appear in that space. However, there're some well known players in that area. We can compare their features however the final choice is still based on the technology used. E.g. if we compare Behat and SpecFlow we definitely can find some gaps and advantages but if we're going to use e.g. C# the Behat is not a subject of interest here. And vice versa.

The more interesting case is when 2 or more engine appear in the same technological area. And it's twice more interesting when these are one of the most popular engines on one of the most popular technology. In this post we'll compare JBehave vs Cucumber against their applicability for Java test automation. We compare them using the same scale as for BDD Engines Comparison post with some additional detalization on features. Additionally, I'd like to mention that we should compare apples to apples so there's no need to dig the Ruby implementation of Cucumber. It's interesting to check the Cucumber Java capabilities against JBehave to make comparison more or less valuable.

All right, we have JBehave, we have Cucumber, so ...

LET THE MORTAL COMBAT COMPARISON BEGIN!!!

Test Automation ROI calculation Part 1 : Simple iterative development model

Introduction

The test automation isn't done just for fun. It has the purpose. Some of the main purposes are:

• Increase testing team capacity
• Minimize testing cycle without changing testing team capacity
• Increase testing coverage without extra resources
• Minimize human factor

But all above and many other benefits can be expressed with the single phase. Test automation should bring the profit. In other words, I have many ways to improve the testing efforts and each of them require some investment. Either I can hire more people or I can invest into some software/hardware which helps me making things faster. But if the investment into some improvement doesn't bring the profit this investment is useless. This is related to test automation as well. Yes, you can buy some expensive tools, you can build the complex infrastructure but if I can do the same in some other way which requires less investment I'll prefer the second option. So, before proposing some test automation efforts we should realize what the advantage we should make with this. One of the wide spread measure is the Return of investment. So, how should we calculate that for test automation project? Let's build some process model to see what we can calculate there.