Web test tools list

Introduction

In this post I'll collect the information about existing web test automation tools (both from vendors and open source). It is represented as the catalogue with basic information rather that some comparison characteristics.

NOTE

The content of this post will be appended/updated from time to time to reflect the most recent changes. Also, some information may be not very precise as it is not so widely accessible. So, any comments related to information corrections are welcome

Sirius: adding Web core functionality

A lot of work were done for Win32 stuff starting from initial core functionality preparation and up to basic operations implementations as well as handling parent/child windows and user input. So, now we can work with Win32 (not in full extand but at least for base operations). One of the further expansion direction is the coverage of the Web functionality. That's the completely new area for Sirius Test Automation Platform and in this post I'll describe the core part creation which mainly affect server side.

Ruby: soap4r - fixing wsdl2ruby for 1.9.3 version

Some time before when I was trying to generate SOAP Ruby client for my project I encountered the problem with soap4r library. The key problem was with Ruby 1.8 and 1.9 versions compatibility. I've found several independent fixes for that:

• soap4r-ruby1.9
• soap4r-ruby19
• rubyjedi-soap4r

and many others. But there were problems with getting them work properly. So, I had to dig into the installed gem code to find out what causes the errors. In particular, I'm getting error like:

[2011-06-09T17:16:01.012268 #10104] INFO -- app: Creating class definition. I, 
[2011-06-09T17:16:01.012268 #10104] INFO -- app: Creates file 'SessionServic e.rb'. F, 
[2011-06-09T17:16:01.014268 #10104] FATAL -- app: Detected an exception. Stop ping ... undefined method collect' for # (NoMethodError) C:/Ruby192/lib/ruby/gems/1.9.1/gems/soap4r-1.5.8/lib/xsd/codegen/gensupport.rb:2 39:intrim_eol' 
C:/Ruby192/lib/ruby/gems/1.9.1/gems/soap4r-1.5.8/lib/xsd/codegen/gensupport.rb:2 27:in format' 
C:/Ruby192/lib/ruby/gems/1.9.1/gems/soap4r-1.5.8/lib/xsd/codegen/commentdef.rb:2 7:indump_comment' 
C:/Ruby192/lib/ruby/gems/1.9.1/gems/soap4r-1.5.8/lib/xsd/codegen/classdef.rb:51: in dump' 
C:/Ruby192/lib/ruby/gems/1.9.1/gems/soap4r-1.5.8/lib/wsdl/soap/classDefCreator.r b:118:inblock in dump_complextype' 
C:/Ruby192/lib/ruby/gems/1.9.1/gems/soap4r-1.5.8/lib/wsdl/soap/classDefCreator.r b:116
...

Actually I was interested in wsdl2ruby utility which was needed for SOAP client generation. So, what should be done to make it working?

Sirius: Win32 - add Dialog handling and basic input

Recently I've added functionality working with main windows. Next step is to apply functionality to interact with dialog boxes and child controls. It's almost the same except these are child windows and we should carry out additional relationship to main window. Additionally, we should add basic ability to send mouse and keyboard input. Thus, we'll complete basic functionality for Win32. All other code to be written should cover some control specifics rather than common functionality.

Java: JBehave integration with JIRA

During BDD engines comparison I noticed that JBehave supports various data sources for the stories but not just text files. This is quite valuable feature as we can share the stories accross the teams. But what if we share that using existing infrastructure? I mean all the tests are typically stored under some tracking system where they can be easily queried from. Such tracking systems should be the place where the tests are written to and updated at. So, why shouldn't we integrate that with the engine so that tests are designed, stored and modified in the tracking system and the engine like JBehave picks them up from there? That would be the tight integration between tests and their automated implementation. In this post I'll show how to make such integration between JIRA and JBehave.

Sirius: Adding new component Demo

Here is the demo showing how to add new component to the Sirius platform. The example demonstrates how to create server module from the scratch and how to apply client for this.

Sirius (Java): writing Win32 client library + Live Demo

Recently I described the creation of Win32 server part and at the end I wrote the function which finds the main window by specified attributes and gets the handle for it. This is quite fundamental thing as once we have a handle of the window we can send any messages to it simulating various GUI interaction actions. At the same time the server side library contains only set of functions as Win32 API is all about the set of functions. But it's too raw for writing tests. It's much more convenient to apply some object model to the window objects. So, it requires some effort on the client side. In this article I'll describe the process of Win32 client API creation. I'll create classes interacting with top level windows and create some sample test to demonstrate how that functionality works.

The final output will contain the functionality interacting with top level Win32 windows as well as some demo showing how to work with it.

Java JNA Win32: when default libraries are not enough

I've been working on implementing Win32 interaction functionality using JNA library and encountered the situation when the library became incomplete. The methods of com.sun.jna.platform.win32.User32 interface which is supplied by the platform library contained only subset of the entire functions of user32.dll library. Actually, I've found that while trying to call GetMenu function. It appeared that there's no such function in the Java class supplied with the JNA library. What to do with that? How to make a Java call for functions from existing DLL?

Sirius: modularize the solution

In the previous post I've added another module to support Win32 operations. Actually, there should be a lot of modules to be added. But I put it directly into the Server module. The more components we cover the more modules I have to include. Thus the server code as well as clients become heavy-weight and entire solution becomes huge. But firstly, it breaks one of the ideas of the Sirius platform. It should have an extensible engine which can plug in the modules on fly. Also, it should be adopted to different operating systems (that's why I chose Java as the main language) but what should the Win32 module do, for instance, on Unix? Nothing. So, in such case I don't need to include it. Or even more, I want to make different complectations for Server side depending on what I actually want to use. For this purpose I have to split our components into functional modules and provide the ability to configure what modules I should include. Also, I would be useful to provide the ability to deliver either entire solution or some parts of it. In this post I'll describe the steps for doing this.

Sirius: core Win32 functionality support

Once we've done with infrastructure it's time to work on development itself. Now I'll start working on the functionality performing interaction with Win32 objects. In this article I'll describe basic preparations for Win32 interactions as well as I'll describe the core functionality to capture the window on screen and perform some manipulations with it. And of course this activity is done in the scope of Sirius development. So, it would be bound to the entire architecture of the platform.

Sirius: Testing setup

Since Sirius is going to be used for testing purposes it's not polite to leave it without testing. Major specifics of this solution is that different parts of entire system are written using different programming languages. At the same time all those parts reflect the same functionality. It means that the tests would be the same for all clients. Well, they would be the same for server part as well (at least most of the core tests). It's not very nice to copy/paste them and then migrate to different languages. So, obviously there's going to be a lot of duplicating work. So, I have to do something to minimize that duplication as well as minimize the maintenance effort in case I want to change some tests. It's valid especially for client side as core part doesn't have any local specific logic. All immediate client methods are actually callers for the same server side methods.

So, there's going to be multiple languages but the same interface for tests. It's not trivial solution but I found that.

BDD instructions writing standards

Introduction

This document describes rules for writing BDD instructions. It is used as a guideline for proper stated and formatted instructions. Natural language instructions used for BDD are not absolutely natural language instructions. They're actually "naturalized" form of some specific command with ability to variate some part. So, from the technical side of BDD the strict rules are still necessary to avoid different forms of absolutely the same phrases but with small differences. It's not essential for understanding but it is critical for technical implementation.

This document contains examples of correct and incorrect phrases. If some phrase is described as incorrect it only means that it violates the standards though it can be properly built from the language point of view. These standards do not teach English. They just describe which subset of possible phrases is taken as acceptable.

And again, standards are designed for uniformity only. There's no strict dogmatum in them. If some phrases seem to be more appropriate but they violate the rules it's OK to adjust rules to current needs rather than restrict needs with existing rules. The main thing is that there should be some uniform rule set which is kept within the project. Current document can be a basis for such standards

Sirius: solution improvements

Overview

The more code is prepared the more items require various corrections and enhancements. Also, some things weren't done initially but they were planned. So, in this article I'll describe various changes applied to the entire project. Here I'll cover the following topics:

1. Adding functionality to stop service endpoints
2. Server side updates for more informative generated code
3. Generate Ruby client from Java server code
4. Migrating solution from Ant to Maven

Sirius: First steps

In the previous article we've created basic skeleton for entire solution. Now it's time to add the content. In this article I'll take sample method, write it's server part and create clients for all programming languages we already prepared infrastructure for. The example would be simple but it's enough to find out how it's better to design entire solution and what is the base flow for adding new functionality. Also it would be clearly seem how it is better to use such solution. So, in this article I'll describe the following:

1. Sample server side method creation (actually that would be the sample of SOAP web-service creation on Java)
2. Java client creation
3. Ruby client creation
4. C# client creation

OK. We're good to go.

Sirius: Dev environment setup

Introduction

All right, I've decided to start development but it's not enough just to write the code. I also should be able to build the packages, test them somehow, provide some documentation and do many other activities. But firstly, I don't want to make a lot of routine work by myself. Let the machine do that for me. Also, I want to be sure that the way the system works for me locally is the same as for any other machine. And eventually, I don't want to keep the source code only locally. It's unsafe as my local machine can be damaged and all information would be lost in this case. So, we need some additional software which makes the development infrastructure and which should be configured in the way convenient to use.

This article is about to describe the infrastructure setup as well as how to configure all this software to point to each other.

Sirius: ice is broken

Pre-phrase

This is the new series of articles dedicated to the Sirius - the new test automation platform development. This is the work I've been doing for several years accumulating different pieces of code with different languages, for different technologies, for different testing types.

The goal of my work is to combine existing solutions under common architecture and represent some unified and extensible design which can be a basis for another brand-new test automation solution or even common platform for delivering test automation types. This is the not the first time I am getting back to this but the more results I get the more different solutions I can integrate under the same framework. I don't expect it to be another widely used platform (I am not even talking about replacement of any vendor tool) but that can be a good prototype. I doubt there were a lot of people who believed in Selenium eight years ago when only some draft version appeared. But take a look at the situation on test automation market now. Selenium became one of the most popular test automation tools and the popularity is growing. Even QTP loses its positions and now it's quite easy to find test automation specialists who are familiar with some popular programming language like Java, C#, Ruby, Python, PHP and many others which were traditionally used for development.

Test Automation ROI calculation Part 1 : Simple iterative development model

Introduction

The test automation isn't done just for fun. It has the purpose. Some of the main purposes are:

• Increase testing team capacity
• Minimize testing cycle without changing testing team capacity
• Increase testing coverage without extra resources
• Minimize human factor

But all above and many other benefits can be expressed with the single phase. Test automation should bring the profit. In other words, I have many ways to improve the testing efforts and each of them require some investment. Either I can hire more people or I can invest into some software/hardware which helps me making things faster. But if the investment into some improvement doesn't bring the profit this investment is useless. This is related to test automation as well. Yes, you can buy some expensive tools, you can build the complex infrastructure but if I can do the same in some other way which requires less investment I'll prefer the second option. So, before proposing some test automation efforts we should realize what the advantage we should make with this. One of the wide spread measure is the Return of investment. So, how should we calculate that for test automation project? Let's build some process model to see what we can calculate there.

Test Automation tools: trends

While making decision which tool is more suitable for test automation we should always take into account the popularity of each specific tool. Why? Because firstly it's about people. What's the point in taking tool which doesn't have users? Thus, we'll have to invest some money into their learning which is extra expences. Secondly, the more popular tool is the more materials are available either in internet or in some other materials. So, we should take a look at some trends to identify which tool is growing with it's popularity and which one is going to be a history.

BDD engines comparison (Cucumber, Freshen, JBehave,NBehave,SpecFlow, Behat)

Introduction

Cucumber is not the only engine supporting natural language instructions. It's just one implementation of natural language instructions interpreter. The actual language to write tests with is called Gherkin. And it has different implementations adopted to different programming language. Thus we have:

• Cucumber for Ruby
• JBehave for Java
• NBehave and SpecFlow for C#
• Freshen for Python
• Behat for PHP

This list isn't complete as there can be many other similar engines which are simply less popular. All of them have some common set of supported features but there're some restrictions and abilities specific to the actual engine. So, the aim of this post is to collect useful features for each listed above engine and present it in some comparable form. Key features to be mentioned are:

Cucumber: How to avoid major mistakes

Recently I came across several posts criticizing Cucumber and approach itself. Some of the links:

• BEHAVIOR-DRIVEN DEVELOPMENT VS. TESTING by James Bach
• Testing like the TSA (here is the criticizm of TDD approach but there is one recomendation here which fits the topic. I mean the recomendation not to use Cucumber)

and many others. Generally speaking the idea is that we should put off our pink sun glasses and look at the real world. Cucumber (and all it's analogs designed for other programming languages) is quite attractive solution providing the ability to write descriptive automated tests. But this attractiveness hides a myriad ways to "shoot your leg". And the more people use the approach of executable specifications the more people have bad experience. Why this happens?