Sirius: core Win32 functionality support

Once we've done with infrastructure it's time to work on development itself. Now I'll start working on the functionality performing interaction with Win32 objects. In this article I'll describe basic preparations for Win32 interactions as well as I'll describe the core functionality to capture the window on screen and perform some manipulations with it. And of course this activity is done in the scope of Sirius development. So, it would be bound to the entire architecture of the platform.

Sirius: Testing setup

Since Sirius is going to be used for testing purposes it's not polite to leave it without testing. Major specifics of this solution is that different parts of entire system are written using different programming languages. At the same time all those parts reflect the same functionality. It means that the tests would be the same for all clients. Well, they would be the same for server part as well (at least most of the core tests). It's not very nice to copy/paste them and then migrate to different languages. So, obviously there's going to be a lot of duplicating work. So, I have to do something to minimize that duplication as well as minimize the maintenance effort in case I want to change some tests. It's valid especially for client side as core part doesn't have any local specific logic. All immediate client methods are actually callers for the same server side methods.

So, there's going to be multiple languages but the same interface for tests. It's not trivial solution but I found that.

BDD instructions writing standards

Introduction

This document describes rules for writing BDD instructions. It is used as a guideline for proper stated and formatted instructions. Natural language instructions used for BDD are not absolutely natural language instructions. They're actually "naturalized" form of some specific command with ability to variate some part. So, from the technical side of BDD the strict rules are still necessary to avoid different forms of absolutely the same phrases but with small differences. It's not essential for understanding but it is critical for technical implementation.

This document contains examples of correct and incorrect phrases. If some phrase is described as incorrect it only means that it violates the standards though it can be properly built from the language point of view. These standards do not teach English. They just describe which subset of possible phrases is taken as acceptable.

And again, standards are designed for uniformity only. There's no strict dogmatum in them. If some phrases seem to be more appropriate but they violate the rules it's OK to adjust rules to current needs rather than restrict needs with existing rules. The main thing is that there should be some uniform rule set which is kept within the project. Current document can be a basis for such standards

Sirius: solution improvements

Overview

The more code is prepared the more items require various corrections and enhancements. Also, some things weren't done initially but they were planned. So, in this article I'll describe various changes applied to the entire project. Here I'll cover the following topics:

1. Adding functionality to stop service endpoints
2. Server side updates for more informative generated code
3. Generate Ruby client from Java server code
4. Migrating solution from Ant to Maven

Sirius: First steps

In the previous article we've created basic skeleton for entire solution. Now it's time to add the content. In this article I'll take sample method, write it's server part and create clients for all programming languages we already prepared infrastructure for. The example would be simple but it's enough to find out how it's better to design entire solution and what is the base flow for adding new functionality. Also it would be clearly seem how it is better to use such solution. So, in this article I'll describe the following:

1. Sample server side method creation (actually that would be the sample of SOAP web-service creation on Java)
2. Java client creation
3. Ruby client creation
4. C# client creation

OK. We're good to go.

Sirius: Dev environment setup

Introduction

All right, I've decided to start development but it's not enough just to write the code. I also should be able to build the packages, test them somehow, provide some documentation and do many other activities. But firstly, I don't want to make a lot of routine work by myself. Let the machine do that for me. Also, I want to be sure that the way the system works for me locally is the same as for any other machine. And eventually, I don't want to keep the source code only locally. It's unsafe as my local machine can be damaged and all information would be lost in this case. So, we need some additional software which makes the development infrastructure and which should be configured in the way convenient to use.

This article is about to describe the infrastructure setup as well as how to configure all this software to point to each other.

Sirius: ice is broken

Pre-phrase

This is the new series of articles dedicated to the Sirius - the new test automation platform development. This is the work I've been doing for several years accumulating different pieces of code with different languages, for different technologies, for different testing types.

The goal of my work is to combine existing solutions under common architecture and represent some unified and extensible design which can be a basis for another brand-new test automation solution or even common platform for delivering test automation types. This is the not the first time I am getting back to this but the more results I get the more different solutions I can integrate under the same framework. I don't expect it to be another widely used platform (I am not even talking about replacement of any vendor tool) but that can be a good prototype. I doubt there were a lot of people who believed in Selenium eight years ago when only some draft version appeared. But take a look at the situation on test automation market now. Selenium became one of the most popular test automation tools and the popularity is growing. Even QTP loses its positions and now it's quite easy to find test automation specialists who are familiar with some popular programming language like Java, C#, Ruby, Python, PHP and many others which were traditionally used for development.

Test Automation ROI calculation Part 1 : Simple iterative development model

Introduction

The test automation isn't done just for fun. It has the purpose. Some of the main purposes are:

• Increase testing team capacity
• Minimize testing cycle without changing testing team capacity
• Increase testing coverage without extra resources
• Minimize human factor

But all above and many other benefits can be expressed with the single phase. Test automation should bring the profit. In other words, I have many ways to improve the testing efforts and each of them require some investment. Either I can hire more people or I can invest into some software/hardware which helps me making things faster. But if the investment into some improvement doesn't bring the profit this investment is useless. This is related to test automation as well. Yes, you can buy some expensive tools, you can build the complex infrastructure but if I can do the same in some other way which requires less investment I'll prefer the second option. So, before proposing some test automation efforts we should realize what the advantage we should make with this. One of the wide spread measure is the Return of investment. So, how should we calculate that for test automation project? Let's build some process model to see what we can calculate there.

Test Automation tools: trends

While making decision which tool is more suitable for test automation we should always take into account the popularity of each specific tool. Why? Because firstly it's about people. What's the point in taking tool which doesn't have users? Thus, we'll have to invest some money into their learning which is extra expences. Secondly, the more popular tool is the more materials are available either in internet or in some other materials. So, we should take a look at some trends to identify which tool is growing with it's popularity and which one is going to be a history.

BDD engines comparison (Cucumber, Freshen, JBehave,NBehave,SpecFlow, Behat)

Introduction

Cucumber is not the only engine supporting natural language instructions. It's just one implementation of natural language instructions interpreter. The actual language to write tests with is called Gherkin. And it has different implementations adopted to different programming language. Thus we have:

• Cucumber for Ruby
• JBehave for Java
• NBehave and SpecFlow for C#
• Freshen for Python
• Behat for PHP

This list isn't complete as there can be many other similar engines which are simply less popular. All of them have some common set of supported features but there're some restrictions and abilities specific to the actual engine. So, the aim of this post is to collect useful features for each listed above engine and present it in some comparable form. Key features to be mentioned are:

Cucumber: How to avoid major mistakes

Recently I came across several posts criticizing Cucumber and approach itself. Some of the links:

• BEHAVIOR-DRIVEN DEVELOPMENT VS. TESTING by James Bach
• Testing like the TSA (here is the criticizm of TDD approach but there is one recomendation here which fits the topic. I mean the recomendation not to use Cucumber)

and many others. Generally speaking the idea is that we should put off our pink sun glasses and look at the real world. Cucumber (and all it's analogs designed for other programming languages) is quite attractive solution providing the ability to write descriptive automated tests. But this attractiveness hides a myriad ways to "shoot your leg". And the more people use the approach of executable specifications the more people have bad experience. Why this happens?

Article: Is it sane to compare Selenium with QTP

Just read article here: Is it sane to compare Selenium with QTP?. Seems like all the article can be minimized to the following part:

Well, when did Selenium claim that it can tests window based applications. Selenium is browser automation framework. Comparing Selenium and QTP on this front is like comparing apples with oranges.

One paragraph says more than all others. Why such questions appear at all?

P.S.: wonder what's the goal of disabling text highlight operation on the page. If I want to copy\paste content I simply can view the page source.

Ruby: retrieve OLE object from HWND

I continue investigating the possibility to create custom auto-clicker. This time it's for my current project, so there'll be no blackjack. But anyway, the problems still appear and I have to resolve them. Just like this time when I've got the way to find Win32 windows by their specific attributes. Ruby contains magic gems like:

• win32-api
  
• windows-api
  

With their help I can easily get an access to Win32 API functions and call them from Ruby. But it works up to the moment when you encounter windows of Internet Explorer_Server class. Their children are HTML elements and they're not accessible directly via API. It's not a problem if we talk about browser. We can use Selenium or Watir to do all we need. But what if we talk about embeded Internet Explorer_Server windows which are placed on non-browser window? One of the ways is to get an access to corresponsing OLE object and hit it's properties and methods. Oh, good idea. But the main question is: how? All we know is just and HWND of this window. There's nothing impossible and this problem has the solution as well.

Selenium IDE: the way to test the locators

Selenium IDE is quite restricted solution and it's not recommended to use it as a solution for full scale test automation. But there're some areas where it helps a lot. Of course it can record sample tests and convert it into the code. That's the main Selenium IDE feature. But not the only one. Selenium IDE is very helpful when we want to test our locators and make sure they point to proper element. In this post I'll try to make step by step instructions how to do that.

Intro: Our own test automation tool

All the time I have to deal with some test automation tool. And all the time I have to live with the difficulties of each tool. If we have some huge vendor tool it's definitely something VBScript/JScript-based with all related restrictions. Or at least there're restrictions of programming language. It's critical because various wizards are not enough to make robust and reliable tests. Sometimes I have to make quite precise tuning to make my solution working the way I need. And even more, such wizards also consume memory and make overall work slower.

Selenium Camp. Impressions and thoughts

First Selenium conference in Europe took place this Saturday the February 26. It was a good chance to see how people use Selenium, some best practices, approaches. Of course, I had a chance to share my knowledge as well since I was one of the presentors.

Making Git working with Hudson

Once again I've started building CI infrastructure for another test automation project. Everything was clear and obvious. Actually, just another "toolset zoo" was involved. There were: PHPUnit, Selenium-RC, PHPDocumentor, Ant, Hudson and Git. Well, everything was obvious and clear. Ant makes all the stuff to prepare all resources and run single command. Hudson was supposed to download resources from Git and run the build.

Behavior-driven development introduction

What is the behavior-driven development approach. It's just another approach to build automated tests, like decomposition, keyword-driven, data-driven, object-based. Key feature of this approach is that each action is represented with some specific sentence describing actions to be done. And it's usually natural-language instruction. So, we actually describe some behaviour of AUT after some actions with it. So, at the highest level test is represented something like that:

Development skills for test automation

The more time I work with test automation the more times I'm assured that there's nothing to do in this area without development vision of the overall solution. Where it is essential?

SOA Testing: what should we know about it

It's obvoius that testing is not limited by clicks on application GUI. All we have as a part of the product should be tested. And sometimes we can interact on non-GUI level. In particular it key feature of services testing.