Test Automation tools: trends

While making decision which tool is more suitable for test automation we should always take into account the popularity of each specific tool. Why? Because firstly it's about people. What's the point in taking tool which doesn't have users? Thus, we'll have to invest some money into their learning which is extra expences. Secondly, the more popular tool is the more materials are available either in internet or in some other materials. So, we should take a look at some trends to identify which tool is growing with it's popularity and which one is going to be a history. We'll use http://www.indeed.com/jobtrends resource for our analysis. We'll be able to see the trend from 2005 up to now to see how popularity changed during that time range. Also, we'll use several groups of tools:

• GUI functional testing
• Non-gui functional testing (various service testing solutions)
• Load\Performance testing

All right. Let's do this!

Gui functional testing

For GUI testing estimations we'll take the following tools:

• QTP
• Selenium
• RFT (IBM Rational Functional Tester)
• Squish
• Microfocus SilkTest
• Smartbear Test Complete
• Parasoft Test
• Ranorex

We'll query indeed.com and get the following graph:

This is the statistics of job posting percentage for the above tools. We definitely see 2 major leaders: QTP and Selenium. But in order to see the trend it's useful to take a look how those values were changed during all that time. The following graph shows such changes:

As you can see the trend is to use free tools rather than vendor ones. It's not surprizing as nobody wants to pay extra money when we have an option not to pay. And after some time free tools become more reliable and more complicated. That explains why they start moving vendors from some piece of market. Anyway, we've identified favorites. But other tools owns some specific segments. Their influence isn't so clearly seen just because the favorites took essential part. So, lets query the statistics showing how other market players go during last 7 years. The job posting percentage looks like:

And changing dynamics respectively is:

From the above graphs we can see the following:

• For a long time (about lst 4 years) SilkTest and IBM Functional Tester were on the same level of popularity and only QTP occupied bigger area.
• TestComplete popularity grows all the time and now it owns same range of market as SilkTest and IBM functional tester. So, they are actually competitors. According to the changes dynamics graph we can see that SilkTest, TestComplete and Functional Tester tend to have linear growth. That indicates that they stabilized their place on market. So, they just at the state of stability
• Squish, Parasoft and Ranorex solutions are quite close and they show positive growth (the growth is even bigger than for tools from previous point) but their area is smaller

Another slice I'd like to make in such exploration is the overview of free tools. Their main feature is that they provide wide range of programming languages to use. Thus, we can use Java, C#, Ruby, PHP, Python which working with Selenium. Watir has different ports to Java, C#. So, it's interesting to see how those tools are used accross the programming languages. The trend is:

From that trend we can see the growing popularity of Java Selenium port. It can be explained by the fact that programmable part of Selenium (Selenium RC) was initially designed for Java and then ported to another languages. Also, Selenium is followed by Watir but Watir isn't so popular and it just occupies some area of market and it can't be considered as some dangerous competitor for Selenium.

Non-Gui functional testing

There's separate segment of testing tools which are targeted to non-GUI testing. It's various service testing tools or similar solutions interacting with applications which aren't represented with some GUI. Major players here are:

• HP Service Test
• Parasoft SOATest
• SoapUI

Actually, this market is not so widely represented as the market of GUI testing tools but anyway there're some trends here as well. Here are the graphs:

As it's seen on graph the most popular solution is SoapUI. Major explanation for this is that it's relatively cheap and basic configuration is available for free. There were the time when Service Test started expanding it's area but yet it's still at the bottom of popularity and it usually exists as the part of QC. One thing which should be mentioned here is that back-end testing is usually done without any specific tools. It's typically some shell scripting or unit-test manner. But it's hard to filter it out to have representative selection. Nevertheless, there's an ability to filter out the growing powershell popularity. The trend shows the following picture:

Generally scripting platforms become more and more popular for back-end testing. It's definitely not the area of dedicated tools.

Load\Performance Testing

This is the separate set of testing tools which require different approach for testing as well as different targets. Major players here are:

• LoadRunner
• IBM Rational Performance Tester
• SilkPerformer
• Parasoft Load Test
• loadUI
• JMeter

One of the noticeable features is that vendor tools are quite expencive which free tools are quite restricted with the protocols supported (usually it's http protocol). For that tools trend looks like:

HP solution is still market leader but it's popularity grows down. It doesn't show any essential grow. The closest free analog (JMeter) is still less popular but growth is seen. Growth isn't very big but it slowly goes up. The trend for Rational Performance tester looks quite surprizing. The absolute values are quite low but it shows quite big shifts. However no drammatical changes in this area is seen. Partially it can be explained by the fact that performance testing is not so widely used as functional testing and performance testing tasks have rather one-time or on demand nature.

Conclusions

The variety of tools is quite big and they all the time compete for some specific market area. The more popular technology under test is used the higher probability that some free solution appears. Also, there's a trend to step away from black-box testing tools and go to the direction of some combination of solutions. That trend is seen in GUI testing when separate libraries become more popular than completed tools. It's seen in non-gui testing where preferance goes to some scripting engines rather than tools. Performance testing area looks more conservative here as technically it's more complicated and we still have a need of complicated infrastructure. So, let's see later where this market moves.