Measure Automated Tests Quality

Introduction

There's one logical recursion I encounter with test automation. Test automation is about developing software targeted to test some software. So, the output of test automation is another software. This is one of the reason for treating the test automation as the development process (which is one of the best practices for test automation). But how are we going to make sure that the software we create for testing is good enough? Indeed, when we develop the software we use testing (and test automation) as one of the tools for checking and measure the quality of the software under test.

So, what about software we create for test automation?

On the other hand we use testing to make sure that the software under test is of acceptable quality. In case of test automation we use another software for this. And in some cases this software becomes complicated as well. So, how can we rely on non-tested software for making any conclusions about the target product we develop? Of course, we can make test automation simple but it's not the common solution. So, we should find some compromise where we use reliable software to check the target software (the system under test). Also, we should find the way to find out how deep testing can be and how we can measure that.

So, main questions which appear here are:

• How can we identify that the automated tests we have are enough to measure quality of end product?
• How can we identify that our tests are really good?
• How can we keep quality control on our automated tests?
• How can we identify if our tests are of acceptable complexity?

In this article I'll try to find out answers to many of those questions.

Sirius: Adding code style checks

We already set up the build process for Sirius as well as we reserved the placeholders for testing. So, further improvement in this area is to extend build with different tests and checkpoints which make more detail verification. This time we'll add more modules which perform static testing for our components and they should fail the build if static checks didn't pass (especially taking into account that it was initially planned).

What would we look for? Basically that's:

• Conformity to coding standards - all code standards are made to provide unified approach of writing code as well as reading and maintaining it.
• Various cases of improper code usage - a lot of actual application errors are results of improper code use or use of potentially dangerous code constructions. So, the earlier we find them the less number of silly mistakes we take during further testing stages

Actually, we've made first approximation for that while we checked the compilation ability. If system isn't compilable it's not working at all. So, we're just making more detailed checks.

Since Sirius modules use different programming languages the tool set will be varying based on that. We'll use the following tool set for that:

Language	Engine
Java	Check Style PMD
C#	StyleCop
Ruby	rubocop

OK. Let's see how it would be added.