Sirius: ruby package installation testing

All Sirius client components are delivered as installable packages. And for Java and C# they are easy to use just by putting references to appropriate binaries. The Ruby packages are easy to install but it's hard to predict in advance whether they reference properly to all necessary resources. That's because any code becomes accessible via adding reference to the path in the file rather than including package name.

The key difference is that local relative paths work OK only for local sources but when you create Ruby package and install it those paths will be applied relatively to completely different folder. So, if you run local unit tests you don't have guarantee that all the code would work in the same fashion as it would work from the gem location. For this purpose, I've added separate class of tests into Sirius Ruby client packages. This class of tests is called installation tests. The main aim of them is to make sure that all modules are properly accessible after the package is installed. And in general it verifies that packages are operational after installation process.

What should be checked

At the moment there're 3 major checkpoints:

1. Recently created package is installable
2. Include files are accessible correctly after package installation
3. Key objects can be created after package is installed

In other words, we perform the following operations with checks:

Operation	Check
Install package from the scratch	Verify that package is viewable in the list of installed packages
Install package from the scratch and include some file from this package	File is found and included without any errors
Install package from the scratch and create some object instance from the classes in installed package	Verify that class is created without any errors

As it's seen these are just small checks however if we do that before any further tests we can get rid of useless runs in case package references are broken or improper.

Implementation

Since the operations to do are pretty simple and mostly related to shell commands execution there's no necessity to use anything more complicated than Test/Unit library as the test engine. So, it makes the test skeleton look like:

require 'test/unit'

class TestInstall < Test::Unit::TestCase

end

Additionally, we should specify the module to install and installation command as it's the same accross all the tests. So, we'll update the code with the following instructions:

  def package
    'sirius-client'
  end
  
  def install
    `gem install #{package} --local ./pkg/#{package}-#{File.read('VERSION').chomp}.gem --no-ri --no-rdoc`
  end

Here the installation is done from the pkg directory which contains locally built gem package.

Simple installation test

Now it's time to make first test where we install package from the scratch. In order to do that we should:

1. Uninstall package
2. Verify the package is removed
3. Install package
4. Verify it's present in the list

So, it's done in the following way:

  def test_modules
    gems = [package]
    
    #Remove package
    gems.each do |gem|
      `gem uninstall #{gem} -a -q`
      output = `gem search #{gem} -l`
      # Verify package is no longer present
      assert_equal nil,output.match("#{gem} [(]"),"The '#{gem}' module wasn't removed"
    end

    # Perform installation
    install

    # Make a console output of the list of installed packages
    output = `gem search #{package} -l`
    gems.each do |gem|
      # Make sure all necessary packages are installed
      assert_not_nil output.match(gem),"The '#{gem}' module wasn't found"
    end
  end

That's it. We're done with the first test. Next test is simpler as we just need to perform installation and then include the file. So, this test looks like:

  def test_inclusions
    install
    assert_nothing_raised do
      require 'sirius'
    end
  end

So, simple but if something's wrong with inclusion paths the test will fail indicating that we should correct all necessary references. The last test is almost the same except additionally we should create new object instance from the package we install. It looks like:

  def test_new_instances
    install
    assert_nothing_raised do
      require 'sirius'
      @client = Sirius::Client::InternalClient.new
    end
  end

If it fails it indicates that there're some runtime errors in the initializer modules. But once all tests are passed without any errors it's high time to trigger unit tests and many other tests verifying functional aspects of the module under test. The sample code can be found here.

Inserting into entire build

That was the basic set of installation tests for Sirius Ruby module. It's quite generic and can be expanded to any other module. At the same time the set of tests can be increased as long as we need some new check. Now it's time to add it into entire build process. For those tests we need the packaging to complete and style checks done. So, firstly, we should add new rake task into the Rakefile with the following content:

Rake::TestTask.new(:install_test) do |task|
  task.name = 'install_test'
  task.libs << "test"
  task.test_files = FileList['tests/install/**/*.rb']
  task.verbose = true
end

This will run our installation tests. And finally, we should group this task and all preceding ones under one name:

task :all => [:rubocop,:package,:install_test]

After that we can trigger the execution using the following command:

rake all

and that will do packaging, style check and installation testing. We can include all other tasks in order to expand the number of build steps.

Summary

That was just another step verifying the quality of the delivery. Actually, when I ran it I've found a number of errors because a lot of problems weren't simply visible while running modules locally. At the same this is real check how users can interact with the library before it goes live.