In the previous post I've added another module to support Win32 operations. Actually, there should be a lot of modules to be added. But I put it directly into the Server module. The more components we cover the more modules I have to include. Thus the server code as well as clients become heavy-weight and entire solution becomes huge. But firstly, it breaks one of the ideas of the Sirius platform. It should have an extensible engine which can plug in the modules on fly. Also, it should be adopted to different operating systems (that's why I chose Java as the main language) but what should the Win32 module do, for instance, on Unix? Nothing. So, in such case I don't need to include it. Or even more, I want to make different complectations for Server side depending on what I actually want to use. For this purpose I have to split our components into functional modules and provide the ability to configure what modules I should include. Also, I would be useful to provide the ability to deliver either entire solution or some parts of it. In this post I'll describe the steps for doing this.

What and how to split

The general approach is to create separate projects for server and clients for each specific functional area. Also, we should reserve base packages to store very basic functionality. General structure after split can be represented with the following diagram:

So,generally all operations remain the same but the entire code is now distributed between different projects and packages. Thus, we'll be able to make targeted changes without impact on any other modules which aren't affected. Also, we should be able to include/exclude any module we need. So, furthermore I'll describe that in details as well as I'll describe the way to implement that.

Splitting the server

Firstly, we should create 2 additional projects:

Sirius-Server-Core - stores code related to core operations
Sirius-Server-Win32 - stores code related to Win32 operations

Secondly, we should copy pom.xml file from the Server project for each newly created one. The only updates here are:

Artifact and group IDs should be renamed to correspond to the current project
Remove innecessary dependencies. E.g. JNA library dependency is needed only for Win32 module while Core project doesn't use it at all

And eventually, we should move the code from Server project to appropriate split projects:

Sirius-Server-Core project should take org.sirius.server.system package
Sirius-Server-Win32 project should take org.sirius.server.win32.* packages

Generally, after the split the server projects look like:

Upload server packages

After above splits the main server part became very small and most of the classes which were supposed to be loaded as service endpoints are now outside the main package. But we still need to load them to make Server work as before. What should be changed then? Actually we should update 2 parts:

Configuration file should reference to the packages paths instead of Local
The service starter code should be updated to load Java packages on-fly

OK. The configuration file updates look like:

Endpoint                                    ,Class                                          ,Package
http://${HOST}:${PORT}/system/directory     ,org.sirius.server.system.DirectoryOperations   ,./Sirius-Server-Core*.jar
http://${HOST}:${PORT}/system/file          ,org.sirius.server.system.FileOperations        ,./Sirius-Server-Core*.jar
http://${HOST}:${PORT}/system/process       ,org.sirius.server.system.ProcessOperations     ,./Sirius-Server-Core*.jar
http://${HOST}:${PORT}/system/system        ,org.sirius.server.system.SystemOperations      ,./Sirius-Server-Core*.jar
http://${HOST}:${PORT}/win32/core/kernel32  ,org.sirius.server.win32.core.Kernel32Lib       ,./Sirius-Server-Win32*.jar
http://${HOST}:${PORT}/win32/core/user32    ,org.sirius.server.win32.core.User32Lib         ,./Sirius-Server-Win32*.jar
http://${HOST}:${PORT}/win32/utils          ,org.sirius.server.win32.Win32Utils             ,./Sirius-Server-Win32*.jar

We should support the ability to define file by mask as the package versions may be varying.

Oce we configured packages we can load them. Generally, the code loading class from the package looks like:

 File location = new File(packageFile);
 URL url[] = { location.getAbsoluteFile().toURI().toURL() };
 ClassLoader loader = new URLClassLoader(url,this.getClass().getClassLoader());
 Class clazz = Class.forName(className,true,loader);

Where packageFile is the location of the package to get class from and className is the name of the class to get instance of. This code can be injected into the startEndPoints method of the Starter class in the following way:

 private String findMatchingFile(String filter){
  File location = new File(filter);
  
  for(String file:location.getParentFile().list() ){
   if(file.matches(location.getName())){
    return file;
   }
  }
  
  return "";
 }¹
 
 public void startEndPoints(ArrayList options, String host,
   String port) throws MalformedURLException {
  for (PackageOptions option : options) {
   ClassLoader loader;²
   if (!option.get_packageLocation().equals("Local")) {
    Log4J.log()
      .info("Uploading binary file:"
        + option.get_packageLocation());

    String packageFile = findMatchingFile(option.get_packageLocation());
    File location = new File(packageFile);
    
    URL url[] = { location.getAbsoluteFile().toURI().toURL() };
    loader = new URLClassLoader(url,this.getClass().getClassLoader());
   }
   else {
    loader = this.getClass().getClassLoader();
   }³
   try {
    String endPoint = option.get_endPoint();
    endPoint = endPoint.replaceAll("\\$\\{HOST}", host);
    endPoint = endPoint.replaceAll("\\$\\{PORT}", port);
    Log4J.log().info("Starting endpoint: " + endPoint);
    Endpoint endpoint = Endpoint.publish(endPoint,
      Class.forName(option.get_className(),true,loader)⁴.newInstance());
    endpoints.add(endpoint);
   } catch (Exception e) {
    Log4J.log().error("Failed publishing server endpoint", e);
   } finally {
    Log4J.log().info("Done...");
   }
  }
 }

The updates here are:

¹ - since we specify files by mask we need a function that will return first matchine file by specified mask
² - we declare the variable storing the class loader we should use in further steps
³ - if we load classes from external package we should initialize URIClassLoader instance. Otherwise the current class loader should be used
⁴ - class is retrieved from the explicitly specified loader

Once this is done we complete our work with server side.

Splitting the client

Code split

The client code split is done in the same fashion as for server side as client should implement interation with some specific server part. So, agains that should be several projects which are 1:1 mapped to corresponding server project. Here is the mapping table showing the project names after split and their correspondence between each other:

Server Project	Java Client Project	Ruby Client Project	C# client project
Sirius-Server	SiriusJavaClient	sirius-ruby-client	SiriusClient
Sirius-Server-Core	SiriusJavaClient-Core	sirius-ruby-client-core	Sirius.Client.Core
Sirius-Server-Win32	SiriusJavaClient-Win32	sirius-ruby-client-win32	Sirius.CSharp.Client.Win32

Packaging changes

The most affected part here is the packaging. Before we made the package for the entire project which contained all. With projects split we should make individual packages. So, we actually copy package scripts. Thus, having small packages we can identify what parts of the client we actually need. At the same time if we want to install package containing all sub-modules we still should refer to main client package. E.g. the command like:

gem install sirius-client-win32

will install only Win32 packages while the command like:

gem install sirius-client

will install all available packages. For this purpose the main package project should reference to other packages as dependencies. E.g. the gem specification for Ruby client should be updated in the following way:

spec = Gem::Specification.new do |s|
  ......
  s.email="kolesnik.nickolay@gmail.com"
  s.add_dependency('sirius-client-core', '>= 0.0')
  s.add_dependency('sirius-client-win32', '>= 0.0')
end

Build and release process changes

All the above changes were reflected in the build process. Firstly, it became more complicated as instead of one build line there're multiple lines now. The changes can be represented with the following diagram:

As it's seen from the above image there're multiple entry points. At the same time the release task is still only one. It means that now we can develop multiple parts independently and if each specific part is at working state the entire release is working.

Summary

What was planned	Done/Failed	Comments/What should be done
Split server into modules	Done
Update server code to upload modules on-fly	Done
Split clients into modules	Done
Re-organize build process for split components	Done

After those changes the Sirius looks more like the platform as now we're free to add many different modules. So, now all we have to extend the coverage on different technologies, languages etc.

Test Automation from inside

Search

Sunday, 13 January 2013

Sirius: modularize the solution