Sirius: solution improvements

Overview

The more code is prepared the more items require various corrections and enhancements. Also, some things weren't done initially but they were planned. So, in this article I'll describe various changes applied to the entire project. Here I'll cover the following topics:

1. Adding functionality to stop service endpoints
2. Server side updates for more informative generated code
3. Generate Ruby client from Java server code
4. Migrating solution from Ant to Maven

Endpoint for service stop

Well, we can start our service and it will listen to specified port waiting for new commands. But if we take a look at the server side from users point of view we'll see that we also need an ability to stop this service. Process stop isn't a way out here as there can be multiple processes with the same name. We should be able to stop specific service which is listening to specific port.

That's why we need dedicated method which will do that for us. Actually, changes are required in 2 directions:

1. We should add the structure which stores references to all Endpoint objects. During stop operation we just iterate the storage and call Endpoint.stop method
2. We should add additional class which should interact as another one web service endpoint where all the stop operations take place

So, let's start from the Starter class. Firstly we'll add the endpoints container (just declare another class variable):

  static ArrayList endpoints = new ArrayList(); 
 

After that we'll update startEndpoints method with the following:

  public void startEndPoints(ArrayList options, String host,
   String port) {
  for (PackageOptions option : options) {
   if (!option.get_packageLocation().equals("Local")) {
    // TODO: Add load class instructions
   }
   try {
    String endPoint = option.get_endPoint();
    endPoint = endPoint.replaceAll("\\$\\{HOST}", host);
    endPoint = endPoint.replaceAll("\\$\\{PORT}", port);
    
    Endpoint endpoint = Endpoint.publish(endPoint, Class.forName(option.get_className())
      .newInstance());
    endpoints.add(endpoint);
   } catch(Exception e){
    ;
   }
  }

Then we need to create new web service endpoint with the method stopping service. For this purpose I create Internal class in the same package as Starter and fill it with the following content:

package org.sirius.server;

import javax.jws.WebService;
import javax.xml.ws.Endpoint;

@WebService
public class Internal {
 
 public void stop(){
  for(Endpoint endpoint:Starter.endpoints){
   endpoint.stop();   
  }
  Starter.endpoints.removeAll(Starter.endpoints);
 }
}

All that's left is just to start internal endpoint somewhere inside the main. So, we should add the following code somewhere into main:

  try {
   endpoints.add(Endpoint.publish("http://" + host + ":" + port
     + "/internal", new Internal()));
  } catch (Exception e) {
   ;
  }

Cool. Now we can stop our service from client side.

Update server side code

If we keep using server side code the way it was done before we'll encounter several problems with client code generation. Major problem is that the names of parameters will not be exported into client code during generation. E.g. if we have server code like:

 public boolean fileExists(String fileName) {
  File file = new File(fileName);
  return file.exists() && file.isFile();
 }

the generated client method will have construction like:

    public boolean fileExists(java.lang.String arg0) throws java.rmi.RemoteException;

So, when we have method which accepts 5, 10 or whatever parameters they all will be non-informative names like arg0, arg1 etc. Despite such generated client code is mostly supposed to be used internally it's not convenient to use it anyway.

Also, C# client has a problem while generating return values for those methods. By default it will generate names like @return which may have problems with compilation.

In order to sort out these problems we can annotate parameters and method itself with WebParam and WebResult annotations respectively. Both of them have name field where we can define which name would be used for representing the parameter or return value. So, the server code with annotations looks like:

 @WebResult(name = "status")
 public boolean fileExists(@WebParam(name = "fileName") String fileName) {
  File file = new File(fileName);
  return file.exists() && file.isFile();
 }

The client code will be reflected with more informative names generated. E.g. the interface method now looks like:

    public boolean fileExists(java.lang.String fileName) throws java.rmi.RemoteException;

This enhancement will seriously help with client generation for Java and C# as well as it would be the basis for Ruby client generation which will be described separately.

Ruby client generation

Previously I encountered the problem with Ruby client generation. It appears that for the recent Ruby versions there's no stable fully automated code generator. That's why I postponed Ruby client part for a while. But this is still serious gap which should be resolved.

Existing Ruby SOAP clients overview

Actually, there're some engines which could be applied for SOAP clients but they have their own gaps. Some of the are:

1. soap4r - it was developped long time ago and was working for 1.8 version. But for 1.9 it became incompatible. Probably it's just because it hasn't been supported for ages. However, for now it's not the best choice
2. httpclient - well, it's universal solution for HTTP requests. But the main gap is that I have to carry out all the SOAP XML transformations by myself. I need something more high-level
3. savon - that's something close to what I want. It provides quite compact interface for SOAP calls which is very useful. The only gap is that I have to write input data structures by myself.

So, I stopped at savon. The methods skeleton is supposed to be quite generic. So, I need some generator which should fill me everything else.

Approach for client generation description

I would have the WSDL as the only source for the input data for generator but I didn't find some data needed for generation. Also, I didn't have a time to dig into the technical details how I could do anything like that. I just decided to find the fastest way to get all data at once. Since I've decorated all server methods with annotations I could make a transformation from server side Java representation to Ruby. E.g. if we take a server side method like:

 @WebResult(name = "status")
 public boolean fileExists(@WebParam(name = "fileName") String fileName) {
  File file = new File(fileName);
  return file.exists() && file.isFile();
 }

to client code like:

          def file_exists(file_name)
            response = @client.request :sys, "fileExists" do
              soap.body = {
                :file_name => file_name
              }
            end
            response.to_hash[:file_exists_response][:status]
          end

The colors in both examples show the mapping between Java and Ruby code. So, there can be one to one mapping between 2 parts. Also, there's some common part for all generated classes which can be placed into generator as is. It is the initialization part which can be represented with the following code:

        class <Class Name>
          attr_accessor(:host, :port, :client)
          def initialize(host = "localhost",port="21212")
            @host = host
            @port = port

            @client = Savon.client do |wsdl|
              wsdl.endpoint = "http://#{@host}:#{@port}/<Endpoint>?wsdl"
              wsdl.namespace = "http://system.server.sirius.org/"
            end
          end

So, general flow to implement generator can be described with the following steps:

1. Retrieve class information
2. Retrieve methods information (walk through annotated methods and read parameters annotation)
3. Transform names where needed and fill all data into Ruby class template to generate class from

Writing generator

Firstly, I've created Generator class apart from the main code but in the same project as the server. In order to collect class and method information in structured way I need 2 additional types. I've added them into Generator class like that:

public class Generator {

 public class MethodData{
  public String name = "";
  public String params[] = {};
  public String returns = "";
 }
 
 public class ClassData{
  public MethodData[] methods = {};
  public String name = "";
 }
}

Another little "whistle" that should be added is the method transforming Java-style names into "snake case" like it's used in Ruby by naming convention. It's the method like:

 public String toSnakeCase(String name){
  name=name.replaceAll("([a-z])([A-Z])", "$1_$2");
  return name.toLowerCase();
 }

Then we should add methods which retrieve class and methods information from the class passed. Since the entire method is quite big there's a reason to split it into 2 methods which are responsible for getting method and class information respectively. Method data is retrieved with the following code:

 public MethodData getMethodInfo(Method method){
  MethodData data = new MethodData();
  data.name = method.getName();

  if(method.isAnnotationPresent(WebResult.class)){
   data.returns = method.getAnnotation(WebResult.class).name() ;
  }
  else {
   data.returns = "";
  }
  
  Annotation[][] annotations = method.getParameterAnnotations();
  
  data.params = new String[annotations.length];
  
  for(int i=0;i<annotations.length;i++){
   for(int j=0;j<annotations[i].length;j++){
    if(annotations[i][j].annotationType().equals(WebParam.class)){
     data.params[i] = ((WebParam)annotations[i][j]).name();
     break;
    }
   }
  }
  
  return data;
 }

The highlighted parts represent the places where we look for specific annotations to get values from. This is the method retrieval part. The class data retrieval part looks like:

 public ClassData getClassInfo(Class clazz){
  ClassData classData = new ClassData();
  
  classData.name = clazz.getSimpleName();
  
  Method[] methods = clazz.getMethods();
  classData.methods = new MethodData[methods.length];
  
  for(int i=0;i<methods.length;i++){
   classData.methods[i] = getMethodInfo(methods[i]);
  }
  return classData;
 }

And eventually, we should collect all those information and fill into the template string. For that purpose we should add another method like:

 public String generateRubyClassCode(Class<?> clazz){
  String classTemplate="        class {CLASSNAME}\r\n" +
    "          attr_accessor(:host, :port, :client)\r\n" +
    "\r\n" +
    "          def initialize(host = \"localhost\",port=\"21212\")\r\n" +
    "            @host = host\r\n" +
    "            @port = port\r\n" +
    "\r\n" +
    "            @client = Savon.client do |wsdl|\r\n" + 
    "              wsdl.endpoint = \"http://#{@host}:#{@port}/system/file?wsdl\"\r\n" +
    "              wsdl.namespace = \"http://system.server.sirius.org/\"\r\n" +
    "            end\r\n" +
    "          end\r\n" +
    "{METHODS}\r\n" + 
    "        end";
  
  String methodTemplate = "\r\n"+
    "   def {METHOD_NAME}({PARAMS})\r\n"+
          "        response = @client.request :sys, \"{METHOD}\" do\r\n"+
          "          soap.body = {\r\n"+
          "{FIELDS}\r\n"+
          "          }\r\n"+
          "        end\r\n"+
          "        response.to_hash[:{METHOD_NAME}_response][:{RESULT}]\r\n"+
          "      end\r\n";
  ClassData classData = getClassInfo(clazz);
  String result = classTemplate.replaceAll("\\{CLASSNAME\\}", classData.name);
  
  String methodText = "";
  
  for(MethodData method:classData.methods){
   String singleText = methodTemplate;
   if(method.returns.equals("")) continue;
   singleText = singleText.replaceAll("\\{METHOD_NAME\\}", toSnakeCase(method.name));
   singleText = singleText.replaceAll("\\{METHOD\\}", method.name);
   singleText = singleText.replaceAll("\\{RESULT\\}", method.returns);
   
   String paramText = "";
   String fieldsText = "";
   
   for(String param:method.params){
    try {
    paramText += toSnakeCase(param)+",";
    fieldsText += "            :"+toSnakeCase(param) +" => " + toSnakeCase(param) + ",\r\n";
    }
    catch(Exception e){
     e.printStackTrace();
    }
   }
   
   singleText = singleText.replaceAll("\\{PARAMS\\}", paramText + "reserved = nil");
   singleText = singleText.replaceAll("\\{FIELDS\\}", fieldsText);

   methodText += "\r\n" + singleText;
  }
  
  result = result.replaceAll("\\{METHODS\\}", methodText);
  
  return result;
 }

The resulting string will contain the generated Ruby code. Of course, it will require some corrections as some default methods will be included as well as we should update the endpoint and namespace if needed. But in general it's much faster than typing the same class manually.

Running generator

Now we can start running generator. E.g. I can create main function and put the code like:

 public static void main(String[] args) {
  Generator gen = new Generator();
  System.out.println(gen.generateRubyClassCode(
   org.sirius.server.system.SystemOperations.class
   )
  );
 }

So, we only have to specify the class we want to generate. The generated code is placed into any templates (I prefer adding those classes into modules) but all those changes are all about customization.

Migrating from Ant to Maven

Reason for migraiton

Actually it was originally planned to use Maven but it required too much setup as well as Ant was much easier to get something working. But at the same time some problems still took place. In particular, there's a necessity to keep external libraries in the repository as well as store them under source control which is not pretty much convenient. So, such dependencies should be removed especially taking into account that similar thing has already been done for Ruby and C# clients.

Another reason for migration is that some modules I need in the future are available only as Maven dependencies. So, those 2 points were enough for me to make a decision to start using Maven instead of Ant for a while.

Configuring Maven

Maven can be downloaded from the official site. After you downloaded the archive you should do the following:

1. Unpack archive to the location where you want to keep Maven installation (further this location will be called MVN_HOME)
2. Add "MVN_HOME\bin" path to path variable
3. Open "MVN_HOME\conf\settings.xml" file and uncomment localRepository node. If necessary, specify custom location. That would be the folder where Maven will export all uploaded modules while resolving dependencies

After local settings are done it's time to set up the Eclipse.

Configuring Eclipse

Firstly, we should install Maven plugin for Eclipse. For this purpose we should configure eclipse to install software from http://download.eclipse.org/technology/m2e/releases update site.

Once installation is done it's time to make Maven global settings. For this purpose we should:

1. Navigate to Window > Preferences menu
2. In the Preferences window navigate to Maven > Installations
3. There would be the list of all available Maven installations. For now there's only one built-in instance.
4. In order to add new installations click on Add button and enter the MVN_HOME path into open dialog. Click OK.
5. The new Maven record appears. Make sure you set check mark on it.
6. In the same dialog navigate to Maven > User Settings
7. In the User settings field enter the address of user settings file (for me it's MVN_HOME\conf\settings.xml)
8. Click OK button to accept all the changes

Project changes

After settings are done we can start updating the projects in Eclipse. I'll start with Server project. Firstly, we can convert the Java project into Maven project. For this purpose we should right-click on project icon on the left pane and select menu Configure > Convert to Maven project.

There will be pom.xml file generated. This is the major build file we should modify. For Server project we'll need 1 dependency: log4j. Also, we should use 4 plugins:

1. maven-compiler-plugin - used for compile task
2. maven-jar-plugin - used to make jar archive
3. maven-assembly-plugin - used for assembly task
4. maven-dependency-plugin - used for uploading dependencies

I won't describe each setting in detail, I'll just list the pom.xml content for that. Here it is:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>
 <groupId>Sirius</groupId>
 <artifactId>Sirius-Server</artifactId>
 <version>${package.version}</version>
 <build>
  <sourceDirectory>src</sourceDirectory>
  <resources>
   <resource>
    <directory>src</directory>
    <excludes>
     <exclude>**/*.java</exclude>
    </excludes>
   </resource>
   <resource>
    <directory>target/dependency</directory>
    <excludes>
     <exclude>**/*.java</exclude>
    </excludes>
   </resource>
  </resources>
  <plugins>
   <plugin>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>2.3.2</version>
    <configuration>
     <source>1.7</source>
     <target>1.7</target>
    </configuration>
   </plugin>
   <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-jar-plugin</artifactId>
    <version>2.3.2</version>
    <configuration>
     <excludes>
      <exclude>*</exclude>
      <exclude>junit/**/*</exclude>
      <exclude>META-INF/maven/**/*</exclude>
     </excludes>
     <archive>
      <manifest>
       <addClasspath>true</addClasspath>
       <addDefaultImplementationEntries>true</addDefaultImplementationEntries>
       <addDefaultSpecificationEntries>true</addDefaultSpecificationEntries>
       <addExtensions>true</addExtensions>
       <classpathLayoutType>repository</classpathLayoutType>
       <mainClass>org.sirius.server.Starter</mainClass>
      </manifest>
     </archive>
    </configuration>
   </plugin>
   <plugin>
    <artifactId>maven-assembly-plugin</artifactId>
    <configuration>
     <descriptorRefs>
      <descriptorRef>-deps</descriptorRef>
     </descriptorRefs>
    </configuration>
   </plugin>
   <plugin>
    <artifactId>maven-dependency-plugin</artifactId>
    <executions>
     <execution>
      <id>unpack-dependencies</id>
      <phase>generate-resources</phase>
      <goals>
       <goal>unpack-dependencies</goal>
      </goals>
     </execution>
    </executions>
   </plugin>
  </plugins>
  <pluginManagement>
   <plugins>
    <!--This plugin's configuration is used to store Eclipse m2e settings 
     only. It has no influence on the Maven build itself. -->
    <plugin>
     <groupId>org.eclipse.m2e</groupId>
     <artifactId>lifecycle-mapping</artifactId>
     <version>1.0.0</version>
     <configuration>
      <lifecycleMappingMetadata>
       <pluginExecutions>
        <pluginExecution>
         <pluginExecutionFilter>
          <groupId>
           org.apache.maven.plugins
          </groupId>
          <artifactId>
           maven-dependency-plugin
          </artifactId>
          <versionRange>
           [2.1,)
          </versionRange>
          <goals>
           <goal>
            unpack-dependencies
           </goal>
          </goals>
         </pluginExecutionFilter>
         <action>
          <ignore></ignore>
         </action>
        </pluginExecution>
       </pluginExecutions>
      </lifecycleMappingMetadata>
     </configuration>
    </plugin>
   </plugins>
  </pluginManagement>
 </build>
 <name>Sirius Server</name>
 <dependencies>
  <dependency>
   <groupId>log4j</groupId>
   <artifactId>log4j</artifactId>
   <version>1.2.9</version>
  </dependency>
 </dependencies>
 <reporting>
  <plugins>
   <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-checkstyle-plugin</artifactId>
    <version>2.9.1</version>
   </plugin>
  </plugins>
 </reporting>
 <url>https://github.com/mkolisnyk/Sirius</url>
 <description></description>
</project>

Once the pom.xml is changed and saved we should update the project. For this purpose right-click on project icon in the left navigation menu and select Maven > Update Project. After that you should see new Maven Dependencies node appeared near JRE System Library dependencies. After that I can remove all the content of lib folder. It's no longer needed.

Java client project requires the same settings except it uses different set of dependencies. So, the dependency section here is:

 <dependencies>
  <dependency>
   <groupId>axis</groupId>
   <artifactId>axis</artifactId>
   <version>1.4</version>
  </dependency>
  <dependency>
   <groupId>org.eclipse.birt.runtime.3_7_1</groupId>
   <artifactId>javax.wsdl</artifactId>
   <version>1.5.1</version>
  </dependency>
 </dependencies>

After the changes are done each of the Java projects can be build using the following command:

mvn package

So, now we're almost done here. All we have to update is Jenkins configuration.

Updating Jenkins

In order to update Jenkins to use Maven we should have Maven plugin install. After that we should do the following:

1. Navigate ot Jenkins > Settings > Configure System
2. Scroll down to Global Settings section to the place where environment variables defined
3. Update PATH variable entry with the path to Maven binaries location (MVN_HOME\bin sub-folder)
4. Scroll down to Maven section and click on Add installation button
5. Specify Maven configuration name (e.g. I set it to the value of "Base")
6. In the MAVEN_HOME field enter the full path to Maven root directory
7. Save changes

Thus we're done with the global settings. Now we're ready to update our build projects. In order to add Maven support to Jenkins project we should do the following (I'll make it based on Sirius Server project):

1. Navigate to Jenkins project page and click on Settings
2. Scroll down to Build Steps section and remove Ant build step
3. Click on Add build step and then Call Maven target menu item
4. In the Maven Version field select the Maven version configured in global settings (for me it's name is "Base")
5. In the Targets field enter the "package" value
6. Under post-build actions we should update path of artifacts to archive. Now it should be Server\target\*.jar
7. Save changes

Now we should be able to run Maven tasks. The same settings should be applied to Sirius Java client project.

Afterwords

For now, we made quite essential improvements. Just some of them:

1. We're unblocked with Ruby client
2. We removed explicit references to libraries as well as we no longer need keeping them in the repository. If we have too much libraries it should eat essential part of the disk space. So, now our solution is more or less light-weight
3. We've applied full control on server side. Thus, we are not only able to start server but also stop it from outside which is convenient for automation

Next steps would be the further solution expansion and testing.