Search

Sunday 15 September 2013

GitHub: organizing and automating test tracking with Java

GitHub provides some tracker where we can keep our tasks. The GitHub API provides the ability to access that resources programmatically. Thus, with the help of API I could integrate JBehave and GitHub so that I can run JBehave tests stored in GitHub tracker. But it's not the only case.

With the help of API I can set the correspondence between tests and features. This gives the ability to make the traceability of tests and features they're applied for. Also, what if we use something different from JBehave, e.g. Cucumber, Sprecflow which use flat files for input? In this article I'll describe how to:

  • Organize tests with GitHub tracker
  • Automatically generate traceability matrix
  • Automatically generate feature files based on the content of the test issues

Organize tests with GitHub tracker

What do we have initially?

First thing we should realize in GitHub tracker is that it is quite simple tracker which is mainly designed to track bugs/tasks but not various complicated resources. That's why there's no different issue types. There's only one entiry: issue.

We cannot distribute issues by types but we still can mark them in a specific way. For this purpose there's labelling functionality. We can setup different labels and use them for indicating different issue types and/or application area.

Another thing is that all issues are entities of the same hierarchial level. It means that you cannot create task and make some sub-tasks. At least explicitly. We can make some implicit check-points inside the task (I'll touch this point later) but it's not the same as you can do via more or less complicated tracking systems. Nevertheless, all tasks can be split by some general group of features which have pre-defined timelines. This groupping entity is called milestone. It can be used for issues groupping.

Organize structure

Based on the above points we can identify the following rules for issues tracking:

  • Milestone is the highest level object representing some specific feature. This item should contain feature description
  • Each specific issue which is related to some feature should be linked to appropriate milestone
  • In order to filter out tests from development tasks or bugs we can use labels
OK. Now we're good to go with the filling data in.

Fill-in the content

For this example I've created Win32 Tab Control support milestone and added some small description like:

As the system user
 I want to be able to interact with Win32 Tab Control
That would be the place-holder for tests.

For better distribution between tests and other task types I'll add the following labels:

  • Test - used for the issues containing test scenarios
  • Bug - used for storing bug information
  • DevTask - used for the issues containing development tasks related to each specific feature
Additionally, we can add various labels representing application modules and other features which uniquely define the area this issue is applied to.

Once we're done with the labelling we can create new issue and fill it in with the test scenario, e.g.:

Scenario: List All page names
  - [ ] User should be able to get the list of all pages

  When I start GUI tests application
  Then I should see the following tabs:
    | Tab Name |
    | Static Text |
    | Edit Page |
    | Rich Text |
    | Buttons |
    | List Box |
    | Combo Box |
    | Scroll Bars |
    | Image Lists |
    | Progress Bar |
    | Sliders |
    | Spinners |
    | Headers |
NOTE
There's one additional line here:
- [ ] User should be able to get the list of all pages
which is rather GitHub wiki markup feature than something that needed for test scenario. Is GitHub issue contains text like that it's interpreted as a check-list item. So, you can split task into smaller sub-tasks and mark their completion inside the issue. But for test scenario it doesn't make effect.

Prepare the code generator

Before we start generating various resources we should identify how it can be done and what should be generated.

Retrieval interface

I'm going to prepare the utility which should be able to generate traceability matrix and feature file based on GitHub issues structure. In both cases the output should contain the following requisites:

  • Header - contains the most common information. Usually overview of the issues
  • Milestone information - general information which is taken from milestone
  • Issue information - contains information taken from issues
  • Labels information - puts information based on labels
  • Footer - contains some disclaimer or some other summaryzing information
So, actually, I have to create something which retrieves that information. It fits the common structure provided by the following interface:
/**
 * 
 */
package sirius.utils.retriever.interfaces;

import java.util.ArrayList;

import org.kohsuke.github.GHIssue;
import org.kohsuke.github.GHMilestone;

/**
 * @author Myk Kolisnyk
 *
 */
public interface IStoryFormatter {
    public String GetHeader(ArrayList issues);
    public String GetMilestone(GHMilestone milestone);
    public String GetIssue(GHIssue issue);
    public String GetLabels(GHIssue issue);
    public String GetFooter(ArrayList issues);
}
and for now we'll create some dummy formatter implementing this interface:
/**
 * 
 */
package sirius.utils.retriever.formatters;

import java.util.ArrayList;

import org.kohsuke.github.GHIssue;
import org.kohsuke.github.GHMilestone;

import sirius.utils.retriever.interfaces.IStoryFormatter;

/**
 * @author Myk Kolisnyk
 *
 */
public class DummyFormatter implements IStoryFormatter {

    /* (non-Javadoc)
     * @see sirius.utils.retriever.interfaces.IStoryFormatter#GetHeader(java.util.ArrayList)
     */
    public String GetHeader(ArrayList issues) {
        // TODO Auto-generated method stub
        return null;
    }

    /* (non-Javadoc)
     * @see sirius.utils.retriever.interfaces.IStoryFormatter#GetMilestone(org.kohsuke.github.GHMilestone)
     */
    public String GetMilestone(GHMilestone milestone) {
        // TODO Auto-generated method stub
        return null;
    }

    /* (non-Javadoc)
     * @see sirius.utils.retriever.interfaces.IStoryFormatter#GetIssue(org.kohsuke.github.GHIssue)
     */
    public String GetIssue(GHIssue issue) {
        // TODO Auto-generated method stub
        return null;
    }

    /* (non-Javadoc)
     * @see sirius.utils.retriever.interfaces.IStoryFormatter#GetFooter(java.util.ArrayList)
     */
    public String GetFooter(ArrayList issues) {
        // TODO Auto-generated method stub
        return null;
    }

    /* (non-Javadoc)
     * @see sirius.utils.retriever.interfaces.IStoryFormatter#GetLabels(org.kohsuke.github.GHIssue)
     */
    public String GetLabels(GHIssue issue) {
        // TODO Auto-generated method stub
        return null;
    }
}
This code is purely generated.

Adding command line parameters

It is convenient to provide command line interface. It can be reworked as the Maven plugin but for now it's enough to have just command line utility. As an input we need the following information:

  • GitHub username
  • GitHub password
  • GitHub repository name
  • Output type - the flag identifying whether we produce traceability matrix or something else
So, generally the command line should look like:
java -jar sirius.utils.retriever-<version>.jar -r <repository> -u <user> -p <password> -t <report_type>
Having this information we can start creating main program part. We'll start with the main program class which gets all those parameters:
/**
 * 
 */
package sirius.utils.retriever;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashMap;
import java.util.LinkedList;
import java.util.List;

import org.kohsuke.github.GHIssue;
import org.kohsuke.github.GHIssueState;
import org.kohsuke.github.GHMilestone;
import org.kohsuke.github.GHRepository;
import org.kohsuke.github.GitHub;

import sirius.utils.retriever.formatters.CucumberFormatter;
import sirius.utils.retriever.formatters.DummyFormatter;
import sirius.utils.retriever.formatters.TraceabilityMatrixFormatter;
import sirius.utils.retriever.interfaces.IStoryFormatter;

/**
 * @author Myk Kolisnyk
 *
 */
public class Program {

    public static final String REPO_NAME="-r"; 
    public static final String USER_NAME="-u";
    public static final String USER_PASS="-p"; 
    public static final String OUTPUT_TYPE="-t"; 
    
    /**
     * @param args
     * @throws IOException 
     */
    public static void main(String[] args) throws IOException {
        String userName="";
        String password="";
        String repository="";
        String outputType = "";
        IStoryFormatter formatter = new DummyFormatter();
        
        HashMap params = new HashMap();

        for (int i = 0; i < (2 * (args.length / 2)); i += 2) {
            if (i < args.length - 1) {
                params.put(args[i], args[i + 1]);
            }
        }

        if (params.containsKey(USER_NAME)) {
            userName = params.get(USER_NAME);
        }
        if (params.containsKey(USER_PASS)) {
            password = params.get(USER_PASS);
        }
        if (params.containsKey(REPO_NAME)) {
            repository = params.get(REPO_NAME);
        }
        if (params.containsKey(OUTPUT_TYPE)) {
            outputType = params.get(OUTPUT_TYPE);
        }
        .....
    }
}
Now we're ready to process the information from the issue tracker.

Implementing base work flow

Now we should go through all issues and report their information based on the formatter we use. The general flow is:

  • Connect to the repository
  • Get all issues both of open and closed status
  • Sort all issues based on Milestone id (thus we group them by milestones as well)
  • Write header
  • Loop through all issues and write appropriate information
  • Write footer
Additionally, I'd like to use only issues which contain Test label. So, let's do it step by step.

Connect to the repository

We have login and password provided as the command line paramenters so we connect to GitHub repository in the following way:

        GitHub client = GitHub.connectUsingPassword(userName, password);
        GHRepository repo = client.getMyself().getRepository(repository);
After that we have an access to all repository items.

Get all issues both of open and closed status

Unfortunately the API provides the ability to get issues by some specific status so I have to make 2 calls instead 1. Generally issues are retrieved with the following code:

        ArrayList issues = new ArrayList();
        issues.addAll(repo.getIssues(GHIssueState.OPEN));
        issues.addAll(repo.getIssues(GHIssueState.CLOSED));

Sort all issues based on Milestone id (thus we group them by milestones as well)

This is the task about sorting structures based on some field value. This is done in 2 steps:

  1. Create comparison class where the comparison criteria is defined:
        public class IssuesComparator implements Comparator {
    
            public IssuesComparator(){;}
            
            /* (non-Javadoc)
             * @see java.util.Comparator#compare(java.lang.Object, java.lang.Object)
             */
            public int compare(GHIssue o1, GHIssue o2) {
                int mileId1,mileId2;
                if(o1.getMilestone()==null){
                    mileId1 = 0;
                }
                else {
                   mileId1 = o1.getMilestone().getNumber();  
                }
                if(o2.getMilestone()==null){
                    mileId2 = 0;
                }
                else {
                   mileId2 = o2.getMilestone().getNumber();  
                }
                return mileId1 - mileId2;
            }
        }
    
  2. Sort array of issues based on the above criteria:
            Program p = new Program();
            IssuesComparator c = p.new  IssuesComparator();
            Collections.sort(issues, c);
    

Write header

    System.out.println(formatter.GetHeader(issues));
Nuff said.

Loop through all issues and write appropriate information

Milestone processing has it's own features. Firstly, if issue doesn't contain milestone defined the milestone should be created. Thus all it's attributes will be null. However, at least we won't have NullPointerException. Each milestone information is displayed only once per all issues within it.

Generally, the code looks like:

        int prevMilestoneId = -1;
        
        for(GHIssue issue:issues){
            GHMilestone milestone = issue.getMilestone();
            
            if(milestone==null){
                milestone = new GHMilestone();                
            }
            
            if(milestone.getNumber() != prevMilestoneId){
                prevMilestoneId = milestone.getNumber();
                System.out.println(formatter.GetMilestone(milestone));
            }
            
            if(issue.getLabels().contains("Test"))
            {
                System.out.println(formatter.GetIssue(issue));
            }
        }

Write footer

    System.out.println(formatter.GetFooter(issues));
Nuff said.

Before we go next

All the above mentioned code can be found here. The only thing I haven't described yet is the formatters initialization. If we copy the code as I described in the chapter all the output will be processed by DummyFormatter. But we have to use more specific formatters in the next chapters.

Automatically generate traceability matrix

Creating formatter

First thing we should do is to create the TraceabilityMatrixFormatter class which implements IStoryFormatter. And them in the main function we should add the following statement:

        if(outputType.equals("trace")){
            formatter = new TraceabilityMatrixFormatter();
        }
It should be placed before any formatter variable usage.

Since we have interface and base flow, we should implement the formatter methods. The formatter class looks like:

/**
 * 
 */
package sirius.utils.retriever.formatters;

import java.util.ArrayList;

import org.kohsuke.github.GHIssue;
import org.kohsuke.github.GHIssueState;
import org.kohsuke.github.GHMilestone;

import sirius.utils.retriever.interfaces.IStoryFormatter;

/**
 * @author Myk Kolisnyk
 *
 */
public class TraceabilityMatrixFormatter implements IStoryFormatter {

    public final String eol = System.getProperty("line.separator");
    
    /* (non-Javadoc)
     * @see sirius.utils.retriever.interfaces.IStoryFormatter#GetHeader(java.util.ArrayList)
     */
    public String GetHeader(ArrayList issues) {
        return "# Tests status" + eol + eol;                
    }

    /* (non-Javadoc)
     * @see sirius.utils.retriever.interfaces.IStoryFormatter#GetMilestone(org.kohsuke.github.GHMilestone)
     */
    public String GetMilestone(GHMilestone milestone) {
        return "## Feature: [" + milestone.getTitle() + "](" + milestone.getUrl() + ")" + eol +
            "```" + eol + milestone.getDescription() + eol + "```" + eol + eol +
           "| Group | Test | Completed |" + eol + 
           "|-------|------|-----------|";
    }

    /* (non-Javadoc)
     * @see sirius.utils.retriever.interfaces.IStoryFormatter#GetIssue(org.kohsuke.github.GHIssue)
     */
    public String GetIssue(GHIssue issue) {
        return "|" + GetLabels(issue) + " | [" + issue.getTitle() + 
                "](" + issue.getUrl() + ") | " +
            ((issue.getState().equals(GHIssueState.CLOSED))?("Yes"):("No")) + 
            "|";
    }

    /* (non-Javadoc)
     * @see sirius.utils.retriever.interfaces.IStoryFormatter#GetFooter(java.util.ArrayList)
     */
    public String GetFooter(ArrayList issues) {
        return "";
    }

    /* (non-Javadoc)
     * @see sirius.utils.retriever.interfaces.IStoryFormatter#GetLabels(org.kohsuke.github.GHIssue)
     */
    public String GetLabels(GHIssue issue) {
        String result = "";
        
        String[] labels = new String[issue.getLabels().size()];
        issue.getLabels().toArray(labels);
        
        for(int i=4;i<labels.length;i+=8){
            result += "[" + labels[i] + "](" + labels[i-2] + ") ";
        }
        
        return result;
    }
}
After that you can run command like:
java -jar sirius.utils.retriever.jar -r Sirius -u %GH_USER% -p %GH_PASS% -t trace
and if all credentials are proper you can get an output like:
# Tests status

## Feature: [Win32 Tab Control support](https://api.github.com/repos/mkolisnyk/Sirius/milestones/1)
```
As the system user
I want to be able to interact with Win32 Tab Control
```

| Group | Test | Completed |
|-------|------|-----------|
|[Test](https://api.github.com/repos/mkolisnyk/Sirius/labels/Test) [Win32](https://api.github.com/repos/mkolisnyk/Sirius/labels/Win32)  | [Win32 Tab Control support. Base Operations](https://github.com/mkolisnyk/Sirius/issues/2) | No|

Integrating formatter output with GitHub wiki pages

Unfortunately GitHub API doesn't contain code interacting with wiki pages. But there's another way to write to there. Wiki pages are stored in separate git repository which can be retrieved separately. To obtain GitHub wiki repository URL you should:

  • Navigate to project Wiki home page
  • Click on Clone URL button
After that the URL will appear in the clipboard. After that you can clone wiki repository into any location. Since it's now the file you can simply redirect the generator output to the required file and commit new changes. Generally it looks like:
java -jar sirius.utils.retriever.jar -r Sirius -u %GH_USER% -p %GH_PASS% -t trace > ../wiki/Traceability-Matrix.md

cd ../wiki/
git pull
git commit -a -m "Traceability matrix update."
git push
Thus we're making changes into the ../wiki/Traceability-Matrix.md file and push those changes to the repository. As the result you'll get newly generated page available at wiki. Something like Traceability Matrix page generated this way.

Automatically generate feature files based on the content of the test issues

Cucumber-like output is done in the same fasion. We have to add the formatter initialization:

        if(outputType.equals("cucumber")){
            formatter = new CucumberFormatter();
        }
After that we should implement another formatter adopter to Cucumber-like constructions:
/**
 * 
 */
package sirius.utils.retriever.formatters;

import java.util.ArrayList;

import org.kohsuke.github.GHIssue;
import org.kohsuke.github.GHIssueState;
import org.kohsuke.github.GHMilestone;

import sirius.utils.retriever.interfaces.IStoryFormatter;

/**
 * @author Myk Kolisnyk
 *
 */
public class CucumberFormatter implements IStoryFormatter {

    public final String eol = System.getProperty("line.separator");
    
    /* (non-Javadoc)
     * @see sirius.utils.retriever.interfaces.IStoryFormatter#GetHeader(java.util.ArrayList)
     */
    public String GetHeader(ArrayList issues) {
        return "";                
    }

    /* (non-Javadoc)
     * @see sirius.utils.retriever.interfaces.IStoryFormatter#GetMilestone(org.kohsuke.github.GHMilestone)
     */
    public String GetMilestone(GHMilestone milestone) {
        return "# " + milestone.getTitle() + eol; 
    }

    /* (non-Javadoc)
     * @see sirius.utils.retriever.interfaces.IStoryFormatter#GetIssue(org.kohsuke.github.GHIssue)
     */
    public String GetIssue(GHIssue issue) {
        return GetLabels(issue) + eol + "Feature: " + issue.getTitle() + eol +
            issue.getBody() + eol;
    }

    /* (non-Javadoc)
     * @see sirius.utils.retriever.interfaces.IStoryFormatter#GetFooter(java.util.ArrayList)
     */
    public String GetFooter(ArrayList issues) {
        return "";
    }

    /* (non-Javadoc)
     * @see sirius.utils.retriever.interfaces.IStoryFormatter#GetLabels(org.kohsuke.github.GHIssue)
     */
    public String GetLabels(GHIssue issue) {
        String result = "";
        
        String[] labels = new String[issue.getLabels().size()];
        issue.getLabels().toArray(labels);
        
        for(int i=4;i<labels.length;i+=8){
            result += "@" + labels[i] + " ";
        }
        
        return result;
    }
}
So, if we run the command like like:
java -jar sirius.utils.retriever.jar -r Sirius -u %GH_USER% -p %GH_PASS% -t cucumber
This command will produce output like:
# Win32 Tab Control support

@Test @Win32 
Feature: Win32 Tab Control support. Base Operations
Scenario: List All page names
  - [x] User should be able to get the list of all pages

  When I start GUI tests application
  Then I should see the following tabs:
    | Tab Name |
    | Static Text |
    | Edit Page |
    | Rich Text |
    | Buttons |
    | List Box |
    | Combo Box |
    | Scroll Bars |
    | Image Lists |
    | Progress Bar |
    | Sliders |
    | Spinners |
    | Headers |
This output can be redirected to any other file and further used by any BDD engine.

Summary

In this article I've described how to organize test tracking using GitHub issue tracker. Also, I've described how to automate some reports generation. And finally, I've made the way to integrate GitHub not only with JBehave as I did it before. Thus, we can have a unified way to perform the same set of tests under different platforms and different programming languages.

No comments:

Post a Comment