GitHub provides some tracker where we can keep our tasks. The GitHub API provides the ability to access that resources programmatically. Thus, with the help of API I could integrate JBehave and GitHub so that I can run JBehave tests stored in GitHub tracker. But it's not the only case.
With the help of API I can set the correspondence between tests and features. This gives the ability to make the traceability of tests and features they're applied for. Also, what if we use something different from JBehave, e.g. Cucumber, Sprecflow which use flat files for input? In this article I'll describe how to:
- Organize tests with GitHub tracker
- Automatically generate traceability matrix
- Automatically generate feature files based on the content of the test issues
Organize tests with GitHub tracker
What do we have initially?
First thing we should realize in GitHub tracker is that it is quite simple tracker which is mainly designed to track bugs/tasks but not various complicated resources. That's why there's no different issue types. There's only one entiry: issue.
We cannot distribute issues by types but we still can mark them in a specific way. For this purpose there's labelling functionality. We can setup different labels and use them for indicating different issue types and/or application area.
Another thing is that all issues are entities of the same hierarchial level. It means that you cannot create task and make some sub-tasks. At least explicitly. We can make some implicit check-points inside the task (I'll touch this point later) but it's not the same as you can do via more or less complicated tracking systems. Nevertheless, all tasks can be split by some general group of features which have pre-defined timelines. This groupping entity is called milestone. It can be used for issues groupping.
Organize structure
Based on the above points we can identify the following rules for issues tracking:
- Milestone is the highest level object representing some specific feature. This item should contain feature description
- Each specific issue which is related to some feature should be linked to appropriate milestone
- In order to filter out tests from development tasks or bugs we can use labels
Fill-in the content
For this example I've created Win32 Tab Control support milestone and added some small description like:
As the system user I want to be able to interact with Win32 Tab ControlThat would be the place-holder for tests.
For better distribution between tests and other task types I'll add the following labels:
- Test - used for the issues containing test scenarios
- Bug - used for storing bug information
- DevTask - used for the issues containing development tasks related to each specific feature
Once we're done with the labelling we can create new issue and fill it in with the test scenario, e.g.:
Scenario: List All page names
- [ ] User should be able to get the list of all pages
When I start GUI tests application
Then I should see the following tabs:
| Tab Name |
| Static Text |
| Edit Page |
| Rich Text |
| Buttons |
| List Box |
| Combo Box |
| Scroll Bars |
| Image Lists |
| Progress Bar |
| Sliders |
| Spinners |
| Headers |
| NOTE |
There's one additional line here:
- [ ] User should be able to get the list of all pageswhich is rather GitHub wiki markup feature than something that needed for test scenario. Is GitHub issue contains text like that it's interpreted as a check-list item. So, you can split task into smaller sub-tasks and mark their completion inside the issue. But for test scenario it doesn't make effect. |
Prepare the code generator
Before we start generating various resources we should identify how it can be done and what should be generated.
Retrieval interface
I'm going to prepare the utility which should be able to generate traceability matrix and feature file based on GitHub issues structure. In both cases the output should contain the following requisites:
- Header - contains the most common information. Usually overview of the issues
- Milestone information - general information which is taken from milestone
- Issue information - contains information taken from issues
- Labels information - puts information based on labels
- Footer - contains some disclaimer or some other summaryzing information
/**
*
*/
package sirius.utils.retriever.interfaces;
import java.util.ArrayList;
import org.kohsuke.github.GHIssue;
import org.kohsuke.github.GHMilestone;
/**
* @author Myk Kolisnyk
*
*/
public interface IStoryFormatter {
public String GetHeader(ArrayList issues);
public String GetMilestone(GHMilestone milestone);
public String GetIssue(GHIssue issue);
public String GetLabels(GHIssue issue);
public String GetFooter(ArrayList issues);
}
and for now we'll create some dummy formatter implementing this interface:
/**
*
*/
package sirius.utils.retriever.formatters;
import java.util.ArrayList;
import org.kohsuke.github.GHIssue;
import org.kohsuke.github.GHMilestone;
import sirius.utils.retriever.interfaces.IStoryFormatter;
/**
* @author Myk Kolisnyk
*
*/
public class DummyFormatter implements IStoryFormatter {
/* (non-Javadoc)
* @see sirius.utils.retriever.interfaces.IStoryFormatter#GetHeader(java.util.ArrayList)
*/
public String GetHeader(ArrayList issues) {
// TODO Auto-generated method stub
return null;
}
/* (non-Javadoc)
* @see sirius.utils.retriever.interfaces.IStoryFormatter#GetMilestone(org.kohsuke.github.GHMilestone)
*/
public String GetMilestone(GHMilestone milestone) {
// TODO Auto-generated method stub
return null;
}
/* (non-Javadoc)
* @see sirius.utils.retriever.interfaces.IStoryFormatter#GetIssue(org.kohsuke.github.GHIssue)
*/
public String GetIssue(GHIssue issue) {
// TODO Auto-generated method stub
return null;
}
/* (non-Javadoc)
* @see sirius.utils.retriever.interfaces.IStoryFormatter#GetFooter(java.util.ArrayList)
*/
public String GetFooter(ArrayList issues) {
// TODO Auto-generated method stub
return null;
}
/* (non-Javadoc)
* @see sirius.utils.retriever.interfaces.IStoryFormatter#GetLabels(org.kohsuke.github.GHIssue)
*/
public String GetLabels(GHIssue issue) {
// TODO Auto-generated method stub
return null;
}
}
This code is purely generated.
Adding command line parameters
It is convenient to provide command line interface. It can be reworked as the Maven plugin but for now it's enough to have just command line utility. As an input we need the following information:
- GitHub username
- GitHub password
- GitHub repository name
- Output type - the flag identifying whether we produce traceability matrix or something else
java -jar sirius.utils.retriever-<version>.jar -r <repository> -u <user> -p <password> -t <report_type>Having this information we can start creating main program part. We'll start with the main program class which gets all those parameters:
/**
*
*/
package sirius.utils.retriever;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashMap;
import java.util.LinkedList;
import java.util.List;
import org.kohsuke.github.GHIssue;
import org.kohsuke.github.GHIssueState;
import org.kohsuke.github.GHMilestone;
import org.kohsuke.github.GHRepository;
import org.kohsuke.github.GitHub;
import sirius.utils.retriever.formatters.CucumberFormatter;
import sirius.utils.retriever.formatters.DummyFormatter;
import sirius.utils.retriever.formatters.TraceabilityMatrixFormatter;
import sirius.utils.retriever.interfaces.IStoryFormatter;
/**
* @author Myk Kolisnyk
*
*/
public class Program {
public static final String REPO_NAME="-r";
public static final String USER_NAME="-u";
public static final String USER_PASS="-p";
public static final String OUTPUT_TYPE="-t";
/**
* @param args
* @throws IOException
*/
public static void main(String[] args) throws IOException {
String userName="";
String password="";
String repository="";
String outputType = "";
IStoryFormatter formatter = new DummyFormatter();
HashMap params = new HashMap();
for (int i = 0; i < (2 * (args.length / 2)); i += 2) {
if (i < args.length - 1) {
params.put(args[i], args[i + 1]);
}
}
if (params.containsKey(USER_NAME)) {
userName = params.get(USER_NAME);
}
if (params.containsKey(USER_PASS)) {
password = params.get(USER_PASS);
}
if (params.containsKey(REPO_NAME)) {
repository = params.get(REPO_NAME);
}
if (params.containsKey(OUTPUT_TYPE)) {
outputType = params.get(OUTPUT_TYPE);
}
.....
}
}
Now we're ready to process the information from the issue tracker.
Implementing base work flow
Now we should go through all issues and report their information based on the formatter we use. The general flow is:
- Connect to the repository
- Get all issues both of open and closed status
- Sort all issues based on Milestone id (thus we group them by milestones as well)
- Write header
- Loop through all issues and write appropriate information
- Write footer
Connect to the repository
We have login and password provided as the command line paramenters so we connect to GitHub repository in the following way:
GitHub client = GitHub.connectUsingPassword(userName, password);
GHRepository repo = client.getMyself().getRepository(repository);
After that we have an access to all repository items.
Get all issues both of open and closed status
Unfortunately the API provides the ability to get issues by some specific status so I have to make 2 calls instead 1. Generally issues are retrieved with the following code:
ArrayList issues = new ArrayList();
issues.addAll(repo.getIssues(GHIssueState.OPEN));
issues.addAll(repo.getIssues(GHIssueState.CLOSED));
Sort all issues based on Milestone id (thus we group them by milestones as well)
This is the task about sorting structures based on some field value. This is done in 2 steps:
- Create comparison class where the comparison criteria is defined:
public class IssuesComparator implements Comparator{ public IssuesComparator(){;} /* (non-Javadoc) * @see java.util.Comparator#compare(java.lang.Object, java.lang.Object) */ public int compare(GHIssue o1, GHIssue o2) { int mileId1,mileId2; if(o1.getMilestone()==null){ mileId1 = 0; } else { mileId1 = o1.getMilestone().getNumber(); } if(o2.getMilestone()==null){ mileId2 = 0; } else { mileId2 = o2.getMilestone().getNumber(); } return mileId1 - mileId2; } } - Sort array of issues based on the above criteria:
Program p = new Program(); IssuesComparator c = p.new IssuesComparator(); Collections.sort(issues, c);
Write header
System.out.println(formatter.GetHeader(issues));
Nuff said.
Loop through all issues and write appropriate information
Milestone processing has it's own features. Firstly, if issue doesn't contain milestone defined the milestone should be created. Thus all it's attributes will be null. However, at least we won't have NullPointerException. Each milestone information is displayed only once per all issues within it.
Generally, the code looks like:
int prevMilestoneId = -1;
for(GHIssue issue:issues){
GHMilestone milestone = issue.getMilestone();
if(milestone==null){
milestone = new GHMilestone();
}
if(milestone.getNumber() != prevMilestoneId){
prevMilestoneId = milestone.getNumber();
System.out.println(formatter.GetMilestone(milestone));
}
if(issue.getLabels().contains("Test"))
{
System.out.println(formatter.GetIssue(issue));
}
}
Write footer
System.out.println(formatter.GetFooter(issues));
Nuff said.
Before we go next
All the above mentioned code can be found here. The only thing I haven't described yet is the formatters initialization. If we copy the code as I described in the chapter all the output will be processed by DummyFormatter. But we have to use more specific formatters in the next chapters.
Automatically generate traceability matrix
Creating formatter
First thing we should do is to create the TraceabilityMatrixFormatter class which implements IStoryFormatter. And them in the main function we should add the following statement:
if(outputType.equals("trace")){
formatter = new TraceabilityMatrixFormatter();
}
It should be placed before any formatter variable usage.
Since we have interface and base flow, we should implement the formatter methods. The formatter class looks like:
/**
*
*/
package sirius.utils.retriever.formatters;
import java.util.ArrayList;
import org.kohsuke.github.GHIssue;
import org.kohsuke.github.GHIssueState;
import org.kohsuke.github.GHMilestone;
import sirius.utils.retriever.interfaces.IStoryFormatter;
/**
* @author Myk Kolisnyk
*
*/
public class TraceabilityMatrixFormatter implements IStoryFormatter {
public final String eol = System.getProperty("line.separator");
/* (non-Javadoc)
* @see sirius.utils.retriever.interfaces.IStoryFormatter#GetHeader(java.util.ArrayList)
*/
public String GetHeader(ArrayList issues) {
return "# Tests status" + eol + eol;
}
/* (non-Javadoc)
* @see sirius.utils.retriever.interfaces.IStoryFormatter#GetMilestone(org.kohsuke.github.GHMilestone)
*/
public String GetMilestone(GHMilestone milestone) {
return "## Feature: [" + milestone.getTitle() + "](" + milestone.getUrl() + ")" + eol +
"```" + eol + milestone.getDescription() + eol + "```" + eol + eol +
"| Group | Test | Completed |" + eol +
"|-------|------|-----------|";
}
/* (non-Javadoc)
* @see sirius.utils.retriever.interfaces.IStoryFormatter#GetIssue(org.kohsuke.github.GHIssue)
*/
public String GetIssue(GHIssue issue) {
return "|" + GetLabels(issue) + " | [" + issue.getTitle() +
"](" + issue.getUrl() + ") | " +
((issue.getState().equals(GHIssueState.CLOSED))?("Yes"):("No")) +
"|";
}
/* (non-Javadoc)
* @see sirius.utils.retriever.interfaces.IStoryFormatter#GetFooter(java.util.ArrayList)
*/
public String GetFooter(ArrayList issues) {
return "";
}
/* (non-Javadoc)
* @see sirius.utils.retriever.interfaces.IStoryFormatter#GetLabels(org.kohsuke.github.GHIssue)
*/
public String GetLabels(GHIssue issue) {
String result = "";
String[] labels = new String[issue.getLabels().size()];
issue.getLabels().toArray(labels);
for(int i=4;i<labels.length;i+=8){
result += "[" + labels[i] + "](" + labels[i-2] + ") ";
}
return result;
}
}
After that you can run command like:
java -jar sirius.utils.retriever.jar -r Sirius -u %GH_USER% -p %GH_PASS% -t traceand if all credentials are proper you can get an output like:
# Tests status ## Feature: [Win32 Tab Control support](https://api.github.com/repos/mkolisnyk/Sirius/milestones/1) ``` As the system user I want to be able to interact with Win32 Tab Control ``` | Group | Test | Completed | |-------|------|-----------| |[Test](https://api.github.com/repos/mkolisnyk/Sirius/labels/Test) [Win32](https://api.github.com/repos/mkolisnyk/Sirius/labels/Win32) | [Win32 Tab Control support. Base Operations](https://github.com/mkolisnyk/Sirius/issues/2) | No|
Integrating formatter output with GitHub wiki pages
Unfortunately GitHub API doesn't contain code interacting with wiki pages. But there's another way to write to there. Wiki pages are stored in separate git repository which can be retrieved separately. To obtain GitHub wiki repository URL you should:
- Navigate to project Wiki home page
- Click on Clone URL button
java -jar sirius.utils.retriever.jar -r Sirius -u %GH_USER% -p %GH_PASS% -t trace > ../wiki/Traceability-Matrix.md cd ../wiki/ git pull git commit -a -m "Traceability matrix update." git pushThus we're making changes into the ../wiki/Traceability-Matrix.md file and push those changes to the repository. As the result you'll get newly generated page available at wiki. Something like Traceability Matrix page generated this way.
Automatically generate feature files based on the content of the test issues
Cucumber-like output is done in the same fasion. We have to add the formatter initialization:
if(outputType.equals("cucumber")){
formatter = new CucumberFormatter();
}
After that we should implement another formatter adopter to Cucumber-like constructions:
/**
*
*/
package sirius.utils.retriever.formatters;
import java.util.ArrayList;
import org.kohsuke.github.GHIssue;
import org.kohsuke.github.GHIssueState;
import org.kohsuke.github.GHMilestone;
import sirius.utils.retriever.interfaces.IStoryFormatter;
/**
* @author Myk Kolisnyk
*
*/
public class CucumberFormatter implements IStoryFormatter {
public final String eol = System.getProperty("line.separator");
/* (non-Javadoc)
* @see sirius.utils.retriever.interfaces.IStoryFormatter#GetHeader(java.util.ArrayList)
*/
public String GetHeader(ArrayList issues) {
return "";
}
/* (non-Javadoc)
* @see sirius.utils.retriever.interfaces.IStoryFormatter#GetMilestone(org.kohsuke.github.GHMilestone)
*/
public String GetMilestone(GHMilestone milestone) {
return "# " + milestone.getTitle() + eol;
}
/* (non-Javadoc)
* @see sirius.utils.retriever.interfaces.IStoryFormatter#GetIssue(org.kohsuke.github.GHIssue)
*/
public String GetIssue(GHIssue issue) {
return GetLabels(issue) + eol + "Feature: " + issue.getTitle() + eol +
issue.getBody() + eol;
}
/* (non-Javadoc)
* @see sirius.utils.retriever.interfaces.IStoryFormatter#GetFooter(java.util.ArrayList)
*/
public String GetFooter(ArrayList issues) {
return "";
}
/* (non-Javadoc)
* @see sirius.utils.retriever.interfaces.IStoryFormatter#GetLabels(org.kohsuke.github.GHIssue)
*/
public String GetLabels(GHIssue issue) {
String result = "";
String[] labels = new String[issue.getLabels().size()];
issue.getLabels().toArray(labels);
for(int i=4;i<labels.length;i+=8){
result += "@" + labels[i] + " ";
}
return result;
}
}
So, if we run the command like like:
java -jar sirius.utils.retriever.jar -r Sirius -u %GH_USER% -p %GH_PASS% -t cucumberThis command will produce output like:
# Win32 Tab Control support
@Test @Win32
Feature: Win32 Tab Control support. Base Operations
Scenario: List All page names
- [x] User should be able to get the list of all pages
When I start GUI tests application
Then I should see the following tabs:
| Tab Name |
| Static Text |
| Edit Page |
| Rich Text |
| Buttons |
| List Box |
| Combo Box |
| Scroll Bars |
| Image Lists |
| Progress Bar |
| Sliders |
| Spinners |
| Headers |
This output can be redirected to any other file and further used by any BDD engine.
Summary
In this article I've described how to organize test tracking using GitHub issue tracker. Also, I've described how to automate some reports generation. And finally, I've made the way to integrate GitHub not only with JBehave as I did it before. Thus, we can have a unified way to perform the same set of tests under different platforms and different programming languages.
No comments:
Post a Comment