Running NUnit tests programmatically

I’m working on a test framework that needs to be run by less-technical testers. The tests are data driven from a spreadsheet (google docs spreadsheet API + gdata.)

Tests will be run locally (for now at least) since there isn’t a test lab available for remote execution, and no CI. I didn’t want to have to require users to install NUnit to execute tests.

At first I started by writing a main() class and rolling my own assertions. But I decided that the parameterized test features of NUnit were worth the effort of a little research. NUnit can, in fact, be run programmatically, though the execution appears less flexible than with other frameworks.

I created a TestRunner class with main function:


using System;
using NUnit.Core;
using NUnit.Framework.Constraints;
using NLog;

namespace oneshore.qa.testrunner
{
    class TestRunner
    {
        Logger log = NLog.LogManager.GetCurrentClassLogger();

        public static void Main(String[] args)
        {
            //get from command line args
            String pathToTestLibrary = "C:\\dev\\oneshore.Tests.DLL"; 

            DateTime startTime = System.DateTime.Now;
            log.Info("starting test execution...");

            TestRunner runner = new TestRunner();
            runner.run(pathToTestLibrary);

            log.Info("...test execution finished");
            DateTime finishTime = System.DateTime.Now;
            TimeSpan elapsedTime = finishTime.Subtract(startTime);
            log.Info("total elapsed time: " + elapsedTime);
        }

        public void run(String pathToTestLibrary)
        {
            CoreExtensions.Host.InitializeService();
            TestPackage testPackage = new TestPackage(@pathToTestLibrary);
            testPackage.BasePath = Path.GetDirectoryName(pathToTestLibrary);
            TestSuiteBuilder builder = new TestSuiteBuilder();
            TestSuite suite = builder.Build(testPackage);
            TestResult result = suite.Run(new NullListener(), TestFilter.Empty);

            log.Debug("has results? " + result.HasResults);
            log.Debug("results count: " + result.Results.Count);
            log.Debug("success? " + result.IsSuccess);
        }
    }
}

Link to gist of this code.

The advantage to running tests this way is that NUnit does not need to be installed (though DLLs for NUnit — nunit.core.dll & nunit.core.interfaces.dll — and any other dependencies like NLog need to be packaged with the TestRunner.) You can still write and execute your tests from NUnit while under development.

One disadvantage is that you don’t have the full test results by using the TestSuiteBuilder to bundle every test it finds into one suite. I’d like to find a way to improve that. You also can’t run more than one test assembly at the same time — you can create a nunit project xml for that — and at that point you might as well bundle the nunit test framework.

For now, my base test class (that each of my Nunit tests inherit from) reports via catching and counting assertion failures and writing to a log file. It can then use the Quality Center integration tool I described in an earlier post, though I’m planning on wiring it all together eventually, so anyone can run tests automatically by clicking on an icon, using a File Picker dialog to select the test library (see upcoming post) and have test results entered in QC.

This will allow distributed parameterized testing that can be done by anyone. I may try to set up a web UI like fitnesse for data driven tests as well.

Updating test results in QC using the QC OTA API explained

Yesterday I cleaned up and posted my example QCIntegration utility on GitHub.

While it works as a standalone tool, some people might not want to wade through the code to understand or modify it. So today, I’m going to try to explain how the OTA API works by recreating the steps as a blog post with explanation in a simple script.

I’ll start with an example using C# and then give an equivalent Python example. I’ll use the same scenario, updating test case results in QC, but if requested, I can also show how to get test steps from a test plan, or read & update defects in QC using the OTA library.

First, create a new project in Visual Studio (or SharpDevelop). You’ll need to add the OTAClient.dll as a reference. It is a COM library and contains the single interface TDConnection.

When searching for the library name it is called the “OTA COM Type Library”. The package is “TDApiOle80.” Since it is a COM library, it needs to use an interop for C#, but this is handled automatically by the IDE.

using TDAPIOLELib;
TDConnection conn = new TDConnection();

Now, let’s create a connection to your Quality Center server. You’ll need to know the URL of your QC Server and have valid login credentials with access to an existing Domain and Project.

Assuming you have quality center installed on your local machine (not a typical setup) you might have the following setup:

string qcUrl = "http://localhost:8080/qcbin";
string qcDomain = "oneshore";
string qcProject = "qa-site";
string qcLoginName = "aaron";
string qcPassword = "secret";

Note: I do not use this same password for my bank account

There are several ways to log in, but I’ll use the simplest here:

tdConn.InitConnectionEx(qcUrl);
tdConn.ConnectProjectEx(qcDomain, qcProject, qcLoginName, qcPassword);

Now you need to find your test sets that need updated. I typically use folder structure that goes something like:

Project – Iteration – Component – Feature

It’s a bit convoluted but here’s the code to get a testSet:

string testFolder = "Root\QASite\Sprint5\Dashboard\Recent Updates";
string testSet = "Recent Updates - New Defects Logged";

TestSetFactory tsFactory = (TestSetFactory)tdConn.TestSetFactory;
TestSetTreeManager tsTreeMgr = (TestSetTreeManager)tdConn.TestSetTreeManager;
TestSetFolder tsFolder = (TestSetFolder)tsTreeMgr.get_NodeByPath(testFolder);
List tsList = tsFolder.FindTestSets(testSetName, false, null);

The parameters for FindTestSets are a pattern to match, whether to match case, and a filter. Since I’m looking for a specific test set, I don’t bother with the other two parameters.
You could easily get a list of all test sets that haven’t been executed involving the recent updates feature by substituting this line:

List tsList = tsFolder.FindTestSets("recent updates", true, "status=No Run");

Now we want to loop through the test set and build a collection of tests to update. Note that we might have more than one test set in the folder and one or more subfolders as well:

foreach (TestSet testSet in tsList)
{
	TestSetFolder tsFolder = (TestSetFolder)testSet.TestSetFolder;
	TSTestFactory tsTestFactory = (TSTestFactory)testSet.TSTestFactory;
	List tsTestList = tsTestFactory.NewList("");

And finally, update each test case status:

    foreach (TSTest tsTest in tsTestList)
    {
        Run lastRun = (Run)tsTest.LastRun;

        // don't update test if it may have been modified by someone else
        if (lastRun == null)
        {
            RunFactory runFactory = (RunFactory)test.RunFactory;
            String date = DateTime.Now.ToString("yyyyMMddhhmmss");
            Run run = (Run)runFactory.AddItem("Run" + date);
            run.Status = "Pass";
            run.Post();
        }
    } // end loop of test cases

} // end outer loop of test sets

Of course you might want to add your actual test results. If you have a dictionary of test names and statuses, you can simply do this:

Dictionary testResults = new Dictionary();
testResults.Add("New defects in Recent Updates are red", "Pass");
testResults.Add("Resolved defects in Recent Updates are green", "Pass");
testResults.Add("Reopened defects in Recent Updates are bold", "Fail");

if (testResults.ContainsKey(tsTest.TestName))
{
    string status = testResults[tsTest.TestName];
    recordTestResult(tsTest, status);
}

That’s all for now. I’ll translate the example into Python tomorrow, but you’ll see it’s really quite straightforward.

Links: Unit Testing and Continous Integration with Flex and AsUnit

Just a bunch of links to tutorials on using AsUnit and continuous integration with Flex Projects:

A post on AsUnit by one of it’s creators,  Luke Baye’s:
http://asserttrue.com/articles/2006/03/10/AsUnit25

An example of a simple TestRunner mxml (AS2):
http://asserttrue.com/articles/2006/10/05/flex2-mxml-project-support-in-asunit

Luke’s post on continous integration:
http://lukebayes.blogspot.com/2005/11/continuous-integration-with-asunit.html

A good tutorial about using AsUnit (but with only a Flash testRunner):
http://www.insideria.com/2008/09/unit-testing-with-asunit.html

Another good tutorial about using AsUnit:
http://www.insideria.com/2008/05/anatomy-of-an-enterprise-flex-10.html

Discussion of one team’s unit test framework requirements:
http://www.eyefodder.com/blog/2006/06/unit_test_frameworks_for_as3_a.shtml

Weaknesses and strengths of FlexUnit and AsUnit:
http://www.eyefodder.com/blog/2006/07/flexunit_asunit_deathmatch_res.shtml

Story of their use of Continuous Integration:
http://www.eyefodder.com/blog/continuous_integration/
http://www.eyefodder.com/blog/2006/05/continuous_integration_with_fl.shtml
http://www.eyefodder.com/blog/2006/05/continuous_integration_with_fl_1.shtml
http://www.eyefodder.com/blog/2006/05/continuous_integration_with_fl_2.shtml
http://www.eyefodder.com/blog/2006/05/continuous_integration_with_fl_3.shtml
http://www.eyefodder.com/blog/2006/05/continuous_integration_with_fl_4.shtml
http://www.eyefodder.com/blog/2006/05/continuous_integration_with_fl_5.shtml
http://www.eyefodder.com/blog/2006/05/continuous_integration_with_fl_6.shtml

A flash-oriented tutorial, but with good AsUnit explanations:
http://marstonstudio.com/2007/07/28/asunit-testing-with-flash-cs3-and-actionscript-3/

Testing tutorials to write

Here are the testing tutorials I’m going to work on:

Selenium
Fit & Fitnesse
Watir
Customizing Bugzilla
Bugzilla features: web services and email_in
TestNG vs. Junit
PHP SimpleTest
Perl Test::Class & WWW::Mechanize
Test Planning, Test Cases & Defects