testdriven.com Wrangling quality out of chaos

Introduction

Perforce Software, founded in 1995, markets and supports Perforce, the Fast Software Configuration Management (SCM) system. Perforce is an award winning SCM system used to version and manage source code and all digital assets. Perforce SCM seamlessly handles distributed development and multi-platform environments and is used by more than 330,000 developers worldwide.

Recently, Perforce adopted Squish for GUI performance testing of their SCM system’s flagship GUI, the Perforce Visual Client (P4V). We discussed Perforce’s use of Squish with Tim Brazil, a Perforce Performance Lab engineer.

Why Squish?

In mid-2009 Tim Brazil chose Squish to round out his testing arsenal and to help ensure that performance remains high as new features and bugfixes are applied to P4V. P4V, written in C++ and using the Qt GUI library, provides a graphical interface on Windows, Mac OS X, Linux, Solaris, and FreeBSD.

One feature in particular Tim liked was Squish’s support for multiple standard scripting languages for writing tests—these currently include JavaScript, Perl, Python, and Tcl. Using a standard scripting language avoids the need for testers to learn an obscure proprietary testing language, and means they can benefit from the large number of third party modules available for standard scripting languages.

Squish provides a complete GUI-based testing IDE capable of recording and playing back GUI tests, but for Tim, it was also Squish’s support for command line use that was particularly appealing.

"I prefer to work in a command-line environment that facilitates the use of scripting languages," Tim explained. "It is evident that froglogic’s approach took engineers like myself into account when they designed the product. For example, features like the envars file, suite.conf file, squishserver, and squishrunner, allow me to design a fairly complex test environment with relatively minimal work."

Tim has many years of experience in the testing field and is a strong proponent of using both automated and manual testing.

"I believe the best software testing solution is to use a good mix of automation and manual tools," said Tim. "By utilizing the strengths of both, producing superior products can be achieved."

"The benefits of automated testing include reliability, repeatability, comprehensiveness, and speed," Tim continues. "Furthermore, automated tests can quickly and dependably navigate through test scenarios and are, therefore, more likely to uncover subtle timing problems." He pointed out that manual testing alone is often both slow and fallible, and that even with a comprehensive test plan at hand it can be difficult to reliably recreate the exact test actions and environment.

Squish at Perforce

Tim uses Squish to test the performance of P4V, specifically on Windows Professional, Mac OS X, and SLED (SUSE Linux Enterprise Desktop). Squish is used to test the nightly build as well as the previous three P4V releases. A daily performance report is generated, identifying performance trends and pinpointing any possible areas of concern as P4V continually evolves.

Each test Tim uses is designed to work independently—if one test fails, it has no effect on subsequent tests. No third party tools are used; instead, a custom Perl script (runner.pl) runs the Squish tests. This script runs each set of Squish P4V tests for each version of P4V on each client machine, accumulating results into an XML file (see the screenshot below). Once the tests are complete, the XML results file is automatically parsed and converted into an HTML report for the test team’s review each morning.

Squish in Practice

We asked Tim what features of Squish he liked most, apart from the multiple scripting language and command-line tools support he’d already mentioned.

Squish’s Object Map came to mind: "At previous companies I have used graphical test tools that were heavily invested in using coordinates to identify objects. This was a maintenance nightmare. Squish’s objects.map is unique and greatly facilitates test readability, robustness, and maintenance."

Squish uniquely identifies application objects such as widgets, using the values of their properties. For every identified object, Squish also creates a corresponding symbolic name—the symbolic name is the one that is normally used inside test scripts. This means that if a developer changes one of an object’s properties, the test engineer only has to update the corresponding object property once in the Object Map to reflect the change. The symbolic name used for the object will continue to work in all the tests where it is used.

In some cases, an object property’s value may vary depending on the platform the application is being run on. Squish’s Object Map can accommodate such challenges since the properties used to identify an object can not only be matched for equality, but also using wildcard or regular expression matches—a feature that Tim has found to be particularly useful.

"One of the biggest challenges I’ve faced with automated GUI testing is the ability to maintain a robust test as the product changes and evolves over time," Tim continues. "This is a problem because GUI application performance can vary on different operating systems. One particular issue we’ve had is being able to identify when a web page or GUI application object is ready to be tested, since load times vary. The wait*() functions Squish provides have been invaluable in helping me tune a test so that it provides consistent behavior when run on various platforms."

In addition to Squish’s documented features, Tim has found Squish’s technical support team very helpful: "I was impressed with froglogic’s support from the moment I started to evaluate Squish. Just because something isn’t stated in the documentation does not mean it cannot be done. Squish’s technical support team has quickly and efficiently provided solutions that work."

Conclusion

Perforce’s Performance Lab depends on the reliability and repeatability of Squish tests to check application performance as well as behavior across multiple platforms. Squish’s usefulness and flexibility has allowed Perforce to rapidly adopt Squish as an integral part of their performance quality monitoring process. This has lead to time and cost savings compared with the previous manual testing, and at the same time ensured that tests are automatically and reliably repeated to ensure product quality.

Making a third party tool an integral part of P4V’s testing process is a significant commitment, but one that Perforce has chosen to make with Squish.

froglogic’s team would like to thank Tim for taking the time to share Perforce’s experience with Squish, and we are looking forward to a continued successful relationship.

More about Squish and froglogic at http://www.froglogic.com

This article shows how to remove duplication from unit tests.

http://www.sqazone.net/modules/news/article.php?storyid=396

This is the story of how I got interested in the MVVM pattern. Not surprisingly, it is all about testing.

http://www.houseofbilz.com/archive/2009/05/27/adventures-in-mvvm-ndash-a-testing-story.aspx

No matter what your opinion is about singletons (I favor against them in most cases), it is important to understand what you can do when you encounter code that is dependent upon them. Especially when it comes to testing.

http://www.houseofbilz.com/archive/2009/05/02/what-to-do-about-those-pesky-singletons.aspx

I was recently asked by someone on Twitter to describe my setup for writing tests against my Silverlight application. I thought I would write them down here in case others might find this information useful.

http://www.houseofbilz.com/archive/2009/03/28/my-silverlight-testing-rig.aspx

System Configurations + Code Revisions = Continuous Integration FTW!

Up to this point, we have tested the service and we have tested the client — both in isolation. We have written unit tests and our code has good coverage. Unfortunately, my clients are not always synchronous. In Silverlight client, for instance, the framework will not permit you to make synchronous service requests. As it turns out, writing tests for asynchronous service clients is not straight-forward. Thankfully, there are some hacks that you can take advantage of to write effective asynchronous tests.

Testing WCF Service Applications (3 of 4) — Mocking the Asynchronous Service

So far, I outlined how to test your WCF service. I simply took advantage or the WCF architecture and tested the service directly outside of the actual service harness. Now I need to set my sights on the client. This becomes a bit more difficult, but I wouldn’t say that it is necessarily hard. I will start by giving a typical textbook example of hooking up to our service, and then I will tell you what is wrong with it. I will continue by modifying the code to be more testable so that the service can be mocked.

Testing WCF Service Applications (Part 2 of 4) — Mocking the Service

It turns out that TDD principles also apply for engineering the space shuttle’s main engine.

This article quotes a report that compares the bottom-up and the top-down approaches for designing jet/rocket engines. It turns out that the bottom-up approach is quite similar to TDD, and likewise successful.

One of the most beautiful things about the WCF framework is the way it was designed to be more testable than ASPX services. When you design your WCF interface, you are mostly just designing an interface with the WCF ServiceContract attributes. The WCF framework uses your interface to determine the actual contract and transport mechanism so you don’t have to.

Read More: Testing WCF Service Applications (1 of 4)

Squish is the leading automated GUI testing tool supporting applications based on cross-platform GUI technologies such as Java Swing/AWT, Java SWT/Eclipse RCP, C++/Qt, Web/HTML/Ajax and more. Squish is renowned for its dedicated toolkit support, use of open scripting languages, great flexibility, and robust test creation and execution.

While the development of the upcoming Squish 4.0 is on-going, Qtrac Ltd.’s Mark Summerfield talked with some of the people behind the product.

In this first interview, Mark asked froglogic’s founders, Harri Porten and Reginald Stadlbauer, to give an overview of Squish 4.0′s features. In the following interviews Mark will have deeper technical discussions with the responsible developers working on specific features.

Read the interview at http://blog.froglogic.com/2008/10/squish-40-interview-talking-with-froglogics-founders/

Although Microsoft will claim that it is "not possible to have a memory leak in managed code", most seasoned .NET developers will laugh at that statement. It turns out that it is very easy to leak memory — just keep a referencing object around longer than the referenced object, and you can leak.

Here is a way to write tests to catch leaks.

http://brian.genisio.org/2008/11/writing-tests-to-catch-memory-leaks-in.html

It’s been over a year now since I have been developing using TDD (Test Driven Development) as my primary development practice. I wanted to reflect on what it has done for me professionally. In reality, the past year has been great for my professional career in many ways.

My Year With TDD

Today I’d like to write about some of Squish’s features which are not so well known but which makes Squish a really powerful GUI testing solution. Squish is a cross-platform, automated GUI testing tool with dedicated support for applications based on Java Swing, AWT, SWT, Eclipse RCP, Web/HTML/AJAX, Qt and many more.Extensibility: When we started to create Squish, our main focus was to support all standard components of the supported GUI technologies (Qt, Swing, SWT, etc.) very well. But soon we found that in nearly every GUI application custom components are used. These are often complex and interactive controls visualizing data in some way. While basic support for all custom components can be provided out-of-the-box (and is provided that way by Squish), there is no way a GUI testing tool can know all internals (such as internal objects) of such custom controls and support that in a robust way.

This made us think about extension capabilities. The first Squish edition where we offered an extension API was Squish for Web- the automted GUI testing tool for web applications. The reason for that is probably that esp. in the world of AJAX GUIs nearly all controls used are custom controls.

This extension allows users to implement a few functions in JavaScript to let Squish know how to query internal information of custom AJAX controls. That way every custom AJAX control can be supported equally well as standard HTML and AJAX controls already known to Squish.

Later we introduced a similar mechanism for our Squish for Java tool, which features automated testing of Java GUIs such as Swing, AWT, SWT and Eclipse RCP. Here we provide a few Java interfaces which you need to implement for your custom Java controls to allow Squish to drill down into the details of your custom controls.

Since these extensions solve many problems of GUI testing, we are now introducing similar extension APIs for our Squish for Qt edition. Also our new Squish edition for testing native Windows applications, which will be part of the upcoming Squish 4.0, will be completely built on this extension mechanism even internally.

Integratability: Many vendors of testing tools want to completely lock their users into their world. So they come with the complete tool set needed to automate the QA. We early realized that GUI testing is just one part of the whole QA process and also saw that most of our users already have automation systems in place for their unit testing. So they usually prefer to integrate Squish into their existing automation system instead of setting up a completely separate automation just for GUI tests.

So we decided to focus on making Squish easy to integrate instead of developing our own test management solution. To allow that we provided easy-to-use command line tools to start and control Squish test runs and generate an easy-to-process XML report from any test run. This allows for easy integration.

In addition we now also offer ready-made integrations for CruiseControl, Maven, Ant, Eclipse TPTP, HP Quality Center and more so Squish can be used as part of the whole test automation regardless of the tools in use.
We will also release our own test management system, which we use internally, eventually. But this will be just another way to do it and not replace our integration. Rather the opposite But we will always ensure that Squish stays easy to integrate since we really think that our users should have the choice what they want to use.

To end this article here are some links with further details on Squish’s extension and integration capabilities:

Squish for Web’s extension API: http://www.froglogic.com/download/book/ug-jsapi.html
Squish for Java’s extension API: http://www.froglogic.com/download/book/ug-javaextapi.html
Squish’s integration plugins:http://www.froglogic.com/download/book/addons.html

Recent article on being test-infected by Chris Corbyn

"Squish has proved to be an excellent and popular replacement for the GUI automation tool which was used in previous automation campaigns." said Shane McCarron from Ericsson AB.We had the pleasure to interview Shane McCarron, a Senior Designer of Ericsson AB, who use froglogic’s Squish for their automated GUI testing effort of several Java GUI applications in different divisions. We talked about their test automation and why they chose Squish over the competition.

Froglogic: What’s your name and position?

Shane: Shane McCarron, Senior Designer.

Froglogic: What’s the name of the company you work for?

Shane: LM Ericsson Ltd.

Froglogic: Can you briefly describe the software you are testing with Squish?

Shane: It is a suite of Java GUI applications used to manage a telecommunications network.

Froglogic: How did you learn about Squish?

Shane: I read about it on a WIKI article which was discussing vendors that provide cross platform GUI automation tools. Squish was said to support multiple Java Virtual Machines (JVMs) which was a key requirement of ours.

Froglogic: When did you start to use Squish?

Shane: In July 2007

Froglogic: What are the main reasons you decided to use Squish for your automated GUI tests?

Shane:

Squish supports testing of multiple JVMs from a single test case
Tests can be written in JavaScript and this was seen as a benefit as there was already some knowledge of this language in our area and we didn’t have to learn a proprietary language
We were impressed with the Spy feature which lets you pick an object on a GUI and then set related synchronisation points in your test script
The step through debugging feature of the IDE was also seen as a very beneficial facility

Froglogic: What are Squish’s main advantages over the competition? Why did you choose Squish over its competitors?

Shane:

Multiple JVM support
JavaScript based test scripts
Ability to integrate existing tests written in other languages e.g. Perl
Modular based development which means commonly used functions can be placed in scripts which are shared by multiple tests
Technical support – this was gauged by our experiences when we had questions during the review process
Competitive cost of the product and its licenses

Froglogic: What’s your favorite feature of Squish?

Shane: The debugger is excellent – when a breakpoint is encountered it is possible to see at that point in time the value of test specific variables and quickly get to the bottom of any problem with the logic of a test.

Froglogic: What is your most wanted feature wish for Squish?

Shane: Formatting and auto-completion of code e.g. in other IDEs when you type a function name and the opening bracket a prompt will appear indicating the parameters that must be passed to the function for that particular call. [Editor's note: froglogic works on a highly improved IDE for Squish 4.0]

Froglogic: Are you satisfied with froglogic’s technical support service?

Shane: Yes. We have found froglogic to be very approachable and knowledgeable. The turnaround time for resolving issues is quite fast too.

Froglogic: What have been your biggest challenges in creating your automated GUI tests with Squish?

Shane: We were early adopters of Squish for Java so there were some teething problems there but I think our main challenge is that the GUIs we are testing are rather complex.

Froglogic: Where do you see the main benefit of automated testing? Shorter testing cycles, decreased use of human resource, better coverage, test reliability, and/or improved product quality?

Shane: We believe automated regression testing to be the quickest way to ensure that the introduction of new functionality has not broken existing core functionality of our GUIs. Where large tests involving many repetitive actions are concerned the automated test has to be considered more reliable too.

Froglogic: Do you use any other 3rd party or internal tools complementing your testing effort?

Shane: Yes for example we use internal Perl scripts to check a database at various points in our tests to validate the actions taking place on the GUI

Froglogic: Is there anything else you’d like to add?

Shane: Squish has proved to be an excellent and popular replacement for the GUI automation tool which was used in previous automation campaigns.

Froglogic: Thank you, Shane, for your time!

More about froglogic and Squish at http://www.froglogic.com.

A few days ago I had a session at a .NET conference in Germany (BASTA Spring 2008). I talked about software quality assurance and especially test driven development. Because of many people asking me about the details and the source code of my talk I wrote an article about TDD (including TDD of Web-UIs) with Visual Studio 2008 and WatiN: Bug Busters – Test Driven Development in .NET.

The article shows a sample application that is developed according to test driven development principles. It shows how to add data access- and business-layer (based on Linq-to-SQL) and how to build unit tests for ASP.NET forms.

I hope you find it interesting to read.

Kind regards,
Rainer

"My Java Swing application has a lot of graphical nterfaces, so it’s impossible to validate all of them by hand. Squish ermits to increase the test coverage without increasing the duration of the validation.", said Vincent Laigle from SAGEM.

We had the pleasure to interview Vincent Laigle, the Validation Team Leader of SAGEM, who use froglogic’s Squish for their automated GUI testing effort of a Java Swing application. We talked about their test automation process and why they chose Squish over HP’s (formerly Mercury) Quick Test Pro.

Froglogic: Can you briefly describe the software you are testing with Squish?

SAGEM: It is a Java program which runs on an embedded Linux platform. It’s best described as a kind of mail software wich is plugged on a radio network.

Froglogic: How did you learn about Squish and when did you start using it?

SAGEM: I learned about it from Aptus. I asked them to search for a Java GUI test tool similar to Quick Test Pro but which runs also on Linux. After a successful evaluation of Squish we started to use it in June 2007.

Froglogic: What are the main reasons you decided to use Squish for your automated GUI tests? What are Squish’s main advantages over the competition?

SAGEM: Squish has several great features. Here is a list of the most important ones for us:

- It works well with Java applications using Swing/AWT (Editor’s note: Squish for Java also supports testing Java SWT and Eclipse RCP applications)

- The ability to modify and customize Squish’s object name generator to influence which objects’ properties to use for generating object names when recording scripts

- Built-in support for data-driven testing and data tables

- Support of easy-to-learn programming languages for test scripts (JavaScript, Python, Perl, Tcl)

Froglogic: Are you satisfied with froglogic’s technical support service?

SAGEM: Yes. The team at froglogic is very responsive and helpful!

Froglogic: What have been your biggest challenges in creating your automated GUI tests?

SAGEM: The software is sold on a kind of palm device with a touch screen. So there is no keyboard but a lot of graphical interfaces and elements.

Our customer often changes his opinion on the presentation of the interfaces. So I have to create test scripts wich can be easely adjusted in the future without the need to redo them for each change in the user interface.

Squish is very good for that: To abstract the test data from the test scripts, I use a lot of data tables. This combined with a few sed and awk scripts allows me to easily adjust the scripts to UI changes (for example to adjust properties in the object names) without the need to re-record the script.

This is something which is not possible with other GUI test tools like HP’s Quick Test Pro!

Froglogic: How many tests cases do you have approximately by now?

SAGEM: By now I have approximately 300 test cases with a total of about 9000 lines of script code and about 3200 lines of data tables. I’ve automated tests for approximately 30% of my application’s functionality so far.

Froglogic: Where do you see the main benefit of automated testing?

SAGEM: My application has a lot of graphical interfaces, so it’s impossible to validate all of them by hand. Squish (or automated tests in general) permit to increase the test coverage without increasing the duration of the validation process.

Froglogic: Is there anything else you’d like to add?

SAGEM: Squish for Java is a very good product, which is very customizable and adaptable. I had big problems with Quick Test Pro because the objects in my applications changing a lot. Quick Test Pro doesn’t permit to modify the object name repositories and generation in a good and automated way. Also it is not possible to easily edit the scripts outside their IDE.

As a result, with Quick Test Pro I had to re-record my scripts over and over again. Squish, on the other hand, permits me to use grep/sed/awk to modify my scripts and object map. Having a flexible tool like Squish saves a lot of time!

Froglogic: Thank you for your time!

JavaWorld published the article Test-Driven GUI Development with FEST, written by yours truly

This article introduces FEST, demonstrates what separates it from other GUI testing frameworks and provides a hands-on introduction to Swing GUI testing with this developer testing library. Video demonstrations and working code examples are included.

At the same time, we just released a new version of FEST (0.4). For more details please see the release announcement.

Regression testing is usually seen as the poorer cousin of "proper" domain-abstracted assertion-based testing. Often rightly so!

However, with the right support in place, I have found that this form of testing can work very well in certain contexts.

This article addresses one such context: testing the view content generated by a web app. I discuss the background, then present a concrete example in the form of a plug-in for Rails.

This article is sharing of my impressions and thoughts about unit testing of J2ME applications and partly an introduction of MoMEUnit – an instance of the xUnit architecture for unit testing of J2ME applications.It is not a secret that using of JUnit test framework for testing J2ME applications is problematic. The main problem is that J2ME is almost JDK 1.0. It doesn’t include reflection API and, of course, it doesn’t support annotations. In this case it is impossible to designate method that implements test via method signature or annotation. In other words, it is impossible to maintain implicit association between the test method and name of test case.

In my humble option, if it is not possible, we shouldn’t do this. To my mind, the solution is not to use “test<TestName>” method name signature to implement tests, and designate only one method:

public void test() throws Throwable

as method for test implementation. In this approach we will not implement tests per method and will not maintain implicit association between the test method name and name of test case, that (recall) is not possible. Instead we will create test per class. in J2ME it is possible to keep implicit association between class name and test case name. Another problem that arises is a sharing of fixture between group of tests. Here is a place where Inheritance can help.

In MoMEUnit we implement different tests by different subclasses of TestCase. Sharing of the same fixture between different tests is realized via inheritance (as mentioned above). The one approach to do this is to create one subclass of TestCase by defining instance variables to store the state of the fixture, overriding setUp() and/or tearDown() methods to initialize fixture and to release resources after test run, implementing test() method. Other tests will extend this test sharing the same fixture. Instance variables, of course, should be declared as at least “protected” to be accessible by subclasses. Another more general approach is to create abstract TestCase subclass by defining instance variables and overriding setUp() and/or tearDown() methods. All test cases will extend this abstract class sharing the same fixture. The second approach is of course more structural but, to my mind, this is a bit a waste of time and memory).

For example instead of classical JUnit TestCase subclass

// sorry for stupid examples.
import junit.framework.TestCase;

public class ArithmeticTest extends TestCase
{

private int arg1;

private int arg2;

public ArithmeticTest( String name)
{
super(name);
}

protected void setUp()
{
this.arg1 = 2;
this.arg2 = 5;
}

public void testAdd()
{
int res = arg1 + arg2;
assertEquals("I don’t know what happened", 7, res);
}

public void testSubtract()
{
int res = arg2 – arg1;
assertEquals("And what again ?", 3, res);
}
}

In MoMEUnit we will have :

//"AddTest.java" file

import momeunit.framework.TestCase;

// Usage of Test suffix of class name is not required. You can use any name.
// This is only for illustration purposes.

public class AddTest extends TestCase
{
protected int arg1; // at least protected to be accessible by subclasses

protected int arg2; // at least protected to be accessible by subclasses

protected void setUp()
{
this.arg1 = 2;
this.arg2 = 5;
}

public void test()
{
int res = arg1 + arg2;
assertEquals("I don’t know what happened", 7, res);
}
}

// "SubtractTest.java" file

// No imports because SubtractTest and AddTest are in the same package

// Usage of Test suffix of class name is not required. You can use any name.
// This is only for illustration purposes.
public class SubtractTest extends AddTest
{
public void test()
{
int res = arg2 – arg1;
assertEquals("And what again ?", 5, res);
}
}

Using inheritance gives the other nice feature. We can define one fixture for a one group of test cases and add additional fixture or change existing one for another group. That can be helpful in some cases. For example

public class AdditionalFetureTest extends OtherTest
{

protected AnObject arg3 = null;

protected void setUp()
{
super.setUp()
this.arg3 = new AnObject( … )
}

public void test()
{
assertTrue("Why additional feature doesn’t work ?",
target.additionalFeature( arg3, … ));
}
}

Maybe it looks like implementing different tests as different classes is cumbersome. To my mind, it is not true. By using IDEs like Eclipse or NetBeans it is almost as simple and easy as creating additional method. Besides, writing class like this

public class MultiplyTest extends AddTest
{
public void test()
{
…
}
}

even by hand is not complex.

Of course implementing different tests as different classes increases a size of J2ME application. But we are just testing. This is not a production J2ME application. Besides, as mentioned above in J2ME it is possible to keep implicit association between test case name and class name but not method name. This enforces to create anonymous classes or use other means that also increases application size. In my humble option test per class in J2ME is an optimal solution.

MoMEUnit has, of course, other changes to JUnit framework, that overcome other restrictions of J2ME platform and minimize memory usage.
MoMEUnit is ,of course, free software. It is available under Common Public License.

Background

Anyone working on any large TDD (Test Driven Development) project knows the drill:

- Make changes to code
- Update with latest version of code base
- Fix inconsistencies and conflicts
- Start running tests
- Make cup of tea / have lunch / go home – depending on how far into the project you are
- Fix one or two test failures – fingers crossed you have not screwed up the other tests
- Try to check in code
- Swear loudly because someone has made changes to the code base that conflict with your changes
- Repeat ad nauseum

OK – I am exaggerating a bit but full end-to-end system tests can take a while on a large system – particularly if you have to reset the system to a known state before each test. If you are attempting to get a release out regularly, fixing the build ready for the next release involves banning commits of new code and days of pain and torment fixing all the build issues.

Dividing the tests into groups (e.g. using TestNG) and building the project from separate projects can help – but in the world of Spring, AOP, proxies, ORM technology and so on – it is very hard to tell which group of tests you need to run to validate a code change.

The solution given here (which has been successfully implemented) goes a long way towards eliminating these problems. It uses the JDK 5 Instrumentation technology to record at run time which tests touch which classes – AND conversely which tests you need to run for each class change. Before anyone screams – What about new classes? What about files that are not classes? I am well aware of these issues – but this technique will find the correct tests 90% of the time – and I rely on the build server to catch the remaining 10%. It certainly beats either not running the tests or waiting a long time for the tests to complete in their entirety.Technology

JUnit was used as the underlying test framework which supports both individual Tests and Test Suites (groups of tests run together).

JDK 5 officially introduced Instrumentation into the Java world. Instrumentation allows classes to be changed as they are loaded for a variety of reasons: to add code to collect coverage statistics, removing not-required log4j messages (http://surguy.net/articles/removing-log-messages.xml), to collect performance statistics and so on.

The instrumentation is performed by code in a JAR file termed an ‘Agent’. The location of the agent JAR is passed in as a JVM argument -javaagent:jarpath[=options ]. The manifest within the JAR file specifies which class to use for the Agent.

On start up the JVM calls the ‘premain’ method of the Agent passing in an Instrumentation object with which the Agent can register an instance of ClassFileTransformer (typically itself):

public static void premain(String options, Instrumentation instrumentation) {
instrumentation.addTransformer(new MyTransformer());
}

Any classes that are loaded by the JVM will now pass through the transform method on MyTransformer before being available to the application.
Unfortunately the JVM specification does not specify a set of tools to perform the bytecode transformations. I chose ASM for performance although there are other frameworks which are simpler to use e.g. BCEL.

Design

Run the tests using JUnit test suites. There are two variants of each Test Suite – one ‘Non Intelligent’ version – which runs all the tests (this is used on the build server) – and one ‘Intelligent’ version which looks to see if there is any stored dependency data and uses that to work out which tests to run.
Each Test Suite has an associated Agent Dataset that is persisted between each run containing:

&#9679; Which classes are used by which test
&#9679; The failed tests from the last run
&#9679; The time of the last run
The Agent Dataset starts out empty before the Test Suite is first run.

Non Intelligent Test Suite Version Workflow:
1. Run all the tests in the test suite

Intelligent Test Suite Version Workflow:
1. Load the Agent Dataset from the last run. If there is none then proceed as per Non Intelligent Version.
2. Inject the old dependency database into the Agent (otherwise we will lose a lot of the dependency information as only a subset of the tests are being run).
3. Create a new set of tests to run. Add all the previously failed tests to it. Iterate through the project classes within the dataset. For each project class – compare the modification date of the class file against the timestamp within the dataset (i.e. when the tests were last run) – if the class file has been modified then add all the tests that touched that class file to the set of tests to run.
4. Run only the tests that are required

Common Workflow (for both Test Suite versions):
1. Instrument all the methods on all the classes within the project to add in a static call back to the ‘record’ method on the Agent passing the name of the containing class.
2. At the beginning of each test make a static call to the Agent to specify which test is currently being run.
3. Use the ‘record’ method on the agent to lot which classes have been used by which tests.
4. At the end of the test suite save the data collected by the Agent together with a timestamp and a list of failed tests.

Implementation

The Agent consisted of the following components:
&#9679; Dependence Database
&#9679; Getters and Setters for the Dependency Database
&#9679; A ‘premain’ method to register an instance of the Agent as a transformer
&#9679; A transformer which uses ASM to add a static callback to ‘record’ on the Agent passing the name of the calling class.
&#9679; A ‘record’ method to record which classes were touched by which tests into the Dependency Database
&#9679; A ‘startTest’ method to tell the Agent which test is currently running

In additions there is a helper class to save and load the Agent Dataset and perform the calculation to determine which tests need to be run.

Due to issues with the coverage tool (Emma) there is also an Ant task to instrument the classes before Emma does its own instrumentation. This is more an issue with Emma than with anything else.

This took about 4 days to implement by someone that had not touched ASM before. The lack of standard toolset or documentation / examples made this harder than it need be. For example JDK 5 uses attributes within the class file to store information for generics – whereas the sample code I was using did not operate with attributes. This was not a hard issue to fix, but lack of documentation made it hard to work out what was happening.

Conclusion

I believe this is an extremely elegant solution in that it is almost completely orthogonal to the rest of the project code (the change to the existing code is minimal – just a few extra test suites, a call to ‘startTest’ on the Agent, and to ‘saveAgentData’ on the helper class at the end of the suite). Using this technique we achieved an up to 10 fold performance improvement in running test suites.
As a way to make TDD more productive and less painful I heartily recommend it.