testdriven.com Wrangling quality out of chaos

Archive for the ‘Tools’ Category

Lessons Learned from the October 2013 SF Selenium Meetup

Lesson 1: When creating single-page applications in JS, its really worth it to do a thorough job of unit testing, because the same tests can be used to test the compatibility of your code across various browsers.


Lesson 2: Don’t fall into the pitfall of a slow build. Design your infrastructure and your tests to be parallelizable from day one. This relatively small investment upfront will save you years of anguish in the future trying to fix your Test Technical Debt.



Taming Legacy Grails Code with Test Generation

04.14.2013 · Posted in Articles, Automatic Testing, Selenium

In his seminal book, Working Effectively with Legacy Code, Michael Feathers defines “legacy code” as code without tests. Unfortunately this is exactly what my team wound up creating towards the end of our Groovy/Grails project. We were hoping that our Selenium tests would cover both the JavaScript in the browser and the Groovy code in the controllers. Unfortunately, due to several issues which are outside the scope of this article, we could not get a robust set of tests that ran fast enough so we decided that the pragmatic thing to do was to abandon the Selenium tests — the project has a rather large QA component so the risk to business was well mitigated. We soon realized that the untested code in the controllers was giving us problems with typos not being found for many days because Groovy 1.8 is a dynamic language.

To quickly rectify the situation we decided to implement a record/playback strategy. In most situations I would be against using record/playback automation tools, by the beautiful thing about software development is that its so contextual and I think this was the right move for the moment.

We wrote a small code generator that produced simple Selenium tests using the HtmlUnitDriver. The generator was triggered by a call in a ‘MyFilters.groovy’ file when the application was tested manually. For every controller/action combination, the generator intercepted the params map and generated the test with the values in the map. The initial assertions were simple — they just checked to see that nothing obviously bad had happened — such as 404 or 500 errors. It took about 1.5 hours to write the generator and then we stepped through the application manually and generated many dozens of tests. Because there was no Ajax involved the tests were 100% reliable and ran in about 90 seconds.

This simple approach allowed us to be sure that when we do refactor our code, we will catch the basic mistakes that a compiler would catch for us. We got an additional bonus when we caught a bug in our sample data that violated a Hibernate constraint.

In our situation, writing a simple record/playback tool was the right move. It took less that a day to implement and it already paid for itself many times over.

Notes from SF Selenium Meetup

01.25.2012 · Posted in Continuous Integration, Events, Selenium

Today I attended an excellent presentation by Denali Lumma at the Selenium SF meetup. Especially pleasant for me was an “Aha!, I told you so” moment when she described Okta Continuous Integration setup that relied on Git Top Branches and Selective testing. When I ran CI for the Google Checkout project, the lowest point of the day was always the “5 checkins went in and the build is broken in non-obvious ways” — time to selectively do a local rollback of each changelist to figure out who broke it. Meanwhile the VCS was locked down. Ouch that hurts! After some time we implemented something we call a Submit Queue which allowed a single changelist to be merged with trunk in a temporary workspace and tests executed on it before it was committed. Some commercial CI servers provide this sort of pre-submit testing. Okta has implemented the same solution using Jenkins and Git Topic Branches — brilliant! Additionally, they have an automated script that does intelligent test selection based on real-time customer usage of system components — double brilliant! Finally, they also scripted the configuration and deployment of the Jenkins server. I confess that I don’t quite understand the context in which they had this need, but I am sure its likewise brilliant.

UquoniTest 2.0 released

11.30.2009 · Posted in Tools

Q-Mentum just released UquoniTest 2.0, an advanced unit testing library for C++, that provides many features that other unit testing frameworks don’t have.

Major features:

* Rich Booleans in assertions for easier checks and more information about failures
* Easy creation of (reusable) test directories
* A powerful mock framework
* Parameterized and template tests
* Automatic registering of tests
* Orthodoxy tests
* Reporting of failing assertions in domain code
* Checking if an expression remains constant throughout all tests
* Custom test listeners and test wrappers
* Abstract test (also multiple levels and multiple inheritance)

Visual C++ 6, 2003, 2005, 2008, MingW gcc, gcc on Linux

More info: UquoniTest.

Qt and Java GUI Test Automation: Many Years of Squish Success at ARM Ltd.

11.12.2009 · Posted in Tools


ARM Ltd., is a well-established company that designs the technology that lies at the heart of many advanced digital products. ARM’s product offering includes RISC microprocessors, graphics processors, enabling software, cell libraries, embedded memories, high-speed connectivity products, peripherals, and development tools.

We had the pleasure of discussing ARM’s use of Squish with Dave Dextor, one of ARM’s Staff Validation Engineers.

Why Squish?

ARM originally evaluated Squish several years ago, and bought their first licenses around 2005. We asked Dave what motivated ARM to choose Squish, and he told us:

The ease of use of Squish along with the object map which can be used to reduce the maintenance of tests were two key factors. And support of multiple operating systems and technologies—especially for Qt and Linux—was also very important.

Squish’s unique object map makes it possible to identify application objects. When application objects change, the object map makes it possible for test engineers to make simple changes to the object map so that objects continue to be correctly identified by test scripts—without having to modify the tests themselves.

Dave also mentioned some of the other features that made Squish attractive to ARM, in particular the ease with which it is possible to do data driven testing, Squish’s support for multiple standard scripting languages, and the fact that Squish has better features than competing tools.

Squish at ARM

ARM started out by using Squish for GUI testing of a proprietary debugger and an Eclipse-based IDE. The debugger was written using Qt and it was when looking for a suitable testing tool for this that ARM first came across Squish. They are now developing a new Eclipse-based debugger in Java and plan to use Squish to provide automated GUI testing for that tool.

The software that Squish is used to test runs on Windows and Linux (specifically, Windows XP, Windows Vista, and Red Hat Enterprise Linux 3 and 4).

Dave told us that one of the biggest challenges ARM had encountered when it came to creating and maintaining tests was keeping the tests current in the face of changes to the software across build cycles. Another challenge was to automate the testing process as much as possible based on the ARM cluster they use as a centralized server for development and testing.

At present, ARM make most use of Squish’s command line tools to provide the maximum amount of test automation possible. Dave said that Squish had been particularly helpful in making testing cycles shorter, and also reducing the amount of human resources needed for testing.

Other than Squish, ARM support their testing process and test automation using in-house developed scripts (in Perl and Python), and proprietary debugger automation software. They also make extensive use of JUnit (a unit testing framework for Java) and CXXTest (a JUnit-like unit testing framework for C and C++).

Regarding froglogic’s technical support, Dave told us:

It’s been very helpful on the few occasions I’ve needed it.


ARM produce cross-platform tools based on Qt and Java technologies. Squish has enabled them to use the same testing tool chain on all the platforms they support and using both the GUI technologies they use—as well as giving their test engineers the freedom to use the scripting languages they prefer. ARM now have several years of experience using Squish, and Dave tells us that in future they plan to use Squish even more than they do now.

froglogic’s team would like to thank Dave for taking the time to share ARM’s experience with Squish, and we are looking forward to a continued successful relationship.

Visit http://www.froglogic.com to learn more about Squish and to get your free and supported evaluation copy!

JetBrains releases YouTrack 1.0, keyboard-centric issue tracker

11.07.2009 · Posted in Tools

JetBrains has released its brand new bug and issue tracker, YouTrack 1.0!

YouTrack helps solve most issue-handling tasks with two simple, keyboard-driven controls: a search box and a command window.

YouTrack key features include:

* Issue reporting with a single shortcut from anywhere within the product.
* Query-based search with query completion and highlighting.
* Powerful commands to quickly modify multiple issues.
* Natural-language-like query and command syntax.
* Extensive keyboard support for creating, editing, and navigating between issues easily.
* AJAX-based user interface for extremely fast issue handling.
* Importing existing projects from JIRA 3.x.
* Integration with external tools, including LDAP, OpenID, and JetBrains TeamCity.

YouTrack 1.0 is available in two editions, Enterprise Edition and Professional Edition. Both editions sell at 50% off before the New Year.

[url=http://www.jetbrains.com/youtrack/?utm_source=testdriven&utm_medium=News&utm_campaign=YouTrack]Learn more and download YouTrack[/url] at JetBrains web site.

FEST-Swing 1.2a1

05.14.2009 · Posted in Java

We are proud to announce the release of FEST-Swing 1.2a1!

FEST-Swing is a Java library that provides a fluent interface for functional Swing GUI testing. This library provides an easy-to-use API that makes creation and maintenance of GUI tests easy.

This is the first out of three alpha and three beta releases planned for version 1.2. This new version focuses on new features and improvements.

More details, please read the release notes.

JUnitMax beta is worth the price

03.07.2009 · Posted in Java

Kent Beck has released JUnitMax for Eclipse as paid beta – currently USD $2/month. JUnitMax is a selective testing tool – it chooses a subset of tests to run so you get feedback very fast. I especially like the fact that the results are written into the same window as compiler failures. Think of JUnitMax as the twin sister of the incremental compiler.

I’ve tested it with the latest version of Apache Commons Collections and it works quite well. Of course its usefulness depends on whether you have fast and reliable tests in the first place, but if your project can benefit from fast tests – you have one less excuse not to write them. :)


Application architecture data published from 2,500 open source projects

03.05.2009 · Posted in Tools

Coverity Inc., the software integrity company, today announced the publication of application architecture data from over 2,500 popular open source software projects at www.scan.coverity.com . As an outgrowth of its contract with the U.S. Department of Homeland Security (DHS), Coverity™ collected this data via the Scan site using the Coverity Architecture Analyzer product. This new, publicly accessible resource includes application architecture files and generated diagrams that will benefit developers planning to incorporate open source packages in their applications, as well as developers that want to learn the architectures of successful projects to improve the structure of their own applications.

"I’ve tried the Coverity Architecture Analyzer application, and it showed me some unexpected references between parts of the code that I want to investigate,” said Volker Lendecke of the Samba Team. “I’m glad to hear the architecture diagrams can be included on our project wiki as documentation to help new developers get up to speed more quickly."

The Scan architecture library is a database of application architecture diagrams from over 2,500 open source projects such as Amanda, NTP, OpenPAM, OpenVPN, Overdose, Perl, PHP, Postfix, Python, Samba, and TCL. Each diagram displays dozens to hundreds of components that comprise a given software project.

According to Forrester Research, “A high-performance scalable architecture depends on continually optimizing your application architecture so that different components built in myriad languages interoperate according to performance best practices. Application architecture is not about "set it and forget it." It requires continuous care and feeding to identify and solve problems, take advantage of new platforms, and handle new loads and new features.” (Best Practices: Attaining and Maintaining Blazing Fast Web Site Performance”, Forrester Research, Inc., February 2009.)

The ability to study a visual presentation of an application’s architecture and related data offers a number of benefits to developers. For example, developers planning to use or build on top of a project in the Scan library can optimize their use of it by comparing architecture plans to other codebases that use the same project. This allows developers to more fully understand the structure and capabilities of the open source software they intend to consume.

For developers writing a plug-in or module for an existing open source package, understanding its architecture helps ensure their design is aligned with the expectations and intent of the target open source application. Finally, developers creating a particular type of application, such as a web server, can study the architecture of similar, successful open source projects to improve and accelerate the development of their own application architecture.

“This new resource is another part of Coverity’s ongoing efforts to provide valuable resources to the software development community, while advancing the state of the art in developing high integrity software,” said David Maxwell, open source strategist for Coverity and NetBSD developer. “Sharing the structural data of a wide variety of application types will be a resource for both open source and commercial developers who are planning to use a specific open source project, as well as developers that are simply curious to learn more about application architectures in general.”

Since 2006, the Scan site has leveraged Coverity’s static analysis technology to analyze 50 million lines of code on a daily basis from more than 250 projects. Since its launch, open source developers have used the Scan site to identify and eliminate over 8,500 security vulnerabilities and quality defects. The new Scan architecture library is publicly available at Coverity’s Scan site http://scan.coverity.com/arch/

The Scan architecture library was created using Coverity Architecture Analyzer. The product automatically maps the relationships between code elements at the function and file levels, identifying the underlying structure of software to help developers identify violations of architectural standards. Coverity Architecture Analyzer requires no change to the source code or build environment.

Open source developers may request inclusion in the Coverity Scan, or complimentary licenses of Coverity Architecture Analyzer for use with the data collected in the Coverity Architecture Database by emailing scan-admin@coverity.com. Trials of Coverity Architecture Analyzer can be downloaded immediately, for use with the Coverity Architecture Library, by registering at http://scan.coverity.com/arch/trial

Developers who participate in the Scan project also receive access to an interactive Coverity Architecture Analyzer application for browsing the library in greater detail.

About Coverity Scan
The Coverity Scan site was developed by Coverity with support from the U.S. Department of Homeland Security as part of the federal government’s Open Source Code Hardening Project. The site divides open source projects into rungs based on the progress each project makes in resolving defects. Projects at higher rungs receive access to additional analysis capabilities and configuration options. Projects are promoted as they resolve the majority of defects identified at their current rung. The Coverity Scan site is freely available to qualified open source projects at: http://scan.coverity.com

About Coverity
Coverity (www.coverity.com ), the software integrity company, is the trusted standard for companies that have a zero tolerance policy for software failures, problems, and security breaches. Coverity’s award winning portfolio of software integrity products helps customers prevent software problems throughout the application lifecycle. Over 100,000 developers and 500 companies including ARM, Phillips, RIM, Rockwell Collins, Samsung and UBS rely on Coverity to help them ensure the delivery of superior software. Coverity is a privately held company headquartered in San Francisco with offices in 6 countries and more than 150 employees.

Ivonna — a Typemock based Asp.Net testing tool.

03.01.2009 · Posted in Tools

Ivonna is an Asp.Net testing tool that allows writing unit tests for your Asp.Net applications. What makes it different is that Ivonna runs in the same process as the test runner.

Ivonna is being developed in partnership with TypeMock and runs on top of the TypeMock Isolator framework.

Site: http://sm-art.biz/Ivonna.aspx

Verde – Turns Brown Code Green

11.13.2008 · Posted in Java

Pillar has announced the availability of a new Software as a Service offering called Verde. We like the challenge of placing automated characterization tests around ancient legacy code. This old "Brown" and brittle code needs a jump start – a chance to turn "Green" again.

This technology produces real Junit code at run time. We have created a recorder that sits inside the JVM and can listen at the Jar, Class, or Method layer. It also produces Mock Objects as needed to create isolation layers as needed.

Please see more at www.pillartechnology.com

XML test case generation from XSD schema using XSLT (open source)

11.13.2008 · Posted in Tools

Our Sourceforge project – camprocessor – is a a toolset that implements the OASIS Content Assembly Mechanism standard. One capability it provides is ability to ingest XSD schema and then generate realistic XML test instances. There are various options to control the generation and provide your own content hinting. All this is built with XSLT using the Saxon processor so is readily extensible.

The main jCAM component is written in Java using the Eclipse framework providing a convenient desktop interface toolset.

Also – you can generate subset schemas by excluding and refining the content model once ingested – to match your specific application. Further xslt scripts generate HTML documentation in business user friendly formats.

You can check out more from the [url=http://www.oasis-open.org/committees/download.php/29661/XSD%20and%20jCAM%20tutorial.pdf]tutorial here[/url].

and the main download site here..


Instinct 0.1.8 Release

07.28.2008 · Posted in Java

Instinct 0.1.8 has been released. This is a maintenance release resolving some minor bugs and packaging issues from the 0.1.7 release earlier this week.Instinct is a Behaviour Driven Development (BDD) framework for Java. Inspired by RSpec, Instinct provides flexible annotation of contexts, specifications and actors; automatic creation of test doubles and test subjects; a state and behaviour expectation API; JUnit test runner integration; Ant support and an IntelliJ IDEA plugin.

Downloads are available from the project site: http://instinct.googlecode.com/

Full list of changes:

Core Features
o Custom specification, before specification and after specification
annotations and naming conventions can be provided via the Context
annotation. This can also be used to turn off naming convention-based
detection (using NoNamingConvention).

o Upgraded to: Functional Java 2.8.

o (Issue 36) Abstract classes are being run as a result of specification runner refactoring.
o (Issue 37) Expected exceptions are being printed to console in Ant
runner even though spec passes.
o (Issue 38) Functional java jar is not included in release zip.

ClassMock – Test Tool for Metadata-Based and Reflection-Based Components

06.26.2008 · Posted in Java

ClassMock is a framework that helps the creation of unit tests for components that use reflection or annotations. In this kind of classes, the behavior is dependent of the class structure. This way, each test case usually works with a different class created specifically for the test. With ClassMock is possible to define and generate classes in runtime, allowing a better test readability and logic sharing between tests.

ClassMock is available at http://classmock.sourceforge.net/

data storm – a new free database viewer for TDD

06.23.2008 · Posted in Tools

Data Storm is a free open source data base browser. While there are millions of data base browsers freely available online, Data Storm is different as it can be launched directly from within your Java code. This enables you to see and modify data that otherwise may not be available to you (e.g. data that will be rolled back).

It is freely available from http://datastorm.sourceforge.net/motivation.html

which also motivates why you need the tool :-)

Pex 0.5 released

06.02.2008 · Posted in Tools

Pex is a project from Microsoft Research that enables *parameterized* unit testing.

Pex generates Unit Tests from hand-written Parameterized Unit Tests through Automated Exploratory Testing based on Dynamic Symbolic Execution.