testdriven.com Wrangling quality out of chaos

It is clear that normal unit-tests represent individual cases only. A specification presents a complete description of a unit’s behavior. Unlike unit-test, sequence diagrams are perfectly readable but unlike sequence diagrams, unit-tests can easily be executed over and over again. The question is whether we could in some way combine sequence diagrams and unit-tests so as to create both the specification and the tests which would help to verify that the implementation fulfills the specification. The rest of this thesis tries to provide an answer.

Author: Peter Dimov
Published: University of Duisburg-Essen Master Thesis, September 28, 2007
link

This paper describes the experience of evolving a domain-specific language embedded in Java over several generations of a test framework. We describe how the framework changed from a library of classes to an embedded language. We describe the lessons we have learned from this experience for framework developers and language designers.

Describes the evolution of jMock from framework to domain-specific language.

Author: Steve Freeman
Presented: OOPSLA 2006
link

Unit testing, a common step in software development, presents a challenge. When produced manually, unit test suites are often insufficient to identify defects. The main alternative is to use one of a variety of automatic unit test generation tools: these are able to produce and execute a large number of test inputs that extensively exercise the unit under test. However, without a priori specifications, developers need to manually verify the outputs of these test executions, which is generally impractical. To reduce this cost, unit test selection techniques may be used to help select a subset of automatically generated test inputs.
Then developers can verify their outputs, equip them with test oracles, and put them into the existing test suite. In this paper, we present the operational violation approach for unit test selection, a black-box approach without requiring a priori specifications.

Authors: Tao Xie, David Notkin (Department of Computer Science & Engineering, University of Washington)
Published: October, 2003
See also:
Jov (Java Unit Test Selection Tool Based on Operational Violations)
Jusc (Java Unit Test Selection Tool Based on Residual Structural Coverage)
link

"Twenty-one years ago, I started writing Testing Computer Software (Kaner, 1988), a book whose second edition (Kaner, Falk, & Nguyen, 1993, 1999) outsold every other book in the field. It didn’t seem destined for the mainstream back then. [..]

This paper for STPCon is my next crack at a paper that reappraises the assumptions about testing and development that I accepted or left alone in 1983, and perhaps the start of a new map for TCS 3.0. Some of the assumptions were probably always wrong. Others work under some circumstances, but not others—and in 21 years evolution of software development, circumstances have changed a lot. Others, I think, retain lasting value."

A thorough review of many assumptions, on:
- The role of testers
- Test planning and documentation
- The practice of testing

Author: Cem Kaner, PhD
Published: Software Test&Performance Conference, December 8, 2004
link

Continuous testing uses excess cycles on a developer’s workstation to continuously run regression tests in the background, providing rapid feedback about test failures as source code is edited. It reduces the time and energy required to keep code well-tested, and prevents regression errors from persisting uncaught for long periods of time. This paper experimentally evaluates the promise of continuous testing, based on a controlled human experiment.

Authors: David Saff & Michael D. Ernst
link

There is a fundamental sea change happening in the industry around us. A move away from prescriptive top down mandates to implement new processes to a developer led viral adoption of Agile Practices. But will this movement hit a glass ceiling and remain in isolated pockets of excellence?

This short paper considers they key issues behind the phenomenon that is quietly ‘infecting’ many of today’s organisations without them even knowing.

Author: Dean Margerison, Dekam Consultancy
Published: June 18, 2004

(via coachspot)
link

Test automation, open source tools and agile methods are three important trends in software development. By integrating these three, a project team at Comcast was able to quickly build and deliver a critical application to its customers. The use of test automation enabled an agile software development lifecycle (SDLC) based largely on XP. The use of open sources tools for automation reduced the project cost while providing additional flexibility. The end result was on-time delivery with solid return on investment (ROI) for the business, and satisfied customers.

Authors: Dan Lavender, Senior Architect, Comcast Media Center
Pete Dignan, President, ProtoTest
Published: Proceedings of STAR West 2003, January, 2003

(via Agile Alliance)
link

This paper compares the two agile software development methodologies: eXtreme programming (XP) and Feature Driven Development (FDD). It focuses on applicability of the methodologies in an academic environment (CSCIE-275 course project) and in regulated environment (health care industry). The paper compares different aspects of project management with XP and FDD: starting from gathering user requirements and up to deployment of the complete applications.

The intended reader should be familiar with at least one of the two development methodologies. This paper is not a description of any of them: it rather highlights their differences and similarities.

Author: Serguei Khramtchenko
Published: Harvard University, May 17, 2004

(via [url=http://agilemovement.it/modules.php?op=modload&name=News&file=article&sid=497]Italian Agile Movement[/url])
link

Sat, an acceptance test system for testing socketbased servers with multiple clients, initially used an XML file to define the tests in a test suite. It was altered to make use of Fit, a testing framework that uses HTML tables for defining tests and reporting any errors. The authors found that the new version was considerably easier to use, and provided a form of "literate testing".

Authors: Rick Mugridge & Ewan Tempero, Department of Computer Science, University of Auckland, New Zealand
Published: 2003
link

Abstract:

"Test-driven development has recently reemerged as a critical enabling practice of the Extreme Programming software development methodology. We ran a case study of this practice at IBM. In the process, a thorough suite of automated test cases was produced after UML design. In this case study, we found that the code developed using a test-driven development practice showed, during functional verification and regression tests, approximately 40% fewer defects than a baseline prior product developed in a more traditional fashion."

Authors: Laurie Williams, E. Michael Maximilien, Mladen Vouk
Published: IEEE 2003, Proceedings of the 14th International Symposium on Software Reliability Engineering (ISSRE’03)
link

We introduce and evaluate a new technique, continuous testing, that uses spare CPU resources to continuously run tests in the background, providing rapid feedback about test failures as as source code is edited. Continuous testing reduced wasted time by 92–98%, a substantial improvement over the other approaches.

Authors: David Saff & Michael D. Ernst
link

Our system, which was developed over a period of several years, was a spaghetti-like system that consisted of C-based applications and scripts and had grown into an unmanageable piece of software that was difficult to improve and maintain. Finally, it became clear that a new, more generic and extensible system was needed: Management decided that replacement was in order. Furthermore, it was decided that a new system would be implemented using Java. At the outset, we decided to base our development process on Extreme Programming and adopt agile practices for our daily work. In this article, I describe those practices, starting with the development process itself, then focus on the two practices that proved the most valuable.

Author: Julius Gawlas
Published: Dr. Dobbs’s Journal, January 2004
link

Despite our best efforts and intentions as educators, student programmers continue to struggle in acquiring comprehension and analysis skills. Students believe that once a program runs on sample data, it is correct; most programming errors are reported by the compiler; when a program misbehaves, shuffling statements and tweaking expressions to see what happens is the best debugging approach. This paper presents a new vision for computer science education centered around the use of test-driven development in all programming assignments, from the beginning of CS1. A key element to the strategy is comprehensive, automated evaluation of student work, in terms of correctness, the thoroughness and validity of the student’s tests, and an automatic coding style assessment performed using industrial-strength tools. By systematically applying the strategy across the curriculum as part of a student’s regular programming activities, and by providing rapid, concrete, useful feedback that students find valuable, it is possible to induce a cultural shift in how students behave.

Authors: Stephen H. Edwards
Published: October, 2003
Source: Companion of the 18th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications
From: The ACM Digital Library
link

Testing is a critical part of good software development, but often gets only minimal coverage in introductory programming courses. Unit testing and selected aspects of test-driven development can be used to improve learning and encourage emphasis on quality and correctness. Tools like JUnit significantly simplify the generation of test cases. An additional benefit for instructors is that these tools can also be used to automate project grading.

Author: Michael Olan
Published: December, 2003
Source: The Journal of Computing in Small Colleges, Volume 19 Issue 2
From: The ACM Digital Library
link

A new approach to teaching software testing is proposed: students use test-driven development on programming assignments, and an automated grading tool assesses their testing performance and provides feedback. The basics of the approach, screenshots of the sytem, and a discussion of industrial tool use for grading Java programs are discussed.

Author: Stephen H. Edwards
Published: October, 2003
Source: Companion of the 18th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications
From: The ACM Digital Library
link

From a project economics point of view, the most important practices of Extreme Programming (XP) are Pair Programming and Test-Driven Development. Pair Programming leads to a large increase in the personnel cost, and Test-Driven Development adds to the development effort. On the other hand, pp can speed the project up, both pp and Tdd can reduce the defect density of the code. Can the increased cost of XP be balanced by its shorter time to market and higher code quality? To answer this question, we construct a new model for the business value of software projects. We then analyze the cost and benefit of XP by applying our model to a realistic sample project. We systematically vary important model parameters to provide a sensitivity analysis. Our analysis shows that the economic value of, XP strongly depends on how large the XP speed and defect advantage really are. We also find that the market pressure is an important factor when assessing the business value of XP., Our study provides clear guidelines for managers when to consider using XP — or better not.

Authors: Matthias M. Müller, Frank Padberg
Published: September, 2003
Source: Proceedings of the 9th European software engineering conference
From: The ACM Digital Library
link

In a software development group of IBM Retail Store Solutions, we built a non-trivial software system based on a stable standard specification using a disciplined, rigorous unit testing and build approach based on the test- driven development (TDD) practice. Using this practice, we reduced our defect rate by about 50 percent compared to a similar system that was built using an ad-hoc unit testing approach. The project completed on time with minimal development productivity impact. Additionally, the suite of automated unit test cases created via TDD is a reusable and extendable asset that will continue to improve quality over the lifetime of the software system. The test suite will be the basis for quality checks and will serve as a quality contract between all members of the team.

Authors: E. Michael Maximilien, Laurie Williams
Source: Proceedings of the 25th international conference on Software engineering
Published: ACM Digital Library, 2003
Membership required
link

Assertions, or more generally “Programming by contract”,
have gained widespread acceptance in the computer science
community as a means for correct program development.
However, the literature lacks an empirical evaluation
of the benefits a programmer gains by using assertions
in his software development. This paper reports two controlled
experiments that close this gap. Both experiments
compare “Programming by contract” to the traditional programming
style without assertions.

Authors: Matthias M. Müller, Rainer Typke, Oliver Hagner
Published: October, 2002
link