testdriven.com Wrangling quality out of chaos

Archive for September, 2014

Test Automation in the Age of Continuous Delivery

09.30.2014 · Posted in Agile Testing, Articles


I spend a lot of my time with clients figuring out the minutia of how to implement optimal test automation strategies that support their transition to Continuous Delivery. The goal is typically to be able to release software after each 2-week iteration (Sprint).

When faced with such a compressed development and testing schedule, most teams quickly realize that they can’t afford to have a separate QA automation group — there is just not enough time for after-the-fact automation and bug investigations that have to be done with that approach. Instead, the teams are adopting “whole team automation” strategies where system, integration and unit tests are owned and developed by the entire team.

The first and most persistent question I hear from folks who are going through this transition is: “who writes the system/integration tests?” This is usually followed by a discussion about the tester’s lack of knowledge of the chosen development language (Java, C#, etc.) and that some cockamamie, XML-based tool will save the day. My advice is to recognize that the fundamental issue is not the tester’s coding skills, but the lack of proper collaboration on the team. Learning to write most test code in Java or C# is not much more difficult than expressing the same script in XML or equivalent QA-only tool — it is difficult is to write ALL of it from scratch and without proper support.

Allow me elaborate: when tests are well-written, using proper layering patterns such as Page Objects for Selenium and using the ‘builder/fluent’ style, its is quite easy to add additional tests or to modify them when requirements change. Once the initial design is laid out — the rest of the code is quite simple to write and when a tester needs help the developer is right there. When testers pair with developers, they become proficient very quickly — pairing is a very efficient way to transfer skills. Now, when you check in code — you are checking in “units of business value” — that is code with the proper amount of tests to mitigate the risks to quality.

Lets take a look at a real-world example:

Imagine an application which has a Web UI and both internal and external RESTful APIs. We chose Cucumber/Selenium for automating the Web UI and rest-assured for automating the web services. How would this work in a fully integrated team?

The Product Owner comes to the planning meeting with a rough draft of Cucumber feature/scenario scripts that support the user stories that may be chosen for the upcoming Sprint. The requirements are reviewed and clarified with developers and testers; additional scenarios are discovered and added to the Cucumber ‘feature’ files. Agreement is reached on what scenarios needs to be automated with system/integration tests.

The developer starts writing the UI code in a story branch. He lays out the fields and controls for a new screen and then generates the Page Objects stubs. She checks in the code. The tester, if available begins the Selenium automation while the developer is working on the JavaScript code for client-side validation and the server-side code driven by rest-assured tests. After some time they both merge their code into the story branch and meet for a pairing session where they do some exploratory testing, review the unit tests and the integration tests and together refactor some of the tests to be more readable. The developer gets additional code reviews while the tester runs a quick exploratory test session based on the code changes for this story. After the system/integration tests pass the pre-commit tests on the build cloud, the code is checked into the master and all of the necessary tests are re-run. Voila! The build is still green The entire process took less than a day and the right level of quality has been achieved. There is joint ownership of the tests and the tests are well written. If they break in the future the developer will have no qualms about fixing them right away or rolling back the commit to keep the build green — “software by accretion” is now part of the culture.