Technology, mountain biking, politics & music.

Archive for the ‘TDD’ Category

CITCON Asia-Pacific 2009

without comments

The location for CITCON Asia-Pacific 2009 has just been announced, and the winner is: Brisbane! Huzzah.

Written by Tom Adams

September 14th, 2008 at 4:45 pm

Posted in Agile,BDD,TDD

Tagged with

Microsoft release Pex

without comments

My googlebot just informed me that Microsoft has released Pex, which “generates Unit Tests from hand-written Parameterized Unit Tests through Automated Exploratory Testing based on Dynamic Symbolic Execution” (what a mouthful!). At first glance, it looks very similar to JUnit’s Theories and Tony’s Reductio. The difference appears to be that Pex generates the tests statically (don’t know how long they hang around) rather than dynamically like Reductio and JUnit. Ah, QuickCheck for Microsofties…

Written by Tom Adams

June 3rd, 2008 at 2:25 pm

Scala with Instinct

with one comment

I was lucky enough to spend that last half of last week in a Scala course run by Tony Morris (contact us at Workingmouse if you’re interested in Scala or the course).

The Scala syntax is very malleable and lends itself well to expressing DSLs. With this in mind, here’s a simple Instinct specification written in Scala (it doesn’t work yet, though I think this is an IDE issue, rather than a limitation)

import com.googlecode.instinct.marker.annotate.Specification
import com.googlecode.instinct.integrate.junit4.InstinctRunner
import org.junit.runner.RunWith

class AnEmptyStack {
    def isEmpty = {}

Once we get the next release out, I’d like to spend some time playing around with using Scala to spec Java objects, there should be no reason it can’t also be used to spec out Scala code also (there may be some other things we need to do).

Here’s some RSpec like (valid) Scala:

describe "An empty stack" = {
    it "is empty" = {}

Written by Tom Adams

January 14th, 2008 at 11:27 am

Posted in Instinct,Java,Scala,TDD

Instinct 0.1.6 Release

without comments

I’m happy to announce the release of Instinct 0.1.6. Thanks to all our new users and especially to the guys at VLC & SAP who’ve helped us apply Instinct in anger, and Sanjiv, who’s come on board with development.

Downloads are available from the project site.

This release includes a raft of updates, most notably auto-creation of specification doubles (mocks, stubs & dummies), automatic reset and verification of mocks, a cleanup of the state-based expectation API, fixes for Eclipse JUnit integration, custom classpath support in the Ant task and inherited contexts.

Here’s the full list of updates:


  • Remove the need for @Context annotation.
  • Automatic creation of specification doubles: mocks, stubs and dummies.
  • Automatic reset and verification of mocks.
  • @BeforeSpecification?, @AfterSpecification?, @Specifications (and naming convention equivalents) can be used across base and subclasses.

Expectation API

  • Make expectations more like natural language. eg. isEqualTo(), doesNotEqual(), etc. Existing code using equalTo(), etc. will need to be updated.
  • Collection checkers: hasTheSameContentAs(Collection) and hasTheSameContentAs(E…). These only check content and not the order of elements.
  • Ensure all “collection” classes (Array, Map, Set, List, String, SharSequence?) have similar size checkers available.
  • Added file checker
  • Better error messages for hasBeanProperty and hasBeanPropertyWithValue.

JUnit integration

  • Fix Eclipse unrooted context.

Ant integration

  • Support for custom classpath.
  • Quiet specification result formatting (only shows errors and pending specs).
  • Use correct project logging level for errors, etc.

jMock integration

  • Support states: Mockery.states(String).


  • Removed reliance on Boost, transferred all relevant Boost classes locally.
  • jMock 2.4.
  • Downgraded to CGLib 2.1.3 (for Maven integration).


  • Miscellaneous NullPointerExceptions? and null related problems in state expectation API.
  • (defect-3) IterableChecker? should have a containsOnly method or something.
  • (defect-8) @BeforeSpecification? does not run if implemented in an abstract base class.
  • (defect-20) CEclipse Junit4 InstinctRunner? shows tests under the “Unrooted Tests” node.
  • (defect-22) Context treeview shows baseclass and subclass when only subclass is run.
  • (defect-23) Overridden specifications run twice.

Written by Tom Adams

December 14th, 2007 at 4:57 pm

Posted in Agile,BDD,Instinct,Java,TDD

Better Testing Through Behaviour

with 3 comments

Just finished my talk on Better Testing Through Behaviour at OSDC 2007. All in all it went pretty well.

The presentation is available as is the paper.

Chris sent me some pictures of my talk, here’s one.

Tom at OSDC 2007

There’s been some posts (good and indifferent) resulting from the talk:

And for those interested, the slides for OSDC 2007 are available on SlideShare, photos on flickr.

Written by Tom Adams

November 27th, 2007 at 1:46 pm

Instinct 0.1.5 Release

without comments

I’m happy to announce the release of Instinct 0.1.5. This release includes a raft of updates, most notable a behaviour expectation API (using jMock 2) and JUnit 4 integration. Here’s the full list of new stuff:


  • Initial cut of pending specifications (doesn’t report correctly in JUnit).
  • Added marking of specifications with naming conventions.

Expectation API

  • Behavioural expectation API (i.e. mocking).
  • Expected exception (simple version, encoded in specification annotation).
  • Add regular expression checking to string checker.
  • Add empty checks to array checker.


  • Added JUnit 4 runner, with @ContextClasses annotation.
  • Moved old mocking code to jMock 2, removed jMock 1.2 dependency.


  • Prototype QuickCheck style properties API (like Popper/JUnit 4 Theories).


  • Fixed Issue 4 – Add newlines between context output in brief runner.
  • Fixed concurrent modification issue with the JUnit runner.


  • Upgraded to jMock 2.2.
  • Upgraded to JUnit 4.4.

Written by Tom Adams

October 17th, 2007 at 9:28 pm

Posted in Agile,BDD,Instinct,Java,TDD

What is Behaviour Driven Development?

with 6 comments

I’ve been reading a lot about BDD of late, mainly from the .Net community and mainly about story-level BDD, mostly driven by RSpec and NBehave. I finally got around to writing my take on BDD, and have had a paper accepted at the Open Source Developers’ Conference coming up in November. The following is an excerpt from my paper, the full text of my submission is available as a PDF.

What is Behaviour Driven Development?

“The act of writing a unit test is more an act of design than of verification. It’s also more an act of documentation than of verification. The act of writing a unit test closes a remarkable number of feedback loops, the least of which is the one pertaining to verification of function.” Robert C. Martin

Behaviour Driven Development (BDD) is a development practice arising out of agile development methodologies, first developed by Dan North in response to the problems experienced using and teaching TDD [10]. At its core, BDD is a refinement to TDD that shifts the emphasis from testing to specification and is in effect the best practices of what developers practising TDD have been doing all along. While this doesn’t seem like a major change, the shift in emphasis from testing to specification brings a number of important flow on effects that positively impact the development of both test and production code. The following sections describe these effects.

Ubiquitous Language

For many, the language used to express a concept has an impact on the way they think about that thing. Goh states that the “abstractions and concepts you use to express yourself in any language … shape the way you think about the problem you are solving” [5]. This is supported in linguistics by the Sapir-Whorf hypothesis which postulates that there is “a systematic relationship between the grammatical categories of the language a person speaks and how that person both understands the world and behaves in it” [18]. For software development, this implies that the language we use to describe the software constructs has an impact on how we create those constructs; by changing the language we affect the code we create. As a concrete example, consider that this line of reasoning is used to justify assigning meaningful names to functions and variables. Also, developers implicitly feel the impact of language when working with a well designed API; an API that provides well named classes and functions within a well designed architecture.

BDD supports using meaningful language in two ways. Firstly, it focuses on “getting the words right”, encouraging good naming of classes, methods and variables. Secondly, it borrows concepts from Domain Driven Development (DDD) to bridge the gap between technical and business artefacts [3]. BDD attempts to capture the behaviour of the domain using a clear and concise syntax that everyone can understand, forming consensus around not only the domain artefacts but also the run-time behaviour of the system [13, 16]. BDD calls the shared language used by all stakeholders the ubiquitous language of the domain.

Design Focus

“A waterfall ‘designer’ starts from an understanding of the problem and builds up some kind of model for a solution, which they then pass on to the implementers. An agile developer does exactly the same, but the language they use for the model happens to be executable source code rather than documents or UML.” Kerry Buckley

TDD advocates acknowledge that one of the most important outcomes of practising TDD is its influence on the design of the resulting code. Test driven code has less defects [7] than non-test driven code and is often more cohesive and less coupled than non-test driven code. However the emphasis on “testing” limits TDD’s uptake and effectiveness in driving design (from both a managerial and technical point of view). BDD acknowledges that design is one of the most important outcomes of testing by providing language and tools that support the creation of well-designed code.

Behaviour Focus

“So if it’s not about testing, what’s it about? It’s about figuring out what you are trying to do before you run off half-cocked to try to do it.” Dave Astels

The difference between testing and specifying may be subtle, but it leads to an increased focus on the behaviour of a piece of code. By focusing on behaviour, BDD frameworks break the traditional 1-to-1 mapping of a unit test class to a production class. The location of the behaviour is no longer important, resulting in test code that is less coupled to its production counterpart and less fragile when production code is refactored. Rather than a 1-to-1 mapping, BDD encourages an M-to-N mapping, allowing as many specification classes as needed to specify the required behaviour. Furthermore, behaviour contexts provide the ability to group related specifications into a single class. Contexts are usually created around different states of an object, such as an empty and non-empty stack. A single context focuses on one thing, thereby reducing the confusion of having multiple states (of a subject) and multiple assertions in a single test.

Figure 1 shows several problems encountered test-driving, and identifies the BDD practices and Instinct features that address the problem.

TDD Problem Matrix

Figure 1. Problems with traditional TDD that BDD and/or Instinct overcome.

BDD Frameworks

Early BDD frameworks focused on acceptance or story-level behaviour. Frameworks such as JBehave [6] allow an application’s behaviour to be described in terms of stories and scenarios. Subsequent frameworks such as RSpec [15] (for Ruby) and NSpec [12] (for C#) focused more on the code level of testing, offering an xUnit equivalent feature set for low level specifications. Both approaches are legitimate, and use ubiquitous language to describe the desired behaviour of the system at the appropriate level.

Story-level frameworks provide APIs to allow business stakeholders (e.g. XP’s Customer, Scrum’s Product Owner, business/functional analyst or traditional QA tester) to understand the behaviour of the system. Typically these frameworks use the “as a/i want/so that” form of story definition and “given/when/then” form of test lifecycle, and also provide an API to plug in the implementation of the acceptance test (the code that supports the wording). Listing 1 shows a story-level specification in Ruby.

Story "transfer to cash account",
%(As a savings account holder
  I want to transfer money from my savings account
  So that I can get cash easily from an ATM) do

  Scenario "savings account is in credit" do
    Given "my savings account balance is", 100
    Given "my cash account balance is", 10
    When "I transfer", 20
    Then "my savings account balance should be", 80
    Then "my cash account balance should be", 30

  Scenario "savings account is overdrawn" do
    Given "my savings account balance is", -20
    Given "my cash account balance is", 10
    When "I transfer", 20
    Then "my savings account balance should be", -20
    Then "my cash account balance should be", 10
Listing 1. Ruby BDD story framework (RBehave) [11]

Code-level frameworks provide APIs that allow developers to design and specify the behaviour of low level components of the system. These frameworks provide xUnit equivalent feature sets and support ubiquitous language by encoding it into the APIs used to describe the expected behaviour of a system, as well as the interaction between the specification code and the BDD framework itself. The APIs used for describing expected behaviour typically take the form of Domain Specific Languages (DSLs) or fluent APIs. Consider the following abridged code samples from the Ruby framework RSpec and the C# framework NSpec.

describe "non-empty Stack" do
    it "should return the top item when sent #peek" do
        @stack.peek.should == @last_item_added

public void NSpecSpecificationExample() {
Listing 2. RSpec & NSpec behaviour specifications.

BDD frameworks also make use of meaningful naming and readability when defining the way a developer interacts with the framework. They provide flexible, explicit and meaningful ways of defining specification methods and classes. Owing to the flexibility of the language, Ruby frameworks set the standard on readability. Java and C# frameworks simulate the freedom provided by more flexible languages by exploiting generics, chained method calls and method naming techniques.

Listing 3 shows the same specification using RSpec syntax and Instinct syntax respectively, highlighting the readability provided by the framework syntax. In these examples the specification should be read as “a non-empty Stack should return the top item when sent peek”.

describe "non-empty Stack" do
    it "should return the top item when sent #peek" do
        @stack.peek.should == @last_item_added

class ANonEmptyStack {
    void shouldReturnTheTopItemWhenSentPeek() {
Listing 3. Behaviour specifications using RSpec and Instinct.

BDD Terminology

As BDD focuses on “getting the words right”, it introduces subtle changes to the traditional TDD vocabulary. BDD frameworks such as Instinct also adopt the terminology introduced by Meszaros [9]. The major concepts introduced in BDD are defined below.

A method/function in which the behaviour of a piece of code is specified. Specifications are executable examples that guide the design process and provide both documentation and tests. Specifications are analogous to test methods in xUnit frameworks.
A context in which a certain behaviour is valid. Contexts can be used to group specifications together and also to set up the state in which expectations hold.
An object that interacts with another object, providing a service to it. Collaborators are usually mocked or stubbed out in specifications in order to verify interactions between them and the subject.
An actor is an object that has a role in a specification. Subjects, dummies, stubs and mocks are all actors.
The class whose behaviour is under scrutiny.
An implementation of an interface (or extension of a class) that is only used for testing. Doubles can be manually created (plain classes that implement an interface) or auto-created via a framework.
Dummy objects are passed around but never actually used, they are usually used just to fill parameters to simplify specifications (where their behaviour may not be important). Dummies will throw exceptions if methods are called on them. They are the simplest form of test double implementation.
Stubs respond to method calls made during a test by providing canned answers. Stubs may record information about calls, such as an email gateway stub that remembers the messages it ‘sent’. Stubs do not fail if methods are called or the order in which they are called (if at all).
Mocks are more advanced stubs, that not only respond to calls made during a test but are also pre-programmed with expectations which form a specification of the calls they are expected to receive. Mocks will throw an exception if they receive a call they weren’t expecting and are checked (called verification) to ensure they received all the calls they expected. Some mocks also verify the order of calls made.
A fixture is a known set of data (or commands to setup that data) that provide the environment for a set of tests. Fixtures work well when you have a bunch of tests that work on similar data reducing the complexity of your testing environment (fixtures can also have a downside when overused between tests that should have independent data).

Further Reading

Read the full paper from which this excerpt was taken.


  1. Apache Ant,, 2007.
  2. Astels, D., A New Look at Test-Driven Development,, 2005.
  3. Bellware, S., Behavior Driven Development is…,, 2007.
  4. Buckley, K., Concurrent design and development – a better spin?,, 2007.
  5. Goh, T., TDD vs BDD,, 2007.
  6. JBehave, 2007,, 2007.
  7. Jeffries, R. and Melnik, G., TDD: The Art of Fearless Programming, IEEE Software,, 2007.
  8. jMock – A Lightweight Mock Object Library for Java,, 2007.
  9. Meszaros, G., xUnit Test Patterns: Refactoring Test Code, Addison-Wesley, 2007.
  10. North, D., Introducing BDD,, 2006.
  11. North, D., Introducing rbehave,, 2007.
  12. NSpec,, 2007.
  13. Ocampo, J., Comment on Adverbs – A Partial Obstruction to BDD,, 2007.
  14. QuickCheck: An Automatic Testing Tool for Haskell,, 2007.
  15. RSpec 1.0.8 Examples,, 2007.
  16. Sahayam, S., Behaviour-Driven Development (BDD) – A first look, Workingmouse Internal Blog (not publically available), 2007.
  17. TestDox,, 2007.
  18. Wikipedia, Sapir–Whorf hypothesis,, 2007.

Written by Tom Adams

October 14th, 2007 at 8:44 pm

Posted in BDD,Instinct,Java,Ruby,TDD

Next Generation Java Testing

without comments

Via Alex via Andy, Hani and Cédric have announced that their book Next Generation Java Testing is available for pre-order.

I was one of the technical reviewers for the book earlier in the year, and I saw it come a long way from where it started to where the reviewers left it, and it seems to have come a far way since. It started out as a fairly TestNG heavy book and hopefully ended as being a good general reference to practical developer testing with the current stable of Java testing tools. I would have liked to have seen extraction and more coverage of some of the common patterns used in testing (edge, statics, external code, new calls, etc.), but hey, it’s not my book!

Interesting, one of the commenters on Alex’s post asks about coverage of QuickCheck and related tools. Inspired by Tony (fellow mouse & developer on ScalaCheck) and Popper/JUnit’s Theories, Chris and I have been working on bringing this to Instinct. Hopefully I’ll get a chance to extend this work in the next few weeks. I see this and BDD as the next big steps in Java testing frameworks.

Written by Tom Adams

September 10th, 2007 at 8:59 pm

Posted in Instinct,Java,TDD


without comments

Paul O’Keeffe and I headed down to Sydney last weekend for the CITCON Asia Pacific conference. The two of us and Paul King managed to get lumped together as the “Brisbane Cluster”, owing to us all sitting together during the opening. Had a great time, somehow I managed to do more talking than listening, hopefully no one fell asleep! I convened or co-convened the following sessions:

After the conference and all the Sydney-siders headed home, we all headed out for a few beers and dinner. Definitely worth the flight down, we’ll all be back next year.

There’s some great photos from Rux and Mark including one of yours truly.

Thanks to the firm for sending me down!

Written by Tom Adams

August 2nd, 2007 at 8:56 pm

Behavior Driven Design with NUnit Presentation

with one comment

Another presentation on BDD with NUnit.

I recently gave a presentation on Behavior Driven Design at the AlamoCoders user group here in San Antonio.

Given the hour constraint I had and the level 100 rating of the presentation, I was pressed to deliver the absolute essentials to get someone up and running with BDD.

One of the more challenging aspects of presenting an entry level BDD presentation is the fact that you have to assume that some people have no exposure to any XUnit type development. The founding principles of TDD (Red, Green, Refactor) must be reiterated before diving into BDD. Once we got passed just what TDD is, the transition to BDD went much more smoothly then I had ever hoped.

If you are interested I have uploaded the presentation and the source to the Los Techies downloads library.

The presentation is also available as a PDF.

Written by Tom Adams

June 21st, 2007 at 4:25 pm

Posted in BDD,TDD