Discussion: TDD and Criteria-Based Test Design

Feature image by Roysneak (Rugby Scrum)

The book for reference in all the post is Introduction to Software Testing, 2nd Edition. 

  • What is “correctness” in agile processes?

All agile methods have an underlying assumption that instead of defining all behaviours with requirements or specifications, we demonstrate some behaviours with specific tests. The software is considered correct if it passes all particular set of tests. But to be honest, no one is sure of what the term correctness mean when applied to a computer program.

  • Do TDD tests do a good job testing the software?

Test Driven development is an agile approach (agile is a mindset not a methodology). So it’s a good tool to be responsive to change, because its focus is create a system that does something as early as possible. TDD allow us to obtain critical feedback quickly as possible. For example today at work something in the backend crashed, but it’s better that if it’s going to fail, that fail as quickly as possible. In conclusion, yes, it’s good depending your focus.

  • Can we automate our tests without TDD?

Yes we can, automation is not unique to TDD

  • Can we use TDD without automating our tests?

Yes we can, but this mean it will be manual, so it will have to spend more time and it will return the human factor. So in other words, it’s a no in disguise.

  • What four structures do we use for test criteria?
  1. Test requirement
  2. Coverage criterion
  3. Minimal test set
  4. Minimum test set
  • What usually prevents our tests from achieving 100% coverage?

As discussed later, there are infinite number of inputs and can’t be explicit enumerated. But we can divide up the input space to maximize the number of faults found per case. To be honest a 100% coverage is not even a realistic or possible goal. Secondly, some requirements can’t be satisfied and are very hard to find (as a purple M&M’s according to the book). Sometimes this is because the existence of dead code that can’t be reached.

Another reason, what does 100% even mean? Where is the criteria for saying how much worst is 99% from 100%.

  • Some organizations in industry who adopt TDD report that it succeeds very well, and others report that it fails. Based on your knowledge of TDD and any experience you have, why do you think it succeeds sometimes but not all?

Even the book knows that the main cost of Agile methods for testing (in this particular case TDD) is that a lot of things are different. So it’s not easy for established teams and companies to change their mindset just like that. Therefore, sometimes TDD fits the project, sometimes it doesn’t. It’s not the methodology but how it’s implemented.

Image credits to New Line Cinema
  • A few software organizations use test criteria and report great success. However, most organizations do not currently use test criteria. Based on your knowledge and experience, why do you think test criteria are not used more?

Based on my experience, test criteria is still used, but not as it used to be. It’s more about following what the user story specifies. The term complete testing or full coverage are poor designed because the potential inputs for most programs are unlimited.

In traditional software development, system requirement are often questionable in terms of how complete and current they are. In agile methods, they are undocumented! So what do system testers do?

Amman, P. and Offutt, J. (2016)

As the book say, there are no definite answer for test criteria.


Amman, P., Offutt, J. (2016) Introduction to Software Testing, 2nd Edition, Cambridge Press Chapter 4 and 5. https://cs.gmu.edu/~offutt/softwaretest/

Cover art by Peter Hoey http://www.peterhoey.com

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s