Discussion: TDD and Criteria-Based Test Design

Feature image by Roysneak (Rugby Scrum)

The book for reference in all the post is Introduction to Software Testing, 2nd Edition. 

  • What is “correctness” in agile processes?

All agile methods have an underlying assumption that instead of defining all behaviours with requirements or specifications, we demonstrate some behaviours with specific tests. The software is considered correct if it passes all particular set of tests. But to be honest, no one is sure of what the term correctness mean when applied to a computer program.

  • Do TDD tests do a good job testing the software?

Test Driven development is an agile approach (agile is a mindset not a methodology). So it’s a good tool to be responsive to change, because its focus is create a system that does something as early as possible. TDD allow us to obtain critical feedback quickly as possible. For example today at work something in the backend crashed, but it’s better that if it’s going to fail, that fail as quickly as possible. In conclusion, yes, it’s good depending your focus.

  • Can we automate our tests without TDD?

Yes we can, automation is not unique to TDD

  • Can we use TDD without automating our tests?

Yes we can, but this mean it will be manual, so it will have to spend more time and it will return the human factor. So in other words, it’s a no in disguise.

  • What four structures do we use for test criteria?
  1. Test requirement
  2. Coverage criterion
  3. Minimal test set
  4. Minimum test set
  • What usually prevents our tests from achieving 100% coverage?

As discussed later, there are infinite number of inputs and can’t be explicit enumerated. But we can divide up the input space to maximize the number of faults found per case. To be honest a 100% coverage is not even a realistic or possible goal. Secondly, some requirements can’t be satisfied and are very hard to find (as a purple M&M’s according to the book). Sometimes this is because the existence of dead code that can’t be reached.

Another reason, what does 100% even mean? Where is the criteria for saying how much worst is 99% from 100%.

  • Some organizations in industry who adopt TDD report that it succeeds very well, and others report that it fails. Based on your knowledge of TDD and any experience you have, why do you think it succeeds sometimes but not all?

Even the book knows that the main cost of Agile methods for testing (in this particular case TDD) is that a lot of things are different. So it’s not easy for established teams and companies to change their mindset just like that. Therefore, sometimes TDD fits the project, sometimes it doesn’t. It’s not the methodology but how it’s implemented.

Image credits to New Line Cinema
  • A few software organizations use test criteria and report great success. However, most organizations do not currently use test criteria. Based on your knowledge and experience, why do you think test criteria are not used more?

Based on my experience, test criteria is still used, but not as it used to be. It’s more about following what the user story specifies. The term complete testing or full coverage are poor designed because the potential inputs for most programs are unlimited.

In traditional software development, system requirement are often questionable in terms of how complete and current they are. In agile methods, they are undocumented! So what do system testers do?

Amman, P. and Offutt, J. (2016)

As the book say, there are no definite answer for test criteria.


Amman, P., Offutt, J. (2016) Introduction to Software Testing, 2nd Edition, Cambridge Press Chapter 4 and 5. https://cs.gmu.edu/~offutt/softwaretest/

Cover art by Peter Hoey http://www.peterhoey.com

The social side of a bug’s life

Logo of a Bugs life credited to Pixar

For this blog post I’ll use the word bug referring to either a fault/error/failure.

Jorge Aranda and Gina Venolia wrote a paper called The Secret life of Bugs.   (click here if you want to read it)In general it talks about how even simple bugs are very complicated to track with just automation, mostly because the human factor.

It’s very funny painful how sometimes when I work on a project, 40% of the time is dedicated to develop a functionality and the rest fixing the secondary effects. If this effect is a bug then I write an issue either on the repository or anything related to the project management. But here is the trick, as humans we believe we can keep track of everything in our minds, but in my experience I have seen a typical scenario:

(I | He | She | They | We ) will write about the bug later or in the next sprint

Something that will be fixed very late or never.

The authors Jorge and Gina have proof of why we can’t fully rely on electronic repositories a 100%. Which is mainly related to coordination problems.

Do we always record the necessary information to understand the whole story of the bug?

According to their studies the researched bugs databases had the next problems:

  • Some bugs in the records are not bugs in strict sense.
  • Some bugs have duplicate records
  • Some bugs exhibit symptoms that are initially seen as different bugs and recorded separately
  • Some bugs do not always die when they are marked as closed
  • Some bugs basic field in the records are incorrect
  • Some bugs have wrong status.
  • Some life of bugs will never be understood without a face to face or personal investigation.

So in the end it is really interesting to see how when they tried to see the track of the life of the bug they reached dead ends. Or found that histories omitted important details of the bug. I believe our social human factor is the key to why we can’t trust 100% information extracted from electronic repositories.

Test a Calc Class

At first, this is our Calc class:

public class Calc {    
static public int add (int a, int b) {
return a + b;

It only has the add method. Hence, we made failing tests for two methods to be implemented: mult and div, for multiplication and division. 

And a failing test for the add method as well.

The input values for the addition are incorrect. Also, the mult and div message are not recognized by the class.

As we can observe, all tests failed…

testAdd failed
testDiv failed
testMult failed

Then, we changed the testAdd input values, so the test won’t fail.

Now the addition will be correct.

Next, we implemented the methods for multiplication and division in the Calc class:

Calc now has three methods

Finally, we ran the tests again and had a successfull execution.

The class recognizes all Calc methods.

Week 4 exercise 3

Retrieved from SWE 437 In Class Exercise #5 JUnit. Code available in the end of the blogpost.

Question 1

Given the 4 @Test methods shown, how many times does the @Before method execute?

Just one time, it will setup the environment for the class we want to test.

Question 2

The contract for equals() states that no exceptions may be thrown. Instead, equals() is supposed to return false if passed a null argument. Write a JUnit test that verifies this property for the EH class.

@Test public void noNPE() {
assertEquals(false, eh1.equals(null));
assertEquals(false, eh2.equals(null));

Question 3

Using the given EH objects, write a test that verifies that equals() returns false if the objects are, in fact, not equal.

@Test public void equalsFalse() {
assertEquals(false, eh1.equals(eh2));
assertEquals(false, eh2.equals(eh3));

Question 4

Using the given EH objects, write a test that verifies that equals() returns true if the objects are, in fact, equal.

@Test public void equalsTrue() {
assertEquals(true, eh1.equals(eh1));
assertEquals(true, eh2.equals(eh2));
assertEquals(true, eh3.equals(eh3));

Question 5

Using the given EH objects, write a test to verify that hashCode() is consistent with equals. This test should fail if hashCode() is commented out (as shown), but pass if hashCode() is implemented.

public void hashConsistent() {
assertEquals(false, eh1.hashCode() == eh2.hashCode());
assertEquals(true, eh2.hashCode() == eh2.hashCode());
assertEquals(true, eh3.hashCode() == eh1.hashCode());

Code repository

4 of 4 test passed


Week 4 Delivery

In this week we didn’t do anything related to the project, otherwise we focused more on the part of getting more involved in testing and we did choose the technologies for testing in JavaScript: Mocha and Chai.

“Mocha is a feature-rich JavaScript test framework running on Node.js and in the browser, making asynchronous testing simple and fun.”


“Chai is a BDD / TDD assertion library for nodeand the browser that can be delightfully paired with any javascript testing framework.”

Resultado de imagen para chaijs

On the other hand, we did meet up and did the team homework, research about testing and its techniques and best practices.

Chapter 3 Team Exercise

Why do testers automate tests? What are the limitations of automation?

Because it reduces the cost of testing and human error, also it makes regression testing easier because it allows a test to be run repeatedly. Excise tasks are candidates for automation.

Although this is an essential process, it has limitations:

  • Continuous adjustment to testing code
  • Cannot be used for testing UX
  • Script reliability depends on programmer
  • Requires silo elimination

How the inheritance hierarchy can affect controllability and observability?

Having a deep inheritance tree of classes could become more complex to test, because the sub classes are dependent of its father, that means, if we modify or update something in the super class, it will affect all its respective children, which means that every test affected by the modification have to be updated for the sub classes. Therefore, inheritance does not guarantee that a method tested in the context of the super class will work correctly or in the same way that in the context of the sub class or sub classes. Thus, this affects controllability because we will need to control every super class and its sub classes, and if wee have a deep inheritance tree, it will be very complicated. On the other hand, it affects observability becase we need to observe a large number of sub classes that actually are doing the same work as the super class.

Develop JUnit tests for the BoundedQueue class.

You can find the code here.

Delete the explicit throw of NullPointerException in the Min program. Verify that the JUnit test for a list with a single null element now fails.

If we delete the line

if (result == null) throw new NullPointerException ("Min.min");

We can see that the test testForSoloNullElement() fails.

6. Consider the following example class. PrimeNumbers has three methods…

(a) A test that does not reach the fault

void testComputePrimesA() {
PrimeNumbers primeNumbers = new PrimeNumbers();

assertEquals("[]", primeNumbers.toString());
// it never enters the while loop

(b) A test that reaches the fault, but does not infect

void testComputePrimesB() {
PrimeNumbers primeNumbers = new PrimeNumbers();

  assertEquals("[2, 3, 5, 7, 11, 13, 17]",

(c) A test that infects the state, but does not propagate

void testComputePrimesC() {
PrimeNumbers primeNumbers = new PrimeNumbers();

assertEquals("[2, 3, 5, 7, 11, 13, 17, 19]",
// it is affected because it doesn't include 19

(d) A test that propagates, but does not reveal

This test is not possible because the faults starts at the 19 (we’re not taking in consideration 9, because it is not prime), all of subsequent primes that ends in ‘9’ won’t be in the result value.

(e) A test that reveals the fault

void testComputePrimesE() {
PrimeNumbers primeNumbers = new PrimeNumbers();

  assertEquals("[2, 3, 5, 7, 11, 13, 17, 19]",

Exercise 7.

The computePrimes is now using the Sieve of Eratosthenes to find the primes. The first false positive result is again 9, it doesn’t print it, to reach the fault it has to compute the first 7 prime numbers. The RIPR model should be used to know if there’s a fault in our system, on the contrary, we could think that our program is running correctly, but we have so many false positives waiting to be propagated.

JUnit and Intelli-J (mac os)

What is Intelli-j? it is basically and IDE for software development, its first release was in the year 2001. It was developed by JetBrains and it’s not based on Eclipse. But i’m pretty sure you aren’t here for this but to to learn how to do a basic setup for J-Unit and Intelli-j community edition. Here is what you’ll need:

  • Mac Os X 10.5 or superior
  • 1GB of RAM as minimum.
  • JDK 1.8

The steps to follow can be found in the next video:

Quick configuration video

If link is broken go here: https://www.youtube.com/watch?v=Z8V7rnvDeB4

Detailed configuration setup video

Link: https://www.youtube.com/watch?v=6k-AA1uR5jY&feature=youtu.be

In general IntelliJ and JUnit 5 configuration is really simple. Mostly because a lot of the Maven integration is done automatically by IntelliJ. So if your goal is to learn how to set up Maven into a Java project, this is not the best method. But if you are looking for simplicity this is a very good choice.

Git repository: https://github.com/kevintroko/JUnitTest

JUnit 5 documentation:https://junit.org/junit5/docs/current/user-guide/