Okay, so assuming that you’ve read part 1 and part 2, you’ll know that I don’t have an issue with TDD. Rather, I have an issue with some of the claims that TDD evangelist use and with the data that’s often produced to support TDD. I feel that the papers used to support TDD are very poor. There is a lack of data points from which to draw meaningful conclusions and in some instances the papers draw conclusions that are simply not supported by the data that they have collected themselves. The Janzen paper is the most egregious example of this.
In this final post, I’ll talk about why TDD is important and what data would be evidence of the benefit of TDD.
Why is Test Driven Development important?
Eighteen months ago, I would have said that TDD was a slam dunk. Now that I’ve taken the time to look at the papers more closely … and actually read more than just the introduction and conclusion … I would say that the only honest conclusion is that TDD results in more tests and by implication, fewer defects. Any other conclusions such as better design, better APIs, simpler design, lower complexity, increased productivity, more maintainable code etc., are simply not supported.
I feel it’s important to emphasize all the studies without exception show a correlation between the use of TDD, an increase in the number of test cases produced, and corresponding improvements such as a decline in the number of defects. As I mention in part 1, I believe this is sufficient for TDD to be recommended as a developer practice. Defects in the code introduce a negative feedback loop. The more defects there are, the more defects will be introduced. And the result is that the cost of change is exponential. Only by actively driving down the defects in our code are we able to deliver new functionality, and modify existing functionality with a reasonably constant cost of change.
Why is TDD important? Because, TDD allows us to break the negative feedback loop and maintain a constant cost of change.
What would a meaningful study include?
Test Driven Development is not new. It’s been practiced for 15-20 years now, and has been mainstream for 10 years now. When I first started looking into the evidence for TDD, I thought there would surely be either (i) a comprehensive (ie no caveats) study, or (ii) that there would be an overwhelming body of data on the benefits of TDD. This has not proven to be the case. So, either the alleged benefits of TDD are intangible, or the alleged benefits of TDD are simply not there. Either option is likely.
A large part of the difficulty in study TDD is that it takes a significant period of time to learn and assimilate. I typically estimate that it takes about 9 months of solid practice, and Bas suggests that it takes 1 year . The reason for this is because TDD fundamentally changes how developers think about writing code. It’s like adopting a healthier lifestyle, only without the granola.
A study that would be convince under these conditions is not easy to find and it may well be impossible. I would suggest that a study which addressed all the exceptions of the previous reports would go a long way to understanding the true benefits (or not) of TDD. My off-the-top-of-my-head list of criteria for such a study, includes (a) a multi year study with a minimum of 3 years consecutive years (b) a study of several teams (c) team sizes must be 7 (+/-2) team members and have (d) at least 4 full time developers. Finally, (e) it needs to be a study of a product in production, as opposed to a study based on student exercises. Given such as study it would be difficult to argue their conclusions, whatever they be.
My personally feeling is that such a study is possible, but unlikely. It would take an exceptionally commit PhD candidate to negotiate such a study and see it through. And, this brings me to my final question which is, given what we currently know and understand should teams and developers adopt TDD?
Should teams and developers adopt TDD?
Yes. I believe that teams and developers should adopt TDD, and they should adopt it solely for the reduced number of defects.
If teams then go on to experience some other unquantifiable benefits of TDD (for example, simpler design, simpler code, increased productivity etc.) that should be considered an added bonus, but should not be a factor in their decision making progress.
 Personal conversation with Bas Vodde.
 Google spreadsheet of the list of papers reviewed.