The benefits of Test Driven Development: Why TDD? (part 3)

Photo Credit: Eivindw

If you have already read the previous two posts on Test Driven Development (TDD), you should probably do so before continuing (part 1, and part 2). I’ll wait here until you’ve read them.

Okay, so assuming that you’ve read part 1 and part 2, you’ll know that I don’t have an issue with TDD. Rather, I have an issue with some of the claims that TDD evangelist use and with the data that’s often produced to support TDD. I feel that the papers used to support TDD are very poor. There is a lack of data points from which to draw meaningful conclusions and in some instances the papers draw conclusions that are simply not supported by the data that they have collected themselves. The Janzen paper is the most egregious example of this.

In this final post, I’ll talk about why TDD is important and what data would be evidence of the benefit of TDD.

Why is Test Driven Development important?

Eighteen months ago, I would have said that TDD was a slam dunk. Now that I’ve taken the time to look at the papers more closely … and actually read more than just the introduction and conclusion … I would say that the only honest conclusion is that TDD results in more tests and by implication, fewer defects. Any other conclusions such as better design, better APIs, simpler design, lower complexity, increased productivity, more maintainable code etc., are simply not supported.

I feel it’s important to emphasize all the studies without exception show a correlation between the use of TDD, an increase in the number of test cases produced, and corresponding improvements such as a decline in the number of defects. As I mention in part 1, I believe this is sufficient for TDD to be recommended as a developer practice. Defects in the code introduce a negative feedback loop. The more defects there are, the more defects will be introduced. And the result is that the cost of change is exponential. Only by actively driving down the defects in our code are we able to deliver new functionality, and modify existing functionality with a reasonably constant cost of change.

Why is TDD important? Because, TDD allows us to break the negative feedback loop and maintain a constant cost of change.

What would a meaningful study include?

Test Driven Development is not new. It’s been practiced for 15-20 years now, and has been mainstream for 10 years now. When I first started looking into the evidence for TDD, I thought there would surely be either (i) a comprehensive (ie no caveats) study, or (ii) that there would be an overwhelming body of data on the benefits of TDD. This has not proven to be the case. So, either the alleged benefits of TDD are intangible, or the alleged benefits of TDD are simply not there. Either option is likely.

A large part of the difficulty in study TDD is that it takes a significant period of time to learn and assimilate. I typically estimate that it takes about 9 months of solid practice, and Bas suggests that it takes 1 year [1]. The reason for this is because TDD fundamentally changes how developers think about writing code. It’s like adopting a healthier lifestyle, only without the granola.

A study that would be convince under these conditions is not easy to find and it may well be impossible. I would suggest that a study which addressed all the exceptions of the previous reports would go a long way to understanding the true benefits (or not) of TDD. My off-the-top-of-my-head list of criteria for such a study, includes (a) a multi year study with a minimum of 3 years consecutive years (b) a study of several teams (c) team sizes must be 7 (+/-2) team members and have (d) at least 4 full time developers. Finally, (e) it needs to be a study of a product in production, as opposed to a study based on student exercises. Given such as study it would be difficult to argue their conclusions, whatever they be.

My personally feeling is that such a study is possible, but unlikely. It would take an exceptionally commit PhD candidate to negotiate such a study and see it through. And, this brings me to my final question which is, given what we currently know and understand should teams and developers adopt TDD?

Should teams and developers adopt TDD?

Yes. I believe that teams and developers should adopt TDD, and they should adopt it solely for the reduced number of defects.

If teams then go on to experience some other unquantifiable benefits of TDD (for example, simpler design, simpler code, increased productivity etc.) that should be considered an added bonus, but should not be a factor in their decision making progress.

[1] Personal conversation with Bas Vodde.
[2] Google spreadsheet of the list of papers reviewed.

, ,

17 Responses to The benefits of Test Driven Development: Why TDD? (part 3)

  1. @SCRUMstudy_ November 30, 2012 at 1:30 am #

    The benefits of TDD: Why TDD? (part 3) – http://t.co/KYfnUHjK

  2. André Orvalho December 5, 2011 at 7:11 pm #

    “The reason for this is because TDD fundamentally changes how developers think about writing code. It’s like adopting a healthier lifestyle, only without the granola.”,

    I am writing an article myself about what we can really know about TDD right now and after reading many opinions on some researches, I was thinking just like these post, but your words come with wisdom.

    I totally agree with this and a simple though comes to my mind, imagine programming when invented, the first technique was TDD, thats all we knew. then somebody comes along and tells us, wouldnt it be better to first program the whole damn thing and then test it? and we all would ask why? and would be trying to support that idea, which would lead us to the same problem we have right now. TDD definitely needs a prove to convince everybody else to change, why lose time in learning something completely different if it does not bring us any clear big advantage.

    “So, either the alleged benefits of TDD are intangible, or the alleged benefits of TDD are simply not there. Either option is likely.”

    than reading what you say here gives me also another idea, maybe making some assumptions.
    Knowing that in general TDD has more tests and that more tests = less bugs, we can think of an intangible quantifier that is the programmers confidence, coming from testing it’s application and see it work. It’s like when a man approaches a woman and he is confident, he is more likely to get her. we all know that from experience. or that when we give a talk, if we know well what we are talking about, we are more confident. if we are more confident, it will make the audience more confident and our ideas make more sense. so is the same for a programmer, when he is confident, he will produce quality code better, hence be more productive. but like said before this is very dependable and intangible since it depends on one’s psychology and we all know how complicated it is on humans.

    I also think that experience comes into play a lot and it does not let me believe any result from researches on universities. because i think to learn and use TDD you need discipline and follow the cycle almost 100%(for me there is always exceptions that confirm the rule) and I can’t believe that students who never used it, or even if they did and being myself one, they will be able to discipline themselves enough to apply TDD correctly. an example is reading this article that talks about the most common mistakes(http://gsd.ime.usp.br/seminars/2010/presentation_mauricio.pdf), which is not a great article, but knowing there is normal people doing them, I imagine what inexperienced students should be doing during those tests(cause mainly they were not supervised on how TDD was employed).

  3. Scrumology April 16, 2010 at 1:27 am #

    Hi Dave, >Not supported by papers and studies, etc. Right.Yes, absolutely. If you'll notice, I have restricted the last three posts on TDD to academic papers and studies. For good or bad, academic peer reviewed papers are the closest that we have to impartial third party opinions. > If you ask people who have to understand and maintain production code, what> do they have to say about the design of code that was test driven vs. code that> was not?Actually, there's very little commentary on the net. Michael James pointed me to probably the most interesting work out there by Keith Braithwaite. Keith's work is great, and it has the same concerns that I've been discussing with the other works … limited dataset and lots of caveats. Keith himself says (emphasis is his): “managed to find the time to look at a /very/ small sample of Java codebases”. Having said that, I'd encourage everyone to read all three posts in the series. Thoughtful stuff.>I've been struck by the fact that most of the researchers and analysts don't >appear to understand what they're observingAgreed. 100%.>They “prove” nothing.I'm not looking for proof. That is unrealistic. When I started this I was looking for either a) a preponderance of data [showing the benefits of TDD], or b) unqualified articles. I'm still looking.>A meta-question comes to mind … Although I agree that there are broader questions to be consider, I still feel that peer review articles are the closes that we currently have to an independent third party. I won't address either of your meta questions in this reply because I view them as being off topic.>With all that in mind, I think there is a point of diminishing returns in> worrying about how many studies exist or how convincing they are.There are two reasons way I think this is important, and why I've devoted 3 posts to the topic even though I detest reading academic articles. They are:1. If the benefits of TDD are not obvious to developers (and I would argue that they are not), there is every likelihood that the TDD meme will cease to gain traction in our community, and may well wither-on-the-vine. This would make me sad, because I have personally learnt a lot about myself from practicing TDD and I would like others to have the same opportunity.2. There is a real risk (both ethically and financially) of promoting the benefits of TDD without being able to support those claims. From my reading, the /only/ claim that can be supported is that TDD reduces defects. I've tried to summarize this in the section titled “Should teams and developers adopt TDD?” above.

  4. Scrumology April 16, 2010 at 2:18 am #

    Dave,I think you may have missed the point that [I understood] Carlton was trying to make. “As a former researcher,” I'm sure Carlton deeply understands the nature of peer review articles … and their limitations. He's expressing the same conclusions as myself: “If you read the peer-reviewed literature, the value of TDD is a mixed bag.”There's a /if/ at that start of that sentence. I agree with your point that there is a difference between what is published in papers, and what practitioners actually experience for themselves. But the huge positive effects on the design etc., that you've experienced are specific to you, and there is no evidence that simply adopting TDD will realize any of those benefits. I personally feel that Lech Madeyski expressed this best as Rules of Thumb: “A principle with a broad application that is not intended to be strictly accurate or reliable in every situation.”

  5. Scrumology April 16, 2010 at 1:31 am #

    Hi Ryan, Dave beat me to the punch. I was going to make essentially the same point as he regarding the value of budgeting. In fact, I'm pretty sure I could effectively argue the con side of the argument for each of the items that you've listed. I can even back up my argument with practitioner experience.

  6. Dave Nicolette April 15, 2010 at 8:33 pm #

    Carlton, Think about it this way for a moment, as an exercise in looking at things from alternative perspectives: The “peers” who review peer-reviewed literature are other researchers. They are not practitioners. I ask you to consider the possibility that the value of TDD isn't necessarily a mixed bag, but rather that the value of peer-reviewed literature as a means of “proof” is a mixed bag. As a practitioner, I would sooner open a donut shop than return to the old way of building software. The main reason isn't merely that there are fewer defects, but because of the huge positive effect on the design, readability, maintainability, and longevity of the resulting code. It's night and day. I don't need a peer-reviewed study to tell me that.Ryan, Have a look at Bjarte Bogsnes' book, Implementing Beyond Budgeting. It isn't universally true that everyone acknowledges the value of budgeting!Dave

  7. Scrumology April 15, 2010 at 6:27 pm #

    Hi Dave, >Not supported by papers and studies, etc. Right.Yes, absolutely. If you'll notice, I have restricted the last three posts on TDD to academic papers and studies. For good or bad, academic peer reviewed papers are the closest that we have to impartial third party opinions. > If you ask people who have to understand and maintain production code, what> do they have to say about the design of code that was test driven vs. code that> was not?Actually, there's very little commentary on the net. Michael James pointed me to probably the most interesting work out there by Keith Braithwaite. Keith's work is great, and it has the same concerns that I've been discussing with the other works … limited dataset and lots of caveats. Keith himself says (emphasis is his): “managed to find the time to look at a /very/ small sample of Java codebases”. Having said that, I'd encourage everyone to read all three posts in the series. Thoughtful stuff.>I've been struck by the fact that most of the researchers and analysts don't >appear to understand what they're observingAgreed. 100%.>They “prove” nothing.I'm not looking for proof. That is unrealistic. When I started this I was looking for either a) a preponderance of data [showing the benefits of TDD], or b) unqualified articles. I'm still looking.>A meta-question comes to mind … Although I agree that there are broader questions to be consider, I still feel that peer review articles are the closes that we currently have to an independent third party. I won't address either of your meta questions in this reply because I view them as being off topic.>With all that in mind, I think there is a point of diminishing returns in> worrying about how many studies exist or how convincing they are.There are two reasons way I think this is important, and why I've devoted 3 posts to the topic even though I detest reading academic articles. They are:1. If the benefits of TDD are not obvious to developers (and I would argue that they are not), there is every likelihood that the TDD meme will cease to gain traction in our community, and may well wither-on-the-vine. This would make me sad, because I have personally learnt a lot about myself from practicing TDD and I would like others to have the same opportunity.2. There is a real risk (both ethically and financially) of promoting the benefits of TDD without being able to support those claims. From my reading, the /only/ claim that can be supported is that TDD reduces defects. I've tried to summarize this in the section titled “Should teams and developers adopt TDD?” above.

  8. Carlton Nettleton April 15, 2010 at 8:16 pm #

    I casually researched the peer-reviewed literature on TDD about a six months ago and came to the same conclusion as you – the evidence was not there andor the examples were thin. As a former researcher, I did not walk away with a warm, fuzzy feeling that TDD is all we say it is. If you read the peer-reviewed literature, the value of TDD is a mixed bag, perhaps no better than other methods.

  9. Ryan Hoegg April 15, 2010 at 7:53 pm #

    I would guess that it is hard to find academic studies that prove that any of the following techniques should be adopted by someone:- Budgeting- Using a filing system- Active listening- Seeking advice- Participating in conversations on the internetHowever, most of us would acknowledge that these techniques are effective, and few would suggest that someone shouldn't adopt them.

  10. Scrumology April 15, 2010 at 7:18 pm #

    Dave,I think you may have missed the point that [I understood] Carlton was trying to make. “As a former researcher,” I'm sure Carlton deeply understands the nature of peer review articles … and their limitations. He's expressing the same conclusions as myself: “If you read the peer-reviewed literature, the value of TDD is a mixed bag.”There's a /if/ at that start of that sentence. I agree with your point that there is a difference between what is published in papers, and what practitioners actually experience for themselves. But the huge positive effects on the design etc., that you've experienced are specific to you, and there is no evidence that simply adopting TDD will realize any of those benefits. I personally feel that Lech Madeyski expressed this best as Rules of Thumb: “A principle with a broad application that is not intended to be strictly accurate or reliable in every situation.”

  11. Scrumology April 15, 2010 at 6:31 pm #

    Hi Ryan, Dave beat me to the punch. I was going to make essentially the same point as he regarding the value of budgeting. In fact, I'm pretty sure I could effectively argue the con side of the argument for each of the items that you've listed. I can even back up my argument with practitioner experience.

  12. Dave Nicolette April 15, 2010 at 1:33 pm #

    Carlton, Think about it this way for a moment, as an exercise in looking at things from alternative perspectives: The “peers” who review peer-reviewed literature are other researchers. They are not practitioners. I ask you to consider the possibility that the value of TDD isn't necessarily a mixed bag, but rather that the value of peer-reviewed literature as a means of “proof” is a mixed bag. As a practitioner, I would sooner open a donut shop than return to the old way of building software. The main reason isn't merely that there are fewer defects, but because of the huge positive effect on the design, readability, maintainability, and longevity of the resulting code. It's night and day. I don't need a peer-reviewed study to tell me that.Ryan, Have a look at Bjarte Bogsnes' book, Implementing Beyond Budgeting. It isn't universally true that everyone acknowledges the value of budgeting!Dave

  13. Carlton Nettleton April 15, 2010 at 1:16 pm #

    I casually researched the peer-reviewed literature on TDD about a six months ago and came to the same conclusion as you – the evidence was not there andor the examples were thin. As a former researcher, I did not walk away with a warm, fuzzy feeling that TDD is all we say it is. If you read the peer-reviewed literature, the value of TDD is a mixed bag, perhaps no better than other methods.

  14. Dave Nicolette April 15, 2010 at 11:14 am #

    “Any other conclusions such as better design, better APIs, simpler design, lower complexity, increased productivity, more maintainable code etc., are simply not supported.”Not supported by papers and studies, etc. Right. Are those claims supported by the direct experience of practitioners? Production support? Customers? If you ask people who have to understand and maintain production code, what do they have to say about the design of code that was test driven vs. code that was not?When reading some of the published studies about TDD, I've been struck by the fact that most of the researchers and analysts don't appear to understand what they're observing when they watch teams in the field, or when they watch groups of students who are involved in the study. Even if their statistical methods are sound, can they reach meaningful conclusions based on a faulty understanding of the topic under investigation? I think academic studies are mainly useful as a mechanism for students to earn degrees, and industry studies are mainly useful as a product for analysts to sell. They “prove” nothing.A meta-question comes to mind, as well: Who is it, exactly, who may or may not be “convinced” by a study? What would those people /not/ be looking at with their own eyes in order to be in a position that they had to rely on published studies to get a sense of the value of TDD (or anything else)?A second meta-question, for those who remain “unconvinced:” Where are the studies that “prove” the value of the methods they are currently using?I think the increased interest in formal studies of contemporary software development practices reflects the penetration of those practices beyond early majority adopters. Late majority adopters typically mimic the behavior of others in their industry. Their adoption of “new” practices is not guided by studies or experience or anything other than copying what their competitors are already doing. I submit that late majority adopters will never be convinced of anything by means of formal studies or by means of direct experience. They will simply wait until the majority of others in their industries have adopted a practice, and then they will copy that behavior. By the same token, early majority adopters and innovators will not wait for “studies” to be published before they try new ideas. There is a significant time lag before studies of new practices become available. With all that in mind, I think there is a point of diminishing returns in worrying about how many studies exist or how convincing they are.

  15. Ryan Hoegg April 15, 2010 at 12:53 pm #

    I would guess that it is hard to find academic studies that prove that any of the following techniques should be adopted by someone:- Budgeting- Using a filing system- Active listening- Seeking advice- Participating in conversations on the internetHowever, most of us would acknowledge that these techniques are effective, and few would suggest that someone shouldn't adopt them.

Trackbacks/Pingbacks

  1. The benefits of TDD (part 2) — Scrumology Pty Ltd - August 23, 2010

    […] SignupSubscribeThe benefits of TDD (part 2)by Kane on March 16, 2010[Update: Part 1, Part 2 and Part 3]In part 1 of this three part series I looked at the evidence supporting Test Driven Development […]

  2. The benefits of TDD are neither clear nor are they immediately apparent — Scrumology Pty Ltd - August 23, 2010

    […] clear nor are they immediately apparentby Kane on February 25, 2010[Update: Part 1, Part 2 and Part 3]Let me be begin this tale at the end, and then go back to the beginning. From personal experience […]