We Don’t Have Time

We all need more time. Where will we find time to write all this test code? There is barely enough time to write the production code we need. There are more lines of test code than production code for the LedDriver. But does that really matter?

If people programmed error-free and at a constant rate, then there is some reason for concern. But people do neither. The time-consuming parts of programming are thinking, problem solving, and confirming solutions. Confirming solutions can be done many ways, Debug Later Programming (DLP) or TDD to name two. The important question is, did writing the test impede or speed your progress?

Many proficient practitioners proclaim that TDD makes them go faster. They report a productive and sustainable pace. The speedup comes from reducing current and future debug times, and from having a cleaner code base with tests as executable documentation.

What if TDD takes a little more time? There are other costs besides development time: customer dissatisfaction, lost sales, warranty repair, defect management, field service...the list goes on. Maybe it is worth it to you and your customers to spend more time and deliver fewer problems to the field. Also, you may become one of those people who goes faster with TDD.

If you look only at the time it takes to get the production code written, you are not looking at the whole job. You still have to get the bugs out. How much time do you currently spend testing and debugging code? The most popular answer I have heard, from polling conference attendees, is 50 percent. That is a lot of time. The first place to look for the time to do TDD is in your current practice. You should be able to trade some reactive debug time for the proactive TDD approach. In the next few sections, we’ll look at some common unit test approaches that could be at least partially replaced by TDD.

Manual Test

If you are manually unit testing, use some of that time. If you are in a legacy code environment, you won’t leave manual testing completely behind, but you could start to develop new code using TDD or write tests for some of the untested legacy code.

The initial investment in manual test may be lower than automating tests, but it is not sustainable; it has a nearly zero future return. A change to manually tested code nullifies the prior manual tests. You have to run the tests again. Because they are manual, we tend to rationalize running only a subset of the tests. When you don’t rerun the right tests, you get the joy and cost of a future bug.

Custom Test Harness

From time to time, we have all written a test main and a few test stubs that exercises newly written code. The test main exercises the code under test, and stubs provide indirect inputs and log their parameters so we can inspect the behavior. You have created a custom test harness.

These tests are very helpful; they improve the quality of the code so that we are integrating better working code into the product. But too often, after integration, the tests fall into disrepair, as all testing moves to the integrated system. The tests fall out of sync with the production code, and the return on investment is diminished. Your custom test harness was helpful for a while.

Often custom test harnesses have a poor return on investment. They often become incompatible and are discarded after very few uses. The custom-crafted test main also takes more effort than writing tests that plug into a test harness like CppUTest or Unity.

Unit Test by Single-Step

Another manual unit test approach is to single-step through the code under test with a debugger. This is a slow and inherently nonrepeatable process. When change comes, as it always does, the single-step unit test has to be repeated. Because it’s a long and tedious process, it is likely to be a less thorough job the second, third, and Nth times through. We’re only human; we make mistakes and miss subtle interactions within the code.

The shelf life of these tests is even worse than the shelf life of the test main approach. Any single change invalidates prior tests. You have to start over and do it again. So, the manual test effort will tend to grow over time. But you can’t afford that time, so you don’t rerun all the needed single-step tests, and what happens? A bug creeps in, costing future effort too.

Documented and Reviewed Unit Test Process

I consulted at a company that had a very well-defined process. Well-defined and big are usually synonymous when it comes to processes. Their process manual was big. Their process police force was big, and they had a big stick.

They were assessed at CMM level three. They had good conformance and enforcement. One area covered by their process was unit testing. The process consisted of first documenting the unit test procedure and then getting the procedure reviewed and approved. Then they had to record evidence of executing the process. I asked the engineer, Dave, how they used this procedure. Here is how that conversation went:

James:

How do you do unit testing?

Dave:

We have a unit testing standard. We write a unit test plan for each function.

James:

Does the unit test plan get reviewed?

Dave:

Yeah, we do a formal technical review on the plan.

James:

When do you perform the unit test plan?

Dave:

We run it before the code has been through its formal technical review.

James:

So if there are holes in the test plan, the reviewer can make suggestions and improvements to fill those holes.

Dave:

Yes, that’s how it goes.

James:

What does the unit test plan look like?

Dave:

We follow the standard template and add the plan as comments before every function in the source code. The plan becomes part of the code. It includes a series of operations performed on the code, checking various conditions. We make sure we check each branch.

James:

What is it like to run the tests?

Dave:

We use the debugger or the emulator and single-step through each statement and verify it does the right thing. We’re really thorough.

James:

It sounds like it. It takes a lot of time I bet.

Dave:

Sure does.

James:

What happens the next time you change that function?

Confidently Dave says:

We do part of the tests over, based on what we changed.

James, knowing the answer to the question:

Does code like this change very often?

Dave says accusingly:

Yes, the systems engineers never can make up their mind.

James:

What happens as there are more changes?

Dave:

We rerun those parts of the unit test affected by the change.

James:

How do you know what part of the plan needs to be rerun?

Dave:

It’s a judgment call.

This big process took a lot of effort. It made everybody feel good because of all the investment in the software quality. Unfortunately, this kind of effort too often returns little on the investment. When the manual unit test process is repeated, the process gets boring, shortcuts naturally follow, and bugs find their way into the code.

I suggest test automation over test documentation. Test automation is the gift that keeps on giving. If you’re using a process that is similar to Dave’s, it’s time to stop! Spend your unit test dollars somewhere else. By the way, the tests are documentation too. Dave’s company had product safety requirements; we settled on reviewing the test cases to help assure the cases were thorough.

Where Do Your Unit Test Dollars Go?

When you look at how to pay for the unit tests that are written as part of TDD, take an honest look at your current process. Maybe your process is ad hoc or you write a test main or you document a unit test procedure or single-step through the code. Those activities cost a lot to implement but have a very limited payback.

You are already paying for unit tests either directly as mentioned or indirectly through long debug cycles. Consider spending some of your unit test effort on TDD rather than your current process. With TDD the tests are run with every change, the tests evolve along with the code, and the investment is returned many times over.