We Don’t Have Time
We all need more time. Where will we find time to
write all this test code? There is barely enough time
to write the production code we need. There are more
lines of test code than production code for the
LedDriver. But does that really
If people programmed error-free and at a constant rate,
then there is some reason for concern. But people do
neither. The time-consuming parts of programming are
thinking, problem solving, and confirming solutions.
Confirming solutions can be done many ways, Debug
Later Programming (DLP) or TDD to
name two. The important question is, did writing the
test impede or speed your progress?
Many proficient practitioners proclaim that TDD makes
them go faster. They report a productive and
sustainable pace. The speedup comes from reducing
current and future debug times, and from having a
cleaner code base with tests as executable
What if TDD takes a little more time? There are other
costs besides development time: customer
dissatisfaction, lost sales, warranty repair, defect
management, field service...the list goes on. Maybe it
is worth it to you and your customers to spend more
time and deliver fewer problems to the field. Also,
you may become one of those people who goes faster
If you look only at the time it takes to get the
production code written, you are not looking at the
whole job. You still have to get the bugs out. How
much time do you currently spend testing and debugging
code? The most popular answer I have heard, from
polling conference attendees, is 50 percent. That is a lot of
time. The first place to look for the time to do TDD
is in your current practice. You should be able to
trade some reactive debug time for the proactive TDD
approach. In the next few sections, we’ll look at
some common unit test approaches that could be at
least partially replaced by TDD.
If you are manually unit testing, use some of that
time. If you are in a legacy code environment, you
won’t leave manual testing completely behind, but you
could start to develop new code using TDD or write
tests for some of the untested legacy code.
The initial investment in manual test may be lower
than automating tests, but it is not sustainable; it
has a nearly zero future return. A change to
manually tested code nullifies the prior manual
tests. You have to run the tests again. Because they
are manual, we tend to rationalize running only a
subset of the tests. When you don’t rerun the right
tests, you get the joy and cost of a future bug.
Custom Test Harness
From time to time, we have all written a test
main and a few test stubs
that exercises newly written code. The test
main exercises the code
under test, and stubs provide indirect inputs and log
their parameters so we can inspect the behavior. You
have created a custom test harness.
These tests are very helpful; they improve the
quality of the code so that we are integrating
better working code into the product. But too often,
after integration, the tests fall into disrepair, as
all testing moves to the integrated system. The
tests fall out of sync with the production code, and
the return on investment is diminished. Your custom
test harness was helpful for a while.
Often custom test harnesses have a poor return on
investment. They often become incompatible and are
discarded after very few uses. The custom-crafted
main also takes more
effort than writing tests that plug into a test
harness like CppUTest or Unity.
Unit Test by Single-Step
Another manual unit test approach is to single-step
through the code under test with a debugger. This is
a slow and inherently nonrepeatable process. When
change comes, as it always does, the single-step
unit test has to be repeated. Because it’s a long
and tedious process, it is likely to be a less
thorough job the second, third, and Nth times
through. We’re only human; we make mistakes and miss
subtle interactions within the code.
The shelf life of these tests is even worse than the
shelf life of the test
approach. Any single change invalidates prior tests.
You have to start over and do it again. So, the
manual test effort will tend to grow over time. But
you can’t afford that time, so you don’t rerun all
the needed single-step tests, and what happens? A
bug creeps in, costing future effort too.
Documented and Reviewed Unit Test Process
I consulted at a company that had a very well-defined process. Well-defined and big are usually
synonymous when it comes to processes. Their process
manual was big. Their process police force was big,
and they had a big stick.
They were assessed at CMM level three. They had good
conformance and enforcement. One area covered by
their process was unit testing. The process
consisted of first documenting the unit test
procedure and then getting the procedure reviewed
and approved. Then they had to record evidence of
executing the process. I asked the engineer, Dave,
how they used this procedure. Here is how that
How do you do unit testing?
We have a unit testing standard. We write a unit
test plan for each function.
Does the unit test plan get reviewed?
Yeah, we do a formal technical review on the plan.
When do you perform the unit test plan?
We run it before the code has been through its
formal technical review.
So if there are holes in the test plan, the
reviewer can make suggestions and improvements to
fill those holes.
Yes, that’s how it goes.
What does the unit test plan look like?
We follow the standard template and add the plan
as comments before every function in the source
code. The plan becomes part of the code. It
includes a series of operations performed on the
code, checking various conditions. We make sure we
check each branch.
What is it like to run the tests?
We use the debugger or the emulator and single-step through each statement and verify it does the
right thing. We’re really thorough.
It sounds like it. It takes a lot of time I bet.
What happens the next time you change that
- Confidently Dave says:
We do part of the tests over, based on what we
- James, knowing the answer to the question:
Does code like this change very often?
- Dave says accusingly:
Yes, the systems engineers never can make up their
What happens as there are more changes?
We rerun those parts of the unit test affected by
How do you know what part of the plan needs to be
It’s a judgment call.
This big process took a lot of effort. It made
everybody feel good because of all the investment in
the software quality. Unfortunately, this kind of
effort too often returns little on the investment.
When the manual unit test process is repeated, the
process gets boring, shortcuts naturally follow, and
bugs find their way into the code.
I suggest test automation over test documentation.
Test automation is the gift that keeps on giving. If
you’re using a process that is similar to Dave’s,
it’s time to stop! Spend your unit test dollars
somewhere else. By the way, the tests are
documentation too. Dave’s company had product safety
requirements; we settled on reviewing the test cases
to help assure the cases were thorough.
Where Do Your Unit Test Dollars Go?
When you look at how to pay for the unit tests that
are written as part of TDD, take an honest look at
your current process. Maybe your process is ad hoc
or you write a test main or you document a unit
test procedure or single-step through the code.
Those activities cost a lot to implement but have a
very limited payback.
You are already paying for unit tests either
directly as mentioned or indirectly through long debug
cycles. Consider spending some of your unit test
effort on TDD rather than your current process. With
TDD the tests are run with every change, the tests
evolve along with the code, and the investment is
returned many times over.