I previously talked about test strategy and I made it clear that in my opinion the test strategy and the test plan are two entirely different things. I’ve had a love-hate affair with test plans over the course of time, mostly because I’ve seen them become an end unto themselves. I’ve seen them become monolithic documents that people who need a sense of control use when they feel they can’t get that control in more effective ways or can’t deal with projects that are more fluid in nature.

But I wonder if I’m too reactionary in nature on this topic or if I’m just reacting to seeing a lot of badly written, rarely read test plans.

I came across the article why do we write a test plan and this intrigued me a bit since it was arguing quite a bit for the notion of a test plan. The article states:

Test plan is the only mechanism to organize test execution and that execution reporting.

This I simply disagree with although I’m not entirely sure if the article is promoting that idea. Another thing said is:

“Test plan” is sometimes (well almost always) seen by management as a central document addressing quality.

This I most certainly agree with and it’s actually one of my concerns because when a test plan is treated as a document that addresses quality, I get very nervous. A test plan, if it addresses anything, addresses testing and the logistics — not the strategy! — of how testing is going to take place. When a test plan is equated with quality I think you get into the “Testing for Quality” mindset.

So then I came across A Question about Test Strategy by James Bach. Interestingly, he says something I said in my previous post: “Test Strategy is the set of ideas that guide test design.” It’s highly doubtful that he and I both happened to use the same wording so it’s highly likely I encountered that phrase of his before, internalized it, and now apparently use it. In that same article Bach also says: a test plan “is the set of ideas that guide a test project.” That was interesting to me but I didn’t know exactly what it meant. Then in Bach’s Test Plan Evaluation Model I found this:

Test projects are the means by which the test strategy is implemented and results delivered.

So, in this view, a test plan guides the test project and those projects are the means by which the test strategy is carried out. That’s resonated at least a tiny bit for me because in my previous post I had said:

The test strategy is the connection between your testing activities and the purpose (or charter) of your testing function.

So if the purpose of the testing function is put in a test plan and then I suppose my thinking sort of aligns with Bach’s.

Okay, so this got me to rethinking my view on test plans a bit. A lot of times people have no trouble saying “we should write a test plan.” What people tend to forget is that there’s a missing qualifier that can be put before the words ‘test plan.’ For example:

• “We should write a good test plan.”
• “We should write an effective test plan.”
• “We should write a short test plan.”
• “We should write a comprehensive test plan.”

But you could also imagine other qualifiers:

• “We should write a complicated test plan.”
• “We should write an unreadable test plan.”
• “We should write an out-of-date test plan.”
• “We should write a bloated test plan.”

Hopefully no one wants to write test plans with the latter qualifiers but, of course, just saying you “should” write a test plan is an empty statement. So my point here is that you shouldn’t blur the distinction between the contents of your test plan with the means by which you actually do the things that your test plan says you will do. The qualifier speaks to the contents; whatever else you do is the means by which you actually carry out your testing and communicate the results of that testing.

What does the qualifier really mean, though? After all, saying we should write a “good test plan” really doesn’t tell me what that means. So what does it mean? Well, that’s what you have to decide as you write such things, I guess. In his TPE model, Bach has this to say:

The test plan is the set of ideas that guide or represent the intended test process.

In my test strategy post, I said that a test strategy expressed a set of choices about your test design — which is an idea I probably stole from James — and now we’re talking about your test process. Keep in mind the distinction. But I think the test plan should focus more on choices that you make.

A test plan should express a set of choices about the test process.

Before I get into those choices, consider that there are two things you are always going to be given: resources and constraints. Resources are not just people. A resource can be a skill in a given technique. A resource can be your knowledge of the underlying technologies of the application or the environment in which it runs. A resource can be a repository of test data. A resource can be a test platform (or a set thereof). A resource can be a test tool. A resource can be testability functions that are built into the application itself (like log files, assertions, etc).

Fred Brooks said something apropos in The Mythical Man-Month:

The challenge and the mission are to find real solutions to real problems on actual schedules with available resources.

So maybe that gives us another heuristic?

When you do any sort of test planning, the goal is to make choices about your test process that will allow you to test within the constraints of the project environment, while exploiting your available resources, enabling you to achieve the charter of your test function.

I think that last point is important because your charter is part of your values. You don’t want to compromise those values due to resources or constraints. Okay, so what choices do you have? What choices do you have to make? Well, I already covered one of them. That’s the test strategy I talked about before. This should tell people the choices you made to answer questions like this:

• “How will we cover the product to find important problems fast?”
• “What specifically will we test?”
• “What techniques will we use to create tests?”
• “How will we recognize bugs when they occur?”

There are also the logistics. That’s a second set of choices. This should tell people the choices you made to answer questions like this:

• “How will we apply resources to fulfill the strategy we’ve outlined?”
• “Who will be testing?”
• “When will testing begin?”
• “Where will testing be done?”
• “What do we need to be successful in testing for this project?”

Did you notice the emphasis on both strategy and logistics? The logistics are getting the right people and resources in the right place at the right time to do testing. But what testing? Well, that’s where the strategy comes in. Many test plans seem to be written with the idea that all you have to do is point a tester or a group of testers to the right environment at the right time and place and everything will be just fine. Bugs will be found and quality will happen.

But does this imply that a strategy can be incorporated into a test plan? Personally, I don’t think so. I think a strategy is a bit too fluid for that. Having a strategy does not just mean that you have a set of test techniques in your toolbox. Having a strategy means you have heuristics for how you know when to use which test technique under which situation. And those situations may change during the course of a project, as new discoveries (some of which will be bugs) are made. All of this, of course, rests on the skill of the testers you have and their ability to utilize the appropriate tools at the appropriate times based on changing considerations.

If you think you can document all that in a test plan, good for you. I personally doubt it can be done well. That strategy that I’m referring to there is going to tell you if you tested effectively and efficiently. All most test plans are going to tell you is that you tested. Or that you were planning to test, anyway. Which brings up a good point: there is another set of choices and this involves work products or test deliverables. This should tell people the choices you made to answer questions like this:

• “How will our work be presented to people who need to know?”
• “What kinds of reports will we produce to show where testing is at?”
• “How will people feel confidence in the reports they are getting?”
• “How can we direct people to the most important problems first?”

All of this can be put down in a test plan but you’ll probably realize that many of your answers to these questions will be largely the same from project to project. Exactly how you carry out some of the activities may differ, but the end results will largely be the same. They are a core part of your strategy. And thus a test plan for each and every project, beyond covering some logistics, may not be all that necessary. Here’s how I think of it:

• A test strategy is how you cover a given application and detect problems with that application.
• A test strategy is the manner in which tests will be designed and executed to support an effective quality assessment.

Test strategy is dynamic; it changes based on the situation. Test plans tend to be written once and pretty much stay that way. So, in those cases, it seems to me that limiting the test plan to the logistics is the smart way to go.

You can see that even though I was trying to talk about test planning it was all I could to do to keep myself from returning to test strategy. So, if it helps, just keep in mind that a test plan is mainly about who is going to be doing the testing, when they are going to do it, and where they are going to do it. It’s largely the logistical elements of your test effort for a given project. That being said, test logistics are really the how and when you apply your resources to execute the test strategy. When the project is in full-swing and developers and testers are working in concert to produce something useful, believe me, it’s the strategy that really matters, not the plan.