Have you ever come across the term “mutation testing”? My experience is that some people might have heard about the concept of mutation testing — perhaps in passing — but rarely have they heard too much substantive about it. Even when I talk with developers and testers, I was initially surprised at how few of them readily accommodated this sort of technique. On the other hand, if there isn’t a lot out there about it, my surprise is unwarranted. That’s even more so the case if the material out there does not talk about how mutation testing can actually be practical.
Category: Testing
Testing: Art and Science
I’ve heard a lot of people in the software industry say that “testing is more art than science.” My opinion is that people who say this most likely know little about art and probably know even less about science.
Testers Write Tools — Sometimes With Gems!
I recently talked about testers writing tools. That particular tool, however, was a simple script. You can definitely improve your career options by being able to write more substantive tools.
As a tester I’ve recently had to start writing some Ruby programs that would serve as effective test frameworks. As it turns out, it was best for me to package up my logic in the form of a Ruby gem so that I could allow others to install it easily along with any dependencies.
Testers Write Tools — Even Simple Ones!
I was in an environment where it was clear some of the testers were struggling with the idea if comparing Excel workbooks: either as actual Excel files or CSV files that were imported into Excel. Expensive automation tools were being used to try to handle some of this. What I immediately thought of is simply: this is why testers do need to have some technical skills in the programmatic arena.
Put Some Heuristics Between Your Strategy and Your Tactics
I like to consider a strategy and tactics distinction when you have a test team that’s striving to operate within the sphere of quality assurance. When you have that situation, then operationally considering strategy and tactics (or logistics, if you prefer) becomes important. The emphasis needs to be on a gradual implementation of good testing practices that then filter up to a full quality assurance process. Since any such quality function, in my opinion, starts with good testing practices of some sort, I’ll mainly make my distinctions here in the context of the idea of testing.
Continue reading Put Some Heuristics Between Your Strategy and Your Tactics
Does Planning For Testing Require a Test Plan?
I previously talked about test strategy and I made it clear that in my opinion the test strategy and the test plan are two entirely different things. I’ve had a love-hate affair with test plans over the course of time, mostly because I’ve seen them become an end unto themselves. I’ve seen them become monolithic documents that people who need a sense of control use when they feel they can’t get that control in more effective ways or can’t deal with projects that are more fluid in nature.
But I wonder if I’m too reactionary in nature on this topic or if I’m just reacting to seeing a lot of badly written, rarely read test plans.
Continue reading Does Planning For Testing Require a Test Plan?
Your Test Strategy Is A Framework
I see a lot of testers — and their managers — misuse the phrase “test strategy.” I say they misuse it not because they don’t agree with some definition I have but rather because I rarely see a good distinction made between test logistics and test strategy. I often see a test strategy equated with a test plan. I hear some testers say that having a test plan is part of your test strategy. I hear others say that it’s the job of the test plan to indicate the test strategy.
So let’s talk about test strategy a bit.
Can I Stress The Load On My Volume?
In my post “Can I Function Test My Units?”, I talked about how testers often promoted a misuse of the phrase “functional testing.” I’ve found the same thing happens with the phrase “performance testing.” What I’ve found is that many testers don’t realize that performance testing, just like functional testing, is a test approach. This means there are various testing techniques that you can apply to the approach. If testers don’t have this kind of focus, I believe that the purpose of performance testing can be lost.
The Art of Testing at Different Levels
Many testers entering the field get nervous when they consider the need to test at different levels of an application, particularly if that involves working with developers. Many testers will enter into environments that claim they are practicing Test-Driven Development (TDD) and/or Behavior-Driven Development (BDD). What I want to do here is show that this stuff isn’t really all that scary and it’s really not that arcane. Further, if you want to be competitive as a tester and continue to add value, it’s in your best interests to learn what it means to test at different levels.
Can I Function Test My Units?
Many testers will tell you that they eventually had to work with their development group to draw the line between unit testing and functional testing. This, of course, implies that there is some sort of a difference between the two. And, in fact, if you go by the literature — professional or otherwise — of the testing industry, there supposedly is a difference.
I’ve just never liked that fact because “functional” is such a broad term. To my way of thinking, unit testing is a form of functional testing. I think testers who use the term “functional” as a separating category within tests are stuck in an “old school” mode of thinking.
Is Quality Really Everyone’s Responsibility?
Who is “responsible” for quality? This is one of those interesting questions that on the surface of it seems really simple to answer depending upon your viewpoint … and yet the question itself hides a few thorny issues. In my experience, most people who claim they are part of a “Quality Assurance Team” are not aware of these issues and thus perpetuate a lot of false thinking regarding the question itself.
It’s very important to think about what this question actually implies. How this question is considered directly ties in with how we think about the notion of responsibility and about how we communicate that notion to others.
Continue reading Is Quality Really Everyone’s Responsibility?
What Makes Testing Complicated?
I think the question of what makes testing a complicated discipline is a good one. It’s one all testers should be able to answer about their profession and, further, answer in a way where the answer couldn’t be applied to just about any other profession. The problem I’ve found is the answers you get are usually simplistic. You’ll hear people say “miscommunication” or “unrealistic schedules.” Well, that’s great but miscommunication can make any profession difficult as can an unrealistic schedule or poor estimating. I bet day traders, doctors, police officers, lawyers, military strategists, nuclear power plant operators and so forth could make the same claim.
Surely we can do better than that, right?
The Value of Testing … Relative to Quality
The perceived value of testing is often related to how test teams believe — and promote — the idea that they “test for quality.” The reason I say that is because I maintain quality is value to someone whose opinion matters. You might want to check out my post on testing for quality where I covered a lot of these points. What I didn’t cover in that post is the value of testing itself. The value of testing can be trickier than you may think. Let’s start with this: people will place different value to the quality they perceive. Do we agree on that? If so, I argue that the value of testing that people perceive will be in direct proportion to how much they believe your testing tells them what they need to know. Different people will need to know different things given that their valuations regarding quality will be different in some cases.
What you end up with is dynamic assessment of value and it’s here that testers can start to think a bit more deeply about how what they do provides value.
“Testing for Quality” and Making Informed Decisions
Sometimes you’ll hear testers talk about how they “test for quality.” My problem with that phrase is that it’s a potentially dangerous one. It’s an even more dangerous expectation unless some ideas of what the phrase actually means are sorted out. Once you have an idea of what “test for quality” actually means, maybe you can craft intelligent processes around the phrase.
If you’re going to do that, make sure you know why you test and make sure you have some idea of what quality means. Without some thinking around those two areas, “testing for quality” is nothing more than something you say to make managers feel good.
Continue reading “Testing for Quality” and Making Informed Decisions
How Big Is Your Testing Umbrella?
I’ve worked with lots of testers who often take on more work than they should because they have the idea that the work they are taking on is what testers should be doing. It’s often not a question of whether they should do this work given their other constraints, but rather just a focus on how they can regardless of those constraints. Granted, many companies seem to foster this by job descriptions that suggest someone needs to be a developer, DBA, business analyst, and tester all in one. A main aspect of my job description as a quality assurance and test practitioner comes down to defining a role, usually separate from the relatively anodyne job descriptions we’re all used to.
Testing By Any Other Name…
In my post Are We Defocusing Our Development and Test Practices? I was talking about the various “driven-development” terms. I had brought up how Behavior Driven Development (BDD) seems to have largely come about because some people felt that the term “test” was a bit too confusing for some people or, at the very least, off putting for some reasons that are not explained very well. There was thus apparently a shift to “examples” that talked about “behavior” and this was somehow different than a test. And this somehow made everything a bit better. But did it? What was it about the terms “test” and “testing” that were just not working? I’d rather people deal with that problem before thinking we need to come up with new acronyms. I say that because, ultimately, the concepts of a “test” and the activity of “testing” is still very much there.
Are We Defocusing Our Development and Test Practices?
As a tester, you’ll hear about behavior-driven development (BDD). Sometimes this goes by acceptance test-driven development (ATDD) or maybe even acceptance test-driven planning (ATDP). You may hear that these are related to test-driven development (TDD). After a while you might start to wonder if these are just different ways of saying the exact same thing.
Continue reading Are We Defocusing Our Development and Test Practices?
What Can Time Travel Teach Us About Testing?
I have long maintained that testing is ultimately about thinking. A lot of your work is going to be predicated upon spotting incongruous elements (whether in documents or application code), resolving potential ambiguities in what you read or see, and dealing with potential inconsistencies or outright contradictions. Sometimes people will frame problems in such a way that your ability to “solve” or deal with the problem effectively is compromised from the start.
A good way to practice your thinking skills is to apply them in an entirely different arena than software testing.
Continue reading What Can Time Travel Teach Us About Testing?
Acceptance Tests Are Just Tests
Some testers have problems conceptualizing acceptance tests as I described them in my post on writing tests from stories. The problem seems to be one of seeing how the concept of an acceptance test applies or translates to concepts they already know, such as test conditions and/or test cases.
Be Acceptable: Write Tests from Stories
In my post on Stories, Features, and Acceptance Tests I talked about a lot but I didn’t really cover the acceptance tests themselves. The tests are quite important. Keep in mind that the goal is to have specifications that are clear, sufficiently testable and falsifiable, and in line with client expectations prior to significant development or testing. In essence, the idea is to work out how a system would be tested as a way to check whether the requirements give you enough information to build the system in the first place. You shouldn’t dive straight away into how to implement something, but rather think about how the finished system will be used — i.e., acceptable use — and then double-check the requirements armed with that knowledge.
Acceptance tests should reflect the customers’ perception of when the application meets their requirements. This does not mean that such tests must be defined by a customer or business analyst but it does often mean that such tests are discussed with them.
But how do you write them? What do they look like? Well, let’s talk about that.