Combination testing involves testing several variables together. As you can imagine, however, this leads into an explosion of tests. Combination testing can be used to make your tests manageable. But you also have to make your strategy for determining combinations manageable. It’s the latter aspect that I’m going to talk about here.
Category: Testing
There Is No “Non-Functional”
I keep running into testers, and others, who like to categorize testing into functional and non-functional aspects. This needs to stop, in my humble opinion.
Path Testing: Independent Paths
This post follows on from the previous two in this series (theory and path coverage). A key theme of these posts is that of modeling paths through an application and then using tests to exercise those paths. What we’ve seen so far is that while many path segments are covered repeatedly, others may only be touched once. You most likely won’t have time to test them all. Along with that, some paths will be more data-dependent than others. So a selection process is critical. Here I’ll talk more about that selection process.
Path Testing: The Coverage
This post continues on from discussing the theory around path testing. Now I want to put some of that theory into practice. While this is geared toward testing, it by necessity jumps into the programmatic side of things. I think this is an important view for testers to have.
Path Testing: The Theory
Path analysis is often considered an activity that is done only by developers. In fact, path analysis is a technique that can be used by both developers and testers. I find many testers, however, who are unaware of this test technique or are uncertain of how to best apply it. So let’s talk about it.
The Craft of Testing
Being a tester with an avid interest (and some practice) in fiction writing, I liken the craft of testing to the craft of writing fiction particularly because I found some correlations in terms of writing fiction and writing tests. (Oh, the opportunities for a few jokes there.) The focus I’m going to be building on here is really a similarity of craft or, more accurately, how there are similar thinking styles that translate between crafts.
The Testing Focus Shift: Back To Its Roots?
There is a large debate ongoing in the testing community about how technical a tester has to be. I actually believe that a tester has to be technical enough to implement test solutions when they are most needed or know how to gather the skills to become technical enough. Pretty vague, huh? The problem is that it really depends on what test solutions you need and the skills needed to create them.
Continue reading The Testing Focus Shift: Back To Its Roots?
Don’t Be So Negative … or Positive
Previously I talked about the distinction between positive and negative tests and why I felt such a distinction was not only unnecessary but also potentially harmful. This is still a topic that interests me because I run into testers who still like to make these distinctions.
What Can Hordes of Zombies Teach Us About Testing?
I already tried to figure out if time travel can teach us about testing and I wondered if military history might also have a few lessons to share. I was marginally surprised that when I saw the recent film World War Z, I found myself thinking about testing.
Continue reading What Can Hordes of Zombies Teach Us About Testing?
Testing Games is Hard!
Actually, testing games is not all that much harder than testing other applications. What’s hard is sometimes being able to figure out how to test them well, given their numerous interfaces and means of interaction. Even harder is sometimes isolating the very specific, but very odd bugs that come up. I recently came across just this issue with a particular bug I found while testing a popular game.
Testers Should Be Cross-Discipline Associative
Cross-Discipline Associative? What does that even mean? I don’t know. It sounds good, though, right? What I mean by it is that testers take information from one area of thought (a discipline) and attempt to apply it to their own area of thought. (Eating my own dog food, I tried this when I talked about ubiquity or looking at military history.)
Being perhaps a little more relevant here, I’ve come to the conclusion that just as we expect developers to learn from testers, more focus on this needs to occur from the other direction. I’ll be the first to admit: my ideas are still forming on this. Or, rather, my ability to express these ideas intelligently are still forming.
Continue reading Testers Should Be Cross-Discipline Associative
A Values-Based Approach to Systemic Test Team Problems
I was recently in a work environment where we had literally thousands and thousands of test cases that were stored in a tool called TestLink. A major problem was that there was very little impetus for the testers to ever really analyze all these tests or to ever question if TestLink was the most effective tool to be used. This was due in part to some members of management who, perhaps in fear of bruising egos or seeming too critical, basically said: “What we’ve done has worked.” When the testers heard that, they basically assumed: “Well, that means our testing has been good.” Eventually I came along and essentially argued that our test repository was a mess and that our tool of choice was not the most effective. Here’s a little of what I learned from that experience.
Continue reading A Values-Based Approach to Systemic Test Team Problems
Enhancing Testing Skills By Playing Games
I started playing a game type that I rarely play: a Massively Multiplayer Online (MMO) game. Specifically, Star Wars: The Old Republic. As I was playing it, I realized that it was actually an interesting game for a tester. I found myself utilizing a lot of the skills that I would as a tester when looking at a business application. (This actually led to me doing some contract testing on the game itself!)
Testing That is Effective, Efficient, and Elegant
There is notion in quality assurance and testing between verification and validation. Verification asks “Are we building the product right?” Validation asks “Are we building the right product?” Some people use this very distinction to draw a line between the activities of quality assurance and the activities of testing.
Continue reading Testing That is Effective, Efficient, and Elegant
Effective Tests, Not Positive and Negative Tests
My opinion is that the “positive” and “negative” distinctions for tests are a faulty conceptual distinction. To me, only “old school” testers talk about “positive testing” and “negative testing.” Pretty bluntly stated, huh? I feel strongly about this because the way the terms are promoted, they take focus away from how a tester should be thinking about testing. That’s my belief anyway. I’ll try to defend that.
Continue reading Effective Tests, Not Positive and Negative Tests
There Is No “Test Phase”
When people talk about the “testing phase” of a project, you already have a problem because that implies testing is an activity that is relegated to a certain phase as opposed to a structural element of the entire project itself.
Test Writers Step in Early
In my last post about what tests are for I talked a bit about the ability of test writers to be nimble in terms of getting quality-focused information — usually as a result collaborative discussions — encoded into readable tests that focus on intent rather than implementation. All of that’s easy to say but test writers sometimes have a hard time figuring out how to exactly do it in the context of the collaboration possibilities in their environment. Here I want to talk a little about the core ideas, backing them up with an example.
What Are Your Tests For?
A key question to ask is what you want to get out of your tests. I’ve already talked about how I think the purpose of testing is to find information and then communicate that information. That’s great and all … but … what do you actually want to get out of the tests themselves? The way you answer that question will shape your testing function to a large extent, particularly when it comes to picking supporting tools, such as for test management or test automation.
Testing’s Brave New World
As a tester you will often encounter environments that desire to practice some form of Behavior-Driven Development (BDD) and testing practices must fit within that. The term “BDD” may be used more or less vaguely. One thing that’s often somewhat common is that people will start to promote so-called “BDD tools.” Examples here would be tools like Cucumber, Spinach, Turnip, SpecFlow, Behat, Lettuce, and so forth. These tools all ask you to adhere to some sort of organization based on terms like “feature”, “story”, “scenario” and so on. What this often ends up doing is totally missing the point.
But what is the point?
Finding Those Hidden Bugs
One area of testing that often gets neglected are the various ways that bugs can be ferreted out by considering how they “hide.” Testing an application is sometimes like walking a mine field, where you have to trigger the mines in order to see where they are hidden. Another analogy might be a virus hunter, who has to find signs of an outbreak by testing the population, some of whom are hiding to cover up the fact that they have a virus. Here I want to present some information about why testers (and developers!) should consider a technique known as hidden-fault analysis and, more importantly, show how the notion of hidden faults can skew test results.