Have you ever come across the term “mutation testing”? My experience is that some people might have heard about the concept of mutation testing — perhaps in passing — but rarely have they heard too much substantive about it. Even when I talk with developers and testers, I was initially surprised at how few of them readily accommodated this sort of technique. On the other hand, if there isn’t a lot out there about it, my surprise is unwarranted. That’s even more so the case if the material out there does not talk about how mutation testing can actually be practical.
Cautionary Tales on Metrics
Some testers — and most managers — like to talk about metrics. One thing that often doesn’t get discussed is what I call metric dissociation. Here’s my thought: metrics should be easily or directly interpretable in terms of product or process features. If they’re not, I think they are dissociated from the actual work that’s being done and thus there’s great danger of the metrics being misleading, at best, or outright inaccurate, at worst.
Thinking and Testing with Executable Specs – Part 5
This is the fifth post in my series of using a tool that supports writing executable specifications. If this is the first post in the series you’re reading, then you are also reading the last post in the series. Which means this post will be almost totally meaningless to you. To catch up you can read parts one, two, three, and four. (Whether those make this post less meaningless is then up to you!)
Continue reading Thinking and Testing with Executable Specs – Part 5
Thinking and Testing with Executable Specs – Part 4
This post is the fourth in my series regarding approaching testing when you are using an executable specification type tool. The first, second and third posts are pretty much necessary reading for this one to make any sense to you at all.
Continue reading Thinking and Testing with Executable Specs – Part 4
Thinking and Testing with Executable Specs – Part 3
This post is the third in my series regarding approaching testing when you are using an executable specification type tool. This one will following on directly from the activities done in the first two posts. (I recommend reading part one and part two if you want to follow along, since these really are interconnected threads.)
Continue reading Thinking and Testing with Executable Specs – Part 3
Thinking and Testing with Executable Specs – Part 2
This post is the second in my series regarding approaching testing when you are using an executable specification type tool. Assuming you want to follow along, the first post is fairly critical to understanding how the project is currently set up.
Continue reading Thinking and Testing with Executable Specs – Part 2
Thinking and Testing with Executable Specs – Part 1
I’ve been doing some presentations and training sessions on using tools like Cucumber. What’s interesting about these is that in many cases people have tried to play around with the tool but have conceptual difficulties with figuring out how to get started in terms of making decisions continuing to using the tool.
As a tester, when introducing yourself to this area of practice, you might read that You’re Cuking It Wrong or that You’re Cuking It Right. It still may not be clear which you are doing. You might read about being imperative versus declarative and wonder which is for you. (Can’t I do both?) You might even come to the conclusion that Cucumber sucks.
Continue reading Thinking and Testing with Executable Specs – Part 1
Testing: Art and Science
I’ve heard a lot of people in the software industry say that “testing is more art than science.” My opinion is that people who say this most likely know little about art and probably know even less about science.
Good Testers Think Operationally
When a professional discipline has a vocabulary for expressing the fundamental concepts on which it’s based, then you have a means for how practitioners in the discipline can make sure they’re talking about the same things. The problem with establishing such a vocabulary is that many people have come to use a variety of different terms or, even in those cases where the terms are the same, the definitions used are different. This is why I like to encourage people to think operationally.
So why are so many testers I meet so bad at this? Is this just not something they’re encouraged to do? Or am I just being too picky about how people use terms?
Starting to Tame Capybara
Capybara is one of those open source testing tools that, when you get to use it, is really nice. However, it’s new enough (or, at least, changing enough) that good documentation on it is a bit sparse when your goal is simply to get started and you don’t have a lot of supporting frameworks in place. Case in point: what if I want to test against a browser that’s displaying my web application? I’m not using Rails. I have no other automated test frameworks in place. I just need to get started as quick as possible to determine if this tool will do what I need.
I’ll show you how I started out here.
Testers Write Tools — Sometimes With Gems!
I recently talked about testers writing tools. That particular tool, however, was a simple script. You can definitely improve your career options by being able to write more substantive tools.
As a tester I’ve recently had to start writing some Ruby programs that would serve as effective test frameworks. As it turns out, it was best for me to package up my logic in the form of a Ruby gem so that I could allow others to install it easily along with any dependencies.
The Ubiquity of Ubiquitousness
I really enjoyed the book Ubiquity: Why Catastrophes Happen by Mark Buchanan. This book doesn’t talk about quality assurance or testing at all, but it does talk about how things can change very quickly and how a certain context can dictate how things change. I definitely recommend the book for anyone working in a software development context because I found I could abstract ideas out of the context that the book talks about and apply them to the idea of change initiatives, faulty software, and the social context of software engineering environments I have worked within.
I’ll distill a little bit of what I got out of the book here, in the hopes of encouraging you to read it.
Action Sets in QTP
Assume you have some data that tells your scripts how to function when they run a particular set of actions. Instead of this logic being stored in a test script, common action sets are stored off in a data source. In this case, I’ll use Excel but you can use any data source at all. One thing I’ve found fairly effective in terms of an approach here is to create a class that contains the methods for building a dictionary that will essentially bring in the “action sets.” Here an action set refers to the data plus the test actions to take with that data against the application.
Testers Write Tools — Even Simple Ones!
I was in an environment where it was clear some of the testers were struggling with the idea if comparing Excel workbooks: either as actual Excel files or CSV files that were imported into Excel. Expensive automation tools were being used to try to handle some of this. What I immediately thought of is simply: this is why testers do need to have some technical skills in the programmatic arena.
Put Some Heuristics Between Your Strategy and Your Tactics
I like to consider a strategy and tactics distinction when you have a test team that’s striving to operate within the sphere of quality assurance. When you have that situation, then operationally considering strategy and tactics (or logistics, if you prefer) becomes important. The emphasis needs to be on a gradual implementation of good testing practices that then filter up to a full quality assurance process. Since any such quality function, in my opinion, starts with good testing practices of some sort, I’ll mainly make my distinctions here in the context of the idea of testing.
Continue reading Put Some Heuristics Between Your Strategy and Your Tactics
Does Planning For Testing Require a Test Plan?
I previously talked about test strategy and I made it clear that in my opinion the test strategy and the test plan are two entirely different things. I’ve had a love-hate affair with test plans over the course of time, mostly because I’ve seen them become an end unto themselves. I’ve seen them become monolithic documents that people who need a sense of control use when they feel they can’t get that control in more effective ways or can’t deal with projects that are more fluid in nature.
But I wonder if I’m too reactionary in nature on this topic or if I’m just reacting to seeing a lot of badly written, rarely read test plans.
Continue reading Does Planning For Testing Require a Test Plan?
Your Test Strategy Is A Framework
I see a lot of testers — and their managers — misuse the phrase “test strategy.” I say they misuse it not because they don’t agree with some definition I have but rather because I rarely see a good distinction made between test logistics and test strategy. I often see a test strategy equated with a test plan. I hear some testers say that having a test plan is part of your test strategy. I hear others say that it’s the job of the test plan to indicate the test strategy.
So let’s talk about test strategy a bit.
Dynamic Execution in QTP
One of the nice things about SilkTest was the use of a dereference operator. I was disappointed to see that QTP didn’t have that. Yet I found you could start to mimic some of the functionality with the Eval function and the Execute statement.
Can I Stress The Load On My Volume?
In my post “Can I Function Test My Units?”, I talked about how testers often promoted a misuse of the phrase “functional testing.” I’ve found the same thing happens with the phrase “performance testing.” What I’ve found is that many testers don’t realize that performance testing, just like functional testing, is a test approach. This means there are various testing techniques that you can apply to the approach. If testers don’t have this kind of focus, I believe that the purpose of performance testing can be lost.
Quality Assurance – Obviously
So it’s obvious we need quality assurance, right? Okay, it’s true that quality is a dynamic concept that is contextual and situational. And, in a way, you can never really know the “actual” quality of an application in some universal sense. This is partly because quality is a value judgment based on the eye of the beholder. And it’s probably true that quality is not open to some absolute definition. (Similar to how other abstract words like justice, beauty, democracy, and so on are not absolute concepts.) But still. We obviously need some way to assure this beast we call quality. We obviously need a team called Quality Assurance.
I mean, it’s obvious. Err … right?