WOPR16 was held in Cleveland, Ohio, USA on April 28-30, 2011, and was hosted by Progressive. Greg McNelly was the Content Owner.
Attendees
Mike Bonar, Ross Collard, Dan Downing, Robert Haley, Paul Holland, Ed King, Felipe Knorr Kuhn, Yury Makedonov, Greg McNelly, Blaine Morgan, Patrick Mulcahy, Liang Peng, Alex Podelko, Eric Proegler, Dave Rabinek, Raymond Rivest, Reid Sheppard, Harsh Sinha, Peg Sonday, Roland Stens, Nishi Uppal
Theme: The Intersection of Performance and Functional Testing
Have you ever seen performance testers scoff in contempt when they discover a batch of functional bugs? Do your programmers roll their eyes when a functional tester reports a performance bug? Performance testing and functional testing are often treated as separate, almost unrelated activities on a testing project – carried out by different personnel, with different skills, with different tools, at different places and times.
Sometimes the opportunities to combine performance and functional testing are obvious. Sometimes coordination is coincidental, even accidental. By sharing experiences about their intersections, we will explore ways to synthesize performance and functional testing more deliberately.
Performance testing is often categorized as a type of non-functional testing – testing that questions not “what” the system does, but rather “how fast” or “how much”. While the separation of those questions may seem clear, the value of the information provided by their answers is commonly co-dependent – a system that does the right thing slowly, or the wrong thing quickly, probably won’t satisfy your customers. Yet several development and testing methodologies, both formal and informal, separate the missions of functional and performance testing, sometimes even suggesting precedence – first make it work, then make it perform. Some may address the potential for introducing one type of bug (functional or performance) while fixing the other type. Experienced testers know that bugs don’t necessarily play by the rules. Some performance bugs may be triggered by functional problems. Some functional bugs may not appear until the system is under stress or load.
Recent WOPRs have delved into heuristics for testing performance and reliability. One could argue that many of the lessons to be derived from the experiences shared would apply equally well to both performance and functional testing. Good functional testers and good performance testers would seem to benefit from many of the same skills and techniques, but the differences between the two types of testing support some very real separations in terms of people, processes, tools and other dimensions in the space of software testing.
At WOPR16, we will explore the relationship and interactions between performance and functional testing. We are looking for experience reports that will advance our community’s understanding of this subject. Here are some questions intended to spark thoughts on the theme:
- How is your performance testing dependent upon functional testing? (and vice versa) Any chicken-and-egg problems? How have you resolved them?
- What resources from functional testing do you reuse for performance testing? (and vice versa)
- During a ‘passing’ performance test, how do you know that the system is not simply malfunctioning very efficiently?
- What surprising functional defect has your performance testing uncovered?
- Does the performance-functional dichotomy benefit or harm your development and testing efforts? Have you experienced a more useful model?
- What heuristics do you employ that bridge the gap, or blur the line, between performance and functional testing?
These experiences may come in all shapes and sizes, perhaps touching on one or more of the following subject areas, or adding something new to the list:
- Test Data
- Testability
- Oracles
- Quality Criteria
- Test Environments / Labs and Tools
- Technical and Testing Skills
- Staffing
- Planning / Scheduling and Project Management
As you prepare your experience, consider the following focusing questions:
- Where or how do performance and functional testing intersect for you?
- How did you discover the intersection
- What are the conditions or forces that helped create it?
- What are the consequences of the intersection?
- Does your intersection create chaos, conflict, or collaboration?
- How has it affected your approach to your work? Is the effect general, or limited to specific situations?
Work Product
Private workshop collaboration space here.