Our last brainstorming exercise was an examination of the ways in which we acknowledge our performance model is likely to be inaccurate, compared to reality. Since much of performance testing centers on “realistic” simulation, and iterative development forces us to test faster with less information, it was instructive to review how we approach simulation fidelity.
Substantial discussions took place under each of these points. These are a “Table of Contents”, at best.
One of these begs some explanation: “Turkey Theory”. Essentially, this refers to the fallacy of assuming that the past completely predicts the future. The example here is the Thanksgiving Turkey, who has the experience of several hundred days to support the proposition that the farmer loves him, feeds him well, and wants him to be healthy and happy. On one particular day, the prediction that today will be the same as yesterday becomes tragically (and deliciously) wrong.
This work is the product of all of the Workshop on Performance and Reliability (WOPR22) attendees. WOPR22 was held May 21-23, 2014, in Malmö, Sweden, on the topic of “Early Performance Testing”. Participants in the workshop included Fredrik Fristedt, Andy Hohenner, Paul Holland, Martin Hynie, Emil Johansson, Maria Kedemo, John Meza, Eric Proegler, Bob Sklar, Paul Stapleton, Andy Still, Neil Taitt, and Mais Tawfik Ashkar.
| Load Models | Environment | Reporting and Projection |
|---|---|---|
| Single User | Different/Less Hardware | Maintenance of Production Environment |
| User Counts | Non-dedicated Test Environment | Over-Simplification |
| Mix of Workflows | Stubbing/ignoring backend and other systems | Generating more reports under the illusion that existing reports are consumed |
| Pacing of user Input & Delays | Background processing and maintenance are not considered | Misuse of statistical analysis |
| Optimized Data - row counts and cardinality | Influence of external systems | Shifting acceptance criteria |
| Very different test data | Impact of marketing/social marketing (point load) | "Turkey Theory" |
| Missing incomplete sessions | Shared infrastructure resources | |
| Not evolving load model and/or data | Variable network models (consider Shunra and LANforge to address) | |
| Organic loads and arrival rates | Warmed Caches | |
| Tool limitations | Inconsistent/dynamically changing configuration, software, and hardware (cloud and/or virtualization) | |
| Tool limitations | ||