Part 2 of 6 · Investing in Software Testing
Buying the right stocks is harder than deciding to invest. The same is true of testing. This article explains the concept of test system fidelity — the degree to which your test system predicts the customer's real experience of quality — and why low-fidelity programs are worse than no program at all.
Read time: ~7 minutes. Written for QA leaders and engineering managers scoping a test strategy.
The investment problem testing actually faces
Part 1 made the financial case for testing as an investment. But just as a dollar in the stock market doesn't automatically compound, a dollar spent on testing doesn't automatically reduce the cost of quality. The return depends entirely on whether the program is finding the defects customers actually care about.
The phrase we use for the full assemblage — environment, data, test cases, tools, and execution processes — is the test system. The question this article addresses is: what makes a test system worth the money?
The answer is fidelity.
What "fidelity" means in a test system
A high-fidelity test system faithfully replicates the behaviors the customer will experience in production. If the product ships a bug that matters to customers, a high-fidelity system catches it before release. If the product works, a high-fidelity system produces a trustworthy confidence signal that lets leadership ship with real information rather than wishful thinking.
A low-fidelity test system does something that looks like testing. It exercises code paths customers don't use, on configurations customers don't run, and reports failures customers wouldn't notice or care about. It produces green dashboards that don't reflect reality. It consumes budget. It generates false confidence, which is arguably worse than no confidence at all — because now the organization is making release decisions on bad data.
Fidelity isn't binary. It's a spectrum. The goal isn't perfection — that's not affordable — but to land as close to the customer's real behavior as budget and schedule allow.
How to waste money on testing
A low-fidelity program wastes budget in three predictable ways:
- Testing features customers don't use. Every test is an opportunity cost. An hour spent validating a screen that gets 0.2% of traffic is an hour not spent on the checkout flow.
- Testing configurations no customer runs. The matrix explosion of OS × browser × region × tier × entitlement is real. A high-fidelity program cuts it down to the configurations representing most customer usage and tests those deeply.
- Reporting problems no customer cares about. Cosmetic bugs, edge-case timing on internal tools, minor log formatting — all real findings, none of them worth release delay. A low-fidelity program treats them the same as revenue-blocking defects. The noise drowns the signal.
The compounding damage is that leadership stops trusting the test function. When "testing" becomes a ritual that blocks releases over non-issues while missing the bug that tanks the next deploy, the program loses credibility — which makes it harder to fund next quarter.
A cautionary case study
Picture a QA team with a slick automated test harness. The product is a multi-OS, multi-database query and reporting system. The harness fires canned queries, compares results against baselines, and runs thousands of tests across a dozen OS/database combinations in a couple of days. By every internal metric — test count, pass rate, coverage breadth — the program looks world-class.
It was a waste of money.
Why? The customer pain wasn't "queries sometimes return wrong results." The query logic was solid. The bugs that reached production — the bugs losing the company deals — were in installation and ancillary tooling. The product was hard to install. The management utilities didn't work. The companion agents crashed.
The test team was perpetuating and expanding a harness that exercised the one part of the product that didn't need it, while ignoring the parts that did. High sophistication, low fidelity. The investment posted a negative return.
The lesson isn't "don't automate." It's: automation amplifies whatever test strategy it implements. If the strategy is wrong, automation makes the wrong thing happen faster and at greater scale.
What high fidelity requires
Fidelity doesn't fall out of a framework. It falls out of paying attention to the customer. Specifically:
- Know the usage profile. What workflows do 80% of customers actually run? What percentage of traffic hits the top 10 features? What configurations dominate the installed base?
- Know the failure modes that matter. Not every defect has the same blast radius. A transient UI glitch and a data-corruption bug both go into the tracker, but they belong in different columns.
- Know the adjacent pain. Installation, upgrade, monitoring, error messages, error recovery — these are the places bugs hide because they're not the "main" product surface, but they dominate customer experience.
- Get the data. Production telemetry, support ticket clustering, churn-exit interviews, sales call-recording keywords, NPS open-text. This is the source material for a test strategy that aligns with reality.
Low-fidelity test strategies are usually the ones built purely from the specification, disconnected from any observation of how the product actually gets used. High-fidelity strategies are grounded in behavior.
The decision this sets up
If fidelity is the goal, then the operational question becomes: how do you pick the right tests? That's the job of quality risk analysis — the subject of Part 3. Risk analysis is the discipline of enumerating what can go wrong, what it would cost, and which of those scenarios are likely enough to deserve test effort. Done well, it's the mechanism that turns an abstract call for "high fidelity" into a concrete, prioritized test plan.
Related resources
- Part 1 — The Cost of Software Quality — the financial foundation this article builds on.
- Part 3 — The Risks to System Quality — the prioritization methodology.
- Quality Risk Analysis Process — the checklist form of the methodology.
- Investing in Software Testing — the leadership deck version of this series.