Investing in Testing, Part 3: The Risks to System Quality

From fidelity to action

Part 2 argued that the best test systems mirror the customer's real experience. That's a goal, not a method. This article provides the method: quality risk analysis — the discipline of enumerating what can go wrong, how much each failure would hurt, and how likely it is, so that testing effort can be deliberately aligned to risk.

What we mean by "quality"

Phil Crosby defined quality as "conformance to requirements." That works if your requirements are good, which in most organizations they aren't. Juran's definition is more durable: fitness for use — a product has quality when it's fit for the uses customers buy it for and set it to. Quality is present when the product exhibits behaviors customers value and avoids behaviors they don't.

This framing matters because it shifts testing from "check this spec" to "predict the customer's experience." Risk analysis is how you turn that shift into a concrete plan.

The four risk categories on every project

Projects face four categories of risk. Project managers who stay employed learn to track all four:

• Financial risks — how might the project overrun budget?
• Schedule risks — how might the project exceed the time box?
• Feature risks — how might we ship the wrong product?
• Quality risks — how might the product fail to be fit for use?

Testing is the instrument that measures quality risk. A rigorous test program turns fuzzy worries about "is it good enough?" into quantified findings the project team can weigh against the other three.

Classes of quality risk

Pattern-matching on known risk categories is far more productive than starting with a blank page. Here's a working set that applies to most systems:

• Functionality — does the software do what it's supposed to do?
• Use cases — do features still work when chained into realistic workflows?
• Robustness — are common errors handled well?
• Performance — does the system respond quickly enough?
• Reliability — does the system stay up under normal and peak load?
• Volume / capacity — does the system hold up at scale?
• Data quality — can the data store be corrupted or accept invalid input?
• Usability — is the interface intuitive and efficient?
• Localization / internationalization — does the system work outside its home locale?
• Security — can attackers damage, corrupt, or exfiltrate?
• Compliance — does the system meet regulatory obligations (GDPR, HIPAA, PCI, SOC 2, state privacy laws, sector-specific rules)?
• Integration / interoperability — do third-party integrations and APIs behave correctly across version changes?
• Observability — can operators detect and diagnose incidents fast enough?
• AI / model behavior — where applicable, are model outputs accurate, consistent, unbiased, and within safety bounds?

Function is the obvious one, and the trap is over-indexing on it. Most catastrophic releases fail on one of the other categories — installation, reliability, data quality, or security.

Prioritizing with three dimensions

Once risks are enumerated, they need to be scored. Three dimensions do the work:

• Severity — how dangerous is a failure in this area? (data loss, legal exposure, safety, revenue blocker, cosmetic)
• Priority — how much does a failure here compromise the value of the product to customers and users?
• Likelihood — what are the odds a user hits this failure, given usage profiles and technical risk?

Scale each one 1 (worst) to 5 (least concerning). The specific scale matters less than using it consistently across the whole analysis.

Three analysis techniques

Pick the lightest technique your organization will actually follow. A well-used informal analysis beats an unused FMEA.

Informal

Best for teams new to the practice, lean organizations, and early-stage products. Three columns:

Build the table, walk it through the product, engineering, support, and sales stakeholders, and socialize the priority column. The table becomes the map for a short, pointed conversation with every stakeholder — one-on-one or as a group — about what the product can and cannot afford to fail on.

ISO-aligned (adapted from ISO/IEC 9126 / 25010)

ISO 9126 (and its successor ISO/IEC 25010) defines a set of quality characteristics. Use them as an enumeration, not as a compliance exercise:

• Functional suitability — completeness, correctness, appropriateness.
• Performance efficiency — time behavior, resource use, capacity.
• Compatibility — coexistence, interoperability.
• Usability — learnability, operability, user error protection, accessibility.
• Reliability — maturity, availability, fault tolerance, recoverability.
• Security — confidentiality, integrity, non-repudiation, accountability, authenticity.
• Maintainability — modularity, reusability, analyzability, modifiability, testability.
• Portability — adaptability, installability, replaceability.

For each characteristic that matters, identify the sub-characteristics that apply to your system, define measurable success criteria, and set priority levels for testing each one.

FMEA (Failure Mode and Effects Analysis)

Best when the cost of missing a defect is high enough to justify rigor — regulated domains, safety-critical systems, enterprise products with large blast radius.

FMEA lists every failure mode, its effects, and a Risk Priority Number (RPN) equal to Severity × Priority × Likelihood. Lower RPN = lower priority. A proper FMEA happens in a room (or a well-facilitated meeting) with all stakeholders so the analysis captures cross-functional perspective, not just the QA team's guesses.

FMEA is heavier than the informal method and pays for itself only when the stakes warrant it. Use it by default for anything that can lose money, injure someone, or make the news.

Two anti-patterns to avoid

Over-decomposing failure modes. "Misspelled user messages" is a useful entry. Trying to enumerate every possible misspelling is not. Stay at the level of abstraction where the entry drives test design decisions.

Asking stakeholders the wrong question. Don't ask "do you want us to make sure this area works?" — you'll get "yes" to everything, which is useless. Instead ask: "How much money, time, and attention would you be willing to give to problems in this area? Would you fund an extra tester here? Would you delay shipping to fix a defect in this category?" Real trade-off questions elicit real priorities.

Translating priority into test effort

After analysis you have a ranked list. Translate the ranking to effort like this:

• High priority — test extensively, with multiple techniques, and monitor in production.
• Medium priority — test broadly but not exhaustively.
• Low priority — cursory testing; accept some escape risk.
• Negligible priority — don't spend on it at all.

This is the point at which "high fidelity" from Part 2 stops being an aspiration and becomes an allocation. Your test budget now has a defensible shape that maps to customer impact.

What comes next

Knowing what to test doesn't tell you how to test it. Different risks call for different techniques — static review, structural unit tests, behavioral end-to-end tests, property-based fuzzing, chaos testing. Part 4 covers the three main technique families and when to use each.

Related resources

• Quality Risk Analysis Process — the printable checklist version of this methodology.
• Risk-Based Testing Webinar — deeper treatment with worked examples.
• Test Estimation Process — how risk priority feeds into test sizing.
• Part 2 — High Fidelity Test Systems.
• Part 4 — The Importance of the Right Technique.