Risk-based testing. What it is, why it works, and a CA pilot that proves it.
Out of the infinite cloud of possible tests, a test team runs a finite number. Risk-based testing decides which ones — in what order, with how much effort, and how to cut when the schedule compresses. This webinar walks through the method, the four benefits, and a six-step pilot at CA that shipped 92 risk items, an adjusted RPN histogram, and a re-allocated test effort that changed how the project shipped.
Abstract
You are going to test less than 100% of the system.
For any complex system, an infinite number of tests exist. You do not have forever. Out of that infinite cloud, the test team must select a finite number. Measured as a percentage of what you could test, your coverage is always zero. So you have to be very, very selective — and very, very smart — about what you run.
Risk-based testing is that selection mechanism. Quality risk is the possibility that the product fails to deliver one or more of its key quality attributes, endangering the outcomes you actually care about. Risk-based testing uses an analysis of those quality risks to prioritize tests, allocate effort, and — when the schedule compresses — drop tests in reverse risk order. It keeps you honest about trade-offs you are already making implicitly.
This webinar walks through the four benefits, then through a six-step pilot at CA that made it real. The pilot finalized 92 risk items, discovered a common clumping problem in the initial RPN distribution, adjusted the impact scale to fix it, and was later published in Better Software magazine. The numbers and the sequence below come from that engagement.
“Release when the risk of delay balances the risk of dissatisfaction.”
— Rex Black, Inc.
Outline
What the talk covers, in order.
Setting the course — infinite tests, finite schedule
Every test plan is an admission that you cannot test everything. The question is whether the selection is deliberate or accidental. Risk-based testing makes it deliberate: every test earns its place by how much risk it reduces.
What risk-based testing actually is
Risk-based testing uses an analysis of quality risks to prioritize tests and allocate testing effort. It involves key business and technical stakeholders so that the focus and sequence of testing line up with what quality means to the people who have to live with the product. It also means managing project risks — events that endanger completion of the project itself — alongside the quality risks.
The four benefits
Running tests in risk order finds defects in severity order. Allocating effort by risk minimizes residual quality risk at release. Measuring by risk tells you the residual quality risk during execution so you can make smart release decisions. And if the schedule compresses, dropping tests in reverse risk order gives you back time with the least possible increase in quality risk.
- Find the scary stuff first.
- Pick the right tests out of the infinite cloud.
- Release when risk of delay balances risk of dissatisfaction.
- Give up tests you worry about the least, not tests you happen to have left.
Does it increase test work?
No — if anything, the long-run effect is to decrease effort. Risk-based testing drives more efficient testing overall. After the initial quality risk analysis, only periodic updates and traceability maintenance are required.
CA pilot — the six activities
CA, a Rex Black, Inc. client, ran a pilot (subsequently followed by two more). The pilot was a six-activity sequence: train the stakeholders, hold a risk analysis session, analyze and refine the results, align testing with the risks, guide the project by risk, and assess the benefits. Each step below is from that pilot.
Training
Participants and stakeholders went through a one-day training covering the principles of risk-based testing, the categories of quality risks, how to analyze quality risks, how to align testing with risk levels, how to document the analysis, how to monitor quality risks during execution, and how to report risk-based results. Training was presentation, discussion, and a worked exercise — not lecture.
Quality risk analysis session
Two sub-sessions. In the first, participants identified risk items: main quality-risk categories went on three whiteboards; participants wrote risk items on sticky notes and posted them under the relevant categories. The sub-session lasted three hours and produced over 100 risk items — plus 11 project risks and 3 other issues as by-products. In the second sub-session, the team rated likelihood and impact for each item; they identified and merged duplicates along the way. At the end of the session, the team had 92 risk items and had rated 40% of them. The test manager finalized the remaining ratings with the participants afterward.
Analyzing and refining — the RPN histogram
With ratings complete, the team calculated an RPN (risk priority number) for each item. On a five-point scale each for likelihood and impact, RPN ranges from 1 (most risky) to 25 (least risky). A common problem: clumping. Clumping shows up when the team skews impact toward worst-case outcomes, or when the scale has poorly defined distinctions. You can check by plotting a histogram — and if you see clumping, you refine the scale and re-rate.
Aligning testing with quality risks
With the analysis settled, the test manager mapped risk items to specifications (the Product Requirements Specification and the Detail Design Specification), mapped them to test cases, re-allocated testing effort based on risk level, and prioritized the test cases within each risk band. An effort-allocation table was customized for the application so that every risk item received at least cursory testing. Mapping risk items to specifications lets you regenerate the analysis quickly when specs change.
Guiding the project based on risk
With finalized RPNs, effort allocations, and traceability, the test manager prioritized execution by RPN. This was a significant change from the previous, purely requirements-based approach. Tests had previously been assigned by tester expertise, which led to availability bottlenecks when a key person was out. Under risk-based testing, those bottlenecks went away — high-priority tests ran early, low-priority tests ran later. Bug management also changed: severity rankings were augmented with risk prioritization, and bug-fix effort focused on the high-risk defects.
Conclusion — benefits and key points
The pilot delivered what risk-based testing typically delivers: intelligent effort allocation within constraints, priority-order bug discovery that optimized fix-time windows, flexible handling of reductions in time or resources, and optimized quality within the constraints the project actually had. Two key points for anyone adopting this: include business users — and possibly customers — in the analysis, and start the risk analysis early in the lifecycle. Bottom line: risk-based testing lets a team deliver increasingly sophisticated products under tightening constraints by prioritizing testing and balancing quality against the other project priorities.
Key takeaways
6 things to remember.
Coverage is always zero
You cannot test everything. The honest question is how you are selecting. Risk-based testing makes selection explicit, argued, and reviewable.
Four benefits, not one
Find severity-first. Minimize residual risk. Know the residual risk in real time. Cut in reverse risk order. You lose all four if you skip the analysis.
Run risk analysis as a workshop
Whiteboards, sticky notes, three hours for identification, a second session for rating. Business plus technical plus test. Do not do it solo at your desk.
Plot the RPN histogram
If the distribution clumps, your scale is wrong — not your product. Adjust the definitions, re-rate, move on.
Map risks to specs and tests
Traceability cuts both ways: when specs change, you know which risks to re-rate; when risks change, you know which test cases to re-prioritize. Miss the mapping and you lose half the benefit.
Priority beats expertise assignment
Assigning tests by tester expertise creates bottlenecks when that expertise is unavailable. Priority-by-RPN routes around those bottlenecks and still finishes the scary stuff first.
Worked examples
One bug. Eight drafts.
RPN clumping — what you'll see in real data
The initial ratings from the CA pilot. Likelihood looked skewed but was reasonable given a mature, stable codebase. Impact was the real problem: half the items were rated 2, which compressed the RPNs into a narrow band. Adjusting the distinction between impact 2 and impact 3 fixed it.
Rating 1 · 5 items
Rating 2 · 9 items
Rating 3 · 25 items
Rating 4 · 39 items
Rating 5 · 26 items
Initial read: skewed toward the high end. Actual: the product was mature and well-established, with a maintainable codebase and experienced developers. The distribution reflected reality — no adjustment needed.
Rating 1 · 10 items
Rating 2 · 52 items
Rating 3 · 32 items
Rating 4 · 8 items
Rating 5 · 2 items
Problem: more than half the items sat at rating 2, which flattened the RPN distribution and made prioritization nearly impossible. Fix: redraw the line between impact 2 and impact 3 with sharper criteria, re-rate the affected items, plot again.
Effort allocation table — how to spend the testing you have
The allocation table from the CA pilot, customized so that every risk item — even the lowest — received at least cursory coverage. Your numbers will differ; the shape of the table will not.
Run a large number of tests that are both broad and deep. Exercise combinations and variations of interesting conditions. This is where the test team spends the bulk of its attention.
Run a medium number of tests that exercise many different interesting conditions. Fewer combinations; still wide coverage of the high-traffic paths.
Run a small number of tests that sample the most interesting conditions. Enough to catch obvious regressions. Not enough to reduce residual risk much further.
Leverage other tests or activities to run a test or two of an interesting condition — but only if it involves a very small investment of time and effort, and only if the opportunity presents itself. If neither condition is met, skip it. That is the entire point of prioritizing by risk.
Closing
Include business users — and potentially customers — in the risk analysis. The people who live with the product know which failures are unacceptable. The test team knows which ones are likely. You need both voices in the room, and you need them there early in the lifecycle, not after the schedule compresses.
Risk-based testing lets a team deliver increasingly sophisticated, complex products within tightening constraints by prioritizing testing and balancing quality with the other priorities the project is managing. It is not a shortcut. It is an honest accounting of what testing can and cannot do — and that honesty is what makes the release decision defensible.
Keep reading
Related pieces.
More for this audience
Articles, guides, and case studies tagged for the same readers.
- Whitepaper
Evaluation Before Shipping: How to Test an AI Application Before It Hits Production
The release-gate playbook for AI features. Covers the five evaluation dimensions, how to build a lean golden set, where LLM-as-judge is trustworthy and where it lies, rollout mechanics with named exit criteria, and the regression suite that keeps a shipped AI feature from quietly rotting in production.
Read → - Whitepaper
Choosing the Right Model (and Knowing When to Switch)
A practical framework for matching LLM model tier to task. Covers the four axes (capability, latency, cost, reliability), cascade routing patterns that cut cost 60 to 80 percent without measurable quality loss, switching costs you did not plan for, and the worked economics at 10K, 100K, and 1M decisions per day.
Read → - Whitepaper
Beyond ISTQB: A Multi-Domain Certification Roadmap for Technical L&D
Most engineering L&D programs over-index on a single certification family, usually ISTQB on the QA side, AWS on the infrastructure side, and under-invest across the rest of the technical domains the org actually needs. This paper covers a multi-domain certification roadmap (QA, AI, cloud, data, security, project management, software engineering) with sequencing logic for each level of the engineering ladder, plus the maintenance discipline that keeps the roadmap relevant as the technology shifts underneath it.
Read → - Guide
The ISTQB Advanced Level path, mapped
The Advanced Level landscape keeps changing — CTAL-TA v4.0 shipped May 2025, CTAL-TM is on v3.0, CTAL-TAE is on v2.0. This guide maps all four core modules, prerequisites, exam formats, sunset dates, and which module a given role should take first. Links directly to the authoritative istqb.org syllabi.
Read → - Whitepaper
Bug Triage: A Cross-Functional Framework for Deciding Which Defects to Fix
Bug triage is the cross-functional decision process that converts raw defect reports into prioritized action. Done well, it optimizes limited engineering capacity against risk; done poorly, it becomes a backlog-management ritual that neither fixes the important defects nor drops the unimportant ones. This whitepaper covers the triage process, the participants, the six action outcomes, the four decision factors, and the governance disciplines that keep triage effective in continuous-delivery environments.
Read → - Whitepaper
Building Quality In: What Engineering Organizations Do from Day One
Testing at the end builds confidence, but the most efficient quality assurance is building the system the right way from day one. This whitepaper covers the upstream disciplines — requirements clarity, lifecycle selection, per-unit programmer practices, and continuous integration — that make system-level testing cheap and fast rather than the only thing holding a release together.
Read →
Where this leads
- Service · Quality engineering
Software Quality & Security
Independent test programs, security testing, and quality engineering for systems where defects cost real money.
Learn more → - Solution
Risk Reduction & Clear Decisions
Quality programs and decision frameworks that shift risk discussions from anecdote to evidence.
Learn more → - Solution
Reliable Software at Scale
Quality engineering programs for organizations whose software is now operationally critical.
Learn more →
Want this talk delivered in-house?
Rex Black, Inc. delivers every talk on this site as a live workshop, a keynote, or a conference session. Tailored to your stack, your team, and your timeline.