TalkRex Black Inc. webinar
Risk-based testing.What it is, why it works, and a CA pilot that proves it.
Out of the infinite cloud of possible tests, a test team runs a finite number. Risk-based testing decides which ones, in what order, with how much effort, and how to cut when the schedule compresses. This webinar walks through the method, the four benefits, and a six-step pilot at CA that shipped 92 risk items, an adjusted RPN histogram, and a re-allocated test effort that changed how the project shipped.
- Slides
- 19
- Pilot risk items
- 92
- RPN range
- 1–25
Abstract
You are going to test less than 100% of the system.
For any complex system, an infinite number of tests exist. You do not have forever. Out of that infinite cloud, the test team must select a finite number. Measured as a percentage of what you could test, your coverage is always zero. So you have to be very, very selective (and very, very smart) about what you run.
Risk-based testing is that selection mechanism. Quality risk is the possibility that the product fails to deliver one or more of its key quality attributes, endangering the outcomes you actually care about. Risk-based testing uses an analysis of those quality risks to prioritize tests, allocate effort, and (when the schedule compresses) drop tests in reverse risk order. It keeps you honest about trade-offs you are already making implicitly.
This webinar walks through the four benefits, then through a six-step pilot at CA that made it real. The pilot finalized 92 risk items, discovered a common clumping problem in the initial RPN distribution, adjusted the impact scale to fix it, and was later published in Better Software magazine. The numbers and the sequence below come from that engagement.
“Release when the risk of delay balances the risk of dissatisfaction.”
,Rex Black Inc.
Outline
What the talk covers, in order.
Setting the course, infinite tests, finite schedule
Every test plan is an admission that you cannot test everything. The question is whether the selection is deliberate or accidental. Risk-based testing makes it deliberate: every test earns its place by how much risk it reduces.
What risk-based testing actually is
Risk-based testing uses an analysis of quality risks to prioritize tests and allocate testing effort. It involves key business and technical stakeholders so that the focus and sequence of testing line up with what quality means to the people who have to live with the product. It also means managing project risks (events that endanger completion of the project itself) alongside the quality risks.
The four benefits
Running tests in risk order finds defects in severity order. Allocating effort by risk minimizes residual quality risk at release. Measuring by risk tells you the residual quality risk during execution so you can make smart release decisions. And if the schedule compresses, dropping tests in reverse risk order gives you back time with the least possible increase in quality risk.
- Find the scary stuff first.
- Pick the right tests out of the infinite cloud.
- Release when risk of delay balances risk of dissatisfaction.
- Give up tests you worry about the least, not tests you happen to have left.
Does it increase test work?
No, if anything, the long-run effect is to decrease effort. Risk-based testing drives more efficient testing overall. After the initial quality risk analysis, only periodic updates and traceability maintenance are required.
CA pilot, the six activities
CA, a Rex Black Inc. client, ran a pilot (subsequently followed by two more). The pilot was a six-activity sequence: train the stakeholders, hold a risk analysis session, analyze and refine the results, align testing with the risks, guide the project by risk, and assess the benefits. Each step below is from that pilot.
Training
Participants and stakeholders went through a one-day training covering the principles of risk-based testing, the categories of quality risks, how to analyze quality risks, how to align testing with risk levels, how to document the analysis, how to monitor quality risks during execution, and how to report risk-based results. Training was presentation, discussion, and a worked exercise, not lecture.
Quality risk analysis session
Two sub-sessions. In the first, participants identified risk items: main quality-risk categories went on three whiteboards; participants wrote risk items on sticky notes and posted them under the relevant categories. The sub-session lasted three hours and produced over 100 risk items, plus 11 project risks and 3 other issues as by-products. In the second sub-session, the team rated likelihood and impact for each item; they identified and merged duplicates along the way. At the end of the session, the team had 92 risk items and had rated 40% of them. The test manager finalized the remaining ratings with the participants afterward.
Analyzing and refining, the RPN histogram
With ratings complete, the team calculated an RPN (risk priority number) for each item. On a five-point scale each for likelihood and impact, RPN ranges from 1 (most risky) to 25 (least risky). A common problem: clumping. Clumping shows up when the team skews impact toward worst-case outcomes, or when the scale has poorly defined distinctions. You can check by plotting a histogram, and if you see clumping, you refine the scale and re-rate.
Aligning testing with quality risks
With the analysis settled, the test manager mapped risk items to specifications (the Product Requirements Specification and the Detail Design Specification), mapped them to test cases, re-allocated testing effort based on risk level, and prioritized the test cases within each risk band. An effort-allocation table was customized for the application so that every risk item received at least cursory testing. Mapping risk items to specifications lets you regenerate the analysis quickly when specs change.
Guiding the project based on risk
With finalized RPNs, effort allocations, and traceability, the test manager prioritized execution by RPN. This was a significant change from the previous, purely requirements-based approach. Tests had previously been assigned by tester expertise, which led to availability bottlenecks when a key person was out. Under risk-based testing, those bottlenecks went away, high-priority tests ran early, low-priority tests ran later. Bug management also changed: severity rankings were augmented with risk prioritization, and bug-fix effort focused on the high-risk defects.
Conclusion, benefits and key points
The pilot delivered what risk-based testing typically delivers: intelligent effort allocation within constraints, priority-order bug discovery that optimized fix-time windows, flexible handling of reductions in time or resources, and optimized quality within the constraints the project actually had. Two key points for anyone adopting this: include business users (and possibly customers) in the analysis, and start the risk analysis early in the lifecycle. Bottom line: risk-based testing lets a team deliver increasingly sophisticated products under tightening constraints by prioritizing testing and balancing quality against the other project priorities.
Key takeaways
6 things to remember.
Coverage is always zero
You cannot test everything. The honest question is how you are selecting. Risk-based testing makes selection explicit, argued, and reviewable.
Four benefits, not one
Find severity-first. Minimize residual risk. Know the residual risk in real time. Cut in reverse risk order. You lose all four if you skip the analysis.
Run risk analysis as a workshop
Whiteboards, sticky notes, three hours for identification, a second session for rating. Business plus technical plus test. Do not do it solo at your desk.
Plot the RPN histogram
If the distribution clumps, your scale is wrong, not your product. Adjust the definitions, re-rate, move on.
Map risks to specs and tests
Traceability cuts both ways: when specs change, you know which risks to re-rate; when risks change, you know which test cases to re-prioritize. Miss the mapping and you lose half the benefit.
Priority beats expertise assignment
Assigning tests by tester expertise creates bottlenecks when that expertise is unavailable. Priority-by-RPN routes around those bottlenecks and still finishes the scary stuff first.
Worked examples
One bug. Eight drafts.
RPN clumping, what you'll see in real data
The initial ratings from the CA pilot. Likelihood looked skewed but was reasonable given a mature, stable codebase. Impact was the real problem: half the items were rated 2, which compressed the RPNs into a narrow band. Adjusting the distinction between impact 2 and impact 3 fixed it.
Rating 1 · 5 items
Rating 2 · 9 items
Rating 3 · 25 items
Rating 4 · 39 items
Rating 5 · 26 items
Initial read: skewed toward the high end. Actual: the product was mature and well-established, with a maintainable codebase and experienced developers. The distribution reflected reality, no adjustment needed.
Rating 1 · 10 items
Rating 2 · 52 items
Rating 3 · 32 items
Rating 4 · 8 items
Rating 5 · 2 items
Problem: more than half the items sat at rating 2, which flattened the RPN distribution and made prioritization nearly impossible. Fix: redraw the line between impact 2 and impact 3 with sharper criteria, re-rate the affected items, plot again.
Effort allocation table, how to spend the testing you have
The allocation table from the CA pilot, customized so that every risk item (even the lowest) received at least cursory coverage. Your numbers will differ; the shape of the table will not.
Run a large number of tests that are both broad and deep. Exercise combinations and variations of interesting conditions. This is where the test team spends the bulk of its attention.
Run a medium number of tests that exercise many different interesting conditions. Fewer combinations; still wide coverage of the high-traffic paths.
Run a small number of tests that sample the most interesting conditions. Enough to catch obvious regressions. Not enough to reduce residual risk much further.
Leverage other tests or activities to run a test or two of an interesting condition, but only if it involves a very small investment of time and effort, and only if the opportunity presents itself. If neither condition is met, skip it. That is the entire point of prioritizing by risk.
Closing
Include business users (and potentially customers) in the risk analysis. The people who live with the product know which failures are unacceptable. The test team knows which ones are likely. You need both voices in the room, and you need them there early in the lifecycle, not after the schedule compresses.
Risk-based testing lets a team deliver increasingly sophisticated, complex products within tightening constraints by prioritizing testing and balancing quality with the other priorities the project is managing. It is not a shortcut. It is an honest accounting of what testing can and cannot do, and that honesty is what makes the release decision defensible.
Keep reading
Related pieces.
Keep reading
Related reading
- Primer
Careers in Software Quality: Roles, Skills, and First Steps
What testers, test automation engineers, quality engineers, and test managers actually do, the skills each role uses, and practical first steps for students and career changers, including the ISTQB Foundation certification.
Read → - Primer
Errors, Defects, and Failures: What a Bug Really Is
Everyone says 'bug', but testers use three precise words: error, defect, and failure. Learn the difference, why it matters, and how severity and priority decide which problems get fixed first.
Read → - Primer
How to Report a Bug So It Actually Gets Fixed
Finding a bug is half the job. Learn how to write a bug report a developer can act on: a clear summary, exact steps to reproduce, expected versus actual results, and the habits professional testers use to make every report count.
Read → - Primer
What Is Software Testing? A Plain-English Introduction
Software testing explained from the beginning: why every program has mistakes in it, what testers actually do all day, and the handful of ideas the whole profession is built on. No experience needed.
Read → - Primer
Your First Test Cases: Designing Tests That Find Problems
A hands-on introduction to test design. Learn what goes into a test case, then use two classic techniques, equivalence partitioning and boundary value analysis, to test a sign-up form the way professionals do.
Read → - Whitepaper
Evaluation Before Shipping: How to Test an AI Application Before It Hits Production
The release-gate playbook for AI features. Covers the five evaluation dimensions, how to build a lean golden set, where LLM-as-judge is trustworthy and where it lies, rollout mechanics with named exit criteria, and the regression suite that keeps a shipped AI feature from quietly rotting in production.
Read →
Practices
Where this leads
- QA & testingSoftware testing
Software quality consulting since 1994: we test the releases that matter, coach your team on the method, and measure how its testing matures.
Book a call →