Since 1994

Risk-based testing.

What it is, why it works, and a CA pilot that proves it · Rex Black, Inc.

You are going to test

less than 100%

of the system.


The question isn't whether you'll make a selection.
It's whether the selection is deliberate or accidental.

REX BLACK, INC. · RISK-BASED TESTING
What this is about

In plain English.


For any real product, there is an infinite cloud of possible tests. You do not have forever. You run a finite number. Measured as a percentage of what you could test, your coverage is always zero.

Risk-based testing is how you select which tests to run, and which to cut first when the schedule compresses. This talk answers four questions:

  • How do you pick the right tests out of an infinite cloud?
  • How do you know when you've tested enough to ship?
  • When the schedule slips, which tests should you drop first?
  • How do you prove to leadership the trade-off was defensible?

Written for test managers and leadership under constrained schedules. Accessible to any business or technical leader who signs off on a release.

REX BLACK, INC. · RISK-BASED TESTING
What risk-based testing is

Quality risk.

Prioritized tests.

Informed trade-offs.

REX BLACK, INC. · RISK-BASED TESTING
Definition

The method in one slide.


Quality risk is the possibility that the product fails to deliver one or more of its key quality attributes.

Risk-based testing:

  • Uses an analysis of quality risks to prioritize tests.
  • Uses that analysis to allocate effort per risk item.
  • Involves business and technical stakeholders so what gets tested lines up with what quality means to the people who live with the product.
  • Also manages project risks (events that endanger the project itself) alongside the quality risks.
REX BLACK, INC. · RISK-BASED TESTING
The four benefits

Find, minimize,

measure, cut.

REX BLACK, INC. · RISK-BASED TESTING
Benefit 1 of 4

Find the scary stuff first.


Running tests in risk order finds defects in severity order.

The first bug you find is the bug that matters most. Every fix window you open gets used on the highest-impact defect in the system.

REX BLACK, INC. · RISK-BASED TESTING
Benefit 2 of 4

Minimize residual risk at release.


Allocating effort by risk concentrates testing where the stakes are highest.

The residual quality risk at the moment of release is lower than any other allocation you could have chosen with the same test budget.

REX BLACK, INC. · RISK-BASED TESTING
Benefit 3 of 4

Know the residual risk during execution.


Measuring by risk tells you where you stand in real time.

You can answer "is this ready?" with evidence instead of opinion. Release when the risk of delay balances the risk of dissatisfaction.

REX BLACK, INC. · RISK-BASED TESTING
Benefit 4 of 4

Cut in reverse risk order.


When the schedule compresses, drop tests you worry about least, not tests you happen to have left on the pile.

The same time back, at the lowest possible increase in residual risk.

REX BLACK, INC. · RISK-BASED TESTING
A common objection

"Doesn't this

increase test work?"

REX BLACK, INC. · RISK-BASED TESTING
Objection answered

No. Decreased long-run effort.


Risk-based testing drives more efficient testing overall.

After the initial quality risk analysis, only periodic updates and traceability maintenance are required. Every subsequent release rides on the same analysis, refined, not rebuilt.

REX BLACK, INC. · RISK-BASED TESTING
Case study

CA pilot.

Six activities.

92 risk items.

REX BLACK, INC. · RISK-BASED TESTING
The pilot at a glance

Six activities.


  1. Train the stakeholders.
  2. Hold the quality risk analysis session.
  3. Analyze and refine the results.
  4. Align testing with the risks.
  5. Guide the project by risk.
  6. Assess the benefits.

Published in Better Software. Followed by two additional pilots at the same client.

REX BLACK, INC. · RISK-BASED TESTING
Activity 1 · Training

One day. Worked exercise.


Not lecture. Presentation, discussion, and a worked exercise.

Covers: the principles; the categories of quality risks; how to analyze them; how to align testing with risk levels; how to document the analysis; how to monitor risks during execution; how to report risk-based results.

REX BLACK, INC. · RISK-BASED TESTING
Activity 2 · Analysis session

Two sub-sessions.

Identify

Three whiteboards for the main quality-risk categories. Sticky notes under the relevant one. Three hours. Over 100 risk items, plus 11 project risks and 3 other issues.

Rate

Likelihood + impact per item. Duplicates identified and merged. 40% rated in-session; test manager finalized the rest with participants afterward. Ended with 92 risk items on the docket.

REX BLACK, INC. · RISK-BASED TESTING
Activity 3 · Analyze & refine

The RPN histogram.


Risk Priority Number = Likelihood × Impact. 1 (most risky) to 25 (least risky) on a 5×5 scale.

Common problem: clumping. Shows up when the team skews impact toward worst-case, or when the scale has poorly defined distinctions.

Check: plot the histogram. Fix: refine the scale, re-rate.

REX BLACK, INC. · RISK-BASED TESTING
Activity 3 · Clumping in real data

The CA histogram.

Likelihood (reasonable)

1 · 5 · 9 · 25 · 39 · 26

Skewed toward high end. Actual: mature product, stable codebase, experienced dev. The distribution reflected reality. No adjustment.

Impact (clumped, fix)

1 · 10 · 52 · 32 · 8 · 2

Over half the items at rating 2. Flattened the RPNs. Redraw the line between impact 2 and 3. Re-rate. Plot again.

REX BLACK, INC. · RISK-BASED TESTING
Activity 4 · Alignment

Allocation of test effort by RPN.

RPN 1–12 · Extensive

Large number of tests. Broad and deep. Combinations and variations. Bulk of test attention.

RPN 13–16 · Broad

Medium number of tests across many interesting conditions. Fewer combinations; still wide coverage.

RPN 17–20 · Cursory

Small number of tests sampling the most interesting conditions. Enough for obvious regressions.

RPN 21–25 · Opportunity

Leverage other tests to run one or two of these, only if investment is small and opportunity presents itself. Otherwise skip it.

REX BLACK, INC. · RISK-BASED TESTING
Activity 5 · Guiding the project

Priority beats expertise assignment.


Previously: tests assigned by tester expertise. Bottleneck every time a key person was out.

Under risk-based testing: priority by RPN. Bottlenecks went away. High-priority tests ran early. Low-priority tests ran later.

Bug management also changed. Severity augmented with risk prioritization. Fix-time focused on the high-risk defects.

REX BLACK, INC. · RISK-BASED TESTING
Activity 6 · Benefits

What the pilot delivered.


  • Intelligent effort allocation within the constraints the project actually had.
  • Priority-order bug discovery that optimized every fix-time window.
  • Flexible handling of reductions in time or resources.
  • Optimized quality within the constraints, not in spite of them.
REX BLACK, INC. · RISK-BASED TESTING
Takeaways

Select deliberately.

Cut honestly.

REX BLACK, INC. · RISK-BASED TESTING
Takeaways · 1 of 2

On the method.


  • Coverage is always zero. The honest question is how you select.
  • Four benefits, not one. Find severity-first. Minimize residual risk. Know real-time risk. Cut in reverse order.
  • Run analysis as a workshop. Whiteboards, sticky notes, business + technical + test.
REX BLACK, INC. · RISK-BASED TESTING
Takeaways · 2 of 2

On running it.


  • Plot the RPN histogram. If it clumps, the scale is wrong, not the product.
  • Map risks to specs and tests. Traceability cuts both ways.
  • Priority beats expertise assignment, routes around every bottleneck.

Release when the risk of delay balances the risk of dissatisfaction.

REX BLACK, INC. · RISK-BASED TESTING
Since 1994

Thank you.

Rex Black, Inc. · rexblack.com/resources/talks/risk-based-testing-webinar