Matching Test Techniques to the Extent of Testing: A Heuristic Framework for Risk-Based Test Design

The problem this framework solves

The Pragmatic Risk Analysis and Management process — from Managing the Testing Process, Pragmatic Software Testing, and Advanced Software Testing: Volume 2 — assigns each risk item an extent of testing, in decreasing order of thoroughness:

• Extensive
• Broad
• Cursory
• Opportunity
• Report bugs only
• None

These extents answer "how much effort against this risk?" They do not prescribe which test-design techniques to apply. That selection depends on:

• The bug hypothesis — what failure modes are suspected for this risk item. (Boris Beizer's coinage; the mental model of what could go wrong.)
• The technology under test — the language, framework, integration points, and infrastructure determine which techniques are even available.
• The test basis — decision tables, state diagrams, use cases, formal specifications are each only usable if they exist.
• The test level — unit, integration, system, system-integration tests favor different technique families.

Within those constraints, however, useful heuristics exist linking extent of testing to test-technique selection. The framework below applies to system and system-integration testing by independent test teams. It adapts to other test levels and organizational structures with minor adjustments.

Extensive

Quality Risk Analysis process template language: "run a large number of tests that are both broad and deep, exercising combinations and variations of interesting conditions."

Because the language explicitly calls for combinational testing, the selected techniques must generate test values that cover combinations:

• Domain analysis or decision tables — appropriate where the mode of interaction between factors is understood (e.g., rules that determine output values from input combinations).
• Classification trees, pairwise testing, or orthogonal arrays — appropriate where the mode of interaction between factors is not understood, or where interaction is not expected to matter (e.g., configuration compatibility).

For each technique selected, the strongest coverage criteria should be applied: all columns in a decision table, including boundary-value analysis and equivalence partitioning on the conditions; full pairwise coverage across all factors; etc. Combinational techniques at strong coverage criteria guarantee deep coverage.

In addition, for all relevant inputs or factors, tests must cover all equivalence partitions and, where applicable, boundary values. This contributes to broad coverage.

Testers should plan to augment the technique-derived test values with values selected using experience-based and defect-based techniques — prior defect patterns for this kind of system, known-problematic inputs, error-guessing hypotheses. This augmentation can occur during test design and implementation, or during execution.

If use cases are available, they should be tested at full normal-and-exception-path coverage.

If state transition diagrams are available, complete state/transition coverage is required, 1-switch (or higher) coverage is recommended, and for safety-related risk items, state-transition-table coverage is also recommended.

In certain cases — safety-critical risks, risks related to key features — the tester may elect to use code coverage measurements and apply white-box techniques to fill code-coverage gaps. In these cases, the white-box layer sits underneath the black-box extensive coverage, not in place of it.

Typical effort proportion: ~50% of total design, implementation, and execution effort goes to the risk items assigned extensive testing.

Broad

Template language: "run a medium number of tests that exercise many different interesting conditions."

The focus is breadth without the deep combinational work of extensive testing:

• Tests cover all equivalence partitions and, where applicable, boundary values.
• Augment with experience-based and defect-based values during design, implementation, or execution.
• If use cases exist, cover all normal and exception paths.
• If state transition diagrams exist, complete state/transition coverage is required; higher switch-coverage levels are used only if the test count stays manageable.
• If decision tables exist, use them — but strive for one test per column rather than per full combinational expansion.

Combinational testing (other than the possible use of decision tables) typically should not be used unless it can be done without generating a large number of test cases.

Typical effort proportion: ~25–35% of total effort goes to the risk items assigned broad testing.

Cursory

Template language: "run a small number of tests that sample the most interesting conditions."

The focus is sampling, not coverage:

• Use equivalence partitioning or boundary-value analysis on the appropriate areas of the system to identify particularly interesting test values. Do not try to cover all partitions or boundaries.
• Augment with experience-based and defect-based values.
• If use cases are available, cover normal paths; exception paths are optional.
• Decision tables may be used, but do not cover columns that represent unusual situations.
• State transition diagrams may be used, but do not visit unusual states or force unusual events.
• Combinational testing (other than possibly decision tables) is not used.

Typical effort proportion: ~5–15% of total effort goes to cursory-level risks.

Opportunity

Template language: "leverage other tests or activities to run a test or two of an interesting condition, but invest very little time and effort."

Opportunity testing is designed-in slack capacity — values inserted into tests that are running for other reasons:

• Experience-based and defect-based techniques are particularly valuable here, since the tester can slot additional test values into the logical flow of other tests without much additional cost.
• Equivalence partitioning or boundary-value analysis can identify the values; the tester does not try to cover all partitions or boundaries, just selects a few interesting ones to slot in.

Typical effort proportion: less than 5% of total effort in aggregate across all opportunity-level risks. Within any given test case, no more than 20% of the effort for that test case should be devoted to opportunity-level risk items, to avoid inadvertently escalating them to cursory or broad.

Report bugs only

Template language: "do not test at all, but, if bugs related to this risk arise during other tests, report those bugs."

No test design, implementation, or execution effort should occur. Any effort spent is a misallocation. The only action is defect reporting if a failure is observed incidentally.

None

Template language: "neither test for these risks nor report related bugs."

No test design, implementation, or execution effort should occur — and, distinct from "report bugs only," no bug reports should be filed if failures are observed incidentally. This extent is rarer than "report bugs only" and usually corresponds to risks explicitly descoped from the release.

Technique selection at a glance

Modern adaptations

The framework is technique-neutral and levels-neutral; it adapts to modern test programs with three specific notes.

API-layer and automated testing

Most technique selection above was articulated in the context of manual system test design. The same framework applies to automated test design, with the technique mapping adjusted:

• Decision tables and domain analysis translate cleanly into parameterized/data-driven tests with table-driven inputs.
• Pairwise testing tools (NIST ACTS, Hexawise, ACTS output directly consumed by test frameworks) generate input combinations for automated suites.
• State transition diagrams translate to state-machine tests in automated suites; 1-switch coverage is directly implementable as test graphs.
• Boundary-value analysis for numeric and size inputs is especially well-suited to property-based testing frameworks (Hypothesis in Python, fast-check in JavaScript, ScalaCheck), which systematically shrink failing inputs to their minimal form.

The effort-allocation heuristics translate, too: if 50% of manual effort should go to extensive-level risks, then 50% of the automated test suite's execution time and maintenance burden should track the same proportion.

Contract testing at the interoperability boundary

For interoperability risks at extensive or broad coverage levels, consumer-driven contract testing (Pact, Spring Cloud Contract) is a modern technique that didn't exist when this framework was first articulated. It fits cleanly at extensive and broad levels as a combinational technique across the integration boundary, and at cursory level as a smoke-test for schema compatibility.

AI-augmented test design

LLM-assisted test case generation — where a model produces candidate test cases from specifications, code, or existing test sets — is a promising augmentation at all coverage levels, with two disciplines:

• Treat model output as input to a human test-design review, not as finished tests. Models are useful for enumeration (brainstorming partitions, generating boundary candidates, listing exception paths) and for translating between representations (decision table → test cases, state diagram → test paths).
• Measure the coverage the generated tests actually achieve against the framework above, not just the number of generated tests. A large volume of generated tests that cluster in a few partitions is not broad coverage — it's cursory coverage at a noisy scale.

Running the framework

A practical application on a real risk analysis:

1. For each quality risk item, the risk analysis has assigned a risk level and, from it, an extent of testing.
2. For each risk item at extensive or broad level, select primary technique(s) from the table above, verify the test basis exists (use cases, state diagrams, decision tables, specifications), and design tests at the specified coverage criterion.
3. For each risk item at cursory or opportunity level, select a sampling or insertion strategy rather than a coverage strategy.
4. Verify the effort proportions at the end of planning — extensive at roughly 50%, broad at 25–35%, cursory at 5–15%, opportunity under 5%, report-bugs-only and none at 0%.
5. If actual proportions deviate substantially from the heuristic, either the risk analysis is producing too many (or too few) risks at a given level, or the test design is over/under-investing against the risk-weighted plan. Either is worth investigating.

The framework is a heuristic, not a rule. A specific program may have good reasons to shift the proportions — regulatory requirements, prior-release defect history, known technology risks. The value is in having a defensible starting point and a clear explanation whenever the actual numbers differ from it.

Related resources

• Quality Risk Analysis — how to produce the risk list and assign the extents this framework consumes.
• Functional Testing — the accuracy/suitability/interoperability framework that sits underneath these technique choices.
• Four Ideas for Improving Test Efficiency — complementary efficiency framework that targets the same goal from a different angle.
• Metrics for Software Testing, Part 3: Project Metrics — how to measure the effort proportions this framework prescribes.