Since 1994

Charting the progress

of system development.

Five bug-data charts every test lead should run every week · Rex Black, Inc.

Bug data is the most honest

dashboard you have.


Status reports are green because the author wants them to be.
Bug data is what it is.


Five charts. One hour a week.
That is the whole method.

REX BLACK, INC. · CHARTING DEFECT DATA
What this is about

In plain English.


If you run or oversee a software project, your bug tracker already has the most honest status report you'll ever get. You just have to plot it.


This talk shows five simple charts that answer four questions:

  • Is the product stabilizing? — or still getting worse every week.
  • Is the dev team keeping up? — or quietly falling behind.
  • Where are the mistakes being made? — requirements, code, environment.
  • Which part of the product is in the most trouble? — where to spend next.

No new tools. One hour a week in a spreadsheet. Written for test managers and delivery leads; accessible to anyone who reviews project status.

REX BLACK, INC. · CHARTING DEFECT DATA
Why chart bug data

Two reasons.

Assess

Product stability. Defect-removal trend. Root-cause distribution. Hot-spot subsystems.

Communicate

Summarize facts and trends — not incident counts. One chart, one argument. Management can act.


Manage the key indicators, not the crisis du jour.

REX BLACK, INC. · CHARTING DEFECT DATA
The worked example

SpeedyWriter.


Three cycles each of component, integration, and system test. Roughly 105 bugs over two months. First customer ship: 13 September.

Component test

7/19 — 25 bugs
7/26 — 20 bugs
8/2 — 5 bugs

Integration test

8/2 — 20 bugs
8/9 — 15 bugs
8/16 — 5 bugs


System test: 8/16 — 10 · 8/23 — 5 · 8/30 — 0. The zero is the first real stability signal.

REX BLACK, INC. · CHARTING DEFECT DATA
The only fields you need

Two columns. Opened date. Closed date.


Everything in this deck comes from those two columns plus a root-cause tag and a subsystem tag at close.


Bug reports approximate underlying bugs linearly — roughly 27% high because of duplicates and rejections. The charts hold up.

REX BLACK, INC. · CHARTING DEFECT DATA
Chart 1 · the flagship

Cumulative opened

vs. cumulative closed.


Run COUNTIF by date. Two running totals. Plot both on the same axis.

REX BLACK, INC. · CHARTING DEFECT DATA
Chart 1 · reading it

The gap is the quality backlog.

Opened curve

Flattens as the system stabilizes and the team finds what it can find.

Closed curve

Converges to the opened curve as the product becomes customer-ready.


Derivative of the opened curve is your daily find rate — also worth plotting. Milestones in the project show up as changes in shape.

REX BLACK, INC. · CHARTING DEFECT DATA
Chart 1 · trouble pattern 1 of 3

Endless bug discovery.


Shape. The opened curve never flattens. Through the planned release date, test is still finding bugs at near-peak rate.


Reading. The product is not stabilizing. Either the test team is exploring net-new territory each cycle, or every build is introducing new defects faster than the old ones are fixed.

Remedy. Stop the feature stream. Hold an integration line. Run a full regression against the last good build. Compare.

REX BLACK, INC. · CHARTING DEFECT DATA
Chart 1 · trouble pattern 2 of 3

Ignored bug reports.


Shape. The opened curve flattens normally. The closed curve lags farther and farther behind. The gap widens instead of closing.


Reading. The development team is not keeping up — understaffed, overcommitted to features, or not incentivized to close.

Remedy. Rebalance dev effort toward close-out, or formally accept the quality backlog as release risk. Do not hide it.

REX BLACK, INC. · CHARTING DEFECT DATA
Chart 1 · trouble pattern 3 of 3

Poor report management.


Shape. Both curves are noisy. The closed curve jumps around. Bugs keep re-opening.


Reading. The bug-report process itself is breaking down — sloppy verification, disputed closes, missing regression tests.

Remedy. Tighten close-out criteria. Add a re-verify step before close. Run the bug-reporting-process checklist with the team.

REX BLACK, INC. · CHARTING DEFECT DATA
Chart 2 · developer responsiveness

Closure period.


The age of a bug at the moment it closes.
Daily and rolling.

REX BLACK, INC. · CHARTING DEFECT DATA
Chart 2 · how to calculate

Four mechanical steps.


  1. For each closed bug: closure_period = closed_date − opened_date.
  2. Sum closure periods by closed_date.
  3. Count the bugs closed on each date.
  4. Daily = sum ÷ count. Rolling = running sum ÷ running count.

All four are pivot-table primitives in any modern spreadsheet.

REX BLACK, INC. · CHARTING DEFECT DATA
Chart 2 · reading it

A stable closure period

is a smooth rolling line.


Daily values bounded at the top by two to three test-cycle durations.


On SpeedyWriter, each cycle is a week — so a well-run project sees daily closure periods mostly under three weeks.

Exploding closure period means dev is not keeping up, regardless of how green the status report is.

REX BLACK, INC. · CHARTING DEFECT DATA
Chart 3 · where the mistakes are

Root cause breakdown.


Tag every bug at close.
Requirements · Design · Coding · Configuration · Environment · Documentation · Test case.

REX BLACK, INC. · CHARTING DEFECT DATA
Chart 3 · what it drives

Course corrections now.

Process improvements next.

During the project

A spike in requirements-driven bugs says the spec process needs a round. A spike in coding-driven bugs says code review needs a round.

After the project

The distribution is the input to the next cycle's development-process improvement plan. Phases, not names.

REX BLACK, INC. · CHARTING DEFECT DATA
Chart 4 · hot spots

Subsystem breakdown.


Where there is one bug,
there is often another.

REX BLACK, INC. · CHARTING DEFECT DATA
Chart 4 · two actionable patterns

Two things fall out.

More testing here

Subsystems with the most bug reports need more test investment. The Pareto holds on almost every system you'll ever ship.

More engineering here

Error-prone subsystems are candidates for development-process improvement too. Preventing upstream is cheaper than finding downstream.

REX BLACK, INC. · CHARTING DEFECT DATA
The takeaways

Five charts.

One hour a week.

REX BLACK, INC. · CHARTING DEFECT DATA
Takeaways · 1 of 2

The shape carries the signal.


  • Totals lie. Curves do not.
  • Learn the three trouble patterns — you'll spot them for the rest of your career.
  • Track the quality backlog (opened − closed), not just the find rate. It trends to zero before release — or nothing else matters.
REX BLACK, INC. · CHARTING DEFECT DATA
Takeaways · 2 of 2

Tag at close. Every bug.


  • Closure period tells you if developers are keeping up.
  • Root cause at close tells you where the mistakes are being made — and what to fix for next release.
  • Subsystem at close tells you where to re-invest test and engineering effort now.

Anything more is a project for the data team.
Anything less is inadequate.

REX BLACK, INC. · CHARTING DEFECT DATA
Since 1994

Thank you.

Rex Black, Inc. · rexblack.com/resources/talks/charting-defect-data