Charting the progress of system development.Five bug-data charts every test lead should run every week.

Why chart bug data at all

Two reasons. First, to assess product, process, and project quality, stability, defect-removal trends, root cause, bug management, and hot spots. Second, to communicate status to peers and management in a way that summarizes key facts, surfaces underlying trends, and gets the point across quickly.

• Summarize key facts and underlying trends, not incident counts.
• Get the point across quickly, one chart, one argument.
• Manage the key indicators, not the crisis du jour.

The SpeedyWriter worked example

SpeedyWriter runs three cycles each of component, integration, and system test. Component test: weeks of 7/19, 7/26, 8/2 (25, 20, 5 bugs. Integration test: 8/2, 8/9, 8/16) 20, 15, 5 bugs. System test: 8/16, 8/23, 8/30, 10, 5, 0 bugs. First customer ship: 13 September. That's ~105 bugs over two months. The tiny system-test-cycle-3 total of zero is the first real stability signal; everything else on the chart is context.

• Two fields from bug tracking carry most of the signal: opened date and closed date.
• Export to a spreadsheet to chart.
• Bug reports approximate underlying bugs linearly (roughly 27% high because of duplicates and rejections), charts hold up.

Chart 1, cumulative opened vs. closed

The flagship chart. The cumulative opened curve flattens as the system stabilizes and the test team finds what it can find. The cumulative closed curve converges to the opened curve as the product becomes customer-ready. A closed curve that tracks the opened curve closely indicates crisp bug management. Milestones in the project show up as changes in the shape of the curves, phase transitions, drop-and-recover events around major merges, and the final close-out.

• In Excel: COUNTIF by date for opened and closed, then two running totals.
• Watch the gap between the two curves; that gap is the quality backlog.
• The derivative of the opened curve is your daily find rate, also worth plotting.

Chart 1, the three trouble patterns

Three distinct failure modes show up unmistakably on the opened/closed chart: endless bug discovery (the opened curve never flattens (the product is not stabilizing), ignored bug reports (a persistent, growing gap between opened and closed) the dev team is not keeping up), and poor report management (the gap is noisy and non-monotonic, reports are opening, closing, and re-opening sloppily). Each has a different remedy, which is the whole point of looking at the pattern rather than the totals.

Chart 2, closure period

Closure period is the age of a bug at the moment it closes. Measures developer responsiveness. Two versions: daily closure period is the average age of all bugs closed on a given day; rolling closure period is the average age of all bugs closed to date. A stable closure period means a smooth rolling line and daily values bounded at the top by two to three test-cycle durations. In the SpeedyWriter example, each cycle is one week, so a well-run project sees daily closure periods mostly under three weeks.

How to calculate closure period

Four mechanical steps. (1) For each closed bug, compute closure period = closed_date − opened_date. (2) Sum closure periods by closed_date. (3) Count the bugs closed on each date. (4) Divide sum by count for the daily closure period; maintain a running total of sum and count for the rolling closure period. All four steps are pivot-table primitives in any modern spreadsheet.

Chart 3, root cause breakdown

Tag every bug with a root-cause code at close, requirements, design, coding, configuration, environment, documentation, test case. The distribution tells you where the mistakes are being made, which drives both course corrections during the project and long-term development-process improvements afterwards. You're not naming names; you're naming phases.

Chart 4, subsystem breakdown

Tag every bug with the affected subsystem. Two actionable patterns fall out. First, the subsystems with the most bug reports should get more testing, where there is one bug, there is often another. Second, the subsystems that are most error-prone are candidates for development-process improvement too, because preventing the additional bugs upstream is cheaper than finding them downstream.