Whitepaper · Quality Foundations · ~9 min read
Every quality conversation in an enterprise engineering program runs on an implicit definition of "quality software." Most of these implicit definitions are incomplete — they cover one dimension of quality and assume the others. The incompleteness shows up in the test program as wasted effort on low-risk areas, blind spots at release, and stakeholder disagreements that don't resolve because the underlying definition was never made explicit.
This whitepaper covers the three-way definition framework — outcomes, characteristics, and knowledge — and how to use it to drive risk-based test planning under the current quality-model standard ISO/IEC 25010:2023. Pairs with the Quality Risk Analysis whitepaper (the process that operationalizes this definition) and the Metrics Part 4 whitepaper (the product-quality metrics that measure against the definition).
Why the definition matters
Ask a room of practitioners, "for the systems you build, what comes to mind when you think about quality?" The answers cluster into three distinct families. Most respondents stay in one family and assume the others. The tests that get designed, the criteria that get written into release gates, and the metrics that get reported at the end of the project are all shaped by which family dominates in the responder's mental model.
When different stakeholders on the same program hold different family definitions — when engineering is thinking about characteristics, the business is thinking about outcomes, and test is thinking about knowledge — the program runs on conflicting assumptions. Test plans drift. Acceptance criteria don't line up. Release conversations become frustrated.
Making the three families explicit, and deciding on a complete definition that addresses all three, is the first step in a defensible quality program.
The three families
Family 1: outcomes — what the result will be
Most responses reduce to one of two definitions:
- The software conforms to its specification. Closely follows Phil Crosby's definition of quality (Quality Is Free, 1979).
- The software fits its various uses and purposes. Closely follows J. M. Juran's definition (Planning for Quality and later Juran on Planning for Quality).
Juran's definition is the stronger of the two for enterprise software. Fully articulated, Juran's definition means: the software has those attributes, characteristics, and behaviors that satisfy its customers, users, and other stakeholders, and has few if any of those attributes, characteristics, and behaviors that dissatisfy them.
Crosby's conformance-to-specification definition sounds reasonable but runs aground when applied to software. Industry research over decades — originally Capers Jones, extended by many others — has consistently shown that a large fraction of defects are introduced during requirements and design specification. Measuring software quality against the specification only is measuring with a flawed yardstick: a system that perfectly implements an incorrect specification is, by definition, useless.
The Juran-style "fit for use" definition does not have this problem. It treats the stakeholder experience of the delivered system as the target, regardless of whether the specification was right.
Family 2: characteristics — what the software will look like
Outcomes are only observable after delivery. Before delivery, the team works against the characteristics the software will have — attributes that, if present, produce the outcomes the stakeholders want.
Some characteristics are easy to surface: reliability, performance, usability, scalability, data integrity. Others are harder — functionality is the most commonly-missed in brainstorming sessions, because it feels so obvious that it doesn't get mentioned until someone points out that "does the thing it's supposed to do" is, in fact, a quality characteristic.
The set of characteristics that matters for a given system depends on the system. A financial system has accuracy and auditability at the top of the list; a consumer application has usability and performance; a healthcare system has accessibility, privacy, and regulatory compliance; an AI-backed system has reasoning accuracy, safety, and explainability. A generic list is useful as a starting prompt, not as the answer.
Family 3: knowledge — how we'll know
Occasionally — in roughly one conversation out of ten — someone responds to the quality question with something like: "software that was thoroughly tested in a way that covered all important quality risks, with few if any blocked tests, critical failures, or high-priority defects at the conclusion of testing."
This response reflects a third dimension. It is not about what the software will be after delivery (outcomes) or what attributes it will have (characteristics) — it is about how the organization will know whether the software has the required quality. This response understands that while testing cannot change the quality of software, testing offers the organization the opportunity to detect and correct quality problems before release, and to build confidence where the system is observed to work properly.
A complete quality definition has all three. Outcomes give the business target. Characteristics give the engineering target. Knowledge gives the testing and release target. An incomplete definition that addresses only outcomes cannot be used to plan engineering work. An incomplete definition that addresses only characteristics cannot be used to design release gates. An incomplete definition that addresses only knowledge produces a test program that is internally rigorous but disconnected from what stakeholders actually care about.
ISO/IEC 25010:2023 — the current reference model
For the characteristics family, the current international standard is ISO/IEC 25010:2023 (published November 2023), which replaced the 2011 edition. The 2023 model identifies nine product-quality characteristics, each with subcharacteristics:
| Characteristic | Selected subcharacteristics |
|---|---|
| Functional suitability | Functional completeness, correctness, appropriateness |
| Performance efficiency | Time behaviour, resource utilization, capacity |
| Compatibility | Co-existence, interoperability |
| Usability | Appropriateness recognizability, learnability, operability, user error protection, user engagement, inclusivity, user assistance, self-descriptiveness |
| Reliability | Faultlessness, availability, fault tolerance, recoverability |
| Security | Confidentiality, integrity, non-repudiation, accountability, authenticity, resistance |
| Maintainability | Modularity, reusability, analysability, modifiability, testability |
| Portability | Adaptability, installability, replaceability |
| Safety | Operational constraint, risk identification, fail safe, hazard warning, safe integration |
The 2023 revision notably added Safety as a top-level characteristic (formerly absorbed into Reliability and Security), expanded Usability to include modern accessibility and user-engagement considerations, and updated Security to reflect current attack-surface thinking.
ISO/IEC 25010:2023 is the reference list many enterprise programs use to structure the "characteristics" part of their quality definition. Two pragmatic notes:
- Not every characteristic matters equally for every system. A static reporting tool has minimal Safety relevance; a medical device has Safety at the top. Weight the characteristics by the system's risk profile.
- Modern systems usually require at least one characteristic not in the standard list. For AI-backed systems, this typically includes explainability, reasoning quality, alignment, and bias/fairness attributes. For distributed systems with emergent behaviour, it may include resilience characteristics beyond those in Reliability. Extend the list explicitly rather than forcing modern concerns into the legacy shape.
An internal two-dozen-item checklist of quality-risk areas — tailored to the organization's typical systems — is a pragmatic companion to the ISO/IEC 25010 list. The two serve different purposes: the ISO model gives the standards-grounded structure; the internal checklist gives the organization-specific prompts.
Using the three-family definition in practice
The three families connect directly to the disciplines of a quality program.
Business-stakeholder conversations (outcomes)
With business stakeholders, the quality conversation is about outcomes: what must be true for customers, users, and other stakeholders to consider the delivered system a success? This is Juran's territory. The deliverable is not a specification; it is a shared understanding of the outcome that specifies, acceptance criteria, and success metrics will all ultimately serve.
Engineering and architecture conversations (characteristics)
With engineers and architects, the quality conversation shifts to characteristics: given the desired outcomes, what attributes must the system have? This is ISO/IEC 25010 territory, plus any domain-specific additions. The deliverable is a prioritized list of quality attributes, weighted by their contribution to the outcomes, and translated into architectural and implementation decisions.
Test-planning and release conversations (knowledge)
With testers, test managers, and release managers, the quality conversation focuses on knowledge: how will the organization know, with enough confidence to ship, that the system has the attributes that will produce the outcomes? This is the territory of test planning, quality risk analysis, coverage, defect data, and release-gate criteria. The deliverable is a test program and a release-gate design that produces the needed confidence within the available effort.
The three conversations are connected. Outcomes define why certain characteristics matter. Characteristics define what attributes the knowledge-producing activities must verify. The test program is therefore not a free-standing activity — it is a derivation from the business outcomes via the characteristic priorities.
Operationalizing the definition: quality risk analysis
In an enterprise program, the bridge between the three-family definition and concrete engineering work is quality risk analysis. For each characteristic that matters for the system, the analysis enumerates the specific ways the characteristic could fail to be present — the quality risks — and assesses the level of each risk based on likelihood (technical considerations) and impact (business considerations).
This is where the definition becomes actionable:
- An outcome that depends on the system's performance under load turns into a characteristic (performance efficiency) and a set of specific risks (response time under concurrency, throughput under resource contention, capacity limits at scale).
- Each risk gets an assessed level, which determines the extent of test effort allocated to mitigating it.
- The resulting test plan is traceable back to the business outcomes through the characteristics that serve them.
Without the three-family definition, the risk analysis drifts — it generates risks against a characteristic list that was never reconciled with the outcomes the stakeholders care about, and the test program covers what was specified rather than what would actually signal success.
Quality cannot be tested in; it must be defined, then built, then verified
A final principle that the three-family definition makes explicit: quality cannot be tested into software at the end of a project. Grinding out as many defects as possible during a pre-release test cycle is, at best, inefficient; at worst, it does not produce the outcomes users will experience as quality. The delightful quality that comes from the best-designed software products is built throughout the lifecycle — in the requirements, the design, the implementation, the review processes, the continuous integration — with testing providing the verification that the built-in quality is in fact present.
The three-family definition belongs at the front of the lifecycle, not at the back. A program that defines quality first, builds against the definition throughout, and verifies against the definition at the end is running on a different engine than a program that defers quality to the test function.
The definition as an artifact
A practical output of this framework is a Quality Definition Document — a short artifact (often one or two pages, sometimes a section of a larger program-quality-plan document) that records:
- Outcomes — the stakeholder-experience targets the system must produce. In business terms, not technical terms. Signed off by the business owner.
- Characteristics — the quality attributes the system must have to produce the outcomes, drawn from ISO/IEC 25010:2023 plus any domain-specific additions, weighted by priority.
- Knowledge — how the organization will know the attributes are present and the outcomes will be produced, including the test program's scope, the release gate's criteria, and the operational monitoring that will continue verification after release.
The document is referenced during risk analysis, test planning, release-gate reviews, and post-release retrospectives. It is updated as the program's understanding of the outcomes and the characteristics evolves. It is not a compliance artifact — it is a reasoning artifact that keeps the program aligned.
A program with a clear Quality Definition Document can answer, for any test activity, release decision, or quality investment: which outcome does this serve, through which characteristic, and what knowledge does it produce? A program without one can only answer in terms of the activity itself — and those answers do not survive contact with stakeholders who are asking the outcome question.
Related resources
- Quality Risk Analysis — the process that operationalizes the three-family definition.
- Metrics for Software Testing, Part 4: Product Metrics — how to measure the characteristics you've decided to test.
- Critical Testing Processes — the framework for building the knowledge-producing activities.
- Investing in Software Testing, Part 1: The Business Case — the ROI framework for the test program that serves the definition.