by T.D. (“Dan”) Nelson:
Studies reporting about the effectiveness of ISO 9001 are not hard to find. These studies commonly purport to convey ISO 9001’s direct impact upon organizational performance, offering insight regarding the effectiveness of the standard and the relative risk of doing business with registered vs. nonregistered organizations. However, these studies are often not as reliable as their statistically valid results may suggest.
The results of these studies are commonly compromised for two reasons. First, they ignore a relevant distinction between “paper quality management systems (QMSs)” and “real QMSs.” Second, the effectiveness of ISO 9001 shouldn’t be determined by directly evaluating organizational performance in the first place. These two related shortcomings bring the results of ISO 9001 effectiveness studies into question.
Real QMSs vs. paper QMSs
Many ISO 9001-registered QMSs don’t demonstrate a process approach as required of the standard. In other words, those companies that have defined their QMSs according to the clauses of ISO 9001 (or by using only the “six required procedures”) are misapplying ISO 9001. They should instead define their processes according to the actual processes working together to produce goods and services.
Unaware of this, registered companies often believe that their standard-based, paper QMSs are real QMSs. In reality, they’re not. They’re sets of documents raised in response to ISO 9001 requirements. These standard-based documents demonstrate a lack of understanding and a failure to apply a process approach.
Standard-based QMSs aren’t really QMSs at all. QMSs are systems of processes delivering goods and services to customers—these are systems of processes, not systems of documents. Documents describing a system of processes producing goods or services don’t constitute that system of processes—although these documents may describe a QMS, they are not themselves the QMS.
Although ISO 9001 requires documented QMSs, it has never intended QMSs to be mere systems of documents. Real QMSs result in goods and services being released to customers. Although following good process documentation may result in outputting quality goods and services, the documents themselves don’t produce goods and services and are thus not a QMS.
A common assumption
The question of ISO 9001 effectiveness entertained by many studies presupposes ISO 9001-registered QMSs are consistently and effectively implemented. Paper QMSs, not being real QMSs in the first place, are thus not effectively implemented QMSs. Yet many paper QMSs achieve and maintain registration.
Unfortunately, because ISO 9001 registration doesn’t distinguish between standard-based (nonconforming) QMSs and process-based (conforming) QMSs, studies of ISO 9001 effectiveness don’t take this distinction into account. Ignorant of the distinction between paper QMSs and real QMSs, the studies only consider the registration itself. The studies then (not unreasonably) assume that if the organization is ISO 9001-registered, then it has a bona fide QMS. By failing to understand the difference between a system of documents and a system of processes—and relying upon ISO 9001 registration to make this distinction—these studies make incorrect assumptions when investigating and reporting about the effectiveness of ISO 9001.
Effectiveness and intent
The studies seem to presume that ISO 9001 is effective if implementation resulted in improved organizational performance. This presumes that ISO 9001 is intended to improve quality performance within any given organization. Although ISO 9001 happens to require improvement, it is and always has been management who manages quality and drives improvement.
It’s not ISO 9001 requirements that are supposed to be implemented in organizations, it’s actually internal processing requirements (or QMS requirements or management’s planned arrangements) that should be implemented—arrangements perhaps complying with ISO 9001. ISO 9001 is not supposed to be implemented by management to ensure quality. Instead, management is supposed to establish (via documented information) its own system for satisfying customers. The system is supposed to demonstrably comply with ISO 9001 requirements (as opposed to being defined by ISO 9001 requirements).
When ISO 9001 is applied to define a QMS, the standard is being misapplied and the result is a dysfunctional, standard-based QMS. ISO 9001 in these cases isn’t being applied as intended to assess suppliers’ abilities; it’s being applied to structure and document organizations’ QMSs. Because it’s not being used as intended (i.e., to assess a QMS), conclusions about its effectiveness are suspect.
Lumping dysfunctional, standard-based QMSs together with sensible, process-based QMSs in a study purporting to report upon ISO 9001 effectiveness is problematic. Because ISO 9001 is not intended as a quality management or operational improvement guide, reporting upon organizations’ quality performance is not reporting upon ISO 9001 effectiveness.
In reality, the effectiveness of ISO 9001 is not accurately evaluated by looking at organizational performance. ISO 9001 registration is intended to provide confidence in a supplier’s abilities; it isn’t designed to impose a QMS on anyone to ensure quality. Assessing quality performance is not assessing ISO 9001 effectiveness; it’s assessing managements’ ability to achieve quality objectives.
It’s kind of like applying a thermometer and drawing conclusions about barometric pressure. Sure, we can report the results, but what do the results have to do with the question? If ISO 9001 requirements were supposed to be implemented in organizations to manage quality, then evaluating quality performance results would indicate the effectiveness of ISO 9001. But since that’s not the case, it doesn’t.
Unrecognized uncertainties affecting the results of ISO 9001 effectiveness studies aren’t considered by the studies’ conclusions. By eliminating these uncertainties, the studies would be more clearly conveying information about the relative risk of doing business with an ISO 9001-registered supplier vs. a non-ISO 9001-registered supplier.
A better question?
Because ISO 9001 registration is intended to provide confidence in an organization’s ability to meet requirements, a question more directly speaking to the effectiveness of ISO 9001 might be: “To what degree is ISO 9001 registration effective in distinguishing between organizations that are likely to consistently provide quality and those that are not?” Or, “Does ISO 9001 effectively distinguish between suppliers with acceptable QMSs and those without acceptable QMSs?”
If the question of the effectiveness of ISO 9001-registered QMSs is to be addressed by a study, it seems a distinction needs to be made between standard-based and process-based QMSs. Although we should be able to rely on ISO 9001 registrations to make this distinction, we can’t. That’s in part why the authors of ISO 9001 are continuing to clarify and emphasize the requirement to apply a process approach—so all registered systems are consistently and sensibly based on their own processes.
Rather than ignoring the distinction between standard-based (paper) QMSs and process-based (real) QMSs, as does current ISO 9001 registration, a more useful study might recognize the difference between the two and report upon the relative effectiveness of standard-based vs. process-based QMSs.
There are two problems here. The first problem is the failure to distinguish between paper QMSs and real QMSs. How can we rely upon the results of a study reporting the effectiveness of a tool, based on users’ experiences with it, when many of the users are using the tool upside down and backward? How can the results of such a study accurately report upon the tool’s effectiveness if the study ignores the fact that the tool is being used improperly by a significant number of users? If a study came out about the effectiveness of a windshield scraper, how much stock would we put in the results of the study if we discovered that most of the people using the scraper were using the handle end to scrap their windshields instead of the blade end of the scraper?
The second problem is that ISO 9001 effectiveness studies are reporting on how effective it is at doing something it’s not intended to do. These studies purportedly show how ISO 9001 has affected organizational performance. Although ISO 9001 is a kind of gauge of organizational performance, it’s not intended to directly improve performance; rather, management is responsible for improving when improvement is needed. A thermometer isn’t designed to raise temperature, but merely to measure temperature.
Expecting ISO 9001 to improve performance is like expecting a thermometer to raise or lower the temperature in a room. The thermometer is merely a measuring device. Its effectiveness isn’t assessed by its ability to raise or lower the temperature in a room, but by its ability to accurately convey what the temperature is. Management is responsible for how well their organizations perform and for improving operations as needed; ISO 9001 is not. Assessing the effectiveness of ISO 9001 by its ability to raise organizational performance is like assessing the effectiveness of a thermometer by its ability to raise room temperature.
About the author
T.D. (“Dan”) Nelson has been closely involved with ISO 9000 since 1994 as a technical writer, quality manager, management representative, consultant, author, and auditor. Nelson also has 12 years of experience as an IRCA-certified QMS lead or principal auditor.
Using a process approach, Nelson has taken scores of clients of various shapes and sizes through registration to ISO 9001:1994/2000/2008 and related sector schemes (e.g., QS-9000, AS9100, ISO 13485, and ISO 17025). He is available for management consulting, training, and coaching, as well as auditor training.