by Andy Hofmann

Sometimes finding a topic on which to base an article is a struggle. One idea after another is considered, but discounted as uninteresting to anyone but the author. Other times, there are too many ideas and picking just one is the challenge. Then there are times when the idea is presented to the author in an unusual coincidence of events. This is one of those times.

A few weeks ago, I joined an online discussion thread focused on methods to measure an internal audit. The thread had a number of terrific ideas including comparing the number of external audit issues raised to those generated internally. There were also ideas concerning the time spent doing the audits compared with the value of the corrective actions/opportunities for improvement (OFI) generated. After several exchanges between the thread originator and the auditing world, it became clear that the discussion was really searching for a way to measure internal audit effectiveness.

Very recently I had the opportunity to attend a lead auditor class for the aerospace industry based on AS9100. The interesting thing about AS9100 is that it requires the auditor to make a determination about management system effectiveness. Although not itself a measure of the value of the internal audits, a system that is audited internally and found conforming and then audited externally and found ineffective speaks volumes. One would have to agree that regardless of any other metric assigned to an internal audit, this would be an important measure.

The convergence of the Internet thread and the AS9100 course I taught provided the topic for this month’s column: measuring internal audit effectiveness.

Most auditors are aware that auditing using the process approach achieves far better results than simply verifying clause requirements. In fact, the authors of AS9100 recognized this because they indicated through AS9101 that each process addresses the majority of the standard’s clauses. For example, auditing the purchasing process results not only in clause 7.4 being brought into play, but also:

• Records
• Documents
• Communication
• Goals and objectives
• Competence and resources
• Legal and regulatory requirements
• Planning
• Contract review
• Product realization
• Measurement
• Improvement
• Nonconforming product
• Data analysis
• Corrective actions
• Preventive actions

The result is that when one completes an audit of all the organization’s processes, the audit standards’ individual requirements have been assessed many different times. Such a broad perspective provides significant data about the organization, its processes, and conformance to the audited standard. This is the first metric for internal audit effectiveness: use of the process approach.

I know what some of you are saying, “Andy, no internal audit would in 2012 be using anything other than the process approach.” Not true. As recently as eight weeks ago I came across not only an audit that focused on conformance with each clause, but the underlying system was also clause-based. What was even more interesting was that the system was based on the structure of ISO 9001 dating back to 1987. That’s right, the system reflected clauses 4.1 to 4.20 by containing 20 sections to the manual. No internal audit comment was made concerning the lack of process identification.

A second metric for internal audit effectiveness would be its ability to keep the system current during changes. Changes in the underlying system standard structure, adding or deleting processes, and changing business goals could be examples. When reviewing internal audits, external auditors often find that audit methods have been in place for many cycles and that the basis for the internal audit protocols haven’t been amended based on new performance objectives, new business, or new processes. This is our second metric for internal audit effectiveness: revision of audit criteria.

Our third metric for internal audit effectiveness would be courtesy of AS9100’s requirements. AS9100 auditors are taught something that isn’t intuitive: It’s possible to be in nonconformance with the requirements of the standard and still have an effective system. Despite not meeting individual clauses or a series of clauses, a system may still satisfy customers and achieve the organization’s stated objectives. The corollary is also true: It’s possible to have full conformance with the standard and yet have an ineffective system. This is because simply creating a system that meets the individual and collective clauses of the standard doesn’t ensure that customers will be satisfied and targets achieved.

Unfortunately, many internal audits focus on simple conformance with the requirements of a standard. Nonconformities are issued when a requirement from one of the clauses doesn’t exist or isn’t being followed. As the internal audit cycles progress, auditors review previous results to figure out how the audit is supposed to be conducted. They see the nonconformities relating to clause requirements and continue to audit in this way. Significant issues concerning process effectiveness are missed. This is a third metric for internal audits: Observations on process and system effectiveness.

In the past months, I have heard the next metric several times from different organizations. For senior management to consider internal audits worth the effort it takes to complete them, internal auditors must find nonconformities. Moreover, managers often want them to find more nonconformities than external auditors found. The rationale for this is difficult to argue with. Management looks at the number of hours invested in the internal audit and compares it with those invested in external audits. If the internal audits invest 100 hours and identify four nonconformities and the external audit invests 40 hours and comes up with ten nonconformities, there’s clearly something wrong.

Although it’s appreciated that the intent of both internal and external audits is to find conformance, the ability to identify weaknesses is key to improvement. ISO 9001 and all subsequent derivatives include internal audits as one of the mechanisms driving continual improvement. Our fourth internal audit metric then would be the number of external audit hours per identified nonconformity vs. number of internal audit hours per identified nonconformity. Any ratio greater than one is good; a number greater than two is excellent.

Our fifth metric relates to another take away from the AS9100 course I taught: Each process defined by the management system must have some kind of measure associated with it. These measures originate with top management and are used to ensure that the personnel conducting the work and the personnel managing the work have a useful reference for their efforts. Linked to the satisfaction of the customers’ requirements, these metrics will usually address price or value, delivery, and quality.

I already addressed the concept of internal audits assessing the effectiveness of a process. This is the goal set for the process. There is a step beyond this that relates to the connection of the individual process goals to those set for the company. In other words, the auditor determines if the achievement of the individual process goals results in the achievement of those for the business. For example, an internal auditor might assess three processes: contracting, purchasing, and shipping and delivery. Each of the audits found that all the teams that perform each process conform to their requirements. Each individual process achieves its goals set by management, which include hours per product, throughput volume, schedule attainment, and right the first time. The auditor concludes that the system conforms to its requirements and that the processes are effective.

However, a review of the performance of the company as a whole shows a different story. Although there are few customer rejections with external PPM of 32, the on-time delivery performance is less than 60 percent against a goal of 98 percent. The external auditor determines that there are contractual schedule requirements that have not been included in production plans. In other words, the target set by the internal schedule is inconsistent with that set by the most important stakeholder, the customer. The relationship between the contracting part of the organization and its scheduling part isn’t effective. Therefore, our fifth metric for an internal audit is its ability to test the interaction of processes.

Finally, our sixth metric is recognition that the appropriate resources are being afforded to the internal audit. The simplest metric to determine this is conformance with the audit schedule. If every planned audit is deferred every time, the resources are clearly not present. It would be useful to quantify this metric, so our final internal audit measure is audits completed on time measured as a fraction (number of audits conducted on time/number of audits scheduled).

Six metrics for internal audit effectiveness:

• Use of the process approach
• Periodic revision of audit criteria
• Observations on process and system effectiveness
• Number of external audit hours per nonconformity vs. number of internal audit hours per nonconformity
• Ability to test the interaction of processes
• Number of audits conducted on time/number of audits scheduled

Only the fourth and sixth items are calculated numbers. The rest are either present or not. At the end of the audit cycle, the lead auditor would need to create a score. Were we to assign a single point each to criteria, we will end up with a total nominal score greater of six. The score for criterion four can range from a fraction of one to some multiple of one. The score for criterion six will be a maximum of one. The ideal result will be a number greater than six.

Weighting of the criteria based on individual company needs is also possible. For example, criterion four might be considered more important than the others. Applying a factor to the number would permit an indication of its relative importance. Thus, if we applied a factor of three to criterion four, the rating scale would have an ideal result greater than eight.

There are three benefits to using this kind of scoring method. First, it measures the ongoing effectiveness of audits. Organizations spend a lot of time training auditors and then providing them the time to perform audits. Having a measure of how well this investment is performing puts audits on the same level of significance as other activities the organization measures.

The second benefit from measuring audit effectiveness is to differentiate between auditors. Those that audit higher-achieving systems can be rewarded and those with less-satisfying results can be provided additional coaching and training. Internal auditors are often supervised by someone other than their regular supervisors, and the audit metric provides a way reward their auditing effort. The third benefit is that the organization will have a great explanation when a third-party auditor asks, “How is internal audit effectiveness determined?”

This method isn’t the only way to determine audit effectiveness. My hope is that through the previous discussion, we stimulate even more robust models for determining internal audit effectiveness.

About the author

Andy Hofmann has been involved with management systems for more than 30 years. He has audited more than 2,500 systems, giving him a unique opportunity view of organizations that are performing well and those that struggle. A regular contributor to American Society for Quality management systems conferences and publications, Hofmann’s intellectual property has received wide acceptance. Currently the president of ICS Certification Services, Hofmann continues to work with management systems professionals throughout North America. He has an MBA from the University of Toronto and is a Certified Engineering Technologist.