by John E. Gray

Audit checklists are the most common tool used by auditors, and for good reason: they provide a multitude of benefits. Audit checklists provide order and organization for planning and executing audits. They also provide rigor to the data-collection process by identifying and organizing information as well as a place to record results. Audit checklists help guide the auditor during the audit and provide a timeline for completing certain audit phases. In addition, well-developed audit checklists can provide auditors clarity, connectivity, and consistency. Clarity helps an auditor know exactly what, when, and where to audit a specific requirement. Connectivity is obtained when the auditor understands how a requirement relates to an organizational objective, and a standard checklist reduces variation by fostering consistency amongst auditors. It’s no wonder that audit checklists are so prominent in auditing. However, for many supplier audit programs, audit checklists are the most misused tool.

Auditors understand that the purpose of an audit drives its scope and type. For instance, a process audit assesses the effectiveness of established procedures; a product audit re-inspects a product after it has gone through final inspection; and a compliance audit evaluates compliance to statutory requirements, standards, regulations, and/or formal contract requirements/specifications. The compliance audit is one of the most prevalent audit types because it provides an immediate assessment using comprehensive audit checklists.

There are many types of compliance audit checklists, such as stove-piped, open-ended, attribute, and variable. The most common are attribute audit checklists, which allow auditors to efficiently document results in a format that is easy to read and use. Attribute-type audit checklists have several variations, such as go/no-go, compliant/noncompliant, or yes/no. There is no gray area, which simplifies decision making. However, when attribute audit checklists are the only audit tool in your tool box and it becomes the focus of your audit function, you can expect some problems. Many of these problems can result in significantly higher audit program costs, increased risk to operations, adversarial supplier-customer relationships, dissatisfied customers, and poor quality of life for auditors. When problems become this systemic they usually have root causes in their design (i.e., checklist architecture).

Many compliance checklists are designed using contractual requirements, which are derived from customer requirements. Customer requirements are analyzed by performing job analysis, which involves getting a bunch of subject matter experts into a room and identifying all the tasks needed to perform a specific function.

The end result is a work breakdown structure that identifies the tasks that become the “shall” statements in the contract statement of work. These “shall” statements are then turned into explicit checklist questions or statements. This is a necessary step in determining costs associated with each service task and it’s where symptoms of questionable checklist design become noticeable, starting with ambiguous contract requirements.

Ambiguous contract requirements lead to bad checklist questions/statements because auditors can’t collect adequate information to assess compliance. Table 1 provides some ambiguous questions from audit checklists.

Table 1: Audit checklists–Ambiguous questions

In question no. 11 what is meant by “timely”? It will more than likely mean one thing to the owner of the requirement but something totally different to the auditor. The same goes with question no. 12. What is “accurate”? What time of day is it to be submitted? Without criteria for what success or failure should look like, the auditor(s) can’t get a go/no-go response from this requirement. Question no. 13 provides a specific task in grounds maintenance and a location of where it’s to be done; however, it also states “but not limited to,” which is nonspecific. Until that information is clear, the supplier can’t fully perform and auditors can’t fully audit. Unfortunately, this issue was prevalent throughout all audit programs I recently researched, but it wasn’t the biggest issue.

The biggest issue was with the work breakdown structure itself. In large-scale government contracts, managers and SMEs are responsible to ensure all requirements and associated checks are accounted for. When the supplier audit checklists mimic the work breakdown structure and auditors are selected based on their technical skills and experience to deploy the audit checklists, functional silos may have been created. The silo perspective leads to several unwanted behaviors: a go/no-go mentality; a continuously expanding checklist in terms of scope, style, and size; lengthy audits; unnecessary recruitment of more auditors; unrealistic policies such as 100-percent inspection on all requirements (no sampling); and a propensity to serve as quality control vs. quality assurance. When SMEs develop audit checklists within their functional silos, be prepared to one day become involved with a very difficult situation. The following case study illustrates this point.

The audit program manager received a checklist from the logistics manager that identified more than 150 questions. When the audit program manager asked which ones were critical to quality assurance (CTQA), the response was that 145 of them were CTQA and, therefore, had to be inspected each month at all 14 sites. This obviously caused the audit program manager concern because he expected the supplier to be performing those checks, not the customer. So he asked the other nine functional area managers the same question. The result was astonishing. Ten functional areas generated 850 questions with 800 CTQA and the expectation that six auditors perform all checks at all 14 sites every month. An effort to establish some guidelines for what CTQA meant the audit program manager asked each functional area what percent of the total contract value did their area represent? The answer totaled 650% of the contract value! If that’s not being territorial and protective, then I don’t know what is. In fact, the second-party supplier audit function at that point began to take on the role of quality control.

When quality control is the focus, it will drive a completely different checklist than if quality assurance had been the focus. The difference is usually in the sheer number of checklist questions/statements and the audit policy used to deploy quality control-centric audit checklists. This is depicted in Table 2.

Table 2: Audit checklists–Impossible checklists

This behavior is a cost driver. Performing that many checks drive up costs for travel, lodging, per diem, and costs associated with escort time, interruptions to operations, time to collect data, analyze results, and report results. One agency was spending more than $1.5 million just on travel costs. The quality of life for auditors was not good either.

Many auditors were spending in excess of 215 days away from home. Most of them had families with young children. Overtime takes its toll. What makes this issue even more dire is that a sampling of 100 checklist questions asked over a five-year period indicated that 97 of them provided evidence that suppliers met or exceeded performance requirements but these questions were asked every visit (six times per year). Even with a quality control focus these questions are candidates for reduced inspection. This and the previous practices are a huge drain on organizational resources and they don’t facilitate the evolution of a supplier’s quality management system.

Quality control is continuous and performed on-site as work is being performed. On the other hand, supplier audits are periodic and usually attribute-focused. In most supplier audit programs studied, the supplier took the view that if the customer wants to inspect to that degree, it becomes the organization’s internal quality control. Eventually, the suppliers reduced their internal quality control inspections. That approach in government contracts (from both parties) is a fraud, waste, and abuse because the Federal Acquisition Regulation clearly states that the supplier is responsible for the quality of products/services—not the government. The government, unless otherwise specified, is paying for the quality control function so why is the supplier performing it? When suppliers’ audits adopt a quality control mentality, the result is bad audit program policies that drive bad audit practices. A good example happened with a federal operations and maintenance contract for tethered aerostat radars.

A 420,000-cubic foot helium-filled aerostat tethered about 15,000 feet above the ground near Rio Grande, Texas, actually broke away from the tether and drifted 400 miles north of San Antonio, where it finally came down. The accident investigation board cited the quality control and assurance functions for lack of focus to critical success factors and associated processes. The report actually stated that government auditors were checklist-bound and paperwork-focused. An analysis of the quality assurance program showed auditors focusing on trivial requirements, such as general purpose vehicle operator care, files maintenance, training records, daily activity reports, and housekeeping.

Auditors told the accident investigation team that it was their audit program policy that when key performance indicators such as equipment mission capable rates were being met, to focus audits on other requirements. Had the auditors used core process or risk-based auditing techniques and tools, they would have found that the aerostat tether-ground system was not being maintained to standards and that the winch truck which held the tether and aerostat to the ground had rusted so badly that the braking system failed and the rapid deflation device used for a breakaway condition also failed due to corrosion. This is an example of a complacent audit function so checklist-bound that it couldn’t see the forest through the trees. Fortunately no one was injured and property damage was minimal.

It goes without saying that audit checklists are not the root cause here but a manifestation of several issues: a silo mentality, ease of the compliance audit (go/no-go technique), mistrust of suppliers, propensity to be quality control and not quality assurance, lack of knowledge on basic audit methodologies and practices (traditional legacy approach), lack of systems thinking, and lack of risk management strategies. Unfortunately, most government audit programs are derived from functionally driven inspection practices. The good news is that most federal supplier audit program managers I’ve talked to want to break out of the legacy inspection-based compliance mentality to more flexible process and risk-based audit programs. To do that they need to look closely at how they perform work breakdown and associated design and deployment of audit checklists. They need to select strategies and tactics that align with organizational structure and culture.

If an organization has a strong hierarchy and values stability and control, it will probably have policies and resources to pursue a supplier-customer relationship that is primarily compliance (quality control) based. If the organization is more team based, market driven, and values flexibility and integration, it would do well to embrace a systems-based audit program. For example, process and risk-based auditing provide more insight into the effectiveness and efficiency of management systems, eliminating wastes and exploiting constraints across the value stream. It would also facilitate the maturity of the supplier’s quality management system and the customer’s quality assurance program. The latter would foster a mutual beneficial supplier-customer relationship, which would influence the design of the audit program and associated tools. Imagine audit tools like the audit checklists that serve the audit function and not the other way around.

About the author

John E. Gray is an organizational strategic planner, audit program manager, certified quality auditor, certified Plexus/coach trainer, and lean facilitator with the U.S. Air Force Acquisition Management and Integration Center. He has more than 15 years’ experience in auditing service suppliers for the Air Combat Command.