Standardization News Search
Feature
Making the Confidence Connection

Conformity Assessment System Design

by Gordon Gillerman

Just as standards connect the expectations of purchasers, users, government, producers and suppliers regarding safety, performance, interoperability and other characteristics, conformity assessment makes the confidence connection for products, services, processes and personnel. This confidence connection facilitates trade with arguably as much or more impact as standards. Standards and conformity assessment affect virtually all commerce. This paper will provide background and context needed to identify and analyze factors that impact the design of conformity assessment systems and explore system design parameters.

Conformity assessment is a general name for a group of activities used to provide confidence in compliance. Conformity assessment is formally defined in the International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) Guide 2:1996, Standardization and Related Activities – General Terminology, as “any activity concerned with determining directly or indirectly that relevant requirements are fulfilled.” The definition will be modified on the publication of what is now draft standard ISO/IEC 17000, Conformity Assessment – Terms and General Principles, as “demonstration that specified requirements relating to a product, process, system, person or body are fulfilled.”

The primary goals of conformity assessment systems are to provide the second party in a transaction (purchaser or user) or government regulators with needed confidence that a product, service, process or person (hereinafter referred to as product for brevity’s sake) meets appropriate requirements and/or standards. This confidence can be needed for many reasons.

Market Needs

Purchasers of computer software need confidence that a particular software package will operate on their particular hardware and operating system reliably. This is an example of a market need. Other market needs include retailers who need to manage their liability exposure on goods that they sell to the public through the public’s confidence that those goods meet accepted safety standards. Manufacturers may need confidence that component parts used in the manufacturing process meet standards for quality, fit and performance. With the rise of just-in-time manufacturing processes, this need for confidence has grown. Just-in-time manufacturing processes often have parts being delivered from distribution directly into production. There is no stock of parts at the factory to fall back on if the incoming parts cannot be used. Non-conforming incoming parts can delay or shut down production and significantly increase costs. These risks can lead manufacturers to the need for a conformity assessment system for incoming components and materials.

Regulatory Needs

Government regulators need confidence that safety, health, environmental and fair commerce requirements are met. Regulations are based on society’s tolerance for perceived risk. Society’s intolerance for some risks inspires regulation so that the government can reduce the risk to tolerable levels. Regulatory conformity assessment needs come in two types: the first relating to safety, health and environment; the second relating to fair commerce. Government requirements for approval of medical devices and drugs are examples of regulatory conformity assessment needs in the safety and health domain. Government requirements for product rating and quantity pertain to fair commerce. State and local government laws having accuracy requirements and conformity assessment systems for measuring instruments used in commerce (such as gasoline dispensers and food scales) are examples of regulatory conformity assessment based on the need for fair commerce.

Factors in Conformity Assessment System Design

Conformity assessment systems come in many types and scales. The needs of the stakeholders, product type, and characteristics that need to be assessed and desired level of confidence in compliance drive the design of conformity assessment systems. There are several basic activities that comprise conformity assessment systems. Some can be used alone. Others are typically activities that are combined in a complete system.

The basic activities of conformity assessment are testing, inspection, supplier’s declaration, certification and registration. These terms all have formal definitions in ISO/IEC Guide 2. The selection of appropriate conformity assessment activities in a system involves consideration of several factors including:

1. The perceived risks associated with non-conformity (see Figure 1);
2. The practical means of evaluating the characteristics of interest.
3. The scale and type of production operation or delivery in the case of a service;
4. The effectiveness of marketplace mechanisms to remove non-conforming products from the market;
5. The effectiveness of penalties for placing non-conforming regulated products in the market; and
6. The effectiveness of systems to recall non-conforming regulated products from the market.

Items 1 through 3 above apply to conformity assessment for both market and regulatory needs. Item 4 is relevant to market-related conformity assessment and items 5 and 6 to regulatory-related conformity assessment. A comprehensive understanding of these factors for a particular product characteristic is needed to develop an effective and efficient conformity assessment system.

Another key to understanding conformity assessment system design is characterizing the involved parties by their relationship to commerce. For this we use simple definitions of the involved parties.

• First party – The manufacturer and/or supplier.
• Second party – The purchaser and/or user.
• Third party – An independent party that has no interest in the transaction between the first and second party.

Government has a unique role in regulation that does not fit neatly into these definitions, but is the second party in procurement.

System Design

The level of risk associated with non-compliance drives decisions on the rigor and independence needed in a conformity assessment system. The greater the perceived risk, the more oversight and independence the conformity assessment system should have. Over-design will add cost to the system and potentially the products assessed. Under-design will result in too little confidence in the product’s compliance and not meet the second party’s or regulator’s needs and impede the acceptance of the product in the market.

The product type, production method and characteristics to be assessed influence the activities that need to be utilized in a system. Some characteristics are easily checked via inspection, such as color and quantity. Other characteristics must be evaluated via testing or process control assessments and audits. For instance, the maximum temperature of an internal component of an electrical appliance under certain operating conditions needs to be evaluated based on results from laboratory tests and could not be readily inspected. Still other characteristics are not evaluated well by inspection or testing and are best evaluated using the assessments and audits of a process control system. The integrity of software systems and environmental compatibility of operations are examples of characteristics that are generally well-suited to a process assessment and audit system.

Supplier’s Declaration of Conformity

Low- to medium-risk areas in which market mechanisms or regulatory action can mitigate the negative consequences associated with non-compliances before those consequences are intolerable to society often rely on conformity assessment by the first party. The supplier undertakes all needed conformity assessment activities. The conformity assessment activities result in a supplier’s declaration of conformity. The declaration may be a formal attestation of conformity or another type such as a brochure or on-product/ package markings. There are ISO/IEC standards that define the needed elements of a formal supplier’s declaration and the technical file that contains the data on which the declaration is based.

An example of supplier’s declaration is the light sensitivity of photographic film expressed in ratings based on test methods in ISO 58000: 1987, Color Negative Films for Still Photography – Determination of ISO Speed. The manufacturer designs and tests film types to establish a light sensitivity rating and declares conformity with the rating on the packaging. Market mechanisms are effective for this situation. Users who experience poor performance due to incorrectly rated film will choose other suppliers, and the supplier of the non-compliant film will have to bring its film into compliance to succeed in the market. Society can tolerate non-compliance in this area since the ramifications do not negatively impact safety, health or environment.

The effectiveness of penalties for placing non-conforming products on the market and the ability to remove or fix non-conforming products that are already in the market are both significant factors in designing regulatory conformity assessment systems. If penalties are an effective deterrent to placing non-conforming products on the market then consideration can be given to reducing the rigor of the conformity assessment appropriately. The effectiveness of penalties is based on several factors, such as how well the penalties are known, the level of enforcement and the impact of the penalties. The ability to recall products from the market can also be considered to potentially reduce the rigor in conformity assessment system design. Some products lend themselves well to recall systems. For example, automobile manufacturers generally can contact purchasers directly to facilitate recalls. They also have an extensive network of service locations to conduct recall-related repairs. So while the perceived risk level may be higher than other products, the effectiveness of the recall system enables first-party conformity assessment.

Testing

Testing may be the most common form of conformity assessment. Laboratories conduct tests and develop data. This test data is used solely or in part to determine whether tested samples demonstrate conformity with requirements. Type testing refers to tests that are conducted on samples representing production. Other testing may be used to check actual production samples or as part of an inspection system. Laboratories can be first, second or third parties.

In some suppliers’ declaration of conformity systems used for regulation, the test data is required to be developed at an accredited third-party laboratory. This presents a hybrid situation where one conformity assessment activity (testing) is conducted by a third party, but the first party uses the test data to determine conformity and make a supplier’s declaration of conformity. Accreditation will be discussed later in this article, but basically accreditation is a type of conformity assessment used to determine compliance of conformity assessment programs and organizations such as laboratories with appropriate operational requirements.

Inspection

Inspection can be performed by first, second or third parties. Generally, inspection systems only demonstrate conformity of the actual products inspected or a lot from which the inspected samples are drawn. Inspection is well-suited to product characteristics that can be readily measured and where production occurs in batches. The supplier can arrange for the inspection of a production batch when needed. However, for products in continuous production, the cost of having an inspector present during production may be restrictive. Inspection is also used to ensure that component parts and materials have been installed correctly. This type of conformity assessment is often applied to structures that must meet regulatory requirements. The inspection may need to take place in phases based on the ability to inspect portions of the structure at certain phases of the construction. Second-party inspections are carried out by manufacturers on the suppliers of critical components and subassemblies that will go into their finished products. Many inspection programs use product markings such as the U.S. Department of Agriculture meat grades or certificates to attest to the conformity of inspected products.

Inspection is also used as part of a more comprehensive conformity assessment system. For example, inspection is often used in the surveillance activities of certification systems.

Third-Party Conformity Assessment

Third-party conformity assessment is often utilized in situations where the need for confidence is higher than a first- or second-party conformity assessment system can provide and where other factors do not reduce the needed rigor and independence. This provides a higher level of confidence in compliance to purchasers and users since the third party’s decision-making process is free from any influence of the business between the first and second parties. Third parties can be laboratories on whose test data determinations of compliance are based: inspection bodies, certification bodies and/or registration bodies.

Certification has two essential characteristics. It is conducted by a third party and includes some form of surveillance activity. Surveillance is a group of activities conducted by a certifier to ensure ongoing compliance once initial compliance has been determined. Post-market surveillance involves the evaluation of certified products from the market to determine if the requirements continue to be met. Pre-market surveillance is the checking of products before they reach the market and may include audits of the supplier’s process control systems and/or inspection of production. In other certification systems, surveillance is accomplished by requiring all or some significant part of the activities used initially to determine compliance to be re-conducted on a periodic basis. This recertification process can take the form of retesting or re-assessing the characteristics of interest at prescribed intervals. Many certification programs use an on-product mark to attest to the conformity of certified products. Other programs use certificates. Most certification programs publish a list of certified products for use by purchasers, users and regulators.

Certification is very useful in situations that involve mass-produced products and characteristics that cannot be readily inspected. Many certification programs focus on product characteristics related to health, safety and protection of the environment.

Certification systems are also used to enhance the purchaser’s ability to compare product attributes, such as the usable volume of a refrigerator or grades of motor oil. In these cases the certification provides confidence that the rated volume or viscosity is based on testing and measurement in accordance with accepted standards. Still other programs certify that products actually come from a certain place, such as potatoes grown in Idaho. These types of certification programs are often developed by suppliers’ trade or professional organizations in response to a market need for reliable information on product characteristics.

In the United States the certification of processes is generally referred to as registration. Registration systems also are conducted by third parties and involve some form of surveillance. Registration systems are commonly used to assess compliance with quality and environmental management systems standards such as the ISO 9000 and 14000 series. The process involves assessing the compliance of written procedures with the management system requirements and then auditing the implementation of the system on an ongoing basis. Registrars issue certificates and publish lists.

The Role of Accreditation — Who Watches the Watchers?

Accreditation systems provide confidence that conformity assessment programs and organizations meet requirements. There are accreditation programs for laboratories, inspection bodies, certifiers and registrars. Accreditation is generally conducted by third parties. ISO/IEC standards and guides along with similar national standards define the operational requirements for conformity assessment. Accreditors use these standards in conjunction with specific technical and programmatic requirements in assessing the compliance of conformity assessment systems. The activities are very similar to those used in process management system registration, but with an additional specific evaluation of technical competence of personnel, facilities, equipment and calibration requirements. Accreditation provides confidence that conformity assessment systems operate with needed integrity and competence.

Accreditation can facilitate a competitive market for conformity assessment by establishing and enforcing the “bar” to which conformity assessment programs must rise for their results to be accepted. This competition can serve to make conformity assessment more efficient, effective and responsive to the needs of its stakeholders. Accreditation can also facilitate the acceptance of products in foreign markets based on conformity assessment conducted domestically. This can reduce the cost of demonstrating compliance for the supplier by limiting the number of conformity assessment providers that they have to utilize to enter multiple markets.

In third-party systems, accreditors watch the performance of the accredited conformity assessment bodies and the conformity assessment bodies watch the suppliers’ performance. The accreditors are the watchers of the watchers. This establishes a type of conformity assessment hierarchy (see Figure 2). Additionally, there are organizations that assess the compliance of the accreditation bodies to the standards that cover their operations. These organizations recognize accreditation programs via assessment and audit similar to the accreditation process itself. Generally, recognition bodies provide confidence that accreditation programs operate with integrity and competence. This may seem like overkill, but the goal of these recognition organizations is to facilitate the acceptance of conformity acceptance results by a larger group of purchasers, users and regulators than accreditors can and potentially create a competitive environment for accreditation with its inherent benefits. Recognition of accreditors is a fairly new concept and on a sector-by-sector basis the market will determine if it provides enough value to thrive.

Another approach to develop confidence among conformity assessment organizations (including accreditors) is peer assessment. Peer assessment is an organized system where staff from one or more conformity assessment bodies assess and audit the competence of other bodies that perform the same conformity assessment function. This approach is used by accreditors, certifiers and laboratories. Most peer assessment systems are aimed at increasing the acceptance of conformity assessment results, rather than facilitating confidence with regulators. Regulators tend to rely on government or qualified third-party accreditation programs.

The Duality of Customers

One of the truly unique aspects of conformity assessment is the fact that there are multiple types of customers with very different needs. Generally, customers are thought of as the entity paying for a good or service. Conformity assessment is definitely a service, but in many third-party conformity assessment business models the paying customer is the supplier. However, the entities in need of the confidence derived from conformity assessment are purchaser, user and regulator. All of these stakeholders are the customers of conformity assessment. Suppliers need efficient and effective conformity assessment that meets the confidence needs of their customers and/or regulators. Purchasers, users and regulators need confidence that the conformity assessment has been carried out with appropriate integrity and competence. These factors along with the increasingly competitive nature of the conformity assessment industry creates a tension that has the potential to improve conformity assessment programs.

Conformity Assessment in the Global Marketplace

The increasing global nature of the market may drive up the value of conformity assessment. As manufacturers, distributors, suppliers, retailers, consumers and regulators interact across the globe, the need grows for conformity assessment that can be conducted at a location convenient for the supplier and accepted by purchasers, users and regulators in far-away markets. The confidence that effective conformity assessment delivers facilitates these international transactions and makes conformity assessment an important tool for global trade.

Conclusion

Many factors need to be identified and analyzed in preparation for system design. Consideration of the characteristics, product type and level of confidence needed are fundamental to effective conformity assessment system design. Well-designed conformity assessment systems provide the needed confidence in a cost-effective manner. Assessing the level of confidence needed is one of the most difficult tasks since the concept is qualitative rather then quantitative. This assessment often guides the selection of the use of first-, second- or third-party conformity assessment. Market needs can often be served by many different types of conformity assessment systems while regulatory needs may require a more regimented approach. Market mechanisms, penalties and recall effectiveness should all be considered. Additionally, the use of accreditation and peer assessment may provide the system with competition and multiple market acceptance that further enhance the value, efficiency and effectiveness of conformity assessment. //

Copyright 2004, ASTM International

Gordon Gillerman is the conformity assessment advisor – homeland security at the National Institute of Standards and Technology in Gaithersburg, Md., designing conformity assessment programs for the U.S. Department of Homeland Security on assignment from Underwriters Laboratories. Gillerman had previously been UL’s manager of governmental services in Washington, D.C., and a staff engineer in the medical device and information technology sectors at UL’s Northbrook, Ill., headquarters.

This article placed third in the World Standards Day Paper Contest on “Standards Connecting the World.” It is reproduced by permission of the Standards Engineering Society. The article was first published in Standards Engineering, the Journal of the Standards Engineering Society, Vol. 56, No. 6, November/December 2004. For subscription or membership information contact SES, 13340 SW 96th Avenue, Miami, Florida 33176.