In the latest Data Points column, statistician Polona Carson, Ph.D., gives an answer.

By Polona Carson

Aug 31, 2023

Participation in interlaboratory comparison studies is of high importance for laboratories. Accredited laboratories, and many laboratories that provide testing within a regulatory framework, have a requirement to demonstrate their performance by comparison with the results of other laboratories. Objectives, protocols, and participants in interlaboratory comparison studies vary, but the overarching goal is the same: To compare participating laboratory results obtained on the same or similar test items against predetermined criteria following known protocols, distribution of samples, reporting of measurement results, evaluation of participants results, and reporting of results.

READ MORE: What are Certified Reference Materials?

Interlaboratory studies can be organized within a larger organization (e.g., a manufacturer with multiple laboratory locations); within a group of laboratories servicing the same project to monitor comparability of internal test facilities and assure that they produce comparable results; or by an external provider as a service to the laboratories that sign in for participation. Navigating different external providers can be challenging as these programs differ in objectives, and consequently, methodology used to evaluate results.

ASTM manages two interlaboratory comparison programs: the Interlaboratory Study Program (ILS) and the Proficiency Testing Program (PTP). Both programs follow well-defined protocols for preparation and distribution of samples, collection of data, and statistical processing of data and reporting. However, the difference is in the aim of the programs

Form and Style for ASTM Standards requires all published methods to include a precision and bias statement. This statement provides insight into the performance of ASTM methods through the assessment of method repeatability, reproducibility, and bias. The role of ASTM in ILS is to support ASTM technical committees with developing precision and bias statements for ASTM test methods that they develop and publish. The need for a study is therefore method-specific and is determined by each ASTM technical committee. The protocols for preparation and distribution of samples and for testing are prepared by the ASTM technical committee organizing an ILS program. Data-collection forms and final report format are provided by the ASTM Interlaboratory Study Program. Participation of laboratories in the ILS program is voluntary and free of charge. Data are analyzed following ASTM standards.

The normative standard for analysis of ILS study data is the standard practice for conducting an interlaboratory study to determine the precision of a test method (E691), which was developed by the committee on quality and statistics (E11). This standard provides general instructions for organizing an ILS as well as methodology for statistical analysis of data. Many committees have their own ILS standards that are based on E691 but adapted to the specifics of test methods within the industries each committee represents. Due to the voluntary nature of this program, the challenges are many, such as assuring a sufficient number of participating laboratories and preparing and distributing samples. From a statistical perspective, the performance of laboratories, as well as the method performance, may not be well known. So data may have a large scatter and evaluation of outliers can be challenging. Although a comparison of laboratories is not a goal of ILS, the report does provide some insight into the performance of each participating laboratory. ASTM currently has about 250 active ILS Programs.

FOR YOU: Data Points: Fundamental Measurement Concepts

Figure 1:Example of ILS evaluation chart

PTP is an accredited proficiency testing (PT) provider that maintains an ISO/IEC 17043-compliant quality system and demonstrates their technical competence through a formal, independent verification by an accreditation body, A2LA. They offer fee-for-service external laboratory quality-control programs that provide an independent assessment of laboratory results by means of interlaboratory comparison against the preestablished criteria that are defined in the design of the PT scheme. Participating laboratories analyze samples using the applicable ASTM method and submit results to the PTP for evaluation. The PTP analyzes participant-submitted data following well-defined procedures and issues several reports presenting coded participant results to assure their anonymity. All reports are aimed at evaluation of participant’s performance:

• Summary table with data for all participants
• Z-chart presenting historical performance of a lab
• QQ plots to present skewness of the data for an analyte
• When the same analyte can be determined using multiple ASTM methods, comparison of method by box and whisker plot
• Company report for companies with multiple laboratories providing comparison of all company labs against other participants

ISO/IEC 17025:2017 requires accredited laboratories demonstrate proficiency by comparison of results with other laboratories. A preferred way to demonstrate proficiency is by regular participation in a PT scheme that is offered by an ISO/IEC 17043-accredited proficiency testing provider. ASTM PTP is one of them and currently offers PT for test methods developed by 13 ASTM committees. When an accredited PT is not available, accredited laboratories can demonstrate proficiency by participation in other interlaboratory comparisons. When available, an ASTM ILS is a good choice. It is a well-established and well-run program that evaluates results following an internationally recognized ASTM standard. To demonstrate proficiency to an accreditation body using an ILS report, a laboratory will need to obtain the code used to anonymize their results.

When neither PT or ILS are available, what can be done? E11 has developed several standards that help design and guide internal evaluation of performance of test methods for an organization or a single laboratory:

• Standard guide for evaluating laboratory measurement practices and the statistical analysis of the resulting
• data (E1323)
• Standard practice for statistical analysis of one-sample and two-sample interlaboratory proficiency testing programs (E2489)
• Standard practice for evaluating equivalence of two testing processes (E2935)
• Standard guide for homogeneity of samples and reference materials used for inter- and intra-laboratory studies (E3264)
• Standard practice for use of control charts in statistical process control (E2587)
• Standard practice for estimating and monitoring the uncertainty of test results of a test method using control chart techniques (E2554)
• Standard guide for measurement systems analysis (MSA) (E2782)

Laboratories can demonstrate their proficiency by one of three means. Participation in an accredited PT program is the best demonstration of proficiency. If this is not available, then other interlaboratory comparison studies can be utilized. ASTM ILS is a very good choice. When neither of these are available, a laboratory can internally organize an ILS or design special studies to demonstrate performance for a test method.

Polona Carson, Ph.D., MBA, is quality management specialist at Neptune and Company, Inc. Carson is also an active member of the committee on quality and statistics (E11) and member at large of the executive subcommittee (E11.90).

John Carson, Ph.D., is senior statistician for Neptune and Co. and the Data Points column coordinator. Carson is a member of the committees on quality and statistics (E11), petroleum products, liquid fuels, and lubricants (D02), air quality (D22), environmental assessment, risk management, and corrective action (E50), and personal protective clothing and equipment (F23).