Bias Management and Continuous Quality Improvements
Committee D02s Proficiency Testing
by R.A. Kishore Nadkarni and W. James Bover
Most company, industry, national and international quality management standards for testing laboratories require implementation of systems or programs to manage precision and bias for test methods. Laboratory precision management is often achieved through the use of statistical quality control practices as described in two standards (1,2) which are under the jurisdiction of ASTM Committee D02 on Petroleum Products and Lubricants.
Although not widely associated with the term bias management, interlaboratory proficiency testing programs (also known as crosscheck, check scheme and round-robin programs) provide one of the most expedient means of assessing the bias of one laboratorys testing relative to other labs in the industry. In fact, the combination of the data derived from statistical quality control and proficiency testing programs can satisfy the relatively new requirements for determining measurement uncertainty. Participation in one or more proficiency testing programs should be considered essential for the right to participate in the marketplace. Likewise, participation in proficiency test programs offers many opportunities for continuous improvements.
There are numerous rationales and benefits for participation in proficiency testing programs such as the ones discussed in this article and offered by ASTM for Committee D02.
They provide laboratories with a statistical quality control tool to compare individual performance for specific (ASTM) test methods and products versus other labs worldwide.
They provide data for monitoring laboratory strengths and weaknesses across numerous test methods and products, thus enabling continuous improvement efforts.
Proficiency test programs can offer data for comparing results and statistics among several test methods measuring the same quality parameter (e.g., five test methods for sulfur).
Residual material from these programs along with the published statistical data can provide useful reference materials (or check standards) for use in control charts or for troubleshooting problems with test methods.
Participation in proficiency test programs can help satisfy parts of the laboratory accreditation requirements for a number of organizations. (3)
Satisfactory performance in a proficiency test program can be used to demonstrate a laboratorys capability to customers.
Proficiency test results and associated statistics can help standard developing organizations such as ASTM validate test method performance under real-world conditions and to enable the continuous improvement process.
D02 Interlaboratory Crosscheck Program
The ASTM board of directors approved the establishment of the Proficiency Testing Program in 1991 partly at the request of Committee D02 on Petroleum Products and Lubricants. (4) D02 launched its first set of PTPs in January 1993 for motor gasoline, aviation turbine (jet) fuel, #2 diesel fuel, and engine lubricating oils. Since then, D02 has expanded its Interlaboratory Crosscheck Program offerings to cover 18 products from crude oil to ultra-low sulfur diesel. Overall, D02s ILCP is considered the largest PTP of its kind in the world for petroleum and lubricant products testing.
The D02 ILCP programs are operated and managed by the ASTM PTP staff with technical direction and oversight provided by D02s Coordinating Subcommittee 92 on Interlaboratory Crosscheck Programs. The ILCP started with four product programs and over a decade later has 18 products covering over 90 standard test methods, with 769 laboratories participating from 68 countries. The ILCPs dramatic growth is portrayed in Figure 1.
Some of these programs are now quite large, covering several hundred laboratories (e.g., 325 for diesel fuel, 228 for engine oil lubricants, 247 for aviation jet fuel, and 204 for #6 fuel oil). Several other programs (motor gasoline, reformulated gasoline, lubricating grease, and ultra low sulfur in diesel) include over 100 laboratories each. We have tracked participation by taking into account the fact that in many cases a single lab participates in more than one product program, so for 2004 we have over 2,000 lab units of participation (see Figure 1).
Figure 2 shows the pattern of multiple program participation across all ILCP programs. Analysis of this data shows that about 55 percent of the labs participate in two or more product programs. It is amazing that about 20 percent of the labs participate in at least four product programs.
From Argentina to Zimbabwe
D02s proficiency test programs are truly an international effort with about 48 percent of the laboratories in 67 countries outside the United States. Figure 3 shows the distribution of the laboratories and the number of programs they participate in by region. For most of the ILCP programs, this level of international participation gives a true measure of global performance to the results. Many international companies have taken advantage of the global reach of the ILCP by having many of their labs participate.
Bulk quantities of product materials are donated by various organizations for each of the product programs in the ILCP. The product received is homogenized and packaged in appropriate individual containers by an ASTM subcontractor. Each laboratory receives one test sample for each test cycle. Twelve of the ILCP programs have three samples distributed per year; four programs are distributed twice a year and the reformulated gasoline and ultra-low sulfur diesel programs are distributed every month. Testing instructions and data report forms are also distributed with the samples. The laboratories analyze their samples for the parameters that they normally perform using standard ASTM test methods. A single determination is made for each test method for a given product sample, except for the low sulfur diesel sample, which requires replicate measurements. The laboratories return their results to ASTM, and in about four to six weeks following the closing date, ASTM sends a statistical summary report to each laboratory. Each report contains the following:
Summary table showing results for all laboratories for each parameter;
Charts showing the distribution of individual test results;
Histograms showing general distribution of results;
Box and whisker charts comparing the results from multiple test methods for the same parameter; and
Laboratory confidentiality, maintained by using codes.
Statistical Data Treatment
All data are analyzed using robust statistics. The means and standard deviations for each data set are calculated using a two-stage robust statistics procedure designed to limit the influence of unusually small or large results, but eliminating only the most egregious among them. Rejected results are clearly identified in the summary table so that the laboratories can take appropriate action. Based on the robust standard deviation, the robust reproducibility is calculated for each test method and these values are compared with the corresponding reproducibility published for the test method. Since only a single analysis is done (except in the case of the ultra-low sulfur diesel program), repeatability cannot be calculated.
Focused on Improvements
From its inception, D02s Coordinating Subcommittee 92 has maintained unwavering focus on satisfying its industry and ASTM stakeholders, and this is the main reason why the ILCP has enjoyed such outstanding success. Over the years, continuing efforts have been undertaken by the CS 92 and ASTM PTP office to improve the program and to provide participants with a value added service. One of the ways that CS 92 maintains its focus is by conducting regular educational and feedback sessions at each of the semi-annual meetings. The increased number of ILCP programs over the years results in part from the input received at these feedback sessions.
In general, participation in the ILCP has helped individual laboratories to improve their testing methodology and techniques, especially in cases when they realize that their data is not quite as good as that produced by their peers. On the other hand, and to the satisfaction of certain company laboratory groups, their own above-average performance has provided confirmation that their quality management policies have benefited them through their superior performance.
As mentioned above, many D02 subcommittees continue to take advantage of the proficiency test program results in helping to understand the variability in the test operations and the relationships between different test methods for the determination of the same parameter. Various subcommittee task groups are actively working on approaches to improve the precision and reduce the bias of their methods. Another tool at the disposal of the subcommittees is to request that additional information concerning specific test methods can be requested of the ILCP participant. For example, when a test method has shown less than satisfactory reproducibility, then the corresponding subcommittee task group has requested that CS 92 survey the participants regarding how they ran the test, the equipment used, etc. This feedback information is then used to support the improvement process.
At the Speed of Electrons
The distribution of program instructions to the participant and the reporting of data to ASTM are in the process of migrating from manual or hardcopy to electronic means. Currently, each lab participant is notified by e-mail about the sample shipment. The instructions and the report forms can be downloaded from the ASTM Web site. The laboratory data is entered on these forms and returned to ASTM. The Web-based data collection phase is being implemented in 2004 for all programs.
Another enhancement being launched in late 2004 is the issuance of custom reports to companies that have at least 15 laboratories participating in a specific product program. These reports will be prepared at the request of the company and with agreement from the laboratories. The reports will present statistics based only on the company lab data and will compare that against the overall statistics from all participating laboratories in that program.
Quality Control Materials
At the suggestion of many laboratories, residual program materials are available for sale to the participants on a first-come, first-serve basis for a modest fee. These materials are intended as check samples with a consensus mean and standard deviation for use in quality control charts and for troubleshooting problems. These materials are not meant for use in instrument calibrations since their assigned mean values may have large uncertainties.
Newsletter and Aids-to-Analyst
CS 92 publishes a newsletter twice a year distributed to all participating laboratories giving information on current activities. A number of times, helpful hints for the analyst about specific tests have been published in this newsletter. A compilation of about 25 of these aids is also available as a booklet free of charge from the PTP office. A number of these aids were later incorporated into the corresponding standard test methods as non-mandatory appendices. It is believed that these aids have proved helpful to practicing analysts in the laboratories to improve performance.
Marketplace Response and New Programs
CS 92 also maintains active liaison with most of the D02 subcommittees and these have also resulted in requests for specific program additions. For example, the reformulated gasoline and ultra low sulfur diesel programs were both specifically requested by their respective subcommittees in response to pending regulatory actions. These two programs are continuing to help industry laboratories develop and maintain correlation between their performance and that of the regulatory laboratories. Each of these programs have a unique feature in that the regulatory laboratories (like those of the U.S. Environmental Protection Agency and California Air Resources Board) that participate in the program have their results reported openly (i.e., not coded) to allow direct comparisons by other participants.
The history of the latest program addition is a good example of ILCP response to industry needs. The U.S. EPA issued a final rule effective June 2006 for highway diesel fuel containing low levels (< 15 mg/kg) of sulfur (see the article in this issue by R.A. Kishore Nadkarni). Since the existing diesel fuel program included only fuel with much higher sulfur levels, laboratory capability at the low sulfur levels was not known. This led to the launching in January 2004 of a new monthly ultra low sulfur in diesel program for product containing less than 15 mg/kg sulfur. In this unique program, duplicate measurements are required so that the laboratory repeatability can also be calculated and the testing is limited to four specific ASTM test methods. (5)
Into the 21st Century
D02s Coordinating Subcommittee 92 and the ASTM Proficiency Test Program department will continue to lead the way for interlaboratory testing to better serve industry in the 21st century. Over the next few years, the ILCP is expected to become completely Web-enabled for all program aspects from notification through report distribution. Existing programs will continue to be modified in response both to participant and D02 subcommittee demands and new programs will be offered in general response to the global marketplace and changing regulatory environment.
Contact David Bradley (phone: 610/832-9681) or Anne McKlindon (phone: 610/832-9688) at ASTM International Headquarters for information on participating in Committee D02s ILCP programs, or visit the program's Web site. See the bibliography for additional description of ILCP programs.
1 D 6792, Guide for Quality Systems In Petroleum Products and Lubricants Testing Laboratories
2 D 6299, Practice for Applying Statistical Quality Assurance Techniques to Evaluate Analytical Measurement System Performance
3 An example is the American Chemistry Councils Product Approval Code of Practice that requires participation in the ASTM D02 lube oil proficiency testing program for the laboratories submitting data for oil approvals to the American Petroleum Institute.
4 D02 has been in the process of developing a broad-based crosscheck program since 1988.
5 D 2622, Test Method for Sulfur in Petroleum Products by Wavelength Dispersive X-ray Fluorescence Spectrometry; D 3120, Test Method for Trace Quantities of Sulfur in Light Liquid Petroleum Hydrocarbons by Oxidative Microcoulometry; D 5453, Test Method for Determination of Total Sulfur in Light Hydrocarbons, Motor Fuels and Oils by Ultraviolet Fluorescence; and D 6920, Test Method for Total Sulfur in Naphthas, Distillates, Reformulated Gasolines, Diesels, Biodiesels, and Motor Fuels by Oxidative Combustion and Electrochemical Detection
David Bradley, ASTM Proficiency Test Programs ASTM Standardization News, p. 18, July 1992.
Jim Bover, Tackling Interlaboratory Variability for the Global Marketplace: Committee D02s Crosscheck Program ASTM Standardization News, p. 56, June 1994.
Jim Bover and Kishore Nadkarni, ASTM, the Petroleum Industry, and Quality Assurance ASTM Standardization News, p. 29, April 1998.
David Bradley, Proficiency Testing Programs ASTM Standardization News, p. 33, June 1999.
Kishore Nadkarni, Does Your Laboratory Measure Up: ASTM Helps Build Proficiency, LubesnGreases, p. 28, April 2001.
Kishore Nadkarni and Kelly Mason, New Program Puts Base Oils to the Test Lubricants World, p. 30, October 2001.
Gary Basilone, Oil Monitoring: Sharpening Our Tools LubesnGreases, p. 30, January 2002.
Bill Tanguay, Confirm Lab Quality with the Crosscheck Program Practicing Oil Analysis, p. 54, July-August 2002.
Anne McKlindon, Two New ASTM Proficiency Programs to be Launched in Coming Months ASTM Standardization News, p. 18., October 2003.
Copyright 2004, ASTM International