Standards and Statistics

Getting to Know Uncertainty

The Role of Uncertainty in ASTM Standard Test Methods

Q: Is estimation of uncertainty required if a laboratory is using a standard ASTM International test method with defined repeatability and reproducibility, and using standard equipment, chemicals and materials?

A. The short answer is that estimation of uncertainty may be required for certification according to International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) 17025, General Requirements for Competence of Testing and Calibration Laboratories, or demanded by a customer. ASTM International test methods are not exempt from such requirements.

To briefly introduce uncertainty, it is a part of a report for a test result. For example, the percent moisture of a sample might be 3.56 ± 0.23. The test result itself is the 3.56. The 0.23 coming after the ± sign is the uncertainty. It gauges how far from the true value for that sample the test result might be. This form of reporting is a longstanding practice in science. ASTM E2586, Practice for Calculating and Using Basic Statistics, includes definitions and examples of standard error and confidence intervals.

There are standard formulas for standard errors and confidence intervals applicable to the parameters of common distributions, such as the normal, binomial, Poisson and chi-square.

In metrology, measurements are often at the end of a long calibration chain. Errors of the calibration must also be accounted for in any realistic estimate of uncertainty. From the 1950s, the management of the National Bureau of Standards (now the National Institute of Standards and Technology) in the United States required uncertainty for calibrations that they provided. In 1993, the Guide to Uncertainty of Measurement (GUM), now maintained by the Joint Committee for Guides in Metrology, was published. When ISO/IEC 17025 was adopted in 1999, it included a requirement that the testing laboratory apply procedures for estimating the uncertainty of measurements they make.

Previous DataPoints articles have described repeatability, reproducibility and intermediate precision. The first two, in particular, are required to be determined through interlaboratory trials for every test method. The effort that goes into their determination is part of what makes ASTM test methods of such high quality, so that the values published in the test methods are valuable guides to the user.

Uncertainty is subtly different from precision. E2655, Guide for Reporting Uncertainty of Test Results and Use of the Term Measurement Uncertainty in ASTM Test Methods, was developed in an attempt to clarify them. When you determine a method’s precision, you characterize the performance of the test method by the variability of results. When you provide uncertainty with a result, you are addressing how close the true value is likely to be to the reported one for that particular result. For example, suppose we have a method with no bias and precision standard deviation . Now, we repeat the determination n times for the same sample using the method and average them (assuming they are all independent). The standard error is the uncertainty (standard error) that would be reported for that sample. In conventional statistical theory, this is explained using the fact that is the standard deviation of a population of average results (of which we have only one), which is again a measure of variability of results, or precision.

It is therefore an entirely reasonable question to ask: Why can’t the precision values given in ASTM test methods fulfill the role? Reproducibility, in particular, would seem to be suitable, since results for different laboratories on a test material are certainly independent.

The outcome of considerable discussion within ASTM committees has been that repeatability and reproducibility results given in ASTM test methods are not suited to be quoted in place of uncertainty. Recall, uncertainty is a measure of how close a particular test result, the product of one laboratory, is to the true value. There is no assurance that any laboratory using the test method will have the same precision and accuracy; some will be better and some worse. Specifications of apparatus and materials in test methods attempt to control uncertainty but cannot guarantee a value. Reproducibility is an average. Therefore, the ASTM policy is given in Form and Style for ASTM Standards, or Blue Book, as follows:

A22.2 It is neither appropriate for, nor the responsibility of, the test method to provide explicit values that a user would quote as their estimate of uncertainty. Uncertainty values must be based on data generated by a laboratory reporting results using the test method.

Form and Style then provides recommendations for test methods to include guidance on uncertainty. Some ASTM test methods now provide a section on how to calculate and report uncertainty for that particular method. For an ASTM test method that has such a section, you would follow it for the particular analysis.

There is also a common general approach, useful in some cases. In this approach, uncertainty is estimated using the site precision standard deviation (long-term within-laboratory standard deviation), such as would be available from the use of check samples. This is neither repeatability nor reproducibility, but a type of intermediate precision. ASTM International Committee D02 on Petroleum Products and Lubricants uses this as general guidance on measurement uncertainty for test methods under its jurisdiction in D6792, Practice for Quality System in Petroleum Products and Lubricants Testing Laboratories. ASTM E2554, Practice for Estimating and Monitoring the Uncertainty of Test Results of a Test Method in a Single Laboratory Using a Control Sample Program, describes an equivalent procedure.

Peter Fortini, Pfizer, Andover, Mass., is vice-chairman of ASTM Committee E11 on Quality and Statistics and a technical contact for E2655.

Dean V. Neubauer, Corning Inc., Corning, N.Y., is an ASTM fellow; he serves as chairman of Committee E11 on Quality and Statistics, chairman of E11.90.03 on Publications and coordinator of the DataPoints column.

This article appears in the issue of Standardization News.