September/October 2008 Reporting Test ResultsDetermining Significant Digits and Rounding Properly
Q: How many significant digits should I use when reporting my test results and comparing those results with specification limits? A. It is essential to understand the use of significant digits when recording, calculating and reporting. A number is a string of digits 0 through 9, either with or without a decimal point, which represents some quantity of interest to us in some system of units. Significant digits are defined in ASTM International standard E29 as any of the figures 0 through 9 that are used with its place value to denote a numerical quantity to some desired approximation, excepting all leading zeros and some trailing zeros in numbers represented without a decimal point. (Click here for full titles of standards cited.) All the digits 1 through 9 are significant in a number. Zeros are a special case. Zeros in between other significant digits are significant. Trailing zeros are significant if the number has a decimal point. Trailing zeros for numbers without a decimal point are the ambiguous case. For example, the observation 0.0304 has three significant digits: 3, 4, and the 0 between them. The value 3040 is an ambiguous case. The final zero is a significant digit if the data are rounded to the nearest unit but is not if the value is rounded to the nearest 10 units. Rounding A number of standards deal with rounding data and comparison with specification limits (see sidebar). We will first consider the general question of working with significant digits in data. Section 7 of ASTM E29 provides guidelines for this purpose. When recording observed values from a scale, such as a burette, ruler or gauge, record all digits that can be read exactly, plus one more that may be interpolated between marks. Many test observations are instrument printouts and these are used as printed. When carrying out calculations, do not perform intermediate rounding. Computations performed with spreadsheets or computer programs hold real numbers in double precision, about 17 digits, so concern about rounding during computation is largely unnecessary. Rounding is used when reporting the test result after all calculations have been completed. When rounding, we replace a result by the nearest value that is a multiple of the rounding interval. The rounding interval to use is a compromise. Too few significant digits cause information to be lost and too many are considered bad style in numerical reporting — showing a lack of understanding of precision. A recommendation for the number of significant digits in results given in ASTM E29 depends on knowing the test method precision, in particular the repeatability standard deviation σ. The recommended rule is that the rounding interval should be between 0.05 σ and 0.5 σ. For example, a test result is 1.45729 and the repeatability standard deviation is 0.0052. You then round the result to 1.457, or to the nearest 0.001, because this rounding interval is between 0.05 σ = 0.00026 and 0.5 σ = 0.0026. A fine point for rounding rules has to do with rounding results when the value is exactly halfway between two rounded values. Most standards for science and technology, led by ASTM E29, use the “fiveeven” rule. Under this rule, for example, the value 98.50, rounded to the nearest unit, becomes 98, to make the last significant digit of the result even. Similarly, 99.50, rounded to the nearest unit, becomes 100. Other standards (for example, ASTM D6026) use the “fiveup” rule, under which numbers exactly halfway between two rounded values round up to the larger value. Under this rule, 98.50 rounds to 99; 99.50 rounds to 100. Formats for computer output often use this rule also. The two practices for comparing a measurement or test result with a specification are the absolute method and the rounding method. Generally, specifications in mechanical industries, especially size dimensions, are interpreted using the absolute method. For example, if a diameter is specified as 30 ± 0.1 mm (absolute method), and measured to the nearest 0.001 mm, then 29.900 is (barely) passing, while 29.899 does not meet the specification. Material property specifications in the process industries are usually interpreted using the rounding method. If the purity of a reagent is specified as 99102 percent (rounding method), then analytical results between 98.5 and 102.5 percent meet the requirement. Several test results may be averaged to obtain the value to be compared with the specification limit. ASTM E29, Manual 7, and SI 10 all give guidance on the rounding of averages. ASTM D3244 also deals with the pooling of information from supplier and customer as well as resolving disputes between them. Tom Murphy, T.D. Murphy Statistical Consulting LLC, is chair of Subcommittee E11.30 on Statistical Quality Control, part of ASTM Committee E11 on Quality and Statistics. Peter Fortini, Wyeth Biotech, is the technical contact for ASTM practice E29 and also a member of Committee E11. Dean Neubauer is the column coordinator and E11.90.03 publications chair.
Statistics play an important role in the ASTM International standards you write, such as the development of precision and bias statements for test methods, running interlaboratory studies, knowing how to round numbers properly and determining sample size. A panel of experts is ready to answer your questions about how to use statistical principles in ASTM standards. Please send your questions to SN Editor in Chief Maryann Gorman, ASTM International, 100 Barr Harbor Drive, P.O. Box C700, W. Conshohocken, PA 194282959.
