Loading...
Loading...
By Katie Daisey
Sep 06, 2024
Interested in learning more about rounding fractions? In the September/October 2024 installment of Data Points, statistics expert Katie Daisey, Ph.D., explains some of the most common methodologies and how ASTM International approaches this topic.
A. There are two topics that need to be addressed when answering this question: rounding and significant digits. Let’s first consider the topic of rounding. In general, the standard practice for using significant digits in test data to determine conformance with specifications (ASTM E29) recommends the use of “bankers rounding,” or the rounding of a number with the last digit followed by 0 to the nearest even number. For example, 5/16 or 0.31250 would round to 0.0312.
There are alternative rounding rules that exist but are not included in E29. One example is rounding upwards (i.e., 0.313). Another example is notating the final digit in subscript (i.e., 0.3125). Note that these rounding examples retain three significant digits. This may not be the case for all examples, as the number of significant digits to retain may be dependent upon the specifics of the particular use case.
When it comes to significant digits, the standard practice for conversion from fractional to decimal notation depends on the context in which the resulting decimal form is being used. Three common use cases are covered in E29: use as a specification limit, use as a test result, and use as an observed value. Each case addresses the significant digits to retain and the rounding method to be used.
For use as a specification limit, E29 4.3 advises to notate absolute or rounded method for specification limits. For the absolute method, 5/16 would be an exact limit and no rounding would take place. So, the reported value would be 0.3125. Under the rounding method, the number of significant digits and rounding method would be agreed upon by the committee (in a standard) or by parties (in a private agreement).
For use as a test result, the number of significant figures should be informed by the precision of the method expressed as the repeatability standard deviation. According to E29 7.4, the “rounding interval” for a test result should be between 0.05 and 0.5 times the repeatability standard deviation. For example, if 5/16 is the test result, and the repeatability standard deviation is 0.03, then 5/16 should be rounded to 0.31 (since 0.01 is between 0.015 and 0.0015) and reported as 0.31 ± 0.03.
For use as an observed value, the number of significant figures used in 5/16 depends directly on the markings on the measuring instrument. Per E29 8.2, “When recording direct measurements, as in reading marks on a buret, ruler, or dial, all digits known exactly, plus one digit which may be uncertain due to estimation, should be recorded.” For example, say a ruler was used and the ruler is marked down to thirty seconds of an inch (i.e., 1/32”). Suppose the observed reading is 10/32”, which reduces to 5/16”. Further suppose that the uncertainty due to estimation is 1/128”, which is 1/4 of the 1/32” marking. This is less than the additional estimation digit allowed by E29 but is reasonable. In fractional notation, this would be noted as 5/16” ± 1/128” and converted to decimal as 0.3125” ± 0.0078125”. The estimation uncertainty is then rounded to 0.008”, indicating that three digits should be retained for a final result of 0.312” ± 0.008”.
Table 1 provides an overall summary of the guidelines in E29.
September / October 2024
