Laboratory-Specific Bias and Its Effect on Precision Statements
Q. How does laboratory bias in an interlaboratory study affect precision statements?
A. The word “bias” has a number of meanings. In ASTM, two kinds of bias come to mind — test method bias and bias(es) associated with one or more laboratories. The focus here will be on laboratory-specific bias.
Laboratory-specific bias differs from an outlying result because it comes from root causes that affect all of a particular laboratory’s round robin results, not just one. It becomes particularly visible in interlaboratory studies (round robins) for precision statement development. Significant bias exists when data from a laboratory are always high or always low for all samples in the ILS (see Figure 1). In some cases, multiple laboratories can exhibit this bias behavior. In very rare instances, they are the closest to the true value. Ideally, the values for each sample obtained by a particular lab should vary randomly above and below the mean of that sample (see Figure 2).
The biases here range up to 6 percent of the mean. There are other cases where the biased laboratory differs from the mean by only a few tenths of a percent.
Is This a Problem?
A study of 85 precision calculations in support of standards developed by ASTM International Committee D02 on Petroleum Products and Lubricants showed that only 20 percent indicated no statistically observable lab bias. Another 20 percent could be characterized as having significantly high bias.
Having one or more biased laboratories can inflate the test method reproducibility (as estimated by D6300, Practice for Determination of Precision and Bias Data for Use in Test Methods for Petroleum Products and Lubricants). In addition, this can also have an adverse effect on the bias of the test method to be recorded in the standard’s precision and bias statement. In the above example, R was increased by 70 percent due to the lab bias effect. Also, significant lab biases reduce the reproducibility degrees of freedom (a measure of the reliability of the reproducibility estimate). In the above example, degrees of freedom decreased from 119 to 11. This effect is highly undesirable because the minimum acceptable limit is 30.
Who Impacts Bias?
The model for obtaining precision does not anticipate human activity. The party responsible for minimizing the bias of a given laboratory is in fact a whole network of people: the individual sampling the bulk for distribution to cooperators; the bottle washer and the manufacturer of disposables; the statistician; the building manager responsible for climate control; chemists at national standards laboratories such as the National Institute of Standards and Technology; the person responsible for administering the ILS; suppliers of solvents and reagents; the analyst who prepares and executes the test; writers of instrument manuals; the ASTM task group, particularly the writers of the method; the subcommittee voters and others. Cause and effect (arrow) diagrams can be used to assemble sources of bias. Values can be assigned and worst-case scenarios developed.
Issues of calibration and recovery factor have been dissected in the literature. Any deficiency at all that affects only one laboratory can contribute to its bias.
How Can Bias Be Minimized?
Sources of bias include, but are not limited to, every step in a test plus sampling plus calculations plus the environment. Subcommittees want to move test methods through the standards development process expeditiously and with due process and can view repeating a round robin with a jaundiced eye. Rather than receive a report about laboratory bias after the conduct of an ILS, subcommittees should take precautionary steps within their control to ensure that there are adequate standardization protocols embedded in the test method that pre-empt this possibility. A few tools to do this are listed here. Since the true value is unknown, the concept presented here is to remove as many sources of bias in the method as possible, such as:
The test method should be written with as few steps as possible, but no fewer. Laboratories should follow the test method as written. Explicitly written procedures that do not require subjective interpretation can reduce time and errors in sample introduction or testing.
How Do We Measure Laboratory Bias?
Occurrences of laboratory bias are seen first by the ILS data analyst. It is assumed that there are no unprocessed checks in the cooperator’s laboratory. The data analyst (in most cases a statistician) can make an Excel plot and inspect the data. If using D6300 to calculate precision, the analyst can note the lab’s F ratio in the analysis of variance (ANOVA) calculation with respect to the critical value. A significant F ratio should arouse curiosity and lead to a rank sum test to identify the laboratory. The results of the analyst’s observations will be reported to the subcommittee sponsor.
So What Is the Lesson?
Assuming that a significant bias is observed for a specific lab, we now engage philosophical beliefs (or biases?). On the one side, one can take the approach that “this is the way it is, the real world.” The precision study is accepted and the subcommittee moves on to other business. On the other side, the offending laboratory can be dropped and the statistics rerun because “the laboratory has had a chance to practice and to run through the test with a known sample. The laboratory asserts that it was in control. We expect proper data to be random normally distributed. The chances of a laboratory deviating this much by chance are infinitesimal.” Or, the laboratory is asked to review their procedures and rerun round robin samples if they are available.
In the extreme case, where numerous laboratories are observed to exhibit bias behavior, this is a strong indicator that the test method lacks sufficient between-laboratory standardization protocol. The test method development task group should be re-engaged and redraft the test method with additional protocols to mitigate the observed bias effects. However, this would introduce delay in test method passage.
Ultimately, the subcommittee will have to vote on the test method draft. The fundamental underlying guiding principle should be: Are the precision and bias adequate for global trade?
David M.G. Lawrey is the retired chief of analytical services for Sun Co., Marcus Hook, Pa., past secretary of Subcommittee D02.03 on Elemental Analysis and a member of Committee D02 on Petroleum Products and Lubricants. An ASTM International fellow and Award of Merit recipient, Lawrey developed the computer program D2PP for calculating the precision of Committee D02 test methods.
Dean Neubauer is the DataPoints column coordinator and E11.90.03 publications chair.