|
Fact vs. Fiction: The Truth About Precision and Bias
by Pat Picariello
Every ASTM test method requires a precision and bias section.
What is it? How do I create one? Read on.
Have you heard any of these terms before—precision, bias, repeatability,
reproducibility, interlaboratory, intralaboratory? Have they sent
shivers running up and down your spine? When the call goes out
at a committee meeting for volunteers to participate in a ruggedness
test or round robin, do you suddenly find your hands glued to
the table?
If you answered yes to any of these questions, you’re reading
the right article (and you’re also the typical ASTM member).
Anyone who has helped create, revise, distribute, or catalog an
ASTM test method has encountered the world of precision and bias.
Every committee has dealt with members whose detailed knowledge
of Form and Style for ASTM Standards (the Blue Book) has generated negative votes because, “The precision
and bias section is insufficient per Section A21 of the Blue Book,”
“The five-year window for a sole reference to repeatability data
has long since closed,” or (the granddaddy of all negatives) “This
is a test method that was developed in 1956–where is the precision
and bias section?!!”
All levity aside, these are the members for which to be thankful.
They understand that the backbone of the ASTM test method is the
precision and bias section. This section is the curriculum vitae
of the test method. It is the pedigree, the provenance, the very
foundation upon which the method rests. It provides validation
for the test method and ensures that it can do what it purports.
What Is Precision?
The Form and Style for ASTM Standards defines precision as follows:
Precision—the closeness of agreement among test results obtained
under prescribed conditions.
A precision statement allows potential users of the test method
to assess, in general terms, its usefulness in proposed applications.
A precision statement is not intended to contain values that can
be duplicated in every user’s laboratory. The statement offers
guidelines regarding the type of variability that can be expected
among test results when the method is used in one or more reasonably
competent laboratories.
There are two measurements that serve to express precision in
the evaluation of a test method. They are commonly referred to
as “repeatability” and “reproducibility” and provide the boundaries
between which precision exists. Simply put, repeatability addresses
variability between independent test results gathered from within
a single laboratory (otherwise known as intralaboratory testing).
Reproducibility addresses variability among single test results
gathered from different laboratories (otherwise known as interlaboratory
testing). Due to the fact that repeatability tends to produce
nominal variability and reproducibility tends to produce appreciable
variability, the boundaries for the precision are clearly established.
What Is Bias?
Bias—a systematic error that contributes to the difference between
the mean of a large number of test results and an accepted reference
value.
When included in a test method, this statement describes the bias
and the methods utilized to provide corrected test results. It
is important to remember that if an accepted reference value is
not available, the bias cannot be established. However, if the
bias is unknown but the direction or bounds of the bias can be
estimated, this information should be included in the bias statement.
The Ruggedness Test
The determination of the variables associated with the performance
of a test method is critical to the ultimate accuracy of the precision
of that method. The ruggedness test is designed to highlight these
variables and allow them to be tightened if necessary. If a ruggedness
test is not performed prior to the actual interlaboratory study,
the participants in the study may never be able to determine if
or where any problems exist. Unlike the interlaboratory study,
the ruggedness test should be confined to a single laboratory
to maximize the visibility of any variables. Examples of variables
to consider include, but are not limited to, sampling, conditioning,
procedure, pressure, temperature, and relative humidity.
There is an ASTM Standard—E 1169 Guide for Conducting Ruggedness Tests—that was specifically
designed to direct you through the steps necessary to complete this
process.
How Is the Data Generated?
The interlaboratory study (ILS) or “round robin” is utilized to
generate the final data used to support the statistical validity
of the test method. Typically, a series of laboratories will run
the test(s) described in the proposed or existing method and all
results will ultimately be compared on an interlaboratory basis.
There are multiple schools of thought regarding the exact number
of participating laboratories necessary to perform a statistically meaningful round
robin. For example, the ASTM standard E 691, Practice for Conducting
an Interlaboratory Study to Determine the Precision of a Test
Method, recommends that a minimum of six laboratories supply test
results for a precision statement. When making this determination,
it is important to remember to include sufficient laboratories to establish a reasonable cross-section of the laboratory
population, to ensure that the poor performance of one or more
laboratories will not skew the study, and to guarantee that the
reproducibility data is meaningful. A good rule of thumb is that
some precision information is always preferable to an absence
of precision information.
At the completion of the round robin and after the formal balloting
of the data into the test method, a final report on the interlaboratory
study shall be provided to ASTM to be retained on file as a research
report. Upon receipt of the data, ASTM will assign a research
report number and a note giving the research report number will
be editorially inserted into the precision and bias section of
the test method. It is important to remember that a research report
should be organized in a specific format. (Guide for the Format of a Research Report)
Why the Specific Research Report Format?
Due to the volume of test methods that have been produced by ASTM
(approximately 5,000), a consistent format for the cataloging
of each research report is a necessity. That format is detailed
in the Guide for the Format of a Research Report. This document identifies sections common to many interlaboratory
studies. All technical committees are encouraged to include as
many of the sections relevant to each interlaboratory study. The
nine sections of the guide, with a brief description of their
content are as follows:
1. Title—A description of the nature of the study.
2. Introduction—Similar to the scope of a standard, this should
include the purpose of the study.
3. Test Method—A copy of the method.
4. List of Participating Laboratories—A list of all participating
labs. It is important to note that prior to the distribution of
the data, each participating lab will receive a code (Lab A, Lab
B, etc.) that will allow them to identify only their data.
5. Interlaboratory Test Program Instructions—All instructions/guidelines
provided to the participants of the testing program.
6. Questionnaires—If questionnaires were distributed, include
a copy with a summary of responses.
7. Data Report Forms—All report forms returned by participating
labs.
8. Statistical Data Summary—A summary of data returned by participating
labs.
9. Research Report Summary—Summary of results, significance, and
conclusions.
Standards as a Resource: E 691 and Others
A question that is frequently asked by the ASTM member is whether
a document exists to aid a committee in conducting a round robin
and ultimately generating a precision statement. The answer to
this question is yes, such a document exists. In fact, multiple
standards that specifically address this issue have been developed
by ASTM technical committees.
The most widely applied of these documents is the ASTM standard E 691, Practice for Conducting an Interlaboratory
Study to Determine the Precision of a Test Method (see ToolBox).
This standard was developed in 1979 (and has been continuously
updated) by Committee E11 on Quality and Statistics. According
to its scope, E 691 “describes the techniques for planning, conducting,
analyzing, and treating the results of an interlaboratory study
(ILS) of a test method. The statistical techniques described in
this practice provide adequate information for formulating the
precision statement of a test method.”
Additionally, E 691 has a companion Windows-based software package
that will aid the member in the calculation and generation of
a precision statement. This software is available from ASTM for
purchase or can be loaned to a member to assist in standards development
activities. Upon completion of that activity, ASTM requires that
the software be returned. Standards that further clarify the issues
discussed in E 691 include E 177, Practice for Use of the Terms
Precision and Bias in ASTM Test Methods, and E 456, Standard Terminology
Relating to Quality and Statistics.
Several committees have chosen to develop statistical standards
that specifically reflect issues faced by their particular industry
or technology. Most of these documents are based on the principles
found in E 691, but have then been tailored to provide a greater
degree of relevance for their technical field. Committees that
have chosen this path include D02 on Petroleum Products and Lubricants
(D 6300, Practice for Determination of Precision and Bias Data
for Use in Test Methods for Petroleum Products and Lubricants),
E15 on Industrial and Specialty Chemicals (E 180, Practice for
Determining the Precision of ASTM Methods for Analysis and Testing
of Industrial Chemicals), D22 on Sampling and Analysis of Atmospheres
(D 3670, Guide for Determination of Precision and Bias of Methods
of Committee D-22), D13 on Textiles (D 2906, Practice for Statements
on Precision and Bias for Textiles), and C09 on Concrete and Concrete
Aggregates (C 670, Practice for Preparing Precision and Bias Statements for Test Methods for Construction Materials).
The point to remember when searching for the appropriate standard
to aid in the determination of precision and bias is that choices
do exist. A thorough search of the standards database located
on the ASTM Web site will provide many options. If questions arise, contact your staff
manager. The staff manager will be able to put you in touch with
the right technical contact to answer your specific questions
and direct you toward the most applicable standard.
As precision and bias are mandatory sections of an ASTM test method,
the utmost care should be taken to ensure that the final data,
as well as the steps that were taken to generate the data, are
as precise and accurate as possible. A test method that is incapable
of doing what it purports can be misleading. Quality precision
and bias statements will help strengthen the validity of the test
method and provide the user of the document with the confidence
to know that the test will work. After all, isn’t that the point?
//
|
|
|
FAQs on P&B
What is the cost of the E 691 software?
ASTM will provide on loan a copy of the E 691 software to an ASTM member for the purpose of working
on a precision statement of a test method. ASTM does require that
when the committee’s work with the software is completed, the
software be returned. If a person, lab, etc., wishes to purchase
E 691 to keep as a permanent resource, the cost of the E 691 software is $99 and the 22-page hard
or electronic copy of the standard is $40.
What should I do with the round robin data? Does ASTM keep it
on file?
Once the round robin has been completed and the information has
been balloted into the test method, the data should be submitted
to ASTM to be kept on file as a research report. A number will
be assigned to the research report and a note will appear in the
precision and bias section of the test method referencing the
specific research report. See the Guide for the Format of a Research Report.
Who absorbs the cost of round robin testing?
In most cases, industry will assume the cost of round robin testing.
The laboratories that volunteer to perform the testing to generate
precision and bias data will do so with all time, effort, and
materials offered on a pro bono basis.
Who can I contact at ASTM Headquarters if I need help understanding
E 691 or using the software?
The staff manager for your technical committee serves as the initial
point of contact for all inquiries. The manager will be able to
refer you to the appropriate resource to answer your question.
You will often be referred to a member of the committee that developed
E 691—Committee E11 on Quality and Statistics.
Can a single company with multiple labs consider each lab as a
separate testing site to reach the required number of labs in
E 691?
Yes. As long as each data set is generated by a laboratory with
its own set of conditions, calibrations, and equipment condition
and is then compared to other laboratories with their own set
of variables, the separate testing site requirement is met.
How long can we wait after developing a test method to include
a P&B statement?
If a committee determines that a delay is necessary to conduct
a round robin, a temporary statement addressing repeatability
conditions is permitted for five years. Included in that statement
shall be language regarding the date of actual availability of
reproducibility data.
Does ASTM require strict compliance with E 691?
ASTM requires strict compliance with Form and Style for ASTM Standards, which mandates that a precision and bias section be included
in all test methods. A means toward achieving that end can be
E 691, or a standard based upon E 691 that has been developed by an ASTM committee.
Can a committee publish a P&B statement if E 691 wasn’t used?
Yes. Section A21.2.7 of the Form and Style for ASTM Standards indicates that a precision statement shall be developed in accordance
with the test program prescribed in
E 691 or by an interlaboratory test program that yields equivalent information.
What happens if we find an old test method without a P&B statement?
Is it grandfathered?
The Form and Style for ASTM Standards mandates that all test methods contain a precision and bias statement.
If a test method is discovered that does not contain this section,
an evaluation should be immediately performed to determine the
utility of the document and steps should be undertaken to address
precision and bias.
What happens if we can’t produce the six labs that E 691 requires?
Will the software still work?
The E 691 software is designed to work optimally Color when data
from a minimum of six laboratories is entered—the greater the number of participating laboratories, the
smaller the chance that lab-to-lab variability will be an issue.
However, it is possible to generate a precision statement with
fewer than six participating laboratories; the bottom line is
that some precision information is preferable to no precision
information.
|
|
|
