March 2000

March SN Contents

ToolBox

Guide for the Format of a Research Report:

ASTM Standard
E 691, Practice for Conducting an Interlaboratory Study to Determine the Precision of a Test Method

Your staff manager can lend you a copy, or purchase from the
ASTM Store.

Your Peers:

Many ASTM members have successfully created valid P&B statements. Network with your fellow committee members and your staff manager for references. In addition, your committee may have developed its own standard for development of P&B statements.

Fact vs. Fiction: The Truth About Precision and Bias

by Pat Picariello

Every ASTM test method requires a precision and bias section. What is it? How do I create one? Read on.

Have you heard any of these terms before—precision, bias, repeatability, reproducibility, interlaboratory, intralaboratory? Have they sent shivers running up and down your spine? When the call goes out at a committee meeting for volunteers to participate in a ruggedness test or round robin, do you suddenly find your hands glued to the table?

If you answered yes to any of these questions, you’re reading the right article (and you’re also the typical ASTM member).

Anyone who has helped create, revise, distribute, or catalog an ASTM test method has encountered the world of precision and bias. Every committee has dealt with members whose detailed knowledge of Form and Style for ASTM Standards (the Blue Book) has generated negative votes because, “The precision and bias section is insufficient per Section A21 of the Blue Book,” “The five-year window for a sole reference to repeatability data has long since closed,” or (the granddaddy of all negatives) “This is a test method that was developed in 1956–where is the precision and bias section?!!”

All levity aside, these are the members for which to be thankful. They understand that the backbone of the ASTM test method is the precision and bias section. This section is the curriculum vitae of the test method. It is the pedigree, the provenance, the very foundation upon which the method rests. It provides validation for the test method and ensures that it can do what it purports.

What Is Precision?

The Form and Style for ASTM Standards defines precision as follows:
Precision—the closeness of agreement among test results obtained under prescribed conditions.

A precision statement allows potential users of the test method to assess, in general terms, its usefulness in proposed applications. A precision statement is not intended to contain values that can be duplicated in every user’s laboratory. The statement offers guidelines regarding the type of variability that can be expected among test results when the method is used in one or more reasonably competent laboratories.

There are two measurements that serve to express precision in the evaluation of a test method. They are commonly referred to as “repeatability” and “reproducibility” and provide the boundaries between which precision exists. Simply put, repeatability addresses variability between independent test results gathered from within a single laboratory (otherwise known as intralaboratory testing). Reproducibility addresses variability among single test results gathered from different laboratories (otherwise known as interlaboratory testing). Due to the fact that repeatability tends to produce nominal variability and reproducibility tends to produce appreciable variability, the boundaries for the precision are clearly established.

What Is Bias?

The Form and Style for ASTM Standards defines bias as follows:

Bias—a systematic error that contributes to the difference between the mean of a large number of test results and an accepted reference value.

When included in a test method, this statement describes the bias and the methods utilized to provide corrected test results. It is important to remember that if an accepted reference value is not available, the bias cannot be established. However, if the bias is unknown but the direction or bounds of the bias can be estimated, this information should be included in the bias statement.

The Ruggedness Test

The determination of the variables associated with the performance of a test method is critical to the ultimate accuracy of the precision of that method. The ruggedness test is designed to highlight these variables and allow them to be tightened if necessary. If a ruggedness test is not performed prior to the actual interlaboratory study, the participants in the study may never be able to determine if or where any problems exist. Unlike the interlaboratory study, the ruggedness test should be confined to a single laboratory to maximize the visibility of any variables. Examples of variables to consider include, but are not limited to, sampling, conditioning, procedure, pressure, temperature, and relative humidity.

There is an ASTM Standard—E 1169 Guide for Conducting Ruggedness Tests—that was specifically designed to direct you through the steps necessary to complete this process.

How Is the Data Generated?

The interlaboratory study (ILS) or “round robin” is utilized to generate the final data used to support the statistical validity of the test method. Typically, a series of laboratories will run the test(s) described in the proposed or existing method and all results will ultimately be compared on an interlaboratory basis.

There are multiple schools of thought regarding the exact number of participating laboratories necessary to perform a statistically meaningful round robin. For example, the ASTM standard E 691, Practice for Conducting an Interlaboratory Study to Determine the Precision of a Test Method, recommends that a minimum of six laboratories supply test results for a precision statement. When making this determination, it is important to remember to include sufficient laboratories to establish a reasonable cross-section of the laboratory population, to ensure that the poor performance of one or more laboratories will not skew the study, and to guarantee that the reproducibility data is meaningful. A good rule of thumb is that some precision information is always preferable to an absence of precision information.

At the completion of the round robin and after the formal balloting of the data into the test method, a final report on the interlaboratory study shall be provided to ASTM to be retained on file as a research report. Upon receipt of the data, ASTM will assign a research report number and a note giving the research report number will be editorially inserted into the precision and bias section of the test method. It is important to remember that a research report should be organized in a specific format. (Guide for the Format of a Research Report)

Why the Specific Research Report Format?

Due to the volume of test methods that have been produced by ASTM (approximately 5,000), a consistent format for the cataloging of each research report is a necessity. That format is detailed in the Guide for the Format of a Research Report. This document identifies sections common to many interlaboratory studies. All technical committees are encouraged to include as many of the sections relevant to each interlaboratory study. The nine sections of the guide, with a brief description of their content are as follows:

1. Title—A description of the nature of the study.
2. Introduction—Similar to the scope of a standard, this should include the purpose of the study.
3. Test Method—A copy of the method.
4. List of Participating Laboratories—A list of all participating labs. It is important to note that prior to the distribution of the data, each participating lab will receive a code (Lab A, Lab B, etc.) that will allow them to identify only their data.
5. Interlaboratory Test Program Instructions—All instructions/guidelines provided to the participants of the testing program.
6. Questionnaires—If questionnaires were distributed, include a copy with a summary of responses.
7.  Data Report Forms—All report forms returned by participating labs.
8. Statistical Data Summary—A summary of data returned by participating labs.
9. Research Report Summary—Summary of results, significance, and conclusions.

Standards as a Resource: E 691 and Others

A question that is frequently asked by the ASTM member is whether a document exists to aid a committee in conducting a round robin and ultimately generating a precision statement. The answer to this question is yes, such a document exists. In fact, multiple standards that specifically address this issue have been developed by ASTM technical committees.

The most widely applied of these documents is the ASTM standard E 691, Practice for Conducting an Interlaboratory Study to Determine the Precision of a Test Method (see ToolBox). This standard was developed in 1979 (and has been continuously updated) by Committee E11 on Quality and Statistics. According to its scope, E 691 “describes the techniques for planning, conducting, analyzing, and treating the results of an interlaboratory study (ILS) of a test method. The statistical techniques described in this practice provide adequate information for formulating the precision statement of a test method.”

Additionally, E 691 has a companion Windows-based software package that will aid the member in the calculation and generation of a precision statement. This software is available from ASTM for purchase or can be loaned to a member to assist in standards development activities. Upon completion of that activity, ASTM requires that the software be returned. Standards that further clarify the issues discussed in E 691 include E 177, Practice for Use of the Terms Precision and Bias in ASTM Test Methods, and E 456, Standard Terminology Relating to Quality and Statistics.

Several committees have chosen to develop statistical standards that specifically reflect issues faced by their particular industry or technology. Most of these documents are based on the principles found in E 691, but have then been tailored to provide a greater degree of relevance for their technical field. Committees that have chosen this path include D02 on Petroleum Products and Lubricants (D 6300, Practice for Determination of Precision and Bias Data for Use in Test Methods for Petroleum Products and Lubricants), E15 on Industrial and Specialty Chemicals (E 180, Practice for Determining the Precision of ASTM Methods for Analysis and Testing of Industrial Chemicals), D22 on Sampling and Analysis of Atmospheres (D 3670, Guide for Determination of Precision and Bias of Methods of Committee D-22), D13 on Textiles (D 2906, Practice for Statements on Precision and Bias for Textiles), and C09 on Concrete and Concrete Aggregates (C 670, Practice for Preparing Precision and Bias Statements for Test Methods for Construction Materials).

The point to remember when searching for the appropriate standard to aid in the determination of precision and bias is that choices do exist. A thorough search of the standards database located on the ASTM Web site will provide many options. If questions arise, contact your staff manager. The staff manager will be able to put you in touch with the right technical contact to answer your specific questions and direct you toward the most applicable standard.

As precision and bias are mandatory sections of an ASTM test method, the utmost care should be taken to ensure that the final data, as well as the steps that were taken to generate the data, are as precise and accurate as possible. A test method that is incapable of doing what it purports can be misleading. Quality precision and bias statements will help strengthen the validity of the test method and provide the user of the document with the confidence to know that the test will work. After all, isn’t that the point? //

FAQs on P&B

What is the cost of the E 691 software?

ASTM will provide on loan a copy of the E 691 software to an ASTM member for the purpose of working on a precision statement of a test method. ASTM does require that when the committee’s work with the software is completed, the software be returned. If a person, lab, etc., wishes to purchase E 691 to keep as a permanent resource, the cost of the E 691 software is $99 and the 22-page hard or electronic copy of the standard is $40.

What should I do with the round robin data? Does ASTM keep it on file?

Once the round robin has been completed and the information has been balloted into the test method, the data should be submitted to ASTM to be kept on file as a research report. A number will be assigned to the research report and a note will appear in the precision and bias section of the test method referencing the specific research report. See the Guide for the Format of a Research Report.

Who absorbs the cost of round robin testing?

In most cases, industry will assume the cost of round robin testing. The laboratories that volunteer to perform the testing to generate precision and bias data will do so with all time, effort, and materials offered on a pro bono basis.

Who can I contact at ASTM Headquarters if I need help understanding E 691 or using the software?

The staff manager for your technical committee serves as the initial point of contact for all inquiries. The manager will be able to refer you to the appropriate resource to answer your question. You will often be referred to a member of the committee that developed
E 691—Committee E11 on Quality a
nd Statistics.

Can a single company with multiple labs consider each lab as a separate testing site to reach the required number of labs in E 691?

Yes. As long as each data set is generated by a laboratory with its own set of conditions, calibrations, and equipment condition and is then compared to other laboratories with their own set of variables, the separate testing site requirement is met.

How long can we wait after developing a test method to include a P&B statement?

If a committee determines that a delay is necessary to conduct a round robin, a temporary statement addressing repeatability conditions is permitted for five years. Included in that statement shall be language regarding the date of actual availability of reproducibility data.

Does ASTM require strict compliance with E 691?

ASTM requires strict compliance with Form and Style for ASTM Standards, which mandates that a precision and bias section be included in all test methods. A means toward achieving that end can be
E 691, or a standard based upon E 69
1 that has been developed by an ASTM committee.

Can a committee publish a P&B statement if E 691 wasn’t used?

Yes. Section A21.2.7 of the Form and Style for ASTM Standards indicates that a precision statement shall be developed in accordance with the test program prescribed in
E 691 or by an interlaboratory test program that yields equivalent information.

What happens if we find an old test method without a P&B statement? Is it grandfathered?

The Form and Style for ASTM Standards mandates that all test methods contain a precision and bias statement. If a test method is discovered that does not contain this section, an evaluation should be immediately performed to determine the utility of the document and steps should be undertaken to address precision and bias.

What happens if we can’t produce the six labs that E 691 requires? Will the software still work?

The E 691 software is designed to work optimally Color when data from a minimum of six laboratories is entered—the greater the number of participating laboratories, the smaller the chance that lab-to-lab variability will be an issue. However, it is possible to generate a precision statement with fewer than six participating laboratories; the bottom line is that some precision information is preferable to no precision information.

Talk to the Editor

Back to Top

Pat Picariello is a manager in the Technical Committee Operations Division.