PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
J AOAC Int. Author manuscript; available in PMC 2013 March 17.
Published in final edited form as:
PMCID: PMC3600167
NIHMSID: NIHMS443717

AOAC INTERNATIONAL Guidelines for Validation of Botanical Identification Methods

1 Scope

The purpose of this document is to provide comprehensive technical guidance for conducting AOAC INTERNATIONAL (AOAC) validation studies for botanical identification methods submitted for AOAC Official Methods of AnalysisSM (OMA) status and/or for Performance Tested MethodsSM (PTM) status. The requirements for single-laboratory validation (SLV) studies, independent validation studies, and collaborative validation studies for those methods are described.

2 Applicability

These guidelines are intended to be applicable to the validation of all candidate botanical identification methods (Appendix A) submitted to AOAC for (1) OMA status through either a collaborative study or an alternative pathway study or (2) PTM certification.

3 Terms and Definitions

3.1 Botanical

Of, or relating to, plants or botany. May also include algae and fungi. May refer to the whole plant, a part of the plant (e.g., bark, woods, leaves, stems, roots, rhizomes, flowers, fruits, seeds, etc.), or an extract of the parts.

3.2 Botanical identification method (BIM)

A method that establishes identity specifications for a botanical material and determines, within a specified statistical limit, a binary test result: YES, the test material is a true example of the target botanical material and meets the identity specifications, or NO, it is not the target botanical. Thus, a BIM answers the question, “Is the test material the same as the target material?” not “What is this material?” In most cases, the method will achieve this goal by comparison of the test material with materials from the inclusivity panel and will return a YES/NO (or, in some cases, a consistent/ nonconsistent) answer.

3.3 Candidate method

The method to be validated or submitted for validation (Appendix A).

3.4 Exclusivity

Ability of a BIM to correctly reject nontarget botanical materials.

3.5 Exclusivity sampling frame (ESF)

A list of practically obtainable nontarget botanical materials that have taxonomic, physical, or chemical composition characteristics similar to the target botanical and must give a negative result when tested by the BIM.

3.6 Exclusivity panel

A subset of the ESF that is selected for the validation study. The identity of these materials should be verified by an appropriate method or process.

3.7 Identity specification (IS)

The morphological, genetic, chemical, or other characteristics that define a target botanical material. Specifications may include, but are not limited to, data from macroscopic, microscopic, genetic (e.g., DNA sequencing), chromatographic fingerprinting (e.g., capillary electrophoresis, gas chromatography, liquid chromatography, or thin-layer chromatography), and spectral fingerprinting (e.g., infrared, near-infrared, nuclear magnetic resonance, ultraviolet/visible absorbance, or mass spectrometry) methods.

3.8 Inclusivity

Ability of a BIM to correctly identify variants of the target material that meet the identity specification.

3.9 Inclusivity sampling frame (ISF)

A list of practically obtainable botanical materials that are expected to give a positive result when tested by the BIM. The inclusivity frame should be sufficiently large that the botanical variation is adequately represented. Sources of variation may include, but are not limited to, species, subspecies, cultivar, growing location, growing conditions, growing season, and post-harvest processing.

3.10 Inclusivity panel

A subset of the ISF that is selected for the validation study. These materials should be authenticated by an appropriate method.

3.11 Laboratory sample

Sample as prepared for sending to the laboratory intended for inspection or testing.

3.12 Nontarget botanical material

Any botanical material that does not meet the identity specification.

3.13 Physical form

Botanical materials exist in a number of physical forms. The form(s) will be specified by the standard method performance requirements (SMPRs).

3.14 Probability of identification (POI)

The expected or observed fraction of test portions at a given concentration that give a positive result when tested by the BIM. A general description is provided in Appendix B.

3.15 Sample

A small portion or quantity, taken from a population or lot that is ideally a representative selection of the whole. Sample homogeneity is usually determined with multiple samples.

3.16 Specified inferior test material (SITM)

A botanical material mixture that has the maximum concentration of target material that is considered unacceptable, as specified by the SMPRs. The BIM must reject this material with a specified minimum level of (1 – POI) with 95% confidence. The ideal BIM would reject the SITM 100% of the time (i.e., accept 0% of the time). The SITM will typically be high-quality target material mixed with the worst-case (for identification) nontarget material.

3.17 Specified superior test material (SSTM)

A botanical material mixture that has the minimum acceptable concentration of the target material, as specified by the SMPR. The BIM must identify this material with a specified minimum level of POI with 95% confidence. The ideal BIM would accept the SSTM 100% of the time. The SSTM will typically be high-quality target material mixed with a small amount of worst-case (for identification) nontarget material.

3.18 Standard method performance requirements (SMPRs)

Performance requirements based on the fitness-for-purpose statement for each method. For BIMs, the SMPRs should include the physical form of the sample, the ISF, the ESF, the SSTM, the SITM, the number of samples for the inclusivity/ exclusivity panels, and the desired probability and confidence limits for the method.

3.19 Target botanical material

The botanical material of interest as described in the identity specification.

3.20 Test portion

The portion of the laboratory sample that is subjected to analysis by the method.

4 Validation Study Guidelines

A validated BIM requires a method validation study that demonstrates its acceptability according to the SMPRs. The guidelines presented here are intended to be applied to any qualitative BIM that returns a binary, YES/NO test result (Appendix A). The guidelines provide technical guidance in validating the method based on the POI model (Appendix B).

4.1 SMPRs

The SMPRs will be prepared by the appropriate AOAC body as per AOAC policy. The SMPRs will specify (1) the target botanical material, (2) the physical form of the material, (3) a list of botanical materials for the ISF/ESF, (4) composition of the SSTM and SITM, (5) maximum POI for the SITM and minimum POI for the SSTM, and (6) the desired probability and confidence limits for the inclusivity/exclusivity and SSTM/SITM measurements.

The SMPRs will consider the nature of the material being tested and determine the necessary breadth and depth of the inclusivity and exclusivity panels. In some cases, a few, very similar exclusivity panel materials may require in-depth testing (more test portions of a smaller group of materials). Conversely, the nature of the material may require greater breadth (fewer test portions of a greater number of materials).

The number of test portions needed should be determined on sound statistical grounds (Appendix C) and subject matter expertise.

4.2 SLV Study

4.2.1 Scope

An SLV study is intended to determine the performance of a candidate method (Appendix A). For validation purposes, the candidate BIM may be regarded as a black box providing a binary, YES/NO test result. The study is designed to evaluate performance parameters for the candidate method including (1) inclusivity/exclusivity, (2) POI for the SSTM and the SITM, and (3) POI as a function of the concentration of the target material (analytical response curve). This last parameter may be optional as specified by the SMPRs.

4.2.2 Inclusivity/Exclusivity Study

The purpose of this study is to confirm the ability of the candidate method to provide positive results (YES answers) for botanical materials on the inclusivity panel and negative results (NO answers) for materials on the exclusivity panel.

4.2.2.1 Inclusivity/Exclusivity Panel Selection

Botanical materials selected from the ISF/ESF will comprise the inclusivity/exclusivity panels. If the ISF/ESF specified by the SMPRs are sufficiently large, a representative subgroup will be selected for the panels by the method validator. Primary requirements for the panel materials are their availability and identity verification by an appropriate method or process. All test portions should be as uniform and homogeneous as possible. The level of replication of the inclusivity/exclusivity panels will be specified in the SMPRs.

4.2.2.2 Study Design

Prepare the test samples in a form appropriate for the candidate method. All test samples will be blinded and randomized so that the analyst(s) cannot know the identity of the samples. Analyze the test samples following the instructions of the candidate method.

4.2.2.3 Data Analysis and Reporting

The data will be analyzed for positive and negative responses. Unexpected results will be investigated, evaluated, and resolved prior to continuing the validation. The data is reported for individual inclusivity/exclusivity material as the number correctly identified. For example, “Of the 30 specific botanical materials of the inclusivity panel that were tested, 28 were identified correctly (gave a positive result) and two were not identified correctly (gave a negative result). Those materials not identified correctly were the following:…” or “Of the 30 specific botanical materials of the exclusivity panel that were tested, 27 were identified correctly (gave a negative result) and three were not identified correctly (gave a positive result). Those not identified correctly were the following:…” The study report should include a table titled “Inclusivity/Exclusivity Panel Results,” which lists all materials tested, their source, origin, and essential characteristics and testing outcome. The implications of each unexpected result should be discussed and evaluated.

4.2.3 SSTM/SITM Study

The purpose of this study is to demonstrate method performance at two concentrations, the SSTM and the SITM.

4.2.3.1 Test Samples

The appropriate amount of a target material is selected from the inclusivity panel and is mixed with an appropriate amount of a nontarget material from the exclusivity panel to produce the SSTM and SITM as specified by the SMPRs. The test materials may be prepared using individual botanical materials from the inclusivity/exclusivity panels or composites of materials from the two panels as specified by the SMPRs.

All test portions should be as uniform and homogeneous as possible. The level of replication of the SSTM and SITM will be specified in the SMPR.

4.2.3.2 Study Design

Prepare the test samples in a form appropriate for the candidate method. All test samples will be blinded and randomized so that the analyst(s) cannot know the identity of the samples. Analyze the test samples following the instructions of the candidate method.

4.2.3.3 Data Analysis and Reporting

The data will be analyzed for positive and negative responses. For the SSTM and the SITM, report the POI results with 95% confidence intervals and the total number tested and the total number correctly identified. Comparison to SMPRs should be made and discussed.

4.2.4 Analytical Response Curve

This study will characterize the POI curve for mixtures of SSTM and SITM.

4.2.4.1 Test Samples

The appropriate amount of a target material is selected from the inclusivity panel and is mixed with an appropriate amount of a nontarget material from the exclusivity panel to produce mixtures with concentrations intermediate between the SSTM and SITM. The test materials shall be prepared using the same target and nontarget botanical material samples used in the SSTM and SITM study. The test materials may also be prepared by mixing appropriate ratios of the SSTM and SITM.

4.2.4.2 Study Design

Prepare the test samples in a form appropriate for the candidate method. All test samples will be blinded and randomized so that the analyst(s) cannot know the identity of the samples. Analyze the test samples following the instructions of the candidate method.

4.2.4.3 Data Analysis and Reporting

The data will be analyzed for positive and negative responses. For each mixture, report the POI results with 95% confidence intervals, the total number of samples tested, and the total number of positive responses. Plot the POI curve and confidence intervals.

4.3 Independent Validation Study

This study is identical to the SLV Study in Section 4.2.

4.4 Collaborative Study

The collaborative study is a route to an Official MethodSM. The purpose of the collaborative study is to estimate the reproducibility and determine the performance of the candidate method among collaborators.

4.4.1 Number of Collaborators

A minimum of 10 independent laboratories reporting valid data is required. The study director should plan on including additional laboratories in the case of invalid data sets.

4.4.2 Number of Tests

Each collaborator receives 12 replicates of each material to be studied. At a minimum these materials will include the SSTM and SITM. Prepare the test samples in a form appropriate for the candidate method. All test samples will be blinded and randomized so that the analyst(s) cannot know the identity of the samples. Analyze the test samples following the instructions of the candidate method.

4.4.3 Data Analysis and Reporting

The data will be analyzed by the laboratory for positive and negative responses. For the SSTM and the SITM, report the POI results with confidence intervals for each laboratory, and for the combined results. Estimate reproducibility as in Appendix C and evaluate compared to the SMPRs.

Acknowledgments

This work was funded by National Institutes of Health, Office of Dietary Supplements.

Appendix A—Candidate Method (or Prevalidation Study)

A.1 Scope

The candidate method must measure appropriate characteristics that are suitable to the question being asked and that will meet predetermined SMPRs. The method may be based on new principles or modifications of an existing method. The identity specifications will be based on morphological, genetic, and/or chemical characteristics, or any other defining feature of the botanical material. The candidate method may use visual inspection, DNA sequencing, instrumental analysis, or any other appropriate measurement. The measured characteristics will collectively provide a single analytical parameter that will be used to determine the final YES or NO result. The analytical parameter may be based on the degree of similarity or the degree of difference of the test sample and the reference material.

A.2 Inclusivity/Exclusivity Panel Selection

The method developer will select representative botanical materials from the ISF and ESF for use as target and nontarget botanical materials, respectively, in development of the method. These materials must be authenticated by an appropriate method.

A.3 Analytical Parameter

The method developer will prepare all the botanical samples in a form appropriate for the candidate method. The developer will analyze the target and nontarget botanical materials using the candidate method and develop an analytical parameter that is suitable for distinguishing between the two sets of materials.

A.4 Probability of Identification (POI)

Target materials will be mixed with systematically increasing amounts of nontarget materials to produce a series of target materials whose concentrations range from 100% to a concentration below the minimum acceptable concentration specified by the SMPRs. The developer will analyze the target and diluted target materials using the candidate method and determine the analytical parameter for each concentration.

A.5 Specific Superior/Inferior Test Materials

Based on the analytical parameters measured for the diluted target materials, a threshold value will be established that will permit positive identification of the minimum acceptable concentration of the target material with the specified confidence (e.g. 95%). The developer will use the threshold to determine a POI for each concentration (Appendix B). The POIs measured for each concentration will be used to construct the POI curve.

A.6 Data Analysis and Reporting

The method developer will document the candidate method and the POI results.

Appendix B—Understanding the POI Model

(See accompanying manuscript, “Probability of Identification: A Statistical Model for the Validation of Qualitative Botanical Identification Methods,” by Robert LaBudde and James M. Harnly, J. AOAC Int. 95, 273–285 (2012). http://dx.doi.org/10.5740/jaoacint.11-266)

Appendix C—Number of Test Portions

See table below. Notes:

  1. Enter the first column with the maximum error fraction tolerated by the SMPR, e.g., 10%.
  2. Select the sample size required by the number of misclassifications to be allowed, e.g., 1 erroneous result gives a sample size of n = 48 for a maximum error probability of 10%.
  3. Allowing more erroneous results increases the sample size required.
  4. The last (AOQL) column indicates the maximum error probability of a method which passes the SMPR for the test. For the example sampling plan indicated, this is 5.4%, approximately ½ of the maximum error probability in the SMPR. Typically the AOQL must be only 50–60% of the SMPR value to reliably pass the validation test. Method developers should take this into account.

Sample Size Required for Proportion

ASSUME:1. Binary outcome (occur / not occur).
2. Constant probability rho of event occurring.
3. Independent trials (e.g., simple random sample).
4. Fixed number of trials N.
INFERENCE:95% confidence interval lies entirely at or BELOW specified maximum rho
DESIRED:Sample size N needed
NOTES:1. Based on modified Wilson score 1-sided confidence interval.
2. AOQL = Average Outgoing Quality Level
Maximum Probability rhoSample Size NMaximum Number Events XMinimum Number Non-events Y1-sided Upper Confidence Limit on rhoExpected Lower Confidence Limit on rhoExpected Upper Confidence Limit on rhoEffective AOQL rho
50%30347.4%0.0%56.1%28.1%
50%102845.9%5.7%51.0%28.3%
50%2061448.4%14.5%51.9%33.2%
50%40142648.0%22.1%50.5%36.3%
50%80324849.2%30.0%51.0%40.5%

45%20257.5%0.0%65.8%32.9%
45%101934.8%0.0%40.4%20.2%
45%2051543.2%11.2%46.9%29.0%
45%40122842.9%18.1%45.4%31.8%
45%80285244.1%25.5%45.9%35.7%

40%50535.1%0.0%43.4%21.7%
40%101934.8%0.0%40.4%20.2%
40%2041637.8%8.1%41.6%24.8%
40%40103037.6%14.2%40.2%27.2%
40%80245639.0%21.1%40.8%30.9%

35%60631.1%0.0%39.0%19.5%
35%101934.8%0.0%40.4%20.2%
35%2031732.2%5.2%36.0%20.6%
35%4093134.9%12.3%37.5%24.9%
35%80215935.0%17.9%36.8%27.3%

30%70727.9%0.0%35.4%17.7%
30%1001021.3%0.0%27.8%13.9%
30%2021826.2%2.8%30.1%16.4%
30%4073329.3%8.7%31.9%20.3%
30%80176329.6%13.7%31.4%22.6%

25%90923.1%0.0%29.9%15.0%
25%1001021.3%0.0%27.8%13.9%
25%2011919.6%0.0%23.6%11.8%
25%4053523.5%5.5%26.1%15.8%
25%80136724.1%9.7%25.8%17.8%

20%1101119.7%0.0%25.9%12.9%
20%2011919.6%0.0%23.6%11.8%
20%2412316.7%0.0%20.2%10.1%
20%3633319.1%2.9%21.8%12.4%
20%4033717.3%2.6%19.9%11.2%
20%4854319.9%4.5%22.2%13.3%
20%6065418.2%4.7%20.1%12.4%
20%7286418.7%5.7%20.4%13.1%
20%80107019.8%6.9%21.5%14.2%

15%2002011.9%0.0%16.1 %8.1%
15%2402410.1%0.0%13.8%6.9%
15%3613511.5%0.0%14.2%7.1%
15%4023814.0%1.4%16.5%8.9%
15%4834514.6%2.1%16.8%9.5%
15%6045614.0%2.6%15.9%9.3%
15%7256713.6%3.0%15.2%9.1%
15%8067413.9%3.5%15.4%9.4%

10%400406.3%0.0%8.8%4.4%
10%481478.8%0.0%10.9%5.4%
10%602589.6%0.9%11.4%6.1%
10%7236910.0%1.4%11.5%6.5%
10%803779.0%1.3%10.5%5.9%

5%600604.3%0.0%6.0%3.0%
5%720723.6%0.0%5.1%2.5%
5%800803.3%0.0%4.6%2.3%
5%901894.8%0.0%6.0%3.0%

Footnotes

This document provides technical protocol guidelines for the AOAC validation of botanical identification methods and/or procedures, and covers terms and their definitions associated with the Performance Tested MethodsSM and Official Methods of AnalysisSM programs.

The guideline working group consisted of James Harnly (Chair, USDA, ARS), Wendy Applequist (Missouri Botanical Garden), Paula Brown (British Columbia Institute of Technology), Steven Caspar (FDA/CFSAN), Peter Harrington (Ohio University), Danica Harbaugh-Reynaud (AuthenTechnologies, LLC), Norma Hill (Alcohol and Tobacco Tax and Trade Bureau Compliance Laboratory), Robert LaBudde (Least Cost Formulations and Old Dominion University), James Neal-Kababick (Flora Research Laboratories), Mark Roman (Tampa Bay Analytical Research), Shauna Roman (Schiff Nutrition International, Inc.), Darryl Sullivan (Covance Laboratories), Barry Titlow (Compound Solutions), and Paul Wehling (General Mills/ Medallion Laboratories).

The guidelines were approved by the AOAC Official Methods Board on October 13, 2011.