Radiologists’ Perceptions of Computer Aided Detection versus Double Reading for Mammography Interpretation

doi:10.1016/j.acra.2010.05.007

Acad Radiol. Author manuscript; available in PMC 2011 October 1.

Published in final edited form as:

Acad Radiol. 2010 October; 17(10): 1217–1226.

doi: 10.1016/j.acra.2010.05.007

PMCID: PMC3149895

NIHMSID: NIHMS208579

Radiologists’ Perceptions of Computer Aided Detection versus Double Reading for Mammography Interpretation

Tracy Onega, PhD,⁽¹⁾ Erin J. Aiello Bowles, MPH,⁽²⁾ Diana L. Miglioretti, PhD,^(2),⁽³⁾ Patricia A. Carney, PhD,⁽⁴⁾ Berta M. Geller, EdD,⁽⁵⁾ Bonnie C. Yankaskas, Ph.D.,⁽⁶⁾ Karla Kerlikowske, MD,⁽⁷⁾ Edward A. Sickles, MD,⁽⁷⁾ and Joann G. Elmore, MD, MPH⁽⁸⁾

⁽¹⁾Dartmouth Medical School, Lebanon, NH

⁽²⁾Group Health Research Institute, Seattle, WA

⁽³⁾University of Washington, Department of Biostatistics, Seattle, WA

⁽⁴⁾Oregon Health & Science University, Portland, OR

⁽⁵⁾University of Vermont, Burlington, VT

⁽⁶⁾University of North Carolina, Chapel Hill, NC

⁽⁷⁾University of California, San Francisco, San Francisco, CA

⁽⁸⁾University of Washington, Departments of Medicine and Epidemiology, Seattle, WA

Correspondence to: Tracy Onega, PhD, Assistant Professor of Community & Family Medicine, Dartmouth Medical School, HB 7927, One Medical Center Dr. Lebanon, NH 03756, Telephone: 603-653-3671, Fax: 603-653-9093, Email: onega/at/dartmouth.edu

The publisher's final edited version of this article is available at Acad Radiol

Publisher's Disclaimer

Abstract

Rationale and Objectives

To examine radiologists’ use and perceptions of computer-aided detection (CAD) and double reading for screening mammography interpretation.

Materials and Methods

A mailed survey of 257 community radiologists participating in the national Breast Cancer Surveillance Consortium assessed perceptions and practices related to CAD and double reading. We used latent class analysis to classify radiologists’ overall perceptions of CAD and double reading based on their agreement or disagreement with specific statements about CAD and double reading.

Results

Most radiologists (64%) reported using CAD for more than half the screening mammograms they interpreted, but only <5% reported double reading that much. More radiologists perceived that double reading improved cancer detection rates compared with CAD (74% vs. 55% reported), while fewer radiologists thought that double reading decreased recall rates compared with CAD (50% vs. 65% reported). Radiologists with the most favorable perceptions of CAD were more likely to think that CAD improved cancer detection rate without taking too much time compared with radiologists with the most unfavorable overall perceptions. In latent class analysis an overall favorable perception of CAD was significantly associated with use of CAD (81%), higher percent of workload in screening mammography (80%), academic affiliation (71%), and fellowship training (58%). Perceptions of double reading that were most favorable were associated with academic affiliation (98%).

Conclusion

Radiologists’ perceptions were more favorable toward double reading by a second clinician than by a computer, although fewer used double reading in their own practice. The majority of radiologists perceived both CAD and double reading at least somewhat favorably, although for largely different reasons.

Keywords: Breast imaging, mammography, computer-aided detection (CAD), double reading

Introduction

We have entered a unique period in medicine in which double reading by a computer may soon replace double reading by a second clinician. This is especially prominent in the field of breast imaging. Computer aided detection (CAD) has been approved by the FDA for use in screening mammography interpretation since 1998 with the goal of serving as the equivalent of a “spell checker” to mark potentially abnormal findings and thus lead to improved breast cancer detection.¹ Since that time, diffusion of CAD in clinical practice has progressed.² CAD use for screening mammography interpretation has been given some positive anecdotal attention from the commercial media and may have increased patient expectations of accurate cancer detection. However, information on the relative benefits and harms of CAD in mammography is limited and often contradictory. CAD has been shown to increase rates of recall and lower specificity in some studies^2,3 and to improve sensitivity and cancer detection rates in others.^3,4–6

Double reading is generally accepted as a practice to improve cancer detection in screening mammography, but this is primarily done in Europe ^7–11 and is, therefore, included in the European Guidelines for Quality Assurance in Breast Cancer Screening and Diagnosis. ^12,13,14 Double reading is not consistently practiced in the U.S. possibly as it demands more radiologist time without reimbursement for the second read. Further, the benefits of double reading depend on how the information from the second radiologist is obtained and used (e.g., whether the second assessment is independent and blinded to that of the first assessment and whether the final medical decision making requires consensus among the two radiologists for recall).

It is thought that the use of CAD has been increasing in the U.S. while double reading has been declining, although numbers are lacking, which is likely to have implications for mammography interpretation variability and performance. This trend is not occurring in other countries, such as the United Kingdom and Norway.^15,16 The differences in trends are likely related to the additional reimbursement for CAD in the U.S., which does not occur in most other countries. Double reading does not generate additional reimbursement in the U.S. but its cost, including additional radiologist time, is included in the budgets of organized screening programs in many other countries.

We studied how U.S. radiologists use double reading and CAD in their clinical practices and how they view these major shifts in the use of computers (CAD) vs. double reading in the medical decision making process. The main objectives were to: 1) provide estimates of current rates of CAD and double reading use for screening and diagnostic mammography in the United States; 2) characterize radiologists’ perceptions about CAD and double reading; 3) examine radiologists’ characteristics that are associated with favorable and unfavorable perceptions towards CAD and double reading; and 4) estimate the similarity in radiologists’ perceptions between CAD and double reading.

Materials and Methods

Data Source and Study Population

The study population consisted of 257 radiologists participating in the Breast Cancer Surveillance Consortium (BCSC), an National Cancer Institute-funded collaboration among seven mammography registries including: 1) Carolina Mammography Registry, North Carolina; 2) Colorado Mammography Project, Denver, Colorado; 3) Group Health Cooperative, Washington; 4) New Hampshire Mammography Network, New Hampshire; 5) New Mexico Mammography Project, New Mexico; 6) San Francisco Mammography Registry, San Francisco, California; and 7) Vermont Breast Cancer Surveillance System, Vermont.^17,18 Radiologists who interpreted mammograms at any of the facilities contributing data to the seven mammography registries from January 2005 to December 2006 (n=364) were invited to participate in a mailed survey in 2006 or 2007.

The 10-page survey was designed to capture radiologists’ demographic and clinical practice information as well as their perceptions of screening and diagnostic mammograms interpreted with CAD and double-reading. Radiologists also indicated the proportion of screening and/or diagnostic mammograms they double read as a first or second reader. To assess perceptions of CAD and double reading, radiologists were asked to agree or disagree with statements related to the question: “What are your thoughts on computer-aided detection (CAD) programs and double reading?” The response categories included: 1) reassures mammographers; 2) reassures patients; 3) improves cancer detection rate; 4) increases recall rate; 5) protects radiologists from malpractice suits; 6) improves profitability of breast imaging; and 7) takes too much time. Radiologists used a 5-point Likert scale (strongly agree, agree, neutral, disagree, and strongly disagree) to respond to the seven statements.

The survey response rate was 71% (257/364). Survey response data were double entered and pooled at the BCSC Statistical Coordinating Center (SCC) for analysis. All study activities were approved by the Institutional Review Boards associated with the seven mammography registries. All procedures were Health Insurance Portability and Accountability Act compliant, and all registries and the Statistical Coordinating Center received a Federal Certificate of Confidentiality and other protection for the identities of women, physicians, and facilities that are subjects of this research.¹⁸

Analysis

To summarize radiologists’ perceptions towards CAD and double reading, we collapsed the perception statement responses into agree (including strongly agree), neutral, or disagree (including strongly disagree). For each statement, we calculated the overall distribution of radiologists who fell into each of these three categories. To create summary measures that captured radiologists’ perceptions towards CAD and double reading, we used latent class analysis to combine these seven items into “perception classes” separately for CAD and for double reading.²⁰ Latent class analysis is an analytic approach used to explain heterogeneity in survey responses involving categorical items.²¹ A latent class is a characteristic that is not directly measured from the survey questions, but can be inferred based on other measured outcomes. For example, radiologists who agreed that CAD reassures mammographers, reassures patients, improves cancer detection rate and profitability, and protects radiologists from malpractice suits are likely to have an overall positive perception of CAD. The latent class analysis estimated overall probabilities that radiologists would belong to each of the latent favorability classes along with the probability that a random individual in a given latent class would have a given response to each perception statement.

Latent class analysis identifies classes that would represent probabilities of radiologists’ overall perceptions of CAD or double reading based on responses to the seven individual perception statements. Using this technique, the number of classes to be used is empirically determined by examining internal validity of response probabilities within classes. We found that a four class latent class model adequately fit the observed response patterns for both CAD and double reading based on a goodness of fit test, whereas a three class model did not. Therefore, we chose to report the results of the four class model. We labeled the classes as “most favorable”, “somewhat favorable”, “neutral or somewhat unfavorable”, and “most unfavorable” perceptions of CAD and double reading. We estimated the probability that a random radiologist in the “most favorable” class “strongly agrees” that CAD reassures mammographers, and similarly for the other perception statements. In this way, we assessed how specific aspects of CAD and double reading related to the overall perception of each method in mammography interpretation. We used latent class regression to explore the associations between radiologist characteristics and the latent classes, separately for CAD and for double reading.²⁰ To assess correspondence of CAD and double reading perceptions within radiologists, we plotted the probabilities of radiologists belonging to each latent class of favorability for perception of CAD by perception of double reading.

Results

Survey respondents were predominantly male (71.6%), and the majority (86.0%) self-reported interpreting an average of more than 1,000 mammograms per year over the past 5 years (Table 1). Most participants were community-based radiologists (81.9%). Over 75% of participating radiologists used CAD for some screening mammography interpretation, whereas only 28% of radiologists used double reading and 22.7% used both. Self-reported mammography practice among the participating radiologists indicated that 41% used CAD for all of their screening mammography interpretations, while 37% used CAD for some, but not all interpretations (Figure 1). Less than 2% of radiologists self-reported using double reading for all screening mammography interpretations and almost three quarters reported no use (Figure 2).

Table 1

Characteristics of radiologists responding to perception statements about CAD and double reading.

Figure 1

Self-reported use of CAD for mammogram interpretation among U.S. community radiologists (n=257).

Figure 2

Self-reported use of double reading for mammogram interpretation among U.S. community radiologists (n=257).

Responses to the seven perception statements about CAD and double reading were given by all but five radiologists for CAD (n=252) and all but 21 for double reading (n=236). Perceptions between CAD and double reading were notably different among the participants (Table 2). The radiologists reported that CAD was less likely than double reading to reassure mammographers (65% VS. 81%), improve cancer detection rate, protect them from medical malpractice and take too much time. The radiologists reported that CAD was more likely than double reading to increase their recall rate and improve profitability of breast imaging.

Table 2

Perceptions of computer aided detection (CAD) and double reading among 257 radiologists surveyed nationally.

Latent class analyses of responses regarding CAD reveal that 36% of radiologists were classified as having most favorable perceptions, 43% had somewhat favorable perceptions, 9% were neutral or somewhat unfavorable, and 12% had most unfavorable perceptions. For double reading, 64% were classified as most favorable, 5% as somewhat favorable, 5% as neutral or somewhat unfavorable, and 28% as most unfavorable using latent class analysis. For both CAD and double reading, radiologists with the most favorable perceptions of CAD and double reading largely agreed with the statements: reassures mammographers, reassures patients, and improves cancer detection rate (Figure 3). Perception statements which were notably different between the most favorable latent analysis classes for CAD versus double reading pertained to profitability and time taken. Among those with the most favorable CAD, 50% agreed that CAD improves profitability (Figure 3a), while less than 5% agreed that double reading does (Figure 3b). Among those with the most favorable perceptions, 98% of radiologists disagreed that CAD takes too much time compared to only 15% for double reading. For double reading, agreement that it takes too much time was the unanimous response among radiologists with the most unfavorable perceptions.

Figure 3

Perception about CAD (Fig. 3a) and double reading (Fig. 3b) on individual survey questions among radiologists who were classified as most favorable vs. most unfavorable by latent class analyses.

Radiologist characteristics associated with favorable and unfavorable perceptions are outlined in Table 3. Women were less likely than men to have the most favorable perceptions towards CAD (24% vs. 43%, respectively), but were more likely have the most favorable perceptions towards double reading (79% vs. 60%, respectively). Radiologists with a primary affiliation with an academic medical center were significantly less likely to have the most favorable perceptions about CAD (17%) relative to radiologists without such an affiliation (36%) or those with an adjunct appointment (70%), but were significantly more likely to have the most favorable perceptions towards double reading (98% for primary affiliation vs. 59% for no affiliation and 69% for adjunct appointment). Interpreting ≤10% or 11–24% of screening exams as part of a radiologist’s entire workload was associated with a more favorable perception for CAD (36% and 41%, respectively) relative to those with more focus on screening mammography (23%). CAD users were between 53% and 60% likely to respond most favorably about CAD, compared to between 12% and 25% who did not use CAD (Table 3). Over 82% of double readers for screening mammograms were likely to view double reading most favorably compared to approximately 63% of those who were not double readers.

Table 3

Prevalence of latent class memberships of radiologists’ perceptions of CAD and double reading by radiologist characteristics

We assessed the correspondence of latent class perceptions of CAD with double reading perceptions for individual radiologists (Figure 4). The probability of favorably perceiving both CAD and double reading was 76.3% (combining most favorable and somewhat favorable). The probability of perceiving CAD favorably but double reading unfavorably was 6.6%. However, 15.9% perceived double reading favorably, but CAD unfavorably. The correspondence of each perception statement for CAD compared to double reading shows strongest agreement for reassuring mammographers and patients, improving cancer detection rates, and increasing recall rate (Table 3). A notable difference is evident in the perception related to interpretive time taken, with 40% disagreeing with the statement that CAD takes too much time while agreeing that double reading does. Another perception that was not congruent for CAD and double reading for individual radiologists was that the interpretive practice improves the profitability of breast imaging; almost 35% of radiologists agreed that CAD improves profitability, but disagreed that double reading improves it (Table 3).

Figure 4

Correspondence of latent class perceptions of CAD with double reading perceptions for individual radiologists (N=236) expressed as probabilities.

Discussion

This study provides one of the only reports of radiologists’ use of CAD and double reading in clinical practice in the U.S. and their perceptions related to use of either the computer or a second radiologist in their interpretive process. Most radiologists reported using CAD, compared to few radiologists who double read, despite the positive perceptions of double reading. We found that radiologists perceived double reading to improve cancer detection rate compared to CAD, but they perceived recall rates to be increased by CAD compared to double reading. While nearly half of radiologists agreed that CAD improves the profitability of breast imaging, very few agreed that double reading improves profitability. While most radiologists thought double reading takes too much time, they believe it does protect radiologists from malpractice suits.

Radiologists’ perception that CAD increases cancer detection rate and recall is likely to stem from the primary function of current-generation CAD programs, which is to provide output solely to alert radiologists about potential cancers, not reassure them by absence of CAD prompts. If used as designed, CAD is supposed to increase recall rate and biopsy rate in order to achieve increased cancer detection rate (that is, the radiologist is supposed to recall every exam judged appropriate for recall before displaying CAD marks, and then to recall additional exams when the CAD prompts are judged to be appropriate). Double reading, especially if used with consensus reading, may achieve the goal of increasing cancer detection rate without affecting recall rate, or of decreasing recall rate without affecting cancer detection rate.^22–24 Most radiologists thought CAD reassured mammographers however, the absence of CAD output could also falsely reassure radiologists. Double reading was perceived even more often than CAD to improve cancer detection rate, but was less often perceived to increase recall rate. These perceptions suggest greater confidence in colleagues’ interpretation (double reading) compared to that of a computer (CAD). The trend for increasing use of CAD rather than double reading will continue to shift clinical interpretation towards computer- rather than colleague-based aids. The view that CAD increases profitability of breast imaging may be related to the additional reimbursement provided when using CAD, and the perception of CAD not taking too much time, unlike double reading.

Radiologists’ perception of CAD and double reading is important for clinical practice at the level of individual radiologists, and also from a larger perspective related to diffusion of technology. As diffusion of CAD increases, it is important to understand radiologists’ perceptions since they are the end users of the technology, yet may not be the primary drivers of its diffusion.

Radiologists’ perception of CAD and double reading may relate to individual clinical practice in several ways. First, the perception that CAD takes less time than double reading may have negative implications if radiologists do actually shorten interpretation time of pre-CAD lesion detection (hence an over-reliance on CAD for lesion detection), because the overall performance of CAD is likely to suffer using this approach. Second, CAD use involves the same radiologist as both the first and second reader. With double reading the second interpreter is another radiologist, and may even include a third-radiologist for arbitration interpretation as in Europe. Double reading, especially with consensus may offer an important opportunity, absent from CAD, for radiologists to learn about the clinical decision making process by directly observing the practice of their peers. Finally, the most favorable perceptions of double reading were strongly related to protection from malpractice suits. Given that breast imaging is one of the highest litigation areas²⁵, this type of reassurance may be very positive for radiologists. CAD and double reading represent tradeoffs for radiologists – with increased cancer detection rate, and reassurance to radiologists and patients to be weighed against increased recall rate, time taken, and profitability

The congruence of radiologists’ perceptions with evidence from the literature is encouraging for clinical practice. The majority of studies show an increase in cancer detection rate for both CAD^3–6 and double reading.^7–11 Of note, however, is that most of the additional cancers detected with CAD were ductal carcinoma in situ (DCIS).²⁶ Some studies of double reading show no increase in recall rate, especially if consensus reading is part of the protocol.^11,27 In this study, academically-affiliated radiologists were more likely to have the most favorable overall view of double reading, but not of CAD. This may reflect the greater opportunities for collegial conferring among academicians and a lesser reliance on technologies for clinical decision making, but this deserves further investigation.

Several limitations and strengths in this study should be noted. First, the distribution of radiologists in some perception classes was small, so the effect should be considered in light of the numbers of radiologists in the classes. Second, latent classes are almost always subject to heterogeneity, so we caution against over-interpretation. Third, we did not collect information on the specific CAD system used by radiologists, so a small potential for differences in perception related to CAD system was possible. Also, we know that methods of double reading can vary, which we did not explicitly address in this study. In a study of community radiologists’ practice patterns, about half of the facilities used a double reading method which considered the second radiologist’s reading final, while the other half reported a variety of methods.²⁸ Therefore, a potential exists for perceptions we report to be based on different methods of double reading. Finally, we were not able to measure the transition of radiologists from double reading to CAD. We did, however, examine perceptions of CAD and double reading according to whether the radiologist used CAD and/or double reading, and found similar trends (data not shown). Strengths of the study include the community-based radiologist participation and geographic distribution of respondents. Also, our high response rate makes selection bias less likely to be prominent.

This survey study is important in understanding how radiologists are using technology in the U.S. and how they perceive technologies and existing protocols. The congruence of radiologists’ perceptions of CAD and double reading performance with evidence from the literature suggests appropriate dissemination of knowledge. This study also suggests that radiologists have a more positive view of obtaining help from a colleague vs. a computer algorithm, but that computer-based input may have more efficiencies and cost advantages. In the U.S., CAD has diffused rapidly, apparently without formal assessment of radiologists’ perceptions, at the same time that double reading has declined. Ultimately, perceptions of these two interpretive practices may influence how they are used, more than whether they are used. An important next step is to more fully isolate the clinical decision making aspects of radiologists’ perceptions and preferences for CAD vs. double reading. One way to do this would be to elicit radiologists’ perceptions and preferences under the scenario in which reimbursement for double reading is equal to that for CAD, or a scenario in which there is no reimbursement for either practice. Understanding radiologists’ views of CAD and double reading helps to make the tradeoffs more apparent and informs clinic practice patterns.

Acknowledgments

This work was supported by the National Cancer Institute [1R01 CA107623, 1K05 CA104699; Breast Cancer Surveillance Consortium (BCSC): U01CA63740, U01CA86076, U01CA86082, U01CA63736, U01CA70013, U01CA69976, U01CA63731, U01CA70040], the Agency for Healthcare Research and Quality (1R01 CA107623), the Breast Cancer Stamp Fund, and the American Cancer Society, made possible by a generous donation from the Longaberger Company’s Horizon of Hope Campaign (SIRGS-07-271-01, SIRGS-07-272-01, SIRGS-07-273-01, SIRGS-07-274-01, SIRGS-07-275-01, SIRGS-06-281-01). The authors had full responsibility in the design of the study, the collection of the data, the analysis and interpretation of the data, the decision to submit the manuscript for publication, and the writing of the manuscript. We thank the participating women, mammography facilities, and radiologists for the data they have provided for this study. A list of the BCSC investigators and procedures for requesting BCSC data for research purposes are provided at: http://breastscreening.cancer.gov/ <http://breastscreening.cancer.gov/>.

Footnotes

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

1. Birdwell RL, Bandodkar P, Ikeda DM. Computer-aided detection with screening mammography in a university hospital setting. Radiology. 2005;236(2):451–457. [PubMed]

2. National Cancer Institute. [accessed July 2009];Computer-Aided Interpretation of Mammograms: Questions and Answers. 2007 http://www.cancer.gov/newscenter/pressreleases/CADmammographyQandA.

3. Taylor P, Potts HW. Computer aids and human second reading as interventions in screening mammography: two systematic reviews to compare effects on cancer detection and recall rate. Eur J Cancer. 2008;44(6):798–807. [PubMed]

4. Freer TW, Ulissey MJ. Screening mammography with computer-aided detection: prospective study of 12,860 patients in a community breast center. Radiology. 2001;220(3):781–6. [PubMed]

5. Houssami N, Given-Wilson R, Ciatto S. Early detection of breast cancer: overview of the evidence on computer-aided detection in mammography screening. J Med Imaging Radiat Oncol. 2009;53(2):171–6. [PubMed]

6. Morton MJ, Whaley DH, Brandt KR, Amrami KK. Screening mammograms: interpretation with computer-aided detection--prospective evaluation. Radiology. 2006;239(2):375–83. [PubMed]

7. Beam C, Hendrick R. Proposition: All mammograms should be double-read. Med Phys. 26(2):115–8. [PubMed]

8. Anttinen I, Pamilo M, Soiva M, Roiha M. Double reading of mammography screening films- one radiologist or two? Clinical Radiology. 1993;48:414–21. [PubMed]

9. Destounis SV, DiNitto P, Logan-Young W, Bonaccio E, Zuley ML, Willison KM. Can computer-aided detection with double reading of screening mammograms help decrease the false-negative rate? Initial experience Radiology. 2004;232(2):578–84. [PubMed]

10. Harvey SC, Geller B, Oppenheimer RG, Pinet M, Riddell L, Garra B. Increase in cancer detection and recall rates with independent double interpretation of screening mammography. AJR Am J Roentgenol. 2003;180(5):1461–7. [PubMed]

11. Blanks RG, Wallis MG, Moss SM. A comparison of cancer detection rates achieved by breast cancer screening programmes by number of readers, for one and two view mammography: results from the UK National Health Service breast screening programme. J Med Screen. 1998;5(4):195–201. [PubMed]

12. Thurfjell EL, Lernevall KA, Taube AAS. Benefit of independent double reading in a population-based mammography screening program. Radiology. 1994;191:241–44. [PubMed]

13. Brown J, Bryan S, Warren R. Mammography screening: an incremental cost effectiveness analysis of double versus single reading of mammograms. British Medical Journal. 1996;312:809–12. [PMC free article] [PubMed]

14. Perry N, Broeders M, Wolf Cd, Törnberg S, Holland R, Karsa Lv, et al. [accessed July 2009];European guidelines for quality assurance in breast cancer screening and diagnosis. Available from URL: Accessed at: http://ec.europa.eu/health/ph_projects/2002/cancer/fp_cancer_2002_ext_guid_01.pdf.

15. Gilbert FJ, Astley SM, Gillan MG, Agbaje OF, Wallis MG, James J, et al. Single reading with computer-aided detection for screening mammography. N Engl J Med. 2008;359(16):1675–84. [PubMed]

16. Hofvind S, Vacek PM, Skelly J, Weaver DL, Geller BM. Comparing screening mammography for early breast cancer detection in Vermont and Norway. J Natl Cancer Inst. 2008;100(15):1082–91. [PMC free article] [PubMed]

17. Breast Cancer Surveillance Consortium. 2009 Oct 05; accessed at: http://breastscreening.cancer.gov.

18. Ballard-Barbarsh R, Taplin SH, Yankaskas BC, Ernster VL, Rosenberg RD, Carney PA, et al. Breast Cancer Surveillance Consortium: A National Mammography Screening and Outcomes Database. American Journal of Roentgenology. 1997;169(4):1001. [PubMed]

19. Carney PA, Geller BM, Moffett H, Ganger M, Sewell M, Barlow WE, et al. Current medicolegal and confidentiality issues in large, multicenter research programs. Am J Epidemiol. 2000;152(4):371–8. [PubMed]

20. Bandeen-Roche KMD, Zeger SL, Rathouz PJ. Latent Variable Regression for Multiple Discrete Outcomes. Journal of the American Statistical Association. 1997;92:1375–86.

21. Lazersfeld PF. Latent Structure Analysis. 1. Boston: Houghton Mifflin; 1968.

22. Thurfjell EL, Lernevall KA, Taube AA. Benefit of independent double reading in a population-based mammography screening program. Radiology. 1994;191(2):241–4. [PubMed]

23. Brown J, Bryan S, Warren R. Mammography screening: an incremental cost effectiveness analysis of double versus single reading of mammograms. BMJ. 1996;312(7034):809–12. [PMC free article] [PubMed]

24. Ciatto S, Del Turco MR, Morrone D, Catarzi S, et al. Independent double reading of screening mammograms. J Med Screen. 1995;2:99–101. [PubMed]

25. Physician Insurers Association of America. Breast Cancer Study. 3. Physician Insurers Association of America; 2002.

26. Fenton JJ, Taplin SH, Carney PA, Abraham L, Sickles EA, D’Orsi C, et al. Influence of computer-aided detection on performance of screening mammography. N Engl J Med. 2007;356(14):1399–409. [PMC free article] [PubMed]

27. Gilbert FJ, Astely SM, McGee MA, Gillan MG, Boggis CR, Griffiths PM, et al. Single reading with computer-aided detection and double reading of screening mammograms in the United Kingdom National Breast Screening Program. Radiology. 2006;241(1):47–53. [PubMed]

28. Hendrick RE, Cutter GR, Berns EA, Nakano C, Egger J, Carney PA, et al. Community-based mammography practice: services, charges, and interpretation methods. Am J Roentgen. 2005;184:433–38. [PMC free article] [PubMed]