PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Acad Radiol. Author manuscript; available in PMC 2012 April 1.
Published in final edited form as:
PMCID: PMC3065970
NIHMSID: NIHMS267116

Using a Tailored Web-based Intervention to Set Goals to Reduce Unnecessary Recall

Abstract

Purpose

To examine whether an intervention strategy consisting of a tailored web-based intervention, which provides individualized audit data with peer comparisons and other data that can affect recall can assist radiologists in setting goals for reducing unnecessary recall.

Methods

In a multi-site randomized controlled study, we used a tailored web-based intervention to assess radiologists' ability to set goals to improve interpretive performance. The intervention provided peer-comparison audit data, profiled breast cancer risk in each radiologist's respective patient populations, and evaluated the possible impact of medical malpractice concerns. We calculated the percentage of radiologists who would consider changing their recall rates, and examined the specific goals they set to reduce recall rates. We describe characteristics of radiologists who developed realistic goals to reduce their recall rates, and their reactions to the importance of patient risk factors and medical malpractice concerns.

Results

Forty-one of 46 radiologists (89.1%) who started the intervention completed it. Thirty-one (72.1%) indicated they would like to change their recall rates and 30 (69.8%) entered a text response about changing their rates. Sixteen of the 30 (53.3%) radiologists who included a text response set realistic goals that would likely result in reducing unnecessary recall. The actual recall rates of those who set realistic goals were not statistically different from those who did not (13.8% vs. 15.1% respectively). The majority of selected goals involved re-reviewing cases initially interpreted as BI-RADS category 0. Over half of radiologists who commented on the influence of patient risk (56.3%) indicated that radiologists planned to pay more attention to risk factors, and 100% of participants commented on concerns radiologists have about malpractice with the primary concern (37.5%) being fear of lawsuits.

Conclusions

Interventions designed to reduce unnecessary recall can succeed in assisting radiologists to develop goals that may ultimately reduce unnecessary recall.

INTRODUCTION

The full potential of mortality reduction by mammography may be limited by the current variability in radiologists’ interpretive performance (115). Sources of variability affecting performance have included fellowship training (15), which is associated with improved performance. Other studies have shown that, for some radiologists, inflated perceptions of breast cancer risk in their patient populations (7) and inflated perceptions of their own malpractice risk (6) are associated with higher than recommended recall rates, which cause harms such as patient anxiety, and excessive false positive biopsies. The potential harms of mammography were a consideration when the U.S. Preventive Services Task Force revised its recommendation for mammography frequency among average risk women aged 50 and older (16).

Prior research suggests that educational interventions may have value, but intervention research that includes rigorous study designs have limitations. The UK National Health Program evaluated a two-week multidisciplinary course with specialist training at high-volume screening sites, which included three sessions per week of interpreting screening mammograms. Radiologists additionally attended routine breast disease-related meetings and received personal and group audit reports that included data on cancer detection rate, recall rate and positive predictive value of biopsy recommendation (PPV2) (17). Performance indices showed a reduction in the recall rate from 7 to 4%, and an increase in the small invasive cancer detection rate from 1.6/1,000 to 2.5/1,000. Unfortunately, mechanisms to track such indices in the U.S. do not typically exist. In another study, performed within a major US health maintenance organization (HMO) (18), 21 radiologists were provided with personal and group audit reports, participated in a self-assessment program, attended case review sessions, and were required to interpret 8,000 mammograms per year per radiologist. Sensitivity improved from 70% to 80%, with a mean cancer detection rate of 7.5/1,000 and a mean recall rate of 7%.

A limitation in the above research is that multiple intervention strategies, such as volume requirements, audit feedback, and case review sessions were tested, which does not allow for assessment of the impact of a single strategy. Other limitations in the HMO study include the small sample size, the lack of a randomized design, and restricted to a single healthcare setting, which reduces its generalizability.

We developed a single intervention strategy that included individualized web-based educational intervention designed to identify areas of interpretive weakness and to assist radiologists with setting goals to reduce unnecessary recall. The intervention has been tested in a randomized controlled design where follow-up data are now being collected. Here we describe characteristics of radiologists who developed realistic goals to reduce their recall rates, the goals they developed, and their reactions to the importance of patient risk factors and medical malpractice concerns when interpreting screening mammography.

METHODS

Four of seven mammography registries that are currently part of the Breast Cancer Surveillance Consortium (BCSC; http://breastscreening.cancer.gov) were included in this study: Group Health Breast Cancer Surveillance Project in Seattle, WA; the New Hampshire Mammography Network; the Vermont Breast Cancer Surveillance System; and the Carolina Mammography Registry. Patient demographic and clinical information is collected at the time mammography is performed at a participating facility. Radiologists' interpretation and follow-up recommendations according to the American College of Radiology's Breast Imaging Reporting and Data System (BI-RADS) (19) are also collected and linked to regional cancer registries and pathology databases to determine cancer outcomes. Data from the registries are annually pooled at the BCSC Statistical Coordinating Center (SCC) located in Seattle, WA for cleaning and analysis.

Each registry and the SCC received IRB approval for either active or passive consenting processes or a waiver of consent to enroll participants, link data, perform analytic studies, and for all activities related to the study described here. All procedures are Health Insurance Portability and Accountability Act (HIPAA) compliant and all registries and the SCC have received a Federal Certificate of Confidentiality and other protection for the identities of women, physicians, and facilities that are subjects of this research (20). In addition, all study-specific activities were reviewed and approved by IRBs associated with each participating site.

Performance data that automatically populated the web-based intervention were derived from the respective mammography registries. The study design included randomizing radiologists to either an early or late intervention group, with the late intervention group serving as controls for assessment of performance measures. Because the study is still in the follow-up data collection phase, data reported here do not include assessment of interpretive performance after the intervention. Thus, the intervention groups are combined in this report.

Radiologist Survey

We invited radiologists from four BCSC registries to complete a previously described mailed survey (15). The survey included questions about radiologist age, years of practice, affiliation with academic medical centers, completion of a breast imaging fellowship, estimates of annual volume of screening and diagnostic mammograms, and percentage of practice time spent in breast imaging. These data were used to characterize the radiologists, but completion of the survey was not required to participate in the web-based intervention.

Web-based Tailored Educational Intervention Data System

Radiologists eligible for the intervention were actively interpreting mammograms at a facility at one of the four participating BCSC registries between January 2006 and September 2007. Because only four of the BCSC registries were involved in the intervention portion of the study, the number of radiologists eligible to take part in the intervention was 196. Sixty-seven eligible radiologists did not complete the radiologist survey and did not consent to the intervention, leaving 129 radiologists who provided data for inclusion in this report. Of these, 55 completed the survey but did not consent to the intervention.

Each consenting radiologist (n=74) was mailed a unique log-in and password and participants were awarded two hours of Category I CME credit. Of the 74 who consented to the intervention, 64 (86.4%) completed the survey, 46 (62.2%) actually logged on to start the intervention and 41 (89.1%) of these completed it. The entire program took an average of 1 hour to complete. Among those radiologists who completed the intervention, 27 (65.9%) were randomized to the early group and 14 (34.2%) to the late group.

The intervention is described in detail elsewhere (21). Briefly, it was designed to encourage radiologists to change performance by illustrating the metrics in clinical performance that could be improved, to enable improvements by showing factors that may influence performance and then reinforce change by assisting radiologists to develop goals that would improve their performance. When a participating radiologist logged on, the system transparently combined mammography registry data provided by the BCSC SCC with content and questions from a machine-readable template prepared by our research team, to produce the final individually tailored intervention. The learning module had three components: 1) Understanding audit statistics for sensitivity, specificity, recall rates, positive predictive value, and cancer detection; 2) Misperceptions radiologists have about women’s risk of breast cancer; 3) Misperceptions about risk of medical malpractice lawsuits related to breast imaging.

Knowledge questions were imbedded into the system, which generated a data file we could use to award CME credits. Radiologists were able to insert their goals for changes they would like to make in their clinical practice, especially regarding recall rates, into a text field at the end of each module. We defined realistic goals as planned actions that, if implemented, would likely result in a change in recall rate toward alignment with national targets. Using this definition, two of the authors (PC and ES) classified each radiologist’s goals as realistic toward improving performance or not. Questions about radiologists' attitudes and perceptions of our web-based intervention program were also embedded into the system.

Data Analyses

Content analysis using the classical methods (22) was done to analyze the text responses of participating radiologists for goals they set to reduce unnecessary recall, and their reactions to the module on patient risk factors and their own risk of medical malpractice. Using a consensus approach, two observes (PAC and EAS) reviewed radiologists responses to goals participants identified as reducing unnecessary recall and classified these as being realistic or not using our agreed upon definition. Realistic goals were further classified using our study methods into four categories: 1) Change interpretive thresholds; 2) Re-review of certain images; 3) Change interpretive practices, and 4) Seek independent second review. We present the actual text for goals that were considered realistic along with radiologists' recall rates for a one-year period (2003–2004). Recall rates were determined by calculating the proportion of exams coded as BI-RADS 0, 4 or 5 or a 3 with a recommendation for immediate work-up among all exams that the radiologist indicated as screening.

We evaluated the distribution of the 41 radiologists’ demographic characteristics, practice types, breast imaging experience, and preferences/attitudes toward CME. We also assessed responses to questions about considering risk factors during mammography interpretation (based on the second module described above) pre- and post-intervention. Finally, we evaluated radiologists’ perceptions of medical malpractice on recall and breast biopsy rates pre- and post-intervention. All results were stratified by whether or not the goals developed by radiologists could realistically improve performance. We tested for statistically significant differences between the two groups using a chi-squared test and Fisher’s exact test when cell sizes were <5. All analyses were conducted using Stata, and p-values <0.05 were considered statistically significant.

RESULTS

Forty-one of 46 radiologists (89.1%) who started the intervention completed it. There were no differences between the characteristics of radiologists who completed the program compared to those who did not (data not shown). A complete assessment of characteristics of radiologists who did and did not participate in the survey is reported elsewhere (21). The demographic characteristics did not differ between radiologists who set goals to reduce their recall rates and those who did not (Table 1). The majority of participants were not affiliated with an academic medical center; only one had fellowship training in breast imaging; most had 10 or more years of experience interpreting breast imaging; and most spent at least 20% of their time in breast imaging. The actual recall rates of those who developed goals were not statistically different from those who did.

Table 1
Radiologist Characteristics* According to Whether or Not Realistic Goals Were Set to Reduce Unnecessary Recall

We also assessed participants' preferences and attitudes toward CME, and determined that the majority (87.2%) preferred instructor-led educational activities, 80% felt CME activities improve interpretive performance, and 88% indicated they would take a free CME course using the Internet (data not shown). No differences were noted for these variables according to whether or not respondents set realistic goals to improve performance.

After viewing their individual indices and the segment of the program illustrating that recall rates above 5–7% do not typically result in more cancer detection, participants were asked if they would consider changing their recall rates and 30 of 41 (73.2%) indicated they would. When asked what goals they would set to reduce their recall, 30 (73.2%) entered a text response in the space provided (Table 2). Fifteen of the 30 (50%) radiologists set realistic goals to reduce recall and one (3.3%) set a goal to increase recall. Thirty-one percent (n=5) indicated they planned to change interpretive thresholds, and 25% (n=4) indicated they would re-review of certain images. Nearly 19% (n=3) planned to change interpretive practices, and 12.5% (n=2) planned to seek an independent second review. Unrealistic goals (n=8, 57.1%) typically included a desire to reduce recall without a mechanism to do so.

Table 2
Radiologists’ Recall Rates and Realistic Goals Set as Part of the Auditing Component of the Tailored Web-based Intervention

We also assessed responses to the breast cancer risk component of the tailored intervention, according to whether or not realistic goals were set to reduce unnecessary recall. No statistically significant differences existed in how radiologists in these two groups consider risk factors when interpreting mammograms at the start of the intervention. This was true for considering age (77% consider among goal setters vs. 78% consider among non-goal setters), breast density (75% among goal setters vs. 44% among non-goal setters), family history (100% among goal setters vs. 94% among non-goal setters) and past breast procedures (58% among goal setters vs. 67% among non-goal setters) as risk factors pre-intervention (data not shown).

Post intervention, when pre-menopausal risk factors were considered, findings were similar in that we found no statistically significant differences between the two groups for the importance of a prior positive mammogram (87% among goal setters vs. 72% among non-goal setters), importance of younger age at menarche (56% among goal setters vs. 32% among non-goal setters), importance of race/ethnicity (80% among goal setters vs. 64% among non-goal setters), importance of higher breast density (100% among goal setters vs. 88% among non-goal setters), importance of family history (100% among goal setters vs. 88% among non-goal setters) and importance of prior benign breast biopsy (94% among goal setters vs. 80% among non-goal setters) as risk factors for premenopausal women. Post-intervention, when radiologists considered risk factors among postmenopausal women, they generally rated them as more important than the same risk factors for premenopausal women, though no significant differences were noted between the two groups (data not shown).

Fifteen radiologists commented on how the use of risk factors should be considered after this module was completed in response to the question: "Is there anything you would do differently as a result of what you have learned?" (Table 3). Six (19.4%) of these were radiologists who set reasonable goals for performance as part of the intervention. The majority of the comments indicated radiologists planned to pay more attention to risk factors (9/16, 56.3%). Another two participants (12.5%) planned to pay less attention to risk, three (18.8%) planned no practice change and two more (12.5%) commented to addressing risk in specific situations. Commenting on risk factors does not appear to be associated with setting specific goals for reducing unnecessary recall.

Table 3
Responses for How Use of Risk Factors May Change Recall Rates As a Result of the Intervention

Prior to the intervention and using a scale where 0=No Change, 5=Work-up Moderately Increased and 10=Work-up Greatly Increased, radiologists who set realistic goals for reducing recall indicated that medical malpractice influences both their recall rates and biopsy recommendations to a greater extent than those who did not set goals to reduce unnecessary recall (Table 4). These scores dropped by half after the intervention, and all radiologists correctly indicated that the risk of a mammography related malpractice suit occurring in the next five years was less than 10% among radiologists working full time in breast imaging. Greater than 85% of radiologists commented that at least half of radiologists are concerned that fear about medical malpractice affects how they interpret mammograms (Table 4).

Table 4
Responses to the Medical Malpractice Component of the Tailored Web-based Intervention According to Whether or Not Realistic Goals Were Set to Reduce Unnecessary Recall

All 41 radiologists who completed the intervention commented on concerns radiologists have about malpractice (Table 5). Of these, 16 (39.0%) set realistic goals, and of these 15 set goals for reducing unnecessary recall and one set a goal to increase recall. Forty of these comments fell into eight relevant categories including (one was considered not applicable): 1) fear of lawsuits 15/40 (37.5%), 2) media sensation 7/40 (17.5%), 3) media exposure 2/40 (5%), 4) media unspecified 3/40 (8%), 5) impact of hearsay 4/40 (10%), 6) fear unspecified 5/40 (12.5%), 7) fear for patients 1/40 (3%), and 8) fear of professional loss 1/40 (3%).

Table 5
Responses for How Risk of Medical Malpractice Change Recall Rates As a Result of the Intervention

DISCUSSION

This study successfully recruited radiologists with recall rates that were, on average, between two and almost three times that of national recommended targets (5 –7%) and above the upper bound of 12% for recently set criteria for recall (23). Radiologists who consented to this study had recall rates that were almost 50% higher than the benchmark recall rate for screening mammography (24). This suggests that radiologists who agreed to take part in this study may have done so to reduce what they acknowledge to be excessive recall rates. In addition, after illustrating to participants during the one-hour CME intervention that recall rates beyond seven percent are not likely to identify many more cancers, nearly seventy-two percent of participants reported considering changing clinical practice to reduce unnecessary recall. Over half of the participants who considered changing their clinical practice developed specific goals that could realistically reduce unnecessary recall. This finding is important because research has shown that an important predictor of actual behavior change is intention to change behavior (25), and successfully setting realistic goals provides such an indication. The average recall rate for those who set realistic goals was 1.3% lower compared to those who did not set reasonable goals, which suggests that those closer to the benchmark were more highly motivated to make a change in practice relative to those whose recall rates were higher.

Among those who set realistic goals, the vast majority chose to undertake a re-review of cases where they felt the likelihood of a cancer being present was very low. Some considered re-reviewing cases they called BI-RADS category "0" or asymmetries. Only a few indicated they would obtain a second opinion as a goal to reduce recall, and, in fact, one participant indicated s/he would cease the practice of asking for a second opinion as s/he felt this was increasing unnecessary recall. This is a very different approach compared to how mammography is practiced outside the US. Interpretation in the UK and elsewhere includes routine double reading, and recall rates that are much lower than in the US (26, 27).

Interestingly, radiologists had strong reactions to the malpractice module that was part of the intervention. The majority of participating radiologists reported the perception that greater than 50% of radiologists are concerned that fear about medical malpractice affects how they interpret mammograms. Before the intervention, more participating radiologists who set realistic goals to reduce unnecessary recall reported that medical malpractice influences both their recall rates and biopsy recommendations than those who did not set goals. Importantly, these scores dropped by half after the intervention, and 100% of radiologists correctly indicated that the probable risk of a mammography related malpractice suit occurring in the next five years was less than 10% among radiologists working full time in breast imaging. In addition, many text comments provided by radiologists underscored the stress that radiologists experience when thinking about malpractice.

The module on breast cancer risk appeared to provide important information to radiologists, especially how risk factors differ among pre- versus postmenopausal women. The pre-intervention questions indicated that radiologists who set realistic goals were more likely to consider breast density a risk factor than those who did not set goals. Post intervention, radiologists who set realistic goals versus those who didn't were more likely to view older age at first birth and older age at mammography among pre-menopausal women as important risk factors. When radiologists considered risk factors among postmenopausal women, they generally rated them as more important than the same risk factors for premenopausal women, which was part of the knowledge content of the module. There were no significant differences in content knowledge between those who did and those who didn't set realistic goals. This finding suggests that the intention to change practice is not necessarily influenced by changes in knowledge. It may also be, as illustrated by the text responses to this module, that radiologists differ in their use of risk factors and any changes in knowledge may not greatly influence practice.

Our prior research on the use of risk factors in interpretation (28) indicates that the majority of radiologists (63.4%) report usually or always using clinical history when interpreting screening mammography; 29.2% reported using it only when they identified an abnormality, and 7.4% reported never or rarely using it. We found that sensitivity, false-positive rate, and PPV did not significantly differ by use of clinical history, however compared to radiologists who rarely use clinical history, radiologists who usually or always use it had a higher false-positive rate associated with either younger patient age, denser breast tissue, or longer screening interval. Participation in our intervention was likely the first time that our study radiologists were presented with the specific risk factors of their patient panel in combination with breast cancer rates showing how low the risk of breast cancer is in the population they actually serve.

The strengths of this study include that we were able to develop an intervention that was automatically populated with data tailored to the each individual radiologist. Over half of the radiologists who took part in the study and who developed goals for reducing unnecessary recall identified goals likely to improve practice. A weakness is that we have not yet assessed the impact of the intervention on actual recall rates, though we plan to do this when follow-up data collection is complete. In addition, recall rates may be overestimated for this group. The definition used to determine recall, though consistent with the ACR BI-RADS manual, may misclassify some diagnostic exams as screening exams because we only used the radiologist’s indication for exams.

In conclusion, interventions designed to reduce unnecessary recall can succeed in assisting radiologists to develop goals that may ultimately reduce unnecessary recall.

Acknowledgements

This work was supported by the National Cancer Institute [1R01 CA107623, 1K05 CA104699; Breast Cancer Surveillance Consortium (BCSC): U01CA63740, U01CA86076, U01CA86082, U01CA63736, U01CA70013, U01CA69976, U01CA63731, U01CA70040], the Agency for Healthcare Research and Quality (1R01 CA107623), the Breast Cancer Stamp Fund, and the American Cancer Society, made possible by a generous donation from the Longaberger Company's Horizon of Hope Campaign (SIRGS-07-271-01, SIRGS-07-272-01, SIRGS-07-273-01, SIRGS-07-274-01, SIRGS-07-275-01, SIRGS-06-281-01).

The collection of cancer data used in this study was supported in part by several state public health departments and cancer registries throughout the United States. For a full description of these sources, please see: http://breastscreening.cancer.gov/work/acknowledgement.html <http://breastscreening.cancer.gov/work/acknowledgement.html>. The authors had full responsibility in the design of the study, the collection of the data, the analysis and interpretation of the data, the decision to submit the manuscript for publication, and the writing of the manuscript. We thank the participating women, mammography facilities, and radiologists for the data they have provided for this study. A list of the BCSC investigators and procedures for requesting BCSC data for research purposes are provided at: http://breastscreening.cancer.gov/ <http://breastscreening.cancer.gov/>.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

REFERENCES

1. Elmore JG, Wells CK, Lee CH, Howard DH, Feinstein AR. Variability in radiologists' interpretations of mammograms. N Engl J Med. 1994;331:1493–1499. [PubMed]
2. Beam CA, Layde PM, Sullivan DC. Variability in the Interpretation of screening mammograms by US radiologists. Arch Intern Med. 1996;156:209–213. [PubMed]
3. Elmore JG, Miglioretti DL, Reisch LM, et al. Screening mammograms by community radiologists: variability in false-positive rates. J Natl Cancer Inst. 2002;94:1373–1380. [PMC free article] [PubMed]
4. Smith-Bindman R, Chu P, Miglioretti DL, et al. Physician predictors of mammographic accuracy. J Natl Cancer Inst. 2005;97:358–367. [PubMed]
5. Carney PA, Elmore JG, Abraham LA, Gerrity MS, Hendrick RE, Taplin SH, Barlow WE, Cutter GR, Poplack SP, D'Orsi CJ. Radiologist Uncertainty and the Interpretation of Screening Mammography. Medical Decision Making. 2004;24(3):255–264. [PMC free article] [PubMed]
6. Elmore JG, Taplin S, Barlow WE, Cutter GR, D’Orsi CJ, Hendrick RE, Abraham LA, Fosse JS, Carney PA. Does Litigation Influence Medical Practice? The Influence of Community Radiologists’ Medical Malpractice Perceptions and Experience on Screening Mammography. Radiology. 2005;236:37–46. [PMC free article] [PubMed]
7. Egger JR, Cutter GR, Carney PA, Taplin SH, Barlow WE, Hendrick RE, D’Orsi, Fosse JS, Abraham L, Elmore JG. Mammographers’ Perception of Women’s Breast Cancer Risk. Medical Decision Making. 2005;25:283–289. [PMC free article] [PubMed]
8. Fenton JJ, Egger J, Carney PA, Cutter G, D’Orsi C, Sickles EA, Fosse J, Abraham L, Taplin SH, Barlow W, Hendrick RE, Elmore JG. Reality Check: Perceived versus Actual Performance of Community Mammographers. AJR. 2006 July;187:42–46. [PMC free article] [PubMed]
9. Miglioretti DL, Smith-Bindman R, Abraham LA, Brenner RJ, Carney PA, Aiello EJ, Buist DSM, Elmore JG. Radiologist Characteristics Associated with Interpretive Performance of Diagnostic Mammography. JNCI. 2007;99:1854–1863. [PMC free article] [PubMed]
10. Taplin SH, Abraham LA, Barlow WE, Fenton JJ, Berns EA, Carney PA, Cutter GR, Sickles EA, D'Orsi C, Elmore JG. Mammography facility characteristics associated with screening mammography interpretive accuracy. JNCI. 2008;100(12):876–887. [PubMed]
11. Geller BM, Aiello Bowles E, Sohng HY, Brenner J, Miglioretti D, Yankaskas B, Carney PA, Cutter G, Elmore JG. Radiologists’ performance and their enjoyment of interpreting screening mammography. AJR. 2009;192:361–369. [PMC free article] [PubMed]
12. Miglioretti DL, Gard CC, Carney PA, Onega T, Buist DSM, Sickles EA, Kerlikowske K, Rosenberg RD, Yankaskas BC, Geller BM, Elmore JG. When radiologists perform best: The learning curve in screening mammography interpretation. Radiology. 2009;253:632–640. [PubMed]
13. Smith-Bindman R, Chu P, Miglioretti DL, Quale C, Rosenberg RD, Cutter G, Geller B, Bacchetti P, Sickles EA, Kerlikowske K. Physician predictors of mammographic accuracy. Journal of the National Cancer Institute. 2005 Mar 2;97(5):358–367. [PubMed]
14. Barlow WE, Chi Chen, Carney PA, Taplin SH, Gutter G, Fosse J, Elmore JG. Accuracy of screening mammography interpretation by characteristics of radiologists. JNCI. 2004;96(24):1840–1850. [PMC free article] [PubMed]
15. Elmore JG, Jackson SL, Abraham L, Miglioretti DL, Carney PA, Geller BM, Yankaskas BC, Kerlikowske K, Onega T, Rosenberg RD, Sickles EA, Buist DSM. Variability in Interpretive Performance of Screening Mammography and Radiologist Characteristics Associated with Accuracy. Radiology. 2009;253:641–651. [PubMed]
17. Perry NM. Breast cancer screening--the European experience. International Journal of Fertility & Women's Medicine. 2004 Sep–Oct;49(5):228–230. [PubMed]
18. Adcock KA. Initiative to Improve Mammogram Interpretation. The Permanente Journal. 2004 Spring;Vol. 8(No. 2)
19. American College of Radiology, Breast Imaging Reporting and Data System (BI-RADS) 2004 Copyright.
20. Carney PA, Geller BM, Moffett H, Ganger M, Sewell M, Barlow WE, Taplin SH, Sisk C, Ernster VL, Wilke HA, Yankaskas B, Poplack SP, Urban N, West MM, Rosenberg RD, Michael S, Mercurio TD, Ballard-Barbash R. Current Medico-legal and Confidentiality Issues in Large Multi-center Research Programs. American Journal of Epidemiology. 2000;152(4):371–378. [PubMed]
21. Carney PA, Geller BM, Abraham L, Sickles SA, Feig SA, Brown D, Cook A, Yankaskas BC, Miglioretti DL, Elmore JG. Feasibility and Satisfaction Associated with Using a Tailored Web-based Intervention for Recalibrating Radiologists Thresholds for Conducting Additional Work-up. (In Review) [PMC free article] [PubMed]
22. Berelson BR. Content Analysis. In: Lindsey G, editor. Handbook of Social Psychology. Vol 1. Reading, MA: Addison-Welsley; 1954.
23. Carney PA, Sickles E, Monsees B, Bassett L, Brenner J, Rosenberg R, Feig S, Browning S, Tran K, Berry J, Kelly M, Miglioretti DL. Identifying Minimally Acceptable Interpretive Performance Criteria for Screening Mammography. Radiology. 2010;255(2):354–361. [PubMed]
24. Rosenberg RD, Yankaskas BC, Abraham L, Sickles EA, Lehman C, Geller BM, Carney PA, Kerlikowske K, Buist DSM, Weaver DL, Barlow WE, Ballard-Barbash R. Performance Benchmarks for Screening Mammography. Radiology. 2006;241(1):55–66. [PubMed]
25. Wakefield J, Herbert CP, Maclure M, Dormuth C, Wright JM, Legare J, Brett-MacLean P, Premi J. Commitment to change statements can predict actual change in practice. Journal of Continuing Education in the Health Professions. 2003;23(2):81–92. [PubMed]
26. Smith-Bindman R, Chu PW, Miglioretti DL, Sickles EA, Blanks R, Ballard-Barbash R, Bobo JK, Lee NC, Wallis M, Patnick J, Kerlikowske K. Comparison of screening mammography in the United States and the United kingdom. JAMA. 2003;290(16):2129–2137. [PubMed]
27. Hofvind S, Vacek PM, Skelly J, Weaver D, Geller BM. Comparing screening mammography for early breast cancer detection in Vermont and Norway. J Natl Cancer Inst. 2008 Aug 6;100(15):1082–1091. [PMC free article] [PubMed]
28. Carney PA, Cook AJ, Miglioretti DL, Feig SA, Bowles EA, Geller BM, Kerlikowske K, Kettler M, Onega T, Elmore JG. Radiologists’ Use of Clinical History and it’s Impact on Interpretive Performance. (In Review)