Feasibility and Satisfaction with a Tailored Web-based Audit Intervention for Recalibrating Radiologists’ Thresholds for Conducting Additional Work-up

doi:10.1016/j.acra.2010.10.011

Acad Radiol. Author manuscript; available in PMC 2012 March 1.

Published in final edited form as:

Acad Radiol. 2011 March; 18(3): 369–376.

Published online 2010 December 30. doi: 10.1016/j.acra.2010.10.011

PMCID: PMC3034778

NIHMSID: NIHMS251542

Feasibility and Satisfaction with a Tailored Web-based Audit Intervention for Recalibrating Radiologists’ Thresholds for Conducting Additional Work-up

Patricia A. Carney, PhD,¹ Berta M. Geller, EdD,² Edward A. Sickles, MD,³ Diana L. Miglioretti, PhD,⁴ Erin J. Aiello Bowles, MPH,⁵ Linn Abraham, MS,⁵ Stephen A. Feig, MD,⁶ David Brown,⁷ Andrea J. Cook, PhD,⁴ Bonnie C. Yankaskas, PhD,⁸ and Joann G. Elmore, MD, MPH⁹

¹Departments of Family Medicine and Public Health and Preventive Medicine, Oregon Health & Science University, Portland, OR

²Departments of Family Medicine and Radiology, Office of Health Promotion Research, University of Vermont, Burlington, VT

³Department of Radiology, University of California, San Francisco, San Francisco, CA

⁴Group Health Research Institute, Biostatistics Unit and University of Washington Department of Biostatistics, Seattle, WA

⁵Group Health Research Institute, Seattle, WA

⁶Department of Radiological Sciences, University of California, Irvine, Irvine, CA

⁷Department of Family Medicine, Oregon Health & Science University, Portland, OR

⁸Department of Radiology, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina

⁹Department of Internal Medicine, University of Washington, Seattle, WA

Correspondence to: Patricia A. Carney, PhD, Professor of Family Medicine, and Public Health and Preventive Medicine - MC-FM, 3181 SW Sam Jackson Park Rd., Oregon Health & Science University, Portland, OR 97239-3098, Telephone: 503-494-9049, Fax: 503-494-2746, Email: carneyp/at/ohsu.edu

The publisher's final edited version of this article is available at Acad Radiol

See other articles in PMC that cite the published article.

Publisher's Disclaimer

Abstract

Purpose

To examine the feasibility of and satisfaction with a tailored web-based intervention designed to decrease radiologists' recommendation of inappropriate additional work-up following a screening mammogram.

Methods

We developed a web-based educational intervention designed to reduce inappropriate recall. Radiologists were randomly assigned to participate in an early intervention group or a late (control) intervention group, the latter of which served as a control for a nine-month follow-up period, after which they were invited to participate in the intervention. Intervention content was derived from our prior research and included three modules: 1) an introduction to audit statistics for mammography performance; 2) a review of data showing radiologists' inflated perceptions of medical malpractice risks related to breast imaging, and 3) a review of data on breast cancer risk among women seen in their practices. Embedded within the intervention were individualized audit data for each participating radiologists obtained from the national Breast Cancer Surveillance Consortium.

Results

Seventy-four radiologists (37.8%; 74/196) consented to the intervention, which was completed by 67.5% (27/40) of those randomized to the early intervention group and 41.2% (14/34) of those randomized to the late (control) group. Thus, a total of 41 (55%) completed the intervention. On average, three log-ins were used to complete the program (range 1–14), which took approximately 1 hour. Ninety-five percent found the program moderately to very helpful in understanding how to calculate basic performance measures. Ninety-three percent found viewing their own performance measures moderately to very helpful, and 83% reported it being moderately to very important to learn that the breast cancer risk in their screening population program was lower than perceived. The percentage of radiologists who reported that the risk of medical malpractice influences their recall rates dropped from 36.3% pre-intervention to 17.8% post-intervention with a similar drop in perceived influence of malpractice risk on their recommendations for breast biopsy (36.4 to 17.3%). Over 75% of radiologists answered the post intervention knowledge questions correctly, and the percent of time spent in breast imaging did not appear to influence responses. The majority (>92%) of participants correctly responded that the target recall rate in the U.S. is 9%. The mean self-reported recall rates were 13.0 for radiologists spending<40% time in breast imaging and 14.9% for those spending > 40% time spent in breast imaging, which was highly correlated with their actual recall rates (0.991; p<0.001).

Conclusions

Radiologists who begin an internet-based tailored intervention designed to help reduce unnecessary recall will likely complete it, though only 55% who consented to the study actually undertook the intervention. Participants found the program useful in helping them understand why their recall rates may be elevated.

INTRODUCTION

Several studies have shown that variability in recall rates in mammography screening is extensive (7–17%), even among high volume readers (1). Similarly, recall rates in the United States (U.S.) are nearly twice as high as those reported in other countries with comparable cancer detection rates. (2, 3). Studies have not fully explained the reasons for high recall rates in the U.S. (4–10. However, the 1992 Mammography Quality Standards Act (MQSA) sought to improve the practice of mammography in the U.S. (11, 12) using a required system to track outcomes associated with mammograms that resulted in a recommendation for biopsy. MQSA requires that each facility track pathology outcomes when biopsy is recommended on the basis of mammography but does not require the tracking of clinical outcomes when additional imaging is recommended (11). Rather, in the U.S., approaches used to implement tracking systems and reviews of additional audit data are left to the discretion of each mammography facility (13, 14).

Very little research has focused on whether auditing systems alone affect recall rates or improve performance. A recent observational study (15) found that among 255 radiologists across the U.S. who completed a survey about medical audits, 91% reported receiving individualized audit reports (in a paper format), which were provided by mammography registries participating in the National Cancer Institute funded Breast Cancer Surveillance Consortium (16). In addition, of these 255 radiologists, 83% reported paying close attention to their audit numbers, 87% found the reports valuable, and 75% felt that audit reports prompted them to improve interpretative performance. However, how radiologists use audit data to determine how their performance can be improved is not known.

The use of Internet-based learning interventions in the health professions (17), especially those that focus on radiologists' perceptual processes, is gaining momentum (18). Compared with passive paper-based learning systems, interactive Internet learning systems allow data manipulation approaches that encourage the radiologists to understand and improve their performance.

We conducted a study to determine the feasibility of and satisfaction with a Continuing Medical Education (CME) accredited tailored interactive web-based intervention that included peer comparison audit data on sensitivity, specificity, positive predictive value (PPV), recall and cancer detection rates. The intervention also allowed radiologists to identify reasons for their own excessive recall rates, such as elevated perceptions of breast cancer risk in their screening population (19) and medical malpractice concerns (20). Undertaking such educational exercises could lead to positive changes in the participating radiologists’ performance by altering thresholds to recall screening patients for subsequent diagnostic work-up. We also examined whether radiologists uptake of the intervention varied according to time spent working in breast imaging.

METHODS

This study involved four of seven mammography registries that are part of the National Cancer Institute-funded Breast Cancer Surveillance Consortium (BCSC; http://breastscreening.cancer.gov), including the Group Health Breast Cancer Surveillance Project in Seattle, WA, the New Hampshire Mammography Network, the Vermont Breast Cancer Surveillance System, and the Carolina Mammography Registry. These registries collect patient demographic and clinical information each time a woman receives a mammogram at a participating facility, including radiologists' interpretation and follow-up recommendations according to the American College of Radiology's Breast Imaging Reporting and Data System (BI-RADS) (21). This information is linked to regional cancer registries and pathology databases to determine cancer outcomes. Data from the registries are annually pooled at the BCSC Statistical Coordinating Center (SCC) in Seattle, WA for cleaning and analysis.

Each registry and the SCC received IRB approval for either active or passive consenting processes or a waiver of consent to enroll participants, link data, perform analytic studies, and for all activities related to the study described here. Participating radiologists also provided informed consent for the intervention study described here. All procedures are Health Insurance Portability and Accountability Act (HIPAA) compliant and all registries and the SCC have received a Federal Certificate of Confidentiality and other protection for the identities of women, physicians, and facilities that are subjects of this research (22). Performance data that automatically populated the web-based intervention were derived from the respective mammography registries.

Radiologist Eligibility and Survey Data Collection

Our study involved a mailed survey (23) completed by consenting radiologists who interpreted screening and diagnostic mammograms at a facility participating in the four BCSC registries. Details of survey development, data collection and quality assessment have been reported previously (15). Survey questions included radiologist age, years of practice, affiliation with an academic medical center, completion of a breast imaging fellowship, self-reported estimates of annual volume of screening and diagnostic mammograms, and percentage of practice time spent in breast imaging. Survey data allowed us to compare characteristics of those who agreed to participate to those who did not. Radiologists eligible for the intervention were actively interpreting mammograms at a facility at any of the four participating BCSC registries between January 2006 and September 2007 (n=196). Radiologists were invited to participate in the CME regardless of their participation in the mailed survey.

Web-based Tailored Educational Intervention Data System

The intervention was designed to illustrate gaps in clinical performance that could be improved, enable improvements by identifying factors that may influence performance and reinforce change by assisting radiologists in developing goals to improve their performance. The interactive nature of our intervention was based primarily upon adult learning theory (24, 25), which posits that adults will: 1) commit to learning when the goals and objectives are considered realistic and important to them; 2) resist learning activities they perceive as an attack on their competence, thus participants need some control over content, method, time and location; 3) need to see that learning is related and relevant to their day-to-day activities; 4) require learning that is structured to allow peer support and reduce the fear of judgment; 5) need to receive feedback on their results; 6) approach learning with a wide range of experiences, knowledge, self-direction, interests, and competencies; and 7) need facilitated transfer of learning, as it is not automatic (25).

We designed and implemented a software system using the Ruby on Rails (26) web development framework, a relational database system, and a specialized extensible markup language (XML) (27) schema to securely present a tailored educational intervention over the web. When a participating radiologist logged on, the system transparently combined mammography data provided by the BCSC with content and questions from a single XML-encoded intervention “template” prepared by our research team to produce the final individually tailored intervention. This system was hosted and monitored by the Oregon Health & Science University’s Advanced Computing Center; all data sent or received by the system was password protected, and secured in transit using 128-bit Transport Layer Security (TLS).

The intervention had three components, the content of each derived from our prior research (8, 9, 19). The first module addressed understanding audit statistics and how they are derived, because our prior research suggested that many radiologists have incorrect perceptions about their own performance, even though these data are provided to them annually (9, 15). The second module addressed radiologists' misperceptions about women’s risk of breast cancer (19), and the third module addressed radiologists' misperceptions about malpractice related to breast imaging (8, 28). Radiologists could click on links embedded in the intervention to read literature related to performance benchmarks for screening and diagnostic mammography as well as relevant published papers from our prior research. The content of each module was initially developed by three authors (PAC, BMG, and JGE) and reviewed by our expert radiologist for the study (EAS). After initial development and revision, the content was pilot tested with several academic and community-based radiologists in regions outside the participating BCSC registries. Knowledge questions were imbedded into the system, which generated a data file we could use to award continuing medical education (CME) credits. Questions about radiologists' attitudes and perceptions of our web-based intervention program were also embedded into the system.

We randomly assigned the consenting radiologists (n=74) to participate in an early intervention group (n=40) or a late intervention group (n=34). The late intervention group served as a control group for a nine-month follow-up period, after which they were also invited to receive the educational intervention. Each consenting radiologist was mailed a unique log-in and password. Participants were awarded up to two hours of Category I CME for participating. The entire program (all three modules combined) took on average about 1 hour to complete. To examine whether responses to the intervention differed by the amount of time the radiologists spend in breast imaging, we stratified responses according to whether they spent more or less than 40% of their time in breast imaging.

Data Analyses

The analyses reported here examined feasibility of and satisfaction with the intervention. We compared characteristics among radiologists who completed the CME program, radiologists who consented to the CME but did not complete the CME program, and radiologists who did not consent. These characteristics were collected on the previously mailed survey; thus we had to exclude radiologists who did not complete the survey from the analyses. We additionally stratified findings according to radiologists’ percent time in breast imaging to determine if uptake of the intervention was influenced by interest in breast imaging. We used chi-squared tests to calculate p-values for differences between the three groups.

Among those radiologists who completed the CME, we evaluated the number of log-ins, time to completion, satisfaction with the CME, and usefulness of CME features (e.g. “How helpful was the 2 × 2 table in helping you understand recall and biopsy yield?”). We compared these outcomes with the self-reported average amount of time radiologists spend in breast imaging (<40% vs. ≥40%). We chose this stratification because it represents two full clinical days per week, indicating more than occasional interpretation of mammography. Finally, we compared radiologists’ knowledge about recall rate and breast cancer risk stratified by the self-reported amount of time they spend in breast imaging. All analyses were unadjusted and conducted using Stata SE 10.1 (StataCorp, College Station, TX). All p-values are two-sided and statistical significance was defined at the 0.05 alpha-level.

RESULTS

Radiologists' screening mammography recall rates between 2003 and 2004 at each of the four participating registries ranged from 13% to 16.5% with a mean of 14.5. Recall rates for the same time period restricted to radiologists eligible for the CME intervention ranged from 11.4% to 14.3% with a mean of 13.2%. This is slightly lower than those of eligible and non-eligible combined, but still indicating room for improvement, since the target recall rate in the US is 9% or less (29). However, even with a mean recall rate of 9%, 50% of radiologists in this study would have recall rates in excess of the recommended value.

Among the four BCSC registries involved in the intervention, the number of eligible radiologists was 196 (Figure 1). One hundred and twenty-two did not consent to the intervention. Of the 74 who consented to the intervention, 46 (62.2%) actually logged on to start the intervention and 41 (89.1%) of these completed it. Among those radiologists who completed the intervention, 27 (65.9%) were randomized to the early group and 14 (34.2%) to the late (control) group.

Figure 1

The only statistical difference noted among radiologists who did and did not consent to participate in the study and then did and did not complete the program was gender (Table1). Female radiologists were more likely to complete the program after consenting than male radiologists. No differences in practice type, breast-imaging experience or preferences toward continuing medical education were noted among eligible radiologists regardless of completing the educational intervention. No differences were noted among radiologists randomized to the early versus late (control) intervention groups for demographic and practice characteristics (data not shown). Most radiologists (87.2%) who completed the intervention reported preferring instructor-led activities, such as lectures or instructor-led conferences versus self-directed activities (33.3%), like reading professional journal articles with CME exercises (These two CME preference questions did not generate mutually exclusive categorical data; thus, responses totaled are greater than 100%). This finding was similar regardless of whether they consented and did not undertake the program or did not consent (Table 1). Greater than 75% of eligible radiologists believe that CME improves interpretive performance, and more than 80% reported they would be interested in a free Category 1 CME program designed to use audit reports to help improve mammography interpretation. Over 85% of eligible radiologists reported they would take a free CME course over the Internet. Thus, interest was high, especially among those who consented to the program but did not complete it.

Table 1

Radiologist Characteristics According to Intervention Status

We noted that both completion factors and satisfaction with the program were similar among those who spend <40% of their time in breast imaging compared with those who spend 40% or more (Table 2). Typically, three log-ins were needed to complete the program (range 1–14), which took approximately 1 hour. Greater than 90% found the program moderately to very helpful in understanding how to calculate basic audit data and a similar percent found the use of a 2 × 2 table moderately to very helpful in understanding recall and biopsy yield (Table 2). Thirty-seven radiologists (92.5%) found seeing their own data displayed in the intervention moderately to very helpful, and 33 (82.5%) reported it being moderately to very important to learn that the breast cancer risk in their patient population is small. No differences were noted among radiologists randomized to the early versus late (control) intervention groups for satisfaction variables (data not shown), though consenting radiologists in the early group were more likely to complete intervention compared to those in the late (control) group.

Table 2

Completion Features and Satisfaction with Web-based Intervention by percent of time spent in breast imaging

Our module on medical malpractice resulted in a decrease in the percentage of radiologists who perceived that having higher recall rates would reduce their risk of medical malpractice from 36.3% pre-intervention to 17.8% post intervention. There was a similar drop in the perception that increased recommendations for breast biopsy would also reduce the risk of medical malpractice (36.3 to 17.3%) (data not shown). We also asked a post-test question regarding the likelihood that a recall rate above 10% would reduce the risk of medical malpractice and 100% of participants indicated, "Not at all" (data not shown).

Seventy-six percent or more of the knowledge questions related to the intervention were answered correctly, and the percent of time spent in breast imaging did not influence responses to the intervention (Table 3). Interestingly, the majority of participants (>92%) responded that the target recall rate was 9% but reported their own mean recall rate was 14.0%. Actual recall rates were lower among those spending ≥ 40% time in breast imaging (13.0 vs. 14.9, p<0.001). Greater than 80% reported correctly that the optimal range of recall (where the plateau exists on a graph displaying cancer detection rates as a function of recall rates) was between 5 – 9%. Ninety percent of respondents indicated their general understanding of breast cancer risk modestly or greatly improved as a result of this intervention. Only one difference was noted between the early and late (control) intervention groups for content knowledge. Self-reported recall rates were much higher in the late (control) intervention group compared to the early group (mean 18.2 vs. 11.8, p=0.015).

Table 3

Knowledge Assessment Associated with Web-based Intervention Content

DISCUSSION

Among eligible radiologists who agreed to participate in this study, about two thirds started the web-based educational intervention and 89% (41/46) of those who started completed it. This suggests that once engaged, the intervention was meaningful to participants. When we examined the characteristics of radiologists who consented to the program versus those who did not consent, we learned that the only difference was gender. Female radiologists were more likely than males to complete the program after consenting. While we learned that it is feasible to develop and implement an educational intervention using a large data repository to individually tailor the intervention to practicing radiologists who are part of the BCSC, our ability to recruit and actively engage radiologist in this activity was disappointing. We know that radiologists are very busy clinicians and perhaps it is too difficult to identify even small sessions of protected time to undertake such an intervention. However, we believe that an interactive intervention is likely to be more effective than paper audit reports.

At the BCSC, and likely elsewhere, these audit reports are provided to either individual radiologists or to the lead radiologist in charge of MQSA audits. Although he/she is responsible for sharing individual data with radiologists interpreting mammography, it is unclear how this process occurs in actual practice. For example, does the radiologist facilitate any discussion about how radiologists might bring their performance data closer to the desirable goals published by the Agency for Health Care Policy and Research and the benchmarks reported by the BCSC (29, 30).

We learned that the majority of participants logged on three to four times to finish the program, so they did not finish in a single session, even though the program took only about one hour to complete. It may be that radiologists completed the program in between interpreting images during their clinical day, resulting in interruptions to the intervention. We learned that the vast majority of radiologists found the audit module activities to be moderately to very helpful, especially when comparing their own data with that of other radiologists in the study. Participants also found it helpful to see that the risk of cancer in their patient panel is very low. This is likely the first time radiologists saw the level of breast cancer risk in their screening population, which was calculated using a risk model developed by BCSC investigators (31) for risk of breast cancer in the next year rather than five years. This risk information is highly relevant to radiologists' assessment of their own annual or biennial screening mammography data.

We were also interested in participants' responses to the malpractice module and their perceptions of the influence of malpractice on their recall and biopsy recommendation rates. Importantly, radiologists' perceptions of the relationship between risk of medical malpractice on their recall rates and breast biopsy recommendations was reduced by half and all participants agreed at the end of the intervention that a recall rate above 10% would not reduce their risk of malpractice. Thus, their perceptions about malpractice have been changed such that they are less likely to affect recall. Also, our knowledge questions clearly indicated that radiologists recognized that their self-reported recall rates (13.0%–14.9%) were substantially higher than the mean recall rate in the U.S. of 9.8%, and that the majority could cite the 5–7% plateau above which additional recall does not result in increased cancer detection. It will be interesting to see if their recall rates change in clinical practice as a result of this intervention. We are now collecting follow-up data to assess this measure between the early and late (control) intervention groups.

The intervention we studied illustrated, for each individual radiologist, his/her performance gap (Module 1) identified modifiable factors that might affect performance and then asked radiologists to develop individualized goals for improvement (Modules 1–3). Our study was innovative in that it was provided over the Internet and when each radiologist logged onto the system, the intervention automatically populated all data fields with his/her clinical performance data and provided peer comparison data to the aggregate of all radiologists. This could not be accomplished without a data collection and data cleaning system that could ascertain and prepare data for interpretation and facilitate each radiologist's use of the data for quality improvement purposes. Though many mammography management software systems are able to calculate audit measures, their data capture is often not complete enough to calculate performance indices with the precision we have using BCSC data, and they cannot provide the performance comparisons the BCSC provides.

The strengths of our study include the participation of radiologists from four states in the U.S., our ability to provide actual clinical performance data in real time, and our linkage to standardized data collected over many years prior to the intervention to assess individual audit data. We could also help the radiologists understand why their recall rates may be unduly high because of perceived medical malpractice risk or inflated perceptions of cancer risk in their patient populations. Weaknesses of the study are that this may not be a representative sample of radiologists across the U.S.. The findings could also be affected by selection bias, though our assessment of the characteristics of those who did and did not consent was reassuring. Though the intervention was designed to address adult learning principles, some aspects of adult learning theory could not be accommodated in our intervention. Such principles include coaching, other kinds of follow-up support and small-group activities, and evaluation that foster sharing and reflection (24), which were beyond the scope of this study. We also have yet to conduct an analysis of the impact of this intervention on clinical performance, though we plan to do this after the follow-up data are collected and analyzed. Lastly, we did not have a control group for some of the knowledge questions that were administered at the post-test period only.

In conclusion, radiologists who begin an internet-based tailored intervention designed to help reduce unnecessary recall in mammography will likely complete it, though only about half who consented to the study actually completed the intervention. Greater than 90% of participants found the intervention useful in helping them understand why their recall rates may be elevated. More research needs to be done to understand how best to engage radiologists in undertaking educational programs on the Internet.

Footnotes

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

REFERENCES

1. Gur D, Sumkin JH, Hardesty LA, Clearfield RJ, Cohen CS, Ganott MA, Hakim CM, Harris KM, Poller WR, Shah R, Wallace LP, Rockette HE. Recall and detection rates in screening mammography: A review of clinical experience: Implications for practice guidelines. Cancer. 2004;100(8):1590–1594. [PubMed]

2. Smith-Bindman R, Chu PW, Miglioretti DL, Sickles EA, Blanks R, Ballard-Barbash R, Bobo JK, Lee NC, Wallis MG, Patnick J, Kerlikowske K. Comparison of Screening Mammography in the United States and the United Kingdom. JAMA. 2003;290:2129–2137. [PubMed]

3. Hofvind S, Vacek PM, Skelly J, Weaver D, Geller BM. Comparing screening mammography for early breast cancer detection in Vermont and Norway. J Natl Cancer Inst. 2008 Aug 6;100(15):1082–1091. [PMC free article] [PubMed]

4. Carney PA, Miglioretti DL, Yankaskas BC, Kerlikowske K, Rosenberg R, Rutter C, Geller BM, Abraham LA, Dignan M, Cutter G, Ballard-Barbash R. Individual and Combined Effects of Breast Density, Age, and Hormone Replacement Therapy Use on the Accuracy of Screening Mammography. Annals of Internal Medicine. 2003;138(3):168–175. [PubMed]

5. Elmore JG, Carney PA, Abraham LA, Barlow WE, Egger JR, Fosse JS, Cutter GR, Hendrick E, D'Orsi CJ, Paliwal P, Taplin SH. The Association Between Obesity and Screening Mammography Accuracy. Archives of Internal Medicine. 2004;164:1140–1147. [PMC free article] [PubMed]

6. Carney PA, Kasales CJ, Tosteson ANA, Weiss JE, Goodrich ME, Poplack SP, Wells WS, Titus-Ernstoff L. Types of Additional Work-up Among Women Undergoing Routine Screening Mammography: The Impact of Age, Breast Density and Hormone Therapy Use. Preventive Medicine. 2004;39:48–55. [PubMed]

7. Barlow WE, Chi chen, Carney PA, Taplin SH, Gutter G, Fosse J, Elmore JG. Accuracy of screening mammography interpretation by characteristics of radiologists. JNCI. 2004;96(24):1840–1850. [PMC free article] [PubMed]

8. Elmore JG, Taplin S, Barlow WE, Cutter GR, D’Orsi CJ, Hendrick RE, Abraham LA, Fosse JS, Carney PA. Does Litigation Influence Medical Practice? The Influence of Community Radiologists’ Medical Malpractice Perceptions and Experience on Screening Mammography. Radiology. 2005;236:37–46. [PMC free article] [PubMed]

9. Fenton JJ, Egger J, Carney PA, Cutter G, D’Orsi C, Sickles EA, Fosse J, Abraham L, Talpin SH, Barlow W, Hendrick RE, Elmore JG. Reality Check: Perceived versus Actual Performance of Community Mammographers. AJR. 2006 July;187:42–46. [PMC free article] [PubMed]

10. Carney PA, Abraham LA, Miglioretti DL, Yarbroff R, Sickles EA, Kasales CJ, Geller BM, Yankaskas BC, Rosenberg RD, Buist DSM, Dignan MB, Kerlikowske K. Factors Associated with Imaging and Procedural Events Used to Detect Breast Cancer Following Screening Mammography. AJR. 2007;188:385–392. [PubMed]

11. U.S. Food & Drug Administration. Mammography Quality Standards Act Regulations. [Accessed March 12, 2009]. http://www.fda.gov/cdrh/mammography/frmamcom2.html.

12. Birdwell RL, Wilcox PA. The mammography quality standards act: benefits and burdens. Breast Dis. 2001;13:97–107. [PubMed]

13. Monsees BS. The Mammography Quality Standards Act. An overview of the regulations and guidance. Radiol Clin North Am. 2000 Jul;38(4):759–772. [PubMed]

14. U.S. Food & Drug Administration. Medical Outcomes Audit General Requirement. [Accessed May 4, 2009]. http://www.fda.gov/CDRH/mammography/robohelp/med_outcomes_audit_gen_req.htm.

15. Elmore JE, Aiello-Bowles E, Geller BM, Oster N, Carney PA, Miglioretti DL, Buist DSM, Kerlikowske K, Sickles EA, Onega T, Rosenburg RD, Yankaskas BC. Radiologists’ Attitudes and Use of Mammography Audit Reports. Academic Radiology. (In Press) [PMC free article] [PubMed]

16. Ballard-Barbash R, Taplin SH, Yankaskas BC, Ernster VL, Rosenberg RD, Carney PA, Barlow WE, Geller BM, Kerlikowske K, Edwards BK, Lynch CF, Urban N, Chrvala CA, Key CR, Poplack SP, Worden JK, Kessler LG. Breast Cancer Surveillance Consortium: A national mammography screening and outcomes database. American Journal of Roentgenology. 1997;169:1001–1008. [PubMed]

17. Cook DA, Levinson AL, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based learning in the health professions. JAMA. 2008;300(10):1181–1196. [PubMed]

18. Taylor PM. A review of research into the development of radiologic expertise: Implications for computer-based training. Acad Radiol. 2007;14(10):1252–1263. [PubMed]

19. Egger JR, Cutter GR, Carney PA, Taplin SH, Barlow WE, Hendrick RE, D’Orsi, Fosse JS, Abraham L, Elmore JG. Mammographers’ Perception of Women’s Breast Cancer Risk. Medical Decision Making. 2005;25:283–289. [PMC free article] [PubMed]

20. Elmore JG, Taplin S, Barlow WE, Cutter GR, D’Orsi CJ, Hendrick RE, Abraham LA, Fosse JS, Carney PA. Does Litigation Influence Medical Practice? The Influence of Community Radiologists’ Medical Malpractice Perceptions and Experience on Screening Mammography. Radiology. 2005;236:37–46. [PMC free article] [PubMed]

22. Carney PA, Geller BM, Moffett H, Ganger M, Sewell M, Barlow WE, Taplin SH, Sisk C, Ernster VL, Wilke HA, Yankaskas B, Poplack SP, Urban N, West MM, Rosenberg RD, Michael S, Mercurio TD, Ballard-Barbash R. Current Medico-legal and Confidentiality Issues in Large Multi-center Research Programs. American Journal of Epidemiology. 2000;152(4):371–378. [PubMed]

23. Elmore JG, Jackson SL, Abraham L, Miglioretti DL, Carney PA, Geller BM, Yankaskas BC, Kerlikowske K, Onega T, Rosenberg RD, Sickles EA, Buist DSM. Variability in Interpretive Performance of Screening Mammography and Radiologist Characteristics Associated with Accuracy. Radiology. 2009;253:641–651. [PubMed]

24. Laidley TL, Braddock CH. Role of adult learning theory in evaluating and designing strategies for teaching residents in ambulatory settings. Advances in Health Sciences Education. 2000;5:43–54. [PubMed]

25. Speck M. Best practice in professional development for sustained educational change. Spring Publishers, ERS Spectrum. 1996:33–41.

26. Ruby on Rails: David Heinemeier Hansson et. al. – http://rubyonrails.org/; Ruby Programming Language: Yukihiro Matsumoto et. al. – http://ruby-lang.org/ **)

27. XML Extensible Markup Language – http://www.w3.org/XML.

28. Dick JF, Yi JP, Gallagher T, Brenner RJ, Yi JP, Reisch-Osterman L, Abraham LA, Miglioretti DL, Carney PA, Cutter G, Elmore JG. Predictors of radiologists’ perceived risk of malpractice lawsuits in breast imaging. AJR. 2009;192:327–333. [PMC free article] [PubMed]

29. Rosenberg RD, Yankaskas BC, Abraham L, Sickles EA, Lehman C, Geller BM, Carney PA, Kerlikowske K, Buist DSM, Weaver DL, Barlow WE, Ballard-Barbash R. Performance Benchmarks for Screening Mammography. Radiology. 2006;241(1):55–66. [PubMed]

30. Bassett LW, Hendrick RE, Bassford TL, et al. Quality Determinants of Mammography: Clinical Practice Guideline No. 13. Rockville, MD: Agency for Health Care Policy and Research, Public Health Service, U.S. Department of Health and Human Services; 1994. Oct, AHCPR Publication No. 95-0632.

31. Barlow WE, White E, Ballard-Barbash R, Vacek PM, Titus-Ernstoff L, Carney PA, Tice JA, Buist D, Geller BM, Rosenberg R, Yankaskas BC, Kerlikowske K. Prospective breast cancer risk prediction model for women undergoing screening mammography. JNCI. 2006;98(17):1204–1214. [PubMed]