The findings from our study suggest that most radiologists participating in the BCSC receive mammography audit reports, review them, consider them valuable, and are prompted by the reports to review missed cancers. The audit reports seen by radiologists in this national consortium varied in content and style and in the use of figures and graphs.
In our highly litigious society, collecting sensitive performance data raises fears of malpractice suits and the potential misuse of statistics16, 17
. However, most states have quality assurance laws that protect these data from legal discovery18
. Care must be taken not only in the production and de-identification of sensitive data, but also in the interpretation and distribution of audit data. While BCSC registries have the ability to pool information across facilities to deal with the issue of radiologists who work at multiple facilities, many facilities in the U.S. cannot do this. In addition, the audit reporting does not adjust for possible differences in characteristics of the patients, which can affect the performance of mammography19
. Few radiologists interpret enough mammograms with cancer to get a precise estimate of their own cancer detection rate or sensitivity. MQSA requires that radiologists read at least 960 mammograms over a two-year period20
; however, the cancer detection rate in the United States is only about 4 per 1,000 screening mammograms in an average-risk population21
. Thus, radiologists who read 500 studies per year may see only one or two cancers annually.
Standardization of audit data and presentation style has the potential to benefit patients through enhanced quality assurance, and to benefit radiologists who work in multiple facilities22,23,24
. Standardization also would clearly facilitate quality assurance research; without a standardized audit report, quality improvement efforts and evaluation research will remain limited in scope. The BI-RADS manual25
offers sample forms to simplify the audit data collection and calculation process. The American College of Radiology also recently introduced the National Mammography Database26
into which U.S. mammography facilities may choose to periodically upload their mammography data and then receive semiannual audit reports with national and demographically-similar-facility comparisons. These reports will not be linked to tumor registries for complete cancer ascertainment, thus, they will not include data on sensitivity or specificity. The BCSC has recently launched a web site where individual radiologists from some BCSC sites can see their outcome audits in comparison to regional and national data. In the future, radiologists who do not practice at a BCSC facility will be able to add their aggregated data to the web site and compare their performance with national benchmarks.
In addition to fulfilling MQSA requirements, mammography audits have the potential to improve interpretive performance and patient outcomes27
. The value of medical audits may be that they provide information to practicing radiologists about their performance relative to that of their peers and national benchmarks. This is important because several studies of physicians' perceptions indicate that they believe they are performing at higher rates than they actually are28, 29
. If used correctly, audits can provide a direct assessment of what the radiology facility and individual radiologist are doing well, and uncover deficiencies in performance, so that radiologists can consider changes they could make to their interpretive practices to improve their performance30
and easily determine if these changes lead to actual improvements.
Several studies have tested multi-component interventions that included audit reports and educational sessions interpreting screening mammograms31-33
. Unfortunately, the specific contribution of receiving audit reports versus other components of the interventions is unknown. In a qualitative study of 25 BCSC radiologists6
, many participants thought customizable, web-based reports would be useful.
Our study has a number of strengths. Mandating the collection of comprehensive audit data, as recommended in the 2005 Institute of Medicine report34
, can be very labor intensive, making it important to see if radiologists are actually using the audit data and if they find the information valuable. Second, our response rate of 71% of radiologists who received the survey is considerably higher than most physicians surveys35
. Third, this study included a diverse group of community-based radiologists who interpret mammograms for women living in seven geographical regions of the United States. Thus, our findings have greater generalizability than a survey restricted to academic radiologists or specialists in breast imaging.
Though there were strengths in this study, there were also limitations. First, it is possible that respondents and non-respondents to the survey differed in their attitudes toward the use of audit reports and reported a biased perspective. However, our high response rate is reassuring, and in previous analyses we found that the interpretive performance of responders to the survey and non-responders is similar11, 36
. We also noted no difference in characteristics between radiologists who did and did not report receiving regular audit reports. A second limitation of this study is that survey data were based on subjective self-report; we did not verify whether reported use of audit data reflected actual review and use, or which radiologists received additional audit data beyond what the BCSC registries supply. The wording of survey questions such as question 6, which asked whether study participants would stop interpreting mammograms if congress mandates more intensive auditing requirements without additional funding, may be particularly prone to a biased response. Third, the audit reports received by BCSC radiologists are likely more detailed than reports received by non-BCSC radiologists due to the BCSC prospective data collection methods, linkage with cancer registries, and follow up on all negative exams. Finally, we were not able to assess the important question of whether use of audit data is associated with improved performance.
Audit reports are one possible means of improving radiologists' interpretative performance, since they may facilitate the review of previous false-negative diagnoses and can shed light on interpretive performance of individual radiologists. Our research indicates that radiologists find mammography audit reports from the BCSC useful, and are prompted by them to review cases with breast cancer diagnosis and improve their performance. Future studies should test whether having audit reports affects interpretive performance and should examine what aspects of the reports radiologists find most useful, and what aspects could be simplified or enhanced to facilitate increased use and benefit.