|Home | About | Journals | Submit | Contact Us | Français|
The U.S. Mammography Quality Standards Act (MQSA) mandates medical audits to track breast cancer outcomes data associated with interpretive performance. The objectives of our study were to assess the content and style of audits and examine use of, attitudes toward, and perceptions of the value that radiologists' have regarding mandated medical audits.
Radiologists (n=364) at mammography registries in seven U.S. states contributing data to the Breast Cancer Surveillance Consortium (BCSC) were invited to participate. We examined radiologists' demographic characteristics, clinical experience, and use, attitudes and perceived value of audit reports from results of a self-administered survey. Information on the content and style of BCSC audits provided to radiologists and facilities were obtained from site investigators. Radiologists' characteristics were analyzed according to whether or not they self-reported receiving regular mammography audit reports. Latent class analysis was used to classify radiologists' individual perceptions of audit reports into overall probabilities of having “favorable,” “less favorable,” “neutral,” or “unfavorable” attitudes toward audit reports.
Seventy-one percent (257 of 364) of radiologists completed the survey; two radiologists did not complete the audit survey question, leaving 255 for the final study cohort. Most survey respondents received regular audits (91%), paid close attention to their audit numbers (83%), found the reports valuable (87%), and felt that audit reports prompted them to improve interpretative performance (75%). Variability was noted in the style, target audience and frequency of reports provided by the BCSC registries. One in four radiologists reported that if congress mandates more intensive auditing requirements but does not provide funding to support this regulation they may stop interpreting mammograms.
Radiologists working in breast imaging generally had favorable opinions of audit reports, which were mandated by Congress; however, almost one in ten radiologists reported that they did not receive audits.
In 1992, the U.S. Congress enacted the Mammography Quality Standards Act (MQSA), which established the first national quality standards for mammography facilities in the United States. MQSA was initiated in response to concerns from the public and medical community about the extensive variability of mammography among facilities1,2. The goal of MQSA is to provide all women living in the U.S. with equal access to quality mammography, regardless of their geographic location. Under MQSA, mammography facilities are required to have a U.S. Food and Drug Administration (FDA)-approved accreditation body review their radiological equipment, personnel qualifications and quality assurance processes every three years to ensure that baseline quality standards are practiced1, 3.
Mammography outcome audits are one of the quality assurance regulations for mammography facilities that fall under MQSA. The basic elements of MQSA's medical audit include: 1) a method to collect follow-up data for positive mammograms (defined as mammograms with final BI-RADS® assessment categories of “Suspicious” or “Highly suggestive of malignancy”); 2) a system to collect pathology results (benign versus malignant) for all biopsies performed among mammograms interpreted as “Suspicious” or Highly suggestive of malignancy”; 3) methods to correlate pathology and mammography results; and 4) review of known false negatives (examinations assessed as “negative,” “benign,” or “probably benign” that became known to the facility as positive for cancer within 12 months of mammography examination).4,5 In addition, at least once every 12 months facilities are required to designate an interpreting physician to review the medical outcomes data and notify other interpreting physicians that their individual results are available for review. FDA regulations do not specifically require that individual radiologists review their outcomes5. Approaches to implement the elements described above are left to the facility's discretion.1, 4, 5
To the best of our knowledge, only one previous qualitative study of 25 U.S. radiologists has explored radiologists' perceptions and use of their mammography audit data6. We had the opportunity to examine the attitudes and use of medical audits among radiologists working in Breast Cancer Surveillance Consortium (BCSC) mammography facilities. Over the last decade mammography facilities in seven U.S. states have submitted data to the BCSC7. The BCSC registries in turn provide data back to the facilities that could fulfill the minimum MQSA medical outcomes audit requirement. BCSC registries have the unique ability to provide substantially more audit data than the minimum MQSA requirements. While the BCSC registries have produced many scientific papers8, the content of the individual registry audit reports has not been reviewed and little is known about radiologists' attitudes and use of the audit data.
Our study goals were to examine whether radiologists report seeing their mammography audit results, and to describe the radiologists' attitudes about the audit and the perceived value of their reports. We also describe the style and content of performance audit data provided by the BCSC to participating facilities.
The study was approved by the IRB of the University of Washington and all seven BCSC sites.
Our survey was mailed to all radiologists who interpreted screening and diagnostic mammograms in 2005-2006 at seven geographically distinct BCSC sites. Details of BCSC practices and survey development have been reported previously7,9, 10-11. We mailed radiologists who were actively interpreting mammograms at a BCSC facility a self-administered survey between January 2006 and September 2007. Seventy-one percent (257 of 364) of radiologists returned the survey with informed consent. A copy of the radiologist survey is available online12.
Survey questions included radiologist age, years of practice, affiliation with academic medical centers, completing a breast imaging fellowship, estimates of annual volume of screening and diagnostic mammograms, and percentage of practice time spent in breast imaging. The survey also included questions on mammography audits including whether radiologists received audit reports showing their performance, what year they began to receive the reports, and how many times per year they receive the audit reports. We excluded two radiologists who had missing data on audit questions, leaving a total of 255 radiologists for analyses. Respondents who reported receiving regular mammography audit reports were then asked to rate the following six statements about audit reports using a 5-point Likert scale (strongly disagree, disagree, neutral, agree, strongly agree):
In addition to the radiologist survey, we collected information on the content and format of reports that BCSC registries provided to facilities between 2003 and 2005. We used the 2003-2005 BCSC reports, as these were the reports that radiologists were most likely to have viewed prior to receipt of the radiologist survey. Data on audit content and format were obtained from the BCSC registry data managers and verified by principal investigators at each BCSC registry.
The results of the content analyses and details of the audit performance reports provided by the seven BCSC sites are presented in Table 1. All reports provided summary data at the facility level, and all but one site reported results for individual radiologists. The reports ranged in length from one to 12 pages of core data, with two sites providing additional pages of pathology data. Three sites used graphics in addition to numeric displays. The frequency of distribution ranged from quarterly to annually. All sites included information on the percentage of screening mammography exams that were BI-RADS category 0 (recommendation for further imaging) and categories 4/5 (recommendation for biopsy/surgical consultation). The other performance data varied among the sites. However, with the information provided in the reports, and if radiologists were so inclined and knew how, they would be able to derive many common measures of performance. Three of the seven sites included results of biopsies performed. Information on cancer outcome data, such as stage, percentage of cancers with minimal disease or node negative, was not available at all sites.
We describe the characteristics of radiologists stratified according to whether or not they self-reported receiving regular mammography audit reports. Among radiologists who reported receiving audit reports, we calculated the distribution of their likert-scale responses to questions on attitudes and perceived value regarding medical audit reports and used chi-squared tests to compare characteristics among radiologists who agreed/strongly agreed with the audit survey statements with those who were neutral, disagreed, or strongly disagreed.
We used latent class analysis13 to describe the overall attitude of radiologists toward audit reports. A latent class is a characteristic that is not directly measured from the study, but can be inferred based on other measured characteristics. Our aim was to identify classes that would represent groups of radiologists' overall perceptions of audit reports (which were not directly measured) based on responses to the six individual attitude statements. We identified four latent classes describing overall attitudes toward audit reports: favorable, less favorable, neutral and unfavorable toward audit reports. For example, using latent class analysis, we estimated the probability that a random radiologist in the “favorable” class “agrees” that audit reports are valuable to their practice. In this way, we assessed how specific aspects of audit reports related to their overall perception. Statistical analyses were performed using fSAS 9.1 (SAS Institute, Cary, NC). The latent class analyses were performed using the PROC LCA add on for SAS, which can be downloaded at http://methodology.psu.edu/index.php/downloads/proclcalta14, 15.
A significance level of p < 0.05 was used to determine statistical significance.
The 255 respondent radiologists interpreted mammograms at 337 mammography facilities in the United States during the study period. Among them, the majority of radiologists (n=233, 91%) reported that they received regular audit reports. Radiologists were more likely to indicate receiving audit reports if they had interpreted mammograms for >10 years or reported interpreting >2000 screening mammograms per year (data not shown). Of the 233 radiologists receiving regular audit data, the frequency was described as once per year (n=155, 67%), more than once per year (n=57, 24%), and the remainder reported seeing their audit reports, but did not respond to the question about frequency (n= 21, 9%).
The 22 radiologists who reported not receiving audit reports were distributed across three BCSC registries (data not shown). Most of the 22 radiologists (N=16, 73%) worked at mammography facilities served by the two BCSC registries that provide audit reports only to facilities, rather than directly to the radiologists.
Among the larger cohort of radiologists (n=255), including those who reported receiving and those who reported not receiving the reports, 26% overall agreed or strongly agreed with the statement that they would stop interpreting mammograms if congress mandated more intensive auditing requirements but did not provide funding to support the regulation. Radiologists who receive audits were more likely to disagree with this statement.
Among the radiologists who reported receiving the reports (N=233), Figure 1 illustrates their attitudes toward audit reports for five of the survey questions and shows that most radiologists in our study agreed or strongly agreed with each statement. Between 74-86% of respondents agreed or strongly agreed that they trusted the accuracy of audit reports, paid attention to them, considered the reports valuable, and were prompted by the reports to review missed cancers and improve their interpretive performance.
Table 2 shows the percentage of radiologists receiving audit reports who agreed or strongly agreed with survey questions regarding attitudes about audit reports, by radiologist characteristics. The only statistically significant differences across radiologist characteristics who did and did not agree or strongly agree was in the survey question “If congress mandates more intensive auditing requirements but does not provide funding to support the regulation, I may stop interpreting mammograms.” Radiologists who reported that they might leave the field of mammography if congress mandated additional requirements were more likely to be younger, with fewer years experience interpreting mammograms, and to self-report a lower total volume of interpretive mammograms in the preceding year. Trends were noted among radiologist characteristics in their responses to the other survey questions, but the differences were not statistically significant. For example, radiologists who were less positive about audits were more likely to be fellowship trained and report annual volume in the lowest range categories.
The results of the latent class analyses are shown in Figures 2a-2d and Appendix 1. The majority of radiologists who received audits (Fig. 2a, 75%) had favorable attitudes toward audit reports; these radiologists primarily trusted the accuracy of reports, paid attention to them, found audit reports to be valuable, used them to review cancers, and reported that audits improved performance. A smaller proportion of radiologists (Fig. 2b, 13%) were less favorable about audit reports; this group trusted the accuracy of the reports, but felt less favorable about whether the reports improve performance. A small proportion of radiologists were neutral about performance reports (Fig. 2c, 8%), though this group agreed that the reports prompted them to review cancers. A very small proportion (Fig. 2d, 3%) were unfavorable toward reports, primarily because they did not trust their accuracy.
The findings from our study suggest that most radiologists participating in the BCSC receive mammography audit reports, review them, consider them valuable, and are prompted by the reports to review missed cancers. The audit reports seen by radiologists in this national consortium varied in content and style and in the use of figures and graphs.
In our highly litigious society, collecting sensitive performance data raises fears of malpractice suits and the potential misuse of statistics16, 17. However, most states have quality assurance laws that protect these data from legal discovery18. Care must be taken not only in the production and de-identification of sensitive data, but also in the interpretation and distribution of audit data. While BCSC registries have the ability to pool information across facilities to deal with the issue of radiologists who work at multiple facilities, many facilities in the U.S. cannot do this. In addition, the audit reporting does not adjust for possible differences in characteristics of the patients, which can affect the performance of mammography19. Few radiologists interpret enough mammograms with cancer to get a precise estimate of their own cancer detection rate or sensitivity. MQSA requires that radiologists read at least 960 mammograms over a two-year period20; however, the cancer detection rate in the United States is only about 4 per 1,000 screening mammograms in an average-risk population21. Thus, radiologists who read 500 studies per year may see only one or two cancers annually.
Standardization of audit data and presentation style has the potential to benefit patients through enhanced quality assurance, and to benefit radiologists who work in multiple facilities22,23,24. Standardization also would clearly facilitate quality assurance research; without a standardized audit report, quality improvement efforts and evaluation research will remain limited in scope. The BI-RADS manual25 offers sample forms to simplify the audit data collection and calculation process. The American College of Radiology also recently introduced the National Mammography Database26 into which U.S. mammography facilities may choose to periodically upload their mammography data and then receive semiannual audit reports with national and demographically-similar-facility comparisons. These reports will not be linked to tumor registries for complete cancer ascertainment, thus, they will not include data on sensitivity or specificity. The BCSC has recently launched a web site where individual radiologists from some BCSC sites can see their outcome audits in comparison to regional and national data. In the future, radiologists who do not practice at a BCSC facility will be able to add their aggregated data to the web site and compare their performance with national benchmarks.
In addition to fulfilling MQSA requirements, mammography audits have the potential to improve interpretive performance and patient outcomes27. The value of medical audits may be that they provide information to practicing radiologists about their performance relative to that of their peers and national benchmarks. This is important because several studies of physicians' perceptions indicate that they believe they are performing at higher rates than they actually are28, 29. If used correctly, audits can provide a direct assessment of what the radiology facility and individual radiologist are doing well, and uncover deficiencies in performance, so that radiologists can consider changes they could make to their interpretive practices to improve their performance30 and easily determine if these changes lead to actual improvements.
Several studies have tested multi-component interventions that included audit reports and educational sessions interpreting screening mammograms31-33. Unfortunately, the specific contribution of receiving audit reports versus other components of the interventions is unknown. In a qualitative study of 25 BCSC radiologists6, many participants thought customizable, web-based reports would be useful.
Our study has a number of strengths. Mandating the collection of comprehensive audit data, as recommended in the 2005 Institute of Medicine report34, can be very labor intensive, making it important to see if radiologists are actually using the audit data and if they find the information valuable. Second, our response rate of 71% of radiologists who received the survey is considerably higher than most physicians surveys35. Third, this study included a diverse group of community-based radiologists who interpret mammograms for women living in seven geographical regions of the United States. Thus, our findings have greater generalizability than a survey restricted to academic radiologists or specialists in breast imaging.
Though there were strengths in this study, there were also limitations. First, it is possible that respondents and non-respondents to the survey differed in their attitudes toward the use of audit reports and reported a biased perspective. However, our high response rate is reassuring, and in previous analyses we found that the interpretive performance of responders to the survey and non-responders is similar11, 36. We also noted no difference in characteristics between radiologists who did and did not report receiving regular audit reports. A second limitation of this study is that survey data were based on subjective self-report; we did not verify whether reported use of audit data reflected actual review and use, or which radiologists received additional audit data beyond what the BCSC registries supply. The wording of survey questions such as question 6, which asked whether study participants would stop interpreting mammograms if congress mandates more intensive auditing requirements without additional funding, may be particularly prone to a biased response. Third, the audit reports received by BCSC radiologists are likely more detailed than reports received by non-BCSC radiologists due to the BCSC prospective data collection methods, linkage with cancer registries, and follow up on all negative exams. Finally, we were not able to assess the important question of whether use of audit data is associated with improved performance.
Audit reports are one possible means of improving radiologists' interpretative performance, since they may facilitate the review of previous false-negative diagnoses and can shed light on interpretive performance of individual radiologists. Our research indicates that radiologists find mammography audit reports from the BCSC useful, and are prompted by them to review cases with breast cancer diagnosis and improve their performance. Future studies should test whether having audit reports affects interpretive performance and should examine what aspects of the reports radiologists find most useful, and what aspects could be simplified or enhanced to facilitate increased use and benefit.
Funding sources: This work was supported by the National Cancer Institute [1R01 CA107623, 1K05 CA104699; Breast Cancer Surveillance Consortium (BCSC): U01CA63740, U01CA86076, U01CA86082, U01CA63736, U01CA70013, U01CA69976, U01CA63731, U01CA70040], the Agency for Healthcare Research and Quality (1R01 CA107623), the Breast Cancer Stamp Fund, and the American Cancer Society, made possible by a generous donation from the Longaberger Company's Horizon of Hope Campaign (SIRGS-07-271-01, SIRGS-07-272-01, SIRGS-07-273-01, SIRGS-07-274-01, SIRGS-07-275-01, SIRGS-06-281-01).
The authors had full responsibility in the design of the study, the collection of the data, the analysis and interpretation of the data, the decision to submit the manuscript for publication, and the writing of the manuscript. We thank the participating women, mammography facilities, and radiologists for the data they have provided for this study. A list of the BCSC investigators and procedures for requesting BCSC data for research purposes are provided at: http://breastscreening.cancer.gov/.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.