Institutional review board exempt status was obtained for this study.
Dictating Preliminary Interpretations On-Call
From January 1 to June 15, 2010, radiology residents and fellows dictated preliminary reports on-call into the Radiology Information System (RIS; GE Centricity® RIS-IC, GE Medical Systems, Waukesha, WI) using two different speech recognition (SR) systems, Centricity Precision Reporting (GE Medical Systems) and RadWhere (Nuance Healthcare Solutions, Burlington, MA). After a resident or fellow signs a report into preliminary status, the report is saved into the RIS and sent to the electronic medical record for referring clinicians to review. Preliminary reports can be changed or updated depending on additional clinical information or images. All versions of draft and preliminary reports stored in the RIS can be accessed using Microsoft® (Redmond, VA) Structured Query Language (MS SQL) queries. At our hospital system, there are approximately eight different types of call shifts ranging from 4 to 12 h in length during which residents and fellows can cover up to three different hospitals: the Hospital of the University of Pennsylvania, Penn-Presbyterian Medical Center, and the Philadelphia VA Medical Center.
Radiology residents and fellows interpreting radiological studies on-call must assign a provider prior to signing the report into preliminary status, which allows the referring physician or service to view the preliminary report in the electronic medical record system. On-call studies are assigned to a generic provider called “Night Radiologist”, which has a corresponding unique provider identification (ID) number defined in the user table within the RIS. We use the provider ID number for “Night Radiologist” to identify all reports interpreted by residents and fellows on-call.
Development of the Core Orion Software Application
The Orion software application was developed using the open-source tools PHP: Hypertext Preprocessor (PHP) and MySQL, a relational database management system (Fig. ). An initial PHP querying script accesses a mirrored RIS database every 90 min and identifies all reports signed out to “Night Radiologist” using the corresponding provider ID number in the report audit table. When a report signed out to “Night Radiologist” is identified, the PHP script uses SQL to query the unique study ID number, accession number, patient information, exam type and code, modality, subspecialty, patient location, date and time that the study is completed in the RIS, date and time of the most recent preliminary interpretation, and all other relevant study information from four tables in the RIS database. The study information is linked across the four tables by a unique exam ID.
Diagrammatic representation of the relational databases, tables, and queries used to monitor resident and fellow discrepancies
All study information queried from the mirrored RIS database is stored in a MySQL table (called “nightrad”) on a separate dedicated server. Additional PHP querying scripts use SQL to query the most recent preliminary and final reports from separate tables in the RIS, which are then stored in the “nightrad” table. A final PHP querying script uses SQL to search the audit table in the RIS and identify all studies in which our automatic electronic notification system to the emergency department was used (Emtrac notification system) and subsequently updates the “nightrad” table to indicate that the Emtrac notification system was used for a particular study. The Emtrac notification system is used for both minor and major discrepancies in reports for studies performed on patients in the emergency department who were discharged prior to the final report being issued. This notification system ensures 100% follow-up of all modified reports, including those with new recommendations.
Once the preliminary and final reports are in the “nightrad” table with the patient and interpreting physician information, two types of PHP identification scripts are used to identify discrepancies. The first PHP identification script (called “MacroType”) identifies unique text strings that are part of four standard report macros that faculty are required to use when reviewing on-call reports. We have simplified the RadPeer scoring system and use three general grades: agreement, minor discrepancy, and major discrepancy. The difference between minor and major discrepancy is that a major discrepancy has the potential to impact patient management or outcome. Each macro has slightly different introductory statement, which allows for detection and categorization using the PHP scripts. The macro introductory statements corresponding to agreement, minor discrepancy, and major discrepancy are listed below:
- “ATTENDING RADIOLOGIST AGREEMENT” indicates agreement without modification
- “ATTENDING RADIOLOGIST ADDITION” indicates agreement with additional text
- “*ATTENDING RADIOLOGIST CHANGE” indicates a minor discrepancy that has no clinical impact
- “**ATTENDING RADIOLOGIST CHANGE” indicates a major discrepancy that may have clinical impact
When the script identifies one of these text strings in the final report, it updates the “MacroType” field in the MySQL table “master” to indicate agreement, minor discrepancy, or major discrepancy.
Automatic Detection of Discrepancies
The second PHP identification script (“AutoDiscrepancy”) evaluates entries in which a standard review macro text string is not identified in the final report, indicating that the faculty member reviewing the report did not use one of the aforementioned report review macros. The “AutoDiscrepancy” script first performs a simple comparison between the preliminary and final reports to determine if there has been a change. If there is no change in text between the preliminary and final reports, the script updates the “AutoDiscrepancyGrade” field of the “master” MySQL table to indicate that the report is unchanged. If there is a change between the preliminary and final report, the “AutoDiscrepancyGrade” script calculates a score based on the percentage of text change and presence of specific words and phrases that indicate discrepancy. Based on a pre-determined threshold, reports are then graded as “agreement” or “discrepancy” in the “AutoDiscrepancyGrade” field of the “master” table. The details of the formula used to determine the likelihood of discrepancy and its development are further discussed in the Results
User Login and Interface
User information is stored in a user table (“rftable”) in the MySQL database on a secure server accessible only via the hospital intranet or virtual private network. Each user is assigned a username and password based on their unique RIS identification number to login to the application, an access setting to determine which modules the user will be able to access, a user designation (resident, fellow, fellowship program director, residency program director, or administrator), specialty (only for fellows), and year of graduation (only for residents). The residency program director has administrator access and can use all Orion modules (Fig. ). Fellowship program directors have mid-level access, which allows them to review specialty-specific on-call studies and performance for fellows within their specialty. Residents and fellows have individual user access and can only review their own reports (Fig. ).
Fig. 2 Administrator (a) and resident/fellow screens (b) demonstrating the various functions of the Orion application. Administrators can review on-call reports by discrepancy type, resident/fellow or accession number, generate summary reports, review report (more ...)
The report review screen (Fig. ) has a navigation bar at the top with all relevant study information immediately below. Grading options and a comment field are located below the study information section. The residency program director and fellowship program directors can grade reports as agreement, minor discrepancy, or major discrepancy based on the preliminary and final reports, which are displayed in tandem as part of a three-column table. The third column is the text differential of the preliminary and final reports with highlighting and strikethrough to more clearly display the changes between the preliminary and final reports. In addition to grading, the residency program director can provide a comment, save the case to a folder (teaching, missed, follow-up), classify the discrepancy (overcall, undercall, misinterpretation), and indicate if the Emtrac notification system and/or required faculty review macros were used appropriately.
Fig. 3 Report review screen demonstrating the navigation bar with option to grade report quality (top), study information, grading/folder options, and comment fields. The preliminary report, final report, and highlighted text differential are displayed in tandem (more ...)
For the residency program director, there is an additional “Grade report quality” button adjacent to the navigation car that links the active study to the report quality module described below under “Evaluating report quality”. Individual residents and fellows can only mark studies as reviewed, provide comments, and save cases in folders. Discrepancy grades, comments, and folder designations for each study are recorded separately for the residency program director, fellowship program directors, and individual residents and fellows in the “master” table. When a report is graded or marked reviewed, it is immediately dropped from the review list, and the next study is loaded and displayed.
Secondary Review of Discrepancies
All studies in which faculty members reviewing an on-call preliminary report used the minor change macro, major change macro, or Emtrac notification system are reviewed by the residency program director (M.H.S.). In addition, all studies identified as “discrepancy” by the “AutoDiscrepancy” script are also reviewed by the residency program director. This ensures that nearly 100% of all resident and fellow discrepancies are reviewed and categorized. The final determination of the type of discrepancy is made by the residency program director. The discrepancy grade is stored in the “master” table and linked to the “nightrad” table using the unique exam ID for the study.
Monitoring Faculty Compliance with Use of Required Macros
The purpose of monitoring compliance is to ensure that faculty members use one of the required macros when they modify a preliminary report. Faculty compliance with using the required macros is determined by using the “MacroType” and “AutoDiscrepancy” script results. Effective compliance is calculate by dividing the number of modified reports without a required macro in the final report text by the total number of modified reports and reported real-time for the last 30, 60, and 90 days. The 30-day report is converted to portable document format (pdf) format and emailed with specific comments to faculty members with less than 90% compliance.
Monitoring Resident Compliance with Reviewing Studies
Radiology residents at our institution do not review on-call cases with faculty at the end of the call shift. There is a resident version of Orion that allows residents to review all minor and major discrepancies and mark them as reviewed. The resident can provide comments about the case, save the case in a folder (missed, teaching, follow-up), and generate a summary report of all cases by type of discrepancy or folder. The residency program director monitors resident compliance with reviewing all minor and major discrepancies through Orion and reviews both resident performance and compliance at the semi-annual review. The resident discrepancy grade is stored in the “master” table and linked to the “nightrad” table using the unique exam ID for the study.
Analyzing Resident and Fellow Performance
The analytics module allows both the residency program director and fellowship program directors to evaluate minor and major discrepancy rates for all residents and fellows in real-time with multiple filter options. Total volume of on-call studies with overall minor and major discrepancy rates can be generated for all residents and fellows for a specified time range and is used for the quarterly clinical quality report for the Department of Radiology. Individual resident minor and major discrepancy rates are used to establish overall performance benchmarks and identify residents who demonstrate a significantly higher major discrepancy rate compared with classmates. Individual resident and fellow minor and major discrepancy rates are generated by modality and subspecialty to target specific areas for improvement. Minor and major discrepancy rates by modality and subspecialty are filtered by residency class to establish performance benchmarks by year of training.
A separate analytics module allows evaluation of major discrepancy rate by time of day and shift length. A schedule table (“nftable”) containing the resident nightfloat call schedule from January 1 to June 15, 2010, was added to the MySQL database along with a separate table (“calltype”) to define the type of nightfloat call (1-week versus 2-week assignments) and shift length (12 h). Modality and subspecialty filters can be employed, and date ranges can be specified.
Generating Summary Reports
Comprehensive summary reports can be generated by the residency program director for a specified time period by discrepancy type, specialty, or resident/fellow. Individual resident summary reports are generated every 6 months, reviewed with residents during the semi-annual review, and placed in the resident’s learning portfolio. Summary reports by specialty are generated every 6 months and forwarded to each subspecialty section chief and fellowship director in an effort to tailor the didactic and case-based curriculum for that specialty to the specific needs of the residents and fellows.
A complete summary report of all major discrepancies is reviewed by the resident missed-case committee every 6 months to identify trends in major discrepancies. Regular resident missed-case conferences are designed to target specific types of misses identified as major discrepancies over the previous 6–12 months. In addition, every PGY-3 resident is required to give a missed-case conference to all residents after taking independent call based on their own discrepancies.
The system maintenance script reports if there are any entries in the MySQL database still signed out to nightrad (not reviewed by a faculty member), as well as entries without a preliminary report, “MacroType”, or “AutoDiscrepancyGrade”. It also reports when the query scripts were last performed and if any errors were encountered. This maintenance report ensures that the query scripts run every day without error and that no issues are encountered with the PHP scripts identifying review macros. It can also identify reports that have not been signed by a faculty member within 24 h, but there are other processes in place at our institution to identify these reports.