Search tips
Search criteria 


Logo of jopHomeThis ArticleASCO JOPSearchSubmitASCO JOP Homepage
J Oncol Pract. 2011 July; 7(4): 222–226.
Published online 2011 June 17. doi:  10.1200/JOP.2011.000237
PMCID: PMC3140442

Implementation of Electronic Checklists in an Oncology Medical Record: Initial Clinical Experience



The quality of any medical treatment depends on the accurate processing of multiple complex components of information, with proper delivery to the patient. This is true for radiation oncology, in which treatment delivery is as complex as a surgical procedure but more dependent on hardware and software technology. Uncorrected errors, even if small or infrequent, can result in catastrophic consequences for the patient. We developed electronic checklists (ECLs) within the oncology electronic medical record (EMR) and evaluated their use and report on our initial clinical experience.


Using the Mosaiq EMR, we developed checklists within the clinical assessment section. These checklists are based on the process flow of information from one group to another within the clinic and enable the processing, confirmation, and documentation of relevant patient information before the delivery of radiation therapy. The clinical use of the ECL was documented by means of a customized report.


Use of ECL has reduced the number of times that physicians were called to the treatment unit. In particular, the ECL has ensured that therapists have a better understanding of the treatment plan before the initiation of treatment. An evaluation of ECL compliance showed that, with additional staff training, > 94% of the records were completed.


The ECL can be used to ensure standardization of procedures and documentation that the pretreatment checks have been performed before patient treatment. We believe that the implementation of ECLs will improve patient safety and reduce the likelihood of treatment errors.


Although accidental overdose is uncommon in radiation oncology, the results are serious for the patient. Recently, the popular press reported on accidents that resulted from failures of equipment, software, and work practices.1 Although these reports have labeled radiation treatments as unsafe, the objective data suggest that errors (or accidents) are fortunately rare. In New York, where one of the reports originated, the error rate based on an estimate of 13.6 million radiation treatments over the period was 0.005%.2 This low rate has been documented by others as well,3 but reported ranges vary from as high as 0.2% without computerized systems to as low as 0.017% with these systems.3 Even though this error rate is small, and most errors are of no consequence, a single error is one too many.

Technology is a double-edged sword. It can function as a valuable adjunct to the human brain, serving to note and ensure completeness.4 However, it can also magnify the scale of errors when it is relied on excessively and taken for granted. Although there have been cases of poorly constructed technology, the majority of errors in radiation oncology result from poor training and individuals not following standard procedures.57

The literature constantly warns about excessive reliance on technology. Technology does not operate by itself and must be supported by departmental procedures.8 Our checklist was modeled on the use of presurgery checklists and can identify many potentially correctable errors to achieve safe and accurate delivery of radiation treatment. The advantage of the electronic checklist (ECL) is that staff are required to document their actions, which can be tracked, audited, and reported in a meaningful manner, thus assisting staff education to achieve higher levels of safety.

Checklists have been used by pilots, NASA engineers, and surgeons.9 Oncologists can do the same. We already use checklists for screening and treating patients in clinical trials. Reporting of adverse effects, quality of life data, and the centralized quality assurance (QA) of clinical trial data are essentially complex and multilevel checklist processes. This approach in clinical trials provides greater uniformity and quality of treatment compared with treatment of the general patient population and is known to enhance outcomes.1013

In the book, “Checklist Manifesto,” Gawande14 describes how faulty memory and distraction can lead to errors in achieving complex tasks for all-or-none processes: preparing an airplane for takeoff, or evaluating a sick person in the emergency room. Checklists provide protection against such failures by providing process verification and instilling discipline to improve performance.14 Certainly the same applies to the complex delivery of radiation therapy, which involves multiple professionals working as a team through a variety of steps to safely and effectively treat the patient's tumor. For this purpose, we created an ECL in the patient's EMR and describe our experience and results with the same.


The software deployed in our department to subserve the Record and Verify function for the linear accelerator (linac) is MOSAIQ (Elekta, Mountain View, CA) This software records all of the treatment-related information (eg, beam angle, monitor units, and so on) and compares it with the desired values based on the treatment plan. Treatment is not permitted to commence until all actual and desired values are in agreement and verified. Once the treatment is delivered, MOSAIQ records the parameters of the delivered treatment. Note that on some newer linacs, MOSAIQ passes information to the treatment unit computer, which is then responsible for ensuring that the desired and actual parameters agree. In addition to the functionality that manages the actual radiation delivery, there is an EMR that provides a clinical cancer database, document and note storage, ordering of medications, assessment of clinical condition, and management of laboratory results, to mention a few.

In our department, the ECL was implemented in January 2009 within the EMR functionality called Clinical Assessments (CA). The CA functionality can be configured as a permanent record in which a user cannot remove an included item from view. It also has the familiar appearance of a spreadsheet, which allows easy checking of pending, completed, and missing items. The CA functionality is also routinely used for the entry of toxicities during patient assessment visits. The entry method can be text or predefined drop-down menu. All entries are time-date stamped and can be configured for mandatory electronic approval. Different levels of security can be assigned to data that are entered, with provisions for multiple signatures if required. An example of the ECL is shown in Table 1.

Recently, recommendations have been published that attempt to create a high-safety environment in radiation oncology departments.15 One of the recommendations has included the performance of an accurate and complete QA test before the delivery of radiation therapy. QA should cover all steps in the treatment delivery process; any step vital to accurate radiation dose delivery should be included. This verification process will be similar to the “time-out” for surgery (ie, pause to manually check parameters before beginning, as recommended by the Joint Commission on Cancer), but inherently more complicated. Although surgery has some possibility of retreat even after the start of the operation, the accuracy of the entire radiation process has to be confirmed before pressing the “beam on” button because radiation overdose cannot be corrected.

The process of creating a custom checklist within MOSAIQ is rather straightforward. First, within the Observation Definitions, a tab view is created. This represents the view that the user will see on the screen. The tab view has a uniquely assigned number that facilitates searching for data within the database by using Crystal Reports (SAP AG, Walldorf, Germany), for example. Under the tab view, a number of items can be added. These include higher level folders and subfolders that allow observations to be organized under specific categories. In our case (Figure 1), the major headings include Dosimetry, Physics, Therapy, and Chart Rounds. Subheadings are located under the Therapy folder and include Chart Review, Pre-Treatment, Pre-Port, and First Day of Treatment. Within each of these, data items are added for a particular group. These data items can take the form of a checkbox (checked when completed), numerical data entry, text entry (note), or drop-down selection. The majority of our checklist consists of items that have a checkbox and sections for notes.

The folders and subfolders reflect the clinical flow of data as the radiation treatment (RT) plan is created, verified, and delivered. On completion of the RT plan, and data entry into MOSAIQ, the dosimetrist completes the Dosimetry heading of the ECL. This necessitates review of the accuracy of plan transfer and ensures that all appropriate approvals are obtained from the physician (ie, radiation prescription) before information is passed on to subsequent groups. The RT plan is passed on to a physicist, who verifies the accuracy of the dose calculation, the plan constraints, and consistency of the RT plan with the physician's intent. Completion of this section lets the therapist know that the plan has been reviewed and that he or she can begin the series of pretreatment checks.

The therapist checklists were designed on the basis of previous paper versions of items that are to be checked before treatment, as well as thorough a review of the processes involved in the initiation of treatment. The first step involves reviewing the RT plan with the dosimetrist who created it. As plans have become more complex in recent years, direct communication between the dosimetrist and the therapist helps ensure that the RT plan will be implemented as the physician intended. After the chart review, the therapists will ascertain that the planning documents have all necessary approvals and that the patient is scheduled within MOSAIQ. The Pre-Port checklist is used by the therapist on the day of a trial run of the treatment plan with the patient on the linac table. The therapist uses the ECL to as a reminder to verify all gantry angles and any other devices that are necessary for treatment. At this time, the therapist may also obtain portal images of the patient to ensure that the patient is positioned properly and that the radiation portal shape is correct. For the first day of treatment, the therapist checks that the physician approved the portal images, takes appropriate photos, and provides notes to document the patient set-up to be used during the course of treatment.

Typically, within the first or second week of treatment, patient charts will be reviewed through chart rounds. The Chart Rounds checklist provides the list of items to be reviewed. This portion, however, is part of ongoing QA for peer review and will not be discussed further.


The use of our checklists has resulted in a more seamless RT plan delivery, and in our opinion, a decreased likelihood for error. Increased efficiency is difficult to document unless historic data are available. Although significant errors are rare with routine QA, we have found that team preparation for RT delivery has benefited in several ways from the discipline of using the ECL.

Easier Patient Set-Up With the Pretreatment Checklist

Our group of radiation oncologists has noted that calls to the linac to check initial patient set-ups ceased completely in 2010. This reduction is due to the therapists systematically reviewing the treatment plan, as indicated on the ECL, and thus having a better understanding of the plan. The result is more efficient use of linac time and less patient anxiety.

Improvement in Safety Through Training

All entries onto the ECL are signed off by the staff member responsible for completing the item. As a result, if errors are found downstream, the identity of responsible staff is known. This provides the opportunity to undertake education with staff to improve checking. In addition, all errors are entered in a hospital database, allowing us to track the number and types of errors that occur.

The MOSAIQ software we used can be configured, but not altered. Unfortunately, there is no capacity to provide alarms to the treating therapist to warn that a given ECL has not been completed before treatment. Human checking, via report or visual perusal, is the only means to verify that checklists have been signed off. We recently reviewed checklist compliance over a 2-month period (December 2010-January 2011). Over this time, on average of 81% of the ECL items were completed for 124 patients starting treatment. These results demonstrate good use of the ECL but also indicate the limitations when software interlocks are not available to prevent treatment without full completion of the ECL. On the basis of these results, we reviewed the use of the ECL with our staff in February 2011. In April 2011, a second review was performed, and we observed that > 94% of the checklist was completed for 85 patients audited. Several of those with missing items were patients who started treatment before staff re-education.

For the physicist and physician, an electronic signature of the prescription is required before treatment, but these signatories assume that the checklists for all other professional groups were completed properly. As with all processes in which checklists must be signed off, downstream staff must trust that the upstream checking has been successfully completed.

Similarly, if a treating linac malfunctions, and patients are moved to another linac with new therapists, our system does not mandate a separate pretreatment checklist encounter. Such a system would be particularly useful for high-dose radiation delivery with techniques like stereotactic radiotherapy.


The Record and Verify software has improved the precision of radiotherapy delivery by confirming that the treatment plan parameters are transferred correctly to linac,16 but it has not removed all human error.6 Our ECL system may increase radiation treatment safety by mandating routine, auditable checking. Similar checklists should be applied for all oncologic care. Chemotherapy delivery has similar requirements for drug handling, premedication, tumor imaging, and laboratory test checking, requiring confirmation and documentation before therapy administration.

The software, however, requires improvement. Electronic alarms and internal locks on treatment delivery must check and indicate that checklists are incomplete, a feature that is presently unavailable. In addition, pop-up screens could inform staff of incomplete checklists earlier in the clinical process. This would encourage behaviors that increase both efficiency and patient safety. We believe that these functionalities are already being planned for future oncology EMR systems (Goldwin J, personal communication, February 2011).

The audit reports that we have created to review staff compliance with checklist use are currently being run monthly and on an as-needed basis. This reporting software possesses the capability for this process to be automated. Software such as Crystal Solutions can generate reports to check progress on a daily basis or provide summary data over longer periods. The time-date stamps allow the timing of each step to be summarized and can increase safety and accountability in the clinic.

Once a process is established on ECLs, it is possible in the future to build in more intricate verification of the flow and acquisition of clinical data that are used or derived from the completion of particular checklist items. For example, when the Prescription Approved checklist is ticked, it should be possible to electronically verify that the prescription is actually approved. This would provide a front-loaded system providing both time-on (ie, automatic process verification by the Treatment Management system) and time-out checking.

We present in this study our initial clinical experience with an ECL for radiation oncology. The ECL was straightforward to implement and provides a logical, standardized means for the various groups involved in patient treatment to ensure that important steps are checked before the initiation of therapy. We believe that such standardization of procedures may improve patient safety. However, the development of the ECL is an ongoing process and needs to be integrated as part of the linac delivery system to prevent treatment unless all items are completed.


All work contained herein was performed at Loyola University Medical Center, Maywood, IL.

Authors' Disclosures of Potential Conflicts of Interest

Although all authors completed the disclosure declaration, the following authors indicated a financial or other interest that is relevant to the subject matter under consideration in this article. Certain relationships marked with a “U” are those for which no compensation was received; those relationships marked with a “C” were compensated. For a detailed description of the disclosure categories, or for more information about ASCO's conflict of interest policy, please refer to the Author Disclosure Declaration and the Disclosures of Potential Conflicts of Interest section in Information for Contributors.

Employment or Leadership Position: None Consultant or Advisory Role: Alexis Andrew Miller, Impac (U), Varian (U) Stock Ownership: None Honoraria: None Research Funding: John C. Roeske, Varian Medical Systems Expert Testimony: None Other Remuneration: None

Author Contributions

Conception and design: Kevin V. Albuquerque, Alexis Andrew Miller, John C. Roeske

Administrative support: Kevin V. Albuquerque

Provision of study materials or patients: Kevin V. Albuquerque, John C. Roeske

Collection and assembly of data: Kevin V. Albuquerque, John C. Roeske

Data analysis and interpretation: Kevin V. Albuquerque, Alexis Andrew Miller, John C. Roeske

Manuscript writing: Kevin V. Albuquerque, Alexis Andrew Miller, John C. Roeske

Final approval of manuscript: Kevin V. Albuquerque, Alexis Andrew Miller, John C. Roeske


1. Bogdanich W. Radiation offers new cures, and ways to do harm. NY Times. 2010 Jan 23;
3. Errors in radiation therapy. Pa Patient Safety Advis. 2009;6:87–92.
4. Kralj B, Iverson D, Hotz K, et al. The impact of computerized clinical reminders on physician prescribing behavior: Evidence from community oncology practice. Am J Med Qual. 2003;18:197–203. [PubMed]
5. Leveson NG, Turner CS. An investigation of the Therac-25 accidents. Computer. 1993;26(7):18–41.
6. Patton GA, Gaffney DK, Moeller JH. Facilitation of radiotherapeutic error by computerized record and verify systems. Int J Radiat Oncol Biol Phys. 2003;56(1):50–57. [PubMed]
7. Fraass BA, Lash KL, Matrone GM, et al. The impact of treatment complexity and computer-control delivery technology on treatment delivery errors. Int J Radiat Oncol Biol Phys. 1998;42:651–659. [PubMed]
8. Baiotto B, Bracco C, Bresciani S, et al. Quality assurance of a record-and-verify system. Tumori. 2009;95:467–472. [PubMed]
9. Haynes AB, Weiser TG, Berry WR, et al. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med. 2009;360:491–499. [PubMed]
10. Burgers JA, Arance A, Ashcroft L, et al. Identical chemotherapy schedules given on and off trial protocol in small cell lung cancer: Response and survival results. Br J Cancer. 2002;87:562–566. [PMC free article] [PubMed]
11. Davis S, Wright PW, Schulman SF, et al. Participants in prospective, randomized clinical trials for resected non-small cell lung cancer have improved survival compared with nonparticipants in such trials. Cancer. 1985;56:1710–1718. [PubMed]
12. Wagner HP, Dingeldein-Bettler I, Berchthold W, et al. Childhood NHL in Switzerland: Incidence and survival of 120 study and 42 non-study patients. Med Pediatr Oncol. 1995;24:281–286. [PubMed]
13. Elting LS, Cooksley C, Nebiyou Bekele BN, et al. Generalizability of cancer clinical trial results – Prognostic differences between participants and nonparticipants. Cancer. 2006;106:2452–2458. [PubMed]
14. Gawande A. New York, NY: Metropolitan Books; 2009. The Checklist Manifesto: How to Get Things Right.
15. Medical Imaging and Technology Alliance: Radiation Dose and Safety.
16. Barthelemy-Brichant N, Sabatier J, Dewé W, et al. Evaluation of frequency and type of errors detected by a computerized record and verify system during radiation treatment. Radiother Oncol. 1999;53:149–154. [PubMed]

Articles from Journal of Oncology Practice are provided here courtesy of American Society of Clinical Oncology