Search tips
Search criteria 


Logo of jamiaAlertsAuthor InstructionsSubmitAboutJAMIA - The Journal of the American Medical Informatics Association
J Am Med Inform Assoc. 2007 May-Jun; 14(3): 264–268.
PMCID: PMC2244885

Evaluation of an Online Platform for Cancer Patient Self-reporting of Chemotherapy Toxicities

Ethan Basch, MD, MSc, a , b , * David Artz, MD, MBA, a , b Alexia Iasonos, PhD, a John Speakman, a Kevin Shannon, c Kai Lin, a Charmaine Pun, c Henry Yong, c Paul Fearn, d Allison Barz, a Howard I. Scher, MD, b Mary McCabe, RN, PhD, e and Deborah Schrag, MD, MPH a , b


The current mechanism for monitoring toxicity symptoms in cancer trials depends on a complex paper-based process. Electronic collection of patient-reported outcomes (PROs) may be more efficient and accurate. An online PRO platform was created including a simple data entry interface, real-time report generation, and an alert system to e-mail clinicians when patients self-report serious toxicities. Feasibility assessment involving 180 chemotherapy patients demonstrated high levels of use at up to 40 follow-up clinic visits per patient over 16 months (85% of patients at any given visit), with high levels of patient and clinician acceptance and satisfaction (>95%). Alerts were used as the basis for delayed chemotherapy treatments, dose modifications, and scheduling changes. These results demonstrate that online patient-reporting is a feasible strategy for chemotherapy toxicity symptom monitoring, and may improve safety and satisfaction with care. Ongoing multi-center research will evaluate the impact of this approach on clinical and administrative outcomes.


Monitoring of patient symptoms during chemotherapy is a cornerstone of oncology practice and is mandated in cancer clinical trials. The standard method for collecting this information includes paper forms filled out by staff members, followed by transcription into local databases. This approach has been criticized as inefficient, error-prone, and not representative of the true patient experience. 1 An alternative method, direct patient reporting of symptoms as electronic “patient-reported outcomes” (PROs), has been suggested to improve the comprehensiveness, quality, and timeliness of symptom data collection. 2–8 PRO collection has been advocated by the Food & Drug Administration, 9–13 and is already standard in other research contexts such as quality of life assessment and non-oncology trials. 14–16 However, there has been scant evaluation of the feasibility or potential benefits of using electronic PROs for toxicity monitoring in cancer patients. 17–19

To address these issues, a multidisciplinary research program was developed at Memorial Sloan-Kettering Cancer Center with the following goals: 1) to create and assess a technology platform for electronic patient self-reporting; 2) to asses the feasibility of using this data gathering method; and 3) to evaluate the impact of this approach on clinical outcomes and administrative efficiency. In the following report we describe the development of this patient self-reporting portal, and the results of sequential studies evaluating the feasibility and clinical impact of this system.


A working group was assembled in 2003 to select a technology platform and content for a patient symptom-reporting portal; to facilitate design and implementation of the system; and to oversee a research program for evaluating the potential clinical and administrative benefits of the system. This group included physicians, nurses, behavioral scientists, and representatives from hospital administration and information services. The group outlined a work flow and timeline based on a set of identified considerations for the creation and scientific evaluation of an electronic PRO platform ([triangle]). The underlying hypothesis was that patient self-reporting may improve the quality and efficiency of symptom toxicity data collection, and may ultimately benefit the well-being and satisfaction of patients.

Table 1
Table 1 Questions and Considerations When Developing an Electronic PRO Questionnaire Platform

Toxicity-related symptom burden was identified as the most pertinent information that can be collected from chemotherapy patients via a PRO approach. Such information is frequently the basis for dose modification or supportive medications, or in the case of clinical trials for regulatory reporting. The mandated instrument for this purpose in oncology trials is the National Cancer Institute’s Common Terminology Criteria for Adverse Events (CTCAE). 20,21 The CTCAE is comprised of over 1000 individual items, each graded on a 5-point ordinal scale, including toxicity-related phenomena such as laboratory values (e.g., anemia) as well as symptoms (e.g., nausea). Each numeric grade is associated with a descriptor, such that grade 1 = mild; grade 2 = moderate; grade 3 = severe; grade 4 = disabling; and grade 5 = death. Grades of 3 and greater are reportable during trials, and may be the grounds for management changes. Although the CTCAE was designed for clinician reporting, the rating scale was readily adaptable for patient self-reporting, 22 and a paper correlation study was conducted to assess the relationship between patient and clinician responses to similar items. 23 The CTCAE items developed to test PRO were added to a master item bank from which electronic questionnaires could be constructed based on the characteristics of a patient population of interest.

For the design of the online patient portal, flexibility was desired to integrate advanced questionnaire techniques, including item response theory and computerized adaptive testing. 24–27 An administrator interface was necessary to control patient and clinician users, as well as a patient interface to securely enter symptom data, view past entered data, and dynamically generate longitudinal symptom reports.

Ease of use for those without prior computer experience was considered essential. Based on user acceptance testing involving 20 patients, a touchscreen design with one item per page was adopted ([triangle]). The longitudinal report was designed to display individual symptom grades for any given day of data entry, as well as temporal relationships to chemotherapy administration. An automated rapid reporting function was included for potentially serious toxicities whereby a designated clinician would be informed electronically in real-time by e-mail whenever a concerning toxicity was self-reported by a patient.

Figure 1
Example Data Entry Screen.

A custom form building application was developed to generate XML-encoded questionnaires using symptoms derived from the item bank, dynamically generate Oracle tables for each questionnaire, and to accommodate integration of item response theory and computerized adaptive testing techniques. To assure security, the data entry interface can be accessed via the Internet outside of the institution’s external firewall, the Web server resides inside an external firewall, the application server resides behind an internal firewall, and all protected health information is stored in a secure database server behind a second internal firewall. Medical record numbers are the only personally identifying information associated with reported symptoms. An automated audit trail records user access to patients’ reported data. We were required by institutional policy to institute a 6-minute time-out for lack of mouse movement and 20-minute absolute system time-out for all users. Touchscreen computers were installed in clinic waiting areas for patients to enter data at appointments.

User acceptance testing with 30 patients found that all participants were able to login, complete, and submit questionnaire forms successfully. Mean duration to complete a questionnaire including 19 items and an open-ended data entry field was 4.7 minutes (median 4.2 minutes). CTCAE items often include lengthy text and take longer per item to read than simple multiple choice items.


To determine the level of access to computers and the Internet in our patient population, an initial baseline paper survey was administered to 443 cancer outpatients and their companions in 2000; 64% of patients and 76% of companions owned computers, and Internet access was available at home to 58% and 68%, respectively. 28 A follow-up survey of 90 outpatients in 2005 found 80% of patients to have regular Web access, 73% with prior Web use, and 65% with e-mail experience. All patients without regular access were older than 75, whereas all patients younger than 75 had regular access. All of those with home access expressed interest in electronic symptom self-tracking. When shown a demonstration, all with Internet experience expressed interest in using the portal.

To assess the feasibility of the portal for relatively ill cancer patients receiving chemotherapy, a single-center study was conducted with 180 cancer patients receiving chemotherapy. 22,29–31 More than 85% of participants logged in at each sequential clinic visit during an eight-week observation period. Two-thirds voluntarily logged in from home computers without prompting. In a subset of 100 patients who were followed over 16 months, similar levels of in-clinic login compliance was seen after up to 40 visits, with no significant attrition over time. Rates of voluntary home logins were high in younger female patients with gynecologic malignancies (66%), but were less impressive in older men with lung cancer (15%). Patient characteristics significantly associated with greater compliance included prior computer experience but not age, sex, education, baseline performance status, or cancer stage.

The impact of electronic transmission of PRO symptom information on clinician behavior was evaluated in 60 cancer patients receiving chemotherapy. Consecutive outpatients being seen in a cancer clinic were approached and invited to participate until 60 with Internet access (at home or office) were enrolled. The demographics of this population were similar to Internet-avid patients at this institution overall in terms of age, education, income, diagnosis, and performance status. 28 All participants were given access to the online self-reporting portal, but no reminders were given to login between clinic visits. During the study period, all patients logged in at least once, with a mean number of logins of 9 (median 8; range 1–30). The mean number of clinic visits during this time period was 3 (range 1–6). Longitudinal reports of prior symptom entries were self-viewed by 55 (92%) participants.

Any time a patient reported a grade 3 (“severe”) or grade 4 (“disabling”) toxicity symptom, an e-mail was sent to the responsible clinical team (nurse and/or physician) with this information. No specific instructions were given to clinicians about how to respond to these alerts. During an eight-week period, 57 grade 3 or 4 toxicities were reported by 25 different patients, of which 42 originated from patients at home ([triangle]). In response to these alerts, 17 clinical actions were taken, including delaying chemotherapy, calling patients, setting up new appointments, and medication changes.

Figure 2
Automated e-mail alerts triggered by patient-reported grade 3/4 (severe/disabling) symptoms, and clinician responses to alerts.

Across studies, patient satisfaction and clinician acceptance of the portal was high overall. Most patients found the portal “easy to use” (96%), “useful” (92%), “improved communication” (85%), and wished to continue using the portal (96%). Among participating physicians, 89% discussed reports with their patients, 78% felt reports “accurately reflect true patient clinical status,” and 89% felt the system was “useful for toxicity monitoring.”


Development of electronic patient interfaces requires consideration of administrative and regulatory requirements including data security and privacy. Administrators may be reluctant to support such systems due to expenses associated with installation and maintenance without tangible returns on investment and concern that clinicians and patients may resist adoption of any new system. It is therefore crucial to demonstrate the feasibility of implementing such systems, as well as the value of portals in terms of clinical and administrative endpoints and satisfaction with care.

Our research program was conceived to identify the optimal approach for building a flexible patient portal, to assess its feasibility, and to measure the potential value of such systems. Completed studies suggest feasibility, high levels of patient satisfaction, clinician acceptance, and willingness of staff to base management decisions on patient-reported information.

Ongoing work is measuring the impact of this approach on hard clinical outcomes. A randomized controlled trial is in progress in which patients are assigned to portal access vs. routine care (no portal). We are measuring differences between groups in the number of canceled or delayed chemotherapy treatments, dose modifications, emergency room visits, unscheduled clinic visits, telephone contacts with staff, quality of life, and satisfaction with care. In addition, 14 cancer centers have agreed to participate in a multi-institutional feasibility assessment through the National Cancer Institute cooperative group mechanism. This study is designed to evaluate barriers to implementing and maintaining the system, training of personnel, patient satisfaction, and clinician acceptance. Results of these studies are anticipated in 2009.

There has been recent attention at the National Cancer Institute to the development of systems for detection and communication of serious adverse events during clinical trials. 32,33 Our results suggest that electronic patient reporting platforms can allow for rapid detection of potentially serious symptoms in real-time, and can elicit clinician responses. In the setting of chemotherapy and cancer clinical trials, during which patients receive toxic therapies, the ability to improve outpatient monitoring may significantly improve clinical outcomes, research data quality, and the patient experience.

Nonetheless, it remains unclear if severely ill or incapacitated patients are capable of consistently self-reporting via this means. Patients in our studies had advanced cancer, but most were sufficiently functional to receive chemotherapy in an ambulatory clinic setting. Therefore other strategies to collect real-time data from homebound patients may be necessary, for example via a multi-tier approach in which those who fail to self-report online receive a call via an interactive voice response system, with a human backup call for those who do not comply.

Beyond issues of feasibility or quality improvement, engaging patients as active participants in their own care conveys the message that symptom control and treatment tolerance are priorities to clinicians and researchers. Harnessing novel technologies to enhance communication and information gathering is therefore a key element in the growth of patient-centered models of care.


1. Gwede CK, Johnson DJ, Daniels SS, Trotti A. Assessment of toxicity in cooperative oncology clinical trials: the long and short of it J Oncol Manag 2002;11(2):15-21. [PubMed]
2. Velikova G, Booth L, Smith AB, Brown PM, Lynch P, Brown JM, et al. Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial J Clin Oncol 2004;22(4):714-724. [PubMed]
3. Science Panel on Interactive Communication and Health Wired for Health and Well-Being: the Emergence of Interactive Health Communication. Washington, DC: US Department of Health and Human Services, US Government Printing Office; 1999. April.
4. Robinson TN, Patrick K, Eng TR, Gustafson D, Science Panel on Interactive Communication and Health An evidence-based approach to interactive health communication: a challenge to medicine in the information age JAMA 1998;280(14):1264-1269. [PubMed]
5. Fowler FJ, Barry MJ, Lu-Yao G, Roman A, Wasson J, Wennberg JE. Patient-reported complications and follow-up treatment after radical prostatectomyThe National Medicare Experience: 1988–1990 (updated June 1993). Urology 1993;42(6):622-629. [PubMed]
6. Sloan JA, Cella D, Hays RD. Clinical significance of patient-reported questionnaire data: another step toward consensus J Clin Epidemiol 2005;58(12):1217-1219. [PubMed]
7. Kim JH, Ferziger R, Kawaloff HB, Sands DZ, Safran C, Slack WV. A web-based rapid prototyping and clinical conversational system that complements electronic patient record system Medinfo 2001;10(Pt 1):628-632.
8. Farzanfar R, Finkelstein J, Friedman RH. Testing the usability of two automated home-based patient-management systems J Med Syst 2004;28(2):143-153. [PubMed]
9. Acquadro C, Berzon R, Dubois D, Leidy NK, Marquis P, Revicki D, et al. Incorporating the patient’s perspective into drug development and communication: an ad hoc task force report of the Patient-Reported Outcomes (PRO) Harmonization Group meeting at the Food and Drug Administration, February 16, 2001 Value Health 2003;6(5):522-531. [PubMed]
10. Morris LA, Miller DW. The regulation of patient-reported outcome claims: need for a flexible standard Value Health 2002;5(4):372-381. [PubMed]
11. U.S. Department of Health and Human Services, Food and Drug Administration Guidance for industry: Patient-reported outcome measures: Use in medical product development to support labeling claims. 2006. Draft guidance released for comment, February 3.
12. Burke L, Patrick D, Scott J, Powers J, Rock E, Monroe S, et al. FDA response to papers toward a final guidance (presentation) FDA Guidance on Patient Reported Outcomes: Discussion, Dissemination, and Operationalization. 2006. 2006 February 25; Chantilly, Virginia.
13. Willke RJ, Burke LB, Erickson P. Measuring treatment impact: a review of patient-reported outcomes and other efficacy endpoints in approved product labels Control Clin Trials 2004;25(6):535-552. [PubMed]
14. Wiklund I. Assessment of patient-reported outcomes in clinical trials: the example of health-related quality of life Fundam Clin Pharmacol 2004;18(3):351-363. [PubMed]
15. Fairclough DL. Patient reported outcomes as endpoints in medical research Stat Meth Med Res 2004;13(2):115-138.
16. Webster K, Cella D, Yost K. The Functional Assessment of Chronic Illness Therapy (FACIT) Measurement System: properties, applications, and interpretation Health Qual Life Outcomes 2003;1(1):79. [PMC free article] [PubMed]
17. Cella D, Peterman A, Hudgens S, Webster K, Socinski MA. Measuring the side effects of taxane therapy in oncology: the functional assessment of cancer therapy-taxane (FACT-taxane) Cancer 2003;98(4):822-831. [PubMed]
18. Le PP, Kohane IS, Weeks JC. Using a pen-based computer to collect health-related quality of life and utilities information Proc Annu Symp Comput Appl Med Care 1995:839-843.
19. Mooney KH, Beck SL, Friedman RH, Farzanfar R. Telephone-linked care for cancer symptom monitoring: a pilot study Cancer Pract 2002;10(3):147-154. [PubMed]
20. National Cancer Institute. Common Terminology Criteria for Adverse Events (CTCAE) v3.0. Available at: Accessed on December 21, 2005.
21. National Cancer Institute, Cancer Therapy Evaluation Program NCI Guidelines—Expedited adverse event reporting requirements for NCI investigational agents. Bethesda, MD: National Cancer Institute; 2001.
22. Basch EAD, Dulko D, Scher K, Sabbatini P, Hensley M, Mitra N, Speakman J, McCabe M, Schrag D. Patient online self-reporting of toxicity symptoms during chemotherapy J Clin Oncol 2005;23(15):3552-3561. [PubMed]
23. Basch E, Iasonos A, McDonough T, Barz A, Culkin A, Kris MG, et al. Patient versus clinician symptom reporting using the National Cancer Institute Common Terminology Criteria for Adverse Events: results of a questionnaire-based study Lancet Oncol 2006;7(11):903-909. [PubMed]
24. Fries JF, Bruce B, Cella D. The promise of PROMIS: using item response theory to improve assessment of patient-reported outcomes Clin Exp Rheumatol 2005;23(5 Suppl 39):S53-S57. [PubMed]
25. Lai JS, Dineen K, Reeve BB, Von Roenn J, Shervin D, McGuire M, et al. An item response theory-based pain item bank can enhance measurement precision J Pain Symptom Manage 2005;30(3):278-288. [PubMed]
26. Yost KJ, Sorensen MV, Hahn EA, Glendenning GA, Gnanasakthy A, Cella D. Using multiple anchor- and distribution-based estimates to evaluate clinically meaningful change on the Functional Assessment of Cancer Therapy-Biologic Response Modifiers (FACT-BRM) instrument Value Health 2005;8(2):117-127. [PubMed]
27. Lai JS, Cella D, Dineen K, Bode R, Von Roenn J, Gershon RC, et al. An item bank was created to improve the measurement of cancer-related fatigue J Clin Epidemiol 2005;58(2):190-197. [PubMed]
28. Basch EM, Thaler HT, Shi W, Yakren S, Schrag D. Use of information resources by patients with cancer and their companions Cancer 2004;100(11):2476-2483. [PubMed]
29. Culkin A, Barz A, Schrag D, Basch E. Electronic symptom reporting by cancer patients. 2006. Oncology Nursing Society 31st Annual Congress; Boston, MA: May.
30. Basch E, Artz D, Dulko D, Scher K, Sabbatini P, Hensley M, et al. Patient self-reporting of toxicity symptoms during chemotherapy. 2005. American Society of Clinical Oncology 2005 Annual Meeting; Orlando, FL: May.
31. Basch E, Barz A, Kris M, Culkin A, Schrag D. Online reporting of toxicity symptoms by lung cancer patients during chemotherapy. 2005. International Society for Pharmacoeconomics and Outcomes Research 11th Annual International Meeting; Philadelphia, PA: May.
32. Goldberg RM, Sargent DJ, Morton RF, Mahoney MR, Krook JE, O’Connell MJ. Early detection of toxicity and adjustment of ongoing clinical trials: the history and performance of the North Central Cancer Treatment Group’s real-time toxicity monitoring program J Clin Oncol 2002;20(23):4591-4596. [PubMed]
33. Sargent DJ, Goldberg RM, Mahoney MR, Hillman DW, McKeough T, Hamilton SF, et al. Rapid reporting and review of an increased incidence of a known adverse event J Natl Cancer Inst 2000;92(12):1011-1013. [PubMed]

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of American Medical Informatics Association