Search tips
Search criteria 


Logo of jcoHomeThis ArticleSearchSubmitASCO JCO Homepage
J Clin Oncol. 2009 April 1; 27(10): 1621–1626.
Published online 2009 March 2. doi:  10.1200/JCO.2008.16.6348
PMCID: PMC2668969

Cancer Patient Assessment and Reports of Excellence: Reliability and Validity of Advanced Cancer Patient Perceptions of the Quality of Care



Consumer perceptions are important measures of the quality of cancer care. This article describes the validation of new measures of the quality of cancer care at the time of diagnosis and treatment for advanced cancer with life-limiting prognosis.


Focus groups, review of guidelines, and an expert panel were used to construct two surveys of the quality of cancer care. A prospective cohort study examined the reliability and validity of three problem scores (ie, counts of the opportunities to improve the quality of care) that examine care at the time of diagnosis and initial treatment.


At the first interview, 58% of 206 cancer patients (54.9% females; 27.5% with lung cancer; 5.4% with pancreatic cancer; 30.4% with colorectal cancer; 18.6% with breast cancer; mean age, 66.6 years) identified one or more concerns with communication about being diagnosed with advanced cancer. At the second interview, 57.0% of the respondents voiced one or more concerns about treatment communication, and 30.2% expressed one or more concerns about the experience of treatment. Each of the problem scores demonstrated both internal consistency with Cronbach's α > .75 and short-term stability of responses in a subsample that had the survey administered twice in 72 hours. Factor analysis largely confirmed the proposed scale structure. All three measures demonstrated moderate correlations suggesting evidence of construct validity.


The three proposed problem scores demonstrate evidence of reliability and validity that warrants further testing to examine their responsiveness and discriminate validity in larger, more generalizable samples.


Donabedian1 noted that, “achieving and producing health and satisfaction, as defined for its individual members by a particular society or subculture, is the ultimate validator of the quality of care.” Increasingly, patients' and family members' perceptions have been recognized as important outcomes of the quality of cancer care. The measurement of consumer perceptions with typical satisfaction measures (eg, completely satisfied to completely dissatisfied) has raised important methodologic concerns. Williams and Calnan2 found that 95% of persons were satisfied with medical care, yet 38% reported that they had difficulty discussing personal problems with their physician and 35% felt that physicians did not spend enough time with them. In response to concerns with satisfaction measures, Cleary et al3 created patient-centered reports, where questions were framed to ask the patient or family member to report as an expert witness on a process of care and whether that process met a certain standard (eg, “Were you told the purpose of your medications in a way that you could understand?”).

Previously, Teno and colleagues developed a survey, the Family Evaluation of Hospice Care, using patient-centered reports to examine bereaved family member perceptions of the quality of dying. This current research builds on that work to develop a survey that reflects the concerns of patients diagnosed with advanced cancer. In this research effort, we used a structured literature review of applicable guidelines, focus groups with consumers, and the advice of an expert panel to develop two surveys that focused on critical time points in the disease trajectory—the experience of being diagnosed with an advanced stage of cancer and beginning treatment for the cancer. In this study, we report the reliability and validity of three measures developed to measure the quality of medical care received at the time of diagnosis of advanced-stage cancer and during initial treatment.


Informed by focus groups, an expert panel, and review of existing guidelines,416 we defined high-quality cancer care based on institutions meeting the following needs of cancer patients: health care providers sensitively communicate with patients about their prognosis and promote shared decision making to the extent that patients want to be involved; patients are educated about what they can expect during treatment, what to monitor, and who they can turn to for help; patients are provided with their desired emotional support and symptom amelioration; patients are treated with respect; cancer treatment sessions support comfort, ensure privacy, and promote healing; and care is coordinated among health care providers. Two of these needs (patients are provided with their desired emotional support and symptom amelioration and patients are treated with respect) have existing measures.17

An expert panel reviewed each item in the developed surveys for relevance to high-quality cancer care and whether the process of care was under the control of a health care provider (either the physicians or other members of their staff). The survey underwent cognitive testing with modification or deletion of items that were not clear to the respondents.

Data Collection

After a pretest in Rhode Island, both surveys were tested on a prospective cohort of persons diagnosed with cancer at an advanced stage that implied a life-limiting illness. Cancer patients with the following diagnoses were recruited from oncologist offices in New Hampshire, Connecticut, and Rhode Island: stage IV pancreatic cancer; inoperable or stage IV non–small-cell lung cancer; stage IV breast cancer that failed first-line treatment; stage IV skin cancer; and stage IV genitourinary cancers that failed first-line treatment. Office staff identified potential patients who were screened by the oncologist, and then patient permission was requested to release their information to researchers at Brown University or Dartmouth Medical School. If that permission was granted, a research assistant either met or called the patient to inquire if the patient would participate in the study. Only 99 of the 299 eligible patients at the Dartmouth site chose to participate because there was another study simultaneously recruiting patients for a randomized controlled trial of a palliative care intervention. At the Brown University data collection site, 119 of the 180 eligible cancer patients agreed to participate in the survey.

The first survey was administered shortly after the patient was diagnosed with or progressed to an advanced stage of cancer. A second survey was administered 1 to 2 months later when the patient had completed at least one treatment cycle. Twenty-three respondents volunteered to have the survey administered again 48 to 72 hours later so we could examine the short-term reliability.

Two types of response tasks were used to measure patients' perceptions of the quality of cancer care at each time period. The first type relied on the methodology of patient-centered reports, which ask patients to report on their perceptions of the quality of care using response categories of “yes/no” or a four-point scale of “always,” “usually,” “sometimes,” and “never.” The second type of response task measured patient unmet needs by rating a need as “less than was needed,” “more than was needed,” or “just the right amount.”

Because high-quality medical care strives for excellence, responses to each question were recoded to create dichotomous measures that indicated opportunities to improve (ie, concerns with care). For example, responses to the question “How often are your cancer care providers available to speak with you when you have a concern about your chemotherapy?” ranged from “always” to “never.” For this particular question, we considered an answer of anything other than “always” as an opportunity to improve and thus dichotomized the variable by assigning a value to 0 to “always” and a value of 1 to all other responses. For the unmet need questions, responses of “less than was needed” and “more than was needed” were considered opportunities to improve and assigned a value of 1. See columns 1 and 2 of Table 1 for item descriptions and rates of concerns. This scoring decision reflects the intended use of the tool for quality improvement purposes.

Table 1.
Survey Items and Rates of Opportunities to Improve That Compose the Three Proposed Problem Scores

Measures: Problem Scores

Three problem scores were created by summing the number of concerns across various items. The first problem score focused on the time of diagnosis of advanced cancer. It consisted of 20 items that focused on communication surrounding the diagnosis of advanced cancer, prognosis, and the initial staging (or restaging) work-up. We created the other two problem scores using the second survey. They consisted of a 15-item score focusing on communication about treatment decision making and a 10-item scale regarding respondents' experiences with chemotherapy or radiation treatment. Higher scores indicated more opportunities to improve the quality of care (ie, more concerns with care). See column 1 of Table 1 to examine the particular items that went into each problem score.

Several additional questions were included for descriptive purposes and for examining the validity of the proposed problem scores. Respondents answered questions regarding the quality of their oncologists' care and their own quality of life using a 5-point likert scale ranging from “excellent” to “poor.” We included the five-item Mental Health Inventory (MHI-5), which has known reliability and validity.18 We also asked respondents to rate particular aspects of cancer care on a scale of 0 (indicating worst care) to 10 (indicating best care). Finally, we collected limited information about particular symptoms, as well as the frequency, severity, and distress of those symptoms.


A correlation analysis examined the psychometric properties of the three proposed problem scores. Cronbach's α was used to examine the internal consistency for each proposed problem score. Depending on the distribution of the problem score, we examined the interitem and item-to-total correlations with either a Pearson or a Spearman correlation coefficient. Factor analysis was used to determine whether items were loaded on the a priori–specified domains of our conceptual model.

We assessed the criterion validity of the problem scores by correlating the problem scores with each of two measures of the rating of the quality of care. The first measure was a single question that examined overall satisfaction using a 5-point scale that ranged from “excellent” to “poor.” We expected to find a moderate correlation with this question based on its use in other validation studies19 and on the knowledge that problem scores asking about a particular aspect of care may not be the key determinant to overall satisfaction. The second measure was a multi-item scale that rated various aspects of the quality of cancer care using a 0 to 10 rating scale from “worst” to “best possible care” and summed across all items. This scale was based on the work of Teno and colleagues in creating the Family Evaluation of Hospice Care.

As a measure of construct validity, we used the Kruskal-Wallis rank sum test to examine whether respondents with more concerns with each problem score had higher levels of emotional distress as measured by the MHI-520 on the second interview. The same test was used to determine whether the problem scores differentiated the quality of care among patients who received their care either in a comprehensive cancer setting or a private office. Finally, the bivariate associations of each problem score were examined with age, education, race, and sex.


A total of 206 cancer patients participated in the first survey that focused on the time of diagnosis. Participants did not differ from nonparticipants in age or race, but there was a slight tendency for more women to participate in the survey (52.9% women v 48.5% men, P = .34). Table 2 lists the characteristics of sample participants. Of the 206 time 1 respondents, 162 patients participated in a second interview 2 months later, 22 patients either died or were too near death to be interviewed, 14 patients refused to be interviewed, two patients left the practice, and the remaining six patients did not participate for other reasons. Twenty-three respondents agreed to retake the survey between 48 and 72 hours with the finding of an overall agreement of 92.1% on the survey items that compose the three proposed problems scores.

Table 2.
Patient Characteristics

As was indicated earlier, Table 1 lists the frequency of concerns for each item that composed the three problem scores. Table 3 lists a number of psychometric properties of the three problem scores. The first set of columns focuses on degree of missing data and the range of reported concerns for each item that composed the problem score. Column 4 shows the average item score (ie, the rate of concern on each question divided by the total number of questions in the problem score), indicating for each item that, on average, approximately one in 10 respondents voiced a concern. The fifth column of Table 3 shows that the majority of respondents identified at least one concern in each of the first two problem scores, whereas less than one third voiced a concern about any item in the third problem score.

Table 3.
Psychometric Properties

The Cronbach's α exceeded .75 for all three problem scores (column 6 of Table 3). In addition, with the exception of the first problem score, factor analysis suggested a unitary solution. The first problem score had a second factor with an eigenvalue of 2.9 composed of the five questionnaire items that focused on communication about prognosis. Finally, the criterion validity was tested with two measures of the rating of the quality of care previously described (results are presented in the last two columns of Table 3). As predicted, the three problem scores showed a moderate correlation with both of these overall measures.

As a test of convergent validity, we examined the association of a higher problem score with emotional distress as measured by MHI-5. For each of the proposed problem scores, persons with a higher rate of concerns reported more distress as measured by the MHI-5. We also examined the ability of the three problem scores to differentiate between two types of practices (comprehensive cancer center v community-based oncologists). There was a trend for more reported concerns with each problem score among cancer patients receiving their care at comprehensive cancer centers compared with community-based physicians. There was no association (all P > .10) found between sex, race, or the level of education and any of the problem scores. However, there was a trend for older persons to voice fewer concerns than younger persons.


The goal of this study was to develop and validate a survey tool to measure quality of care concerns of patients with advanced-stage cancer at the time of diagnosis and beginning of treatment. Typical measures of consumer perceptions of the quality of care do not address the specific needs of this group and do not use newer survey methodologies to provide more actionable information to guide quality improvement efforts. Many of the enclosed items are important for all cancer patients at the time of diagnosis and treatment. This preliminary validation provides evidence of reliability and validity that warrants further testing to develop a measurement strategy that ensures that the patient's voice informs the assessment of the quality of cancer care.

Previous surveys that examined the cancer patient's perceptions of the quality of care did not use newer survey methodologies and/or did not addressed the unique needs of this population. These surveys focused on information needs2124 or particular types of cancer25 or used only rating scales.26 Only the Ambulatory Oncology Patient Satisfaction Survey27,28 used newer question response tasks, as patient-centered reports, to examine quality of cancer care among all cancer patients. Unlike the Ambulatory Oncology Patient Satisfaction Survey, we chose to develop a survey that focused on the needs of cancer patients with advanced disease at two critical time points in the disease trajectory—the experience of being diagnosed with an advanced stage of cancer and beginning treatment for that stage.

Previous research highlighted the importance of communication and providing the cancer patient with adequate information about his or her prognosis and the treatment plan. The majority of respondents expressed one or more concerns with communication at the time of diagnosis or ongoing treatment. In contrast, only one third of respondents voiced any concerns about the experience of cancer treatment. However, this still represents a tangible target for improvement. Although Sandov et al27 found a higher rate of concerns using patient-centered reports from a more generalizable sample of ambulatory cancer patients in Canada, our sample with lower rates of concern was based on oncologist offices that volunteered to participate in this study.

This preliminary study provides evidence of internal consistency and validity of the proposed three problem scores. Our findings further support reports that persons with more unmet needs for information experience a higher rating of psychological distress compared with patients with fewer unmet information needs.24 Specifically, we found that persons who reported a higher rate of concerns with the quality of care reported a higher level of psychological distress as measured by the MHI-5.

There are important limitations to bear in mind when interpreting this study. Because of a competing protocol at one of the recruitment sites, many patients refused participation in this study at that site. On the basis of available demographic information, women were slightly more likely to participate in this study, and only a small number of minority patients participated. Focus groups were conducted with African Americans in the development of the survey to ensure that their perspective was included in the development of the instrument, but recruitment of patients for the cohort study at potential locations was not possible. The examination of the association of race and other demographic characteristics with the proposed problem scores is preliminary and limited by the number of patients and sites that participated in the study. Thus, the finding that cancer patient perceptions differed between a comprehensive cancer center and private office should be replicated among a larger sample of sites and patients. However, these preliminary results do provide evidence that warrants the further testing of this instrument in more generalizable samples and the translation of the instrument into Hispanic languages for further validation.

High-quality cancer care should address both the curing and caring aspects of care.29 At times, health care providers and patients with advanced cancer that implies a life-limiting prognosis are faced with, “Hoping for the best, but preparing for the worst.”30 Especially at this time, medical care must be patient centered, understanding the needs of the patients and their families, as well as their goals of care. To achieve this vision, we need surveys that collect reliable and valid data about consumer perceptions as one of the measures of the quality of cancer care. This study introduces a set of three problem scores that shows promise. Further studies will help solidify the usefulness of these measures.


We thank the following offices for their participation in the Cancer Assessments and Reports of End-of-Life Treatment study: Vincent A. Armenio, MD; Illya Blokh, MD; Jason Haldas, MD; Richard M. Hellman, MD; Vanessa M. Johnson, MD, MPH; Plakyil J. Joseph, MD; Donald R. Joyce, MD; Robert E. Knisley, MD; Robert Legare, MD; Leslie C. Lockridge, MD; Gabriela B. Masko, MD; Ahmed Nadeem, MD; Nicklas B.E. Oldenburg, MD; Kathy Ann Radie-Keane, MD; Sundaresan T. Sambandam, MD; David B. Stoll, MD, FACP; Anthony F. Testa, MD; Kathy P. Theall, MD; and Scott A. Triedman, MD.

Supported by National Cancer Institute Grant No. CA097358 R01.

Authors' disclosures of potential conflicts of interest and author contributions are found at the end of this article.

The Acknowledgment is included in the full-text version of this article, available online at It is not included in the PDF version (via Adobe® Reader®).


The author(s) indicated no potential conflicts of interest.


Conception and design: Joan M. Teno, Julie C. Lima, Kathleen Doyle Lyons

Manuscript writing: Joan M. Teno, Julie C. Lima, Kathleen Doyle Lyons


1. Donabedian A. Evaluating the quality of medical care. Milbank Mem Fund Q. 1966;44(suppl 1):166–206. [PubMed]
2. Williams SJ, Calnan M. Convergence and divergence: Assessing criteria of consumer satisfaction across general practice, dental and hospital care settings. Soc Sci Med. 1991;33:707–716. [PubMed]
3. Cleary PD, Edgman Levitan S, Roberts M, et al. Patients evaluate their hospital care: A national survey. Health Aff Millwood. 1991;10:254–267. [PubMed]
4. Griffin JP, Nelson JE, Koch KA, et al. End-of-life care in patients with lung cancer. Chest. 2003;123(suppl 1):312S–331S. [PubMed]
5. Sloan PA, Vanderveer BL, Snapp JS, et al. Cancer pain assessment and management recommendations by hospice nurses. University of Kentucky, Lexington, Kentucky. J Pain Symptom Manage. 1999;18:103–110. [PubMed]
6. Coackley A, Hutchinson T, Saltmarsh P, et al. Assessment and management of fatigue in patients with advanced cancer: Developing guidelines. Int J Palliat Nurs. 2002;8:381–388. [PubMed]
7. Lee J, Marchbank A, Goldstraw P. Implementation of the British Thoracic Society recommendations for organising the care of patients with lung cancer: The surgeon's perspective. Ann R Coll Surg Engl. 2002;84:304–308. [PMC free article] [PubMed]
8. Scottish Intercollegiate Guidelines Network (SIGN) Management of patients with lung cancer. A national clinical guideline.
9. American Cancer Society and National Comprehensive Cancer Network. Distress treatment guidelines for patients.
10. Peterson R, Whitman H, Smith J. Oncology Nursing Society Multicultural Outcomes: Guidelines for Cultural Competence. Pittsburgh, PA: Oncology Nursing Society; 1999.
11. Working Group of the Ethical Issues in Medicine Committee of the Royal College of Physicians. Principles of pain control in palliative care for adults. J R Coll Physicians. 2000;34:350–352. [PubMed]
12. Cancer care during the last phase of life. J Clin Oncol. 1998;16:1986–1996. [PubMed]
13. Department of Health Clinical Outcomes Group, Cancer Guidance Subgroup. Guidance on Commissioning Cancer Services: Improving Outcomes in Lung Cancer—The Manual. London, United Kingdom: National Health Service; 1998.
14. Forcina JM. Re: National Institutes of Health State-of-the-Science Conference Statement: Symptom Management in Cancer: Pain, Depression, and Fatigue, July 15-17, 2002. J Natl Cancer Inst. 2004;96:1109–1110. [PubMed]
15. University of Iowa Gerontological Nursing Interventions Research Center. Acute pain management in the elderly.
16. National Comprehensive Cancer Network/American Cancer Society. Advanced cancer and palliative care: treatment guidelines for patients.
17. Teno JM, Clarridge B, Casey V, et al. Validation of Toolkit After-Death Bereaved Family Member Interview. J Pain Symptom Manage. 2001;22:752–758. [PubMed]
18. Rumpf HJ, Meyer C, Hapke U, et al. Screening for mental health: Validity of the MHI-5 using DSM-IV axis I psychiatric disorders as gold standard. Psychiatry Res. 2001;105:243–253. [PubMed]
19. Hays RD, Shaul JA, Williams VS, et al. Psychometric properties of the CAHPS 1.0 survey measures: Consumer Assessment of Health Plans Study. Med Care. 1999;37:MS22–MS31. [PubMed]
20. Berwick DM, Murphy JM, Goldman PA, et al. Performance of a five-item mental health screening test. Med Care. 1991;29:169–176. [PubMed]
21. Thomas R, Kaminski E, Stanton E, et al. Measuring information strategies in oncology: Developing an information satisfaction questionnaire. Eur J Cancer Care (Engl) 2004;13:65–70. [PubMed]
22. Jones R, Pearson J, McGregor S, et al. Cross sectional survey of patients' satisfaction with information about cancer. BMJ. 1999;319:1247–1248. [PMC free article] [PubMed]
23. Liekweg A, Eckhardt M, Taylor SC, et al. Psychometric assessment and application of a questionnaire measuring patient: Satisfaction with information on cancer treatment. Pharm World Sci. 2005;27:96–103. [PubMed]
24. Llewellyn CD, Horne R, McGurk M, et al. Development and preliminary validation of a new measure to assess satisfaction with information among head and neck cancer patients: The satisfaction with cancer information profile (SCIP) Head Neck. 2006;28:540–548. [PubMed]
25. Brett J, Austoker J. Development and validation of the EBS: A measure to assess women's experience at all stages of the breast screening process. J Public Health (Oxf) 2004;26:79–83. [PubMed]
26. Bredart A, Razavi D, Robertson C, et al. A comprehensive assessment of satisfaction with care: Preliminary psychometric analysis in an oncology institute in Italy. Ann Oncol. 1999;10:839–846. [PubMed]
27. Sandoval GA, Brown AD, Sullivan T, et al. Factors that influence cancer patients' overall perceptions of the quality of care. Int J Qual Health Care. 2006;18:266–274. [PubMed]
28. Watson DE, Mooney D, Peterson S. Patient experiences with ambulatory cancer care in British Columbia, 2005/06.
29. McCorkle R. Response to Oberst. Cancer. 1984;53:2373–2375.
30. Teno JM. Quality of Care and Quality Indicators for End-of-Life Cancer Care: Hope for the Best, Yet Prepare for the Worst. Washington, DC: National Academy Press; 2001. pp. 96–131.

Articles from Journal of Clinical Oncology are provided here courtesy of American Society of Clinical Oncology