|Home | About | Journals | Submit | Contact Us | Français|
Clinical practice guidelines can be used to help develop measures of quality of cancer care. This paper describes the use of a Cancer Care Quality Measurement System (CCQMS) for monitoring these measures for colorectal cancer in the Veterans Health Administration.
The CCQMS assessed practice guideline concordance primarily based on colon (14 indicators) and rectal (11 indicators) cancer care guidelines of the National Comprehensive Cancer Network (NCCN). Indicators were developed with input from VHA stakeholders with the goal of examining the continuum of diagnosis, neoadjuvant therapy, surgery, adjuvant therapy, and survivorship surveillance and/or end-of-life care. In addition, 9 measures of timeliness of cancer care were developed. The measures/indicators formed the basis of a computerized data abstraction tool that produced reports on quality of care in real-time as data were entered.
The tool was developed for a 28-facility learning collaborative, the Colorectal Cancer Care Collaborative (C4), aimed at improving CRC care quality. Data on 1,373 incident stage I-IV CRC cases were entered over approximately 18 months. Data were used to target and monitor quality improvement activities. The primary opportunity for improvement involved surveillance colonoscopy and services in patients after curative intent treatment.
NCCN guidelines were successfully used to develop a measurement system for VHA research-operations quality improvement partnership.
The Veterans Health Administration (VHA) is the largest provider of cancer care in the United States. With approximately 5.5 million patients receiving care under the auspices of 153 medical centers in 2008, the VHA has the largest fully-integrated delivery system in the country.1 The system treats approximately three percent of U.S. cancer cases (>43,000 in 2005). As in the rest of the country,2 colorectal cancer (CRC) is the third most common cause of cancer in the VHA.3 There were >4,600 new cases of CRC entered into the Veterans Affairs (VA) Central Cancer Registry in 2005 (11% of VHA cancer cases).
Recognizing the need to reduce the time from positive screening to diagnosis and enhance the use of guideline-concordant care, the VHA began the Colorectal Cancer Care Collaborative (C4) quality improvement program in 2005. Between February 2007 and April 2008, C4 conducted the Colorectal Cancer Treatment Improvement Collaborative project. As part of this quality improvement effort, the Cancer Care Quality Measurement System (CCQMS) was developed to collect data on cancer care. CCQMS produces reports on the degree of guideline-concordant care based on the National Comprehensive Cancer Network (NCCN) practice guidelines as well as timeliness of CRC care.4, 5 The measurement tool was developed to evaluate the continuum of CRC care, from diagnosis through treatment and surveillance.
While the existence of this tool has been previously noted as part of summaries of the C4 initiative,4, 5 the purpose of this paper is to describe the development of specific quality indicators and CCQMS computerized tool and use of the CCQMS among the 28 facilities participating in the quality improvement collaborative.
The development and use of the CCQMS was approved by the Institutional Review Board of the Durham Veterans Affairs (VA) Medical Center. The tool was used by VA medical centers that volunteered to participate in the 28-facility VHA Colorectal Cancer Treatment Improvement Collaborative.
Based on the quality improvement collaborative methodology developed by the Institute of Healthcare Improvement in the 1990’s,6 the VHA conducted a collaborative focused on enhancing the guideline concordance of care, timeliness of care, and patient experience with care among Veterans with CRC. The collaborative has been fully described elsewhere.4, 5 Briefly, a core feature of collaboratives is utilizing rapid cycle quality improvement – Plan-Do-Study-Act cycles – to test improvements to the care system using data related to quality of care.7, 8 VHA Office of Quality and Performance (since renamed Office of Informatics and Analytics) invited each VHA regional Veterans Integrated Service Network (VISN) to nominate facilities for participation. Participating medical centers had to commit to both quality improvement and measurement activities in a participation contract signed by the medical center director, chief of staff, and chief nursing officer. Teams consisting of physicians, nurses, other clinicians, and non-clinician administrators planned and worked with facilities to conduct tests of change. In the C4 treatment collaborative, teams learned from each other through two in-person learning sessions, monthly conference calls, an email listserv, calls on special topics (e.g. ensuring surveillance colonoscopies for CRC survivors), and examples of improvement tools collaboratively developed with collaborative staff. Central to the collaborative was the use of data to target improvement activities and monitor changes. These data were collected using the CCQMS.4, 5
The CCQMS was developed with the intent of identifying facility-level and VHA-wide deviations from established standards of care to better enable cancer care quality improvement efforts. The data abstraction tool was written in C-Sharp (C#) programming language using the Microsoft.net framework and was comprised of approximately 230 data elements. Staff at the 28 facilities participating in the C4 treatment collaborative entered information pertaining to newly diagnosed colorectal cancer patients who received care at their VA medical center.
During the collaborative, the tool was housed within the VA-protected computer environment (i.e. behind the VA firewall). To assure that data security standards were met, all data were stored on a centralized VA server with no data stored on local computers or networks. When an abstractor answered an individual question, data were transferred to the centralized server. Additional features of the tool include: 1) a system for indicating whether case data require updating/completion; 2) the ability to search for entered cases; 3) a data verification feature; 4) helpful hints for specific questions; 5) printable reports updated in real time as data are entered; 6) ability to create reports by year of diagnosis; 7) links to key printable documents about the CCQMS (i.e. directions, question list, data dictionary, quality indicator definitions); and 8) a centralized e-mail helpdesk that operated during the collaborative.
CCQMS data were used to calculate 25 quality indicators based on the NCCN treatment guidelines for colon cancer9 and rectal cancer10 treatment. In addition, the CCQMS data abstraction tool calculated the number of elapsed days for time intervals between major components of cancer care, serving as a basis for nine timeliness of care measures.
Quality indicators were developed with the input of VHA constituencies such as the VHA Oncology Field Advisory Committee and teams participating in the C4 treatment improvement collaborative. They are based primarily on the guidelines of the NCCN. Founded in 1995, NCCN is consortium of 21 academic cancer centers that develops guidelines on a number of cancers.11 Guidelines provide screening and treatment algorithms for each stage of cancer and are reviewed yearly. They represent a combination of evidence-based recommendations in areas for which peer-reviewed evidence is available and consensus-based recommendations for areas in which the evidence base is not fully developed.12 The first guidelines for colon and rectal cancer screening and treatment were published in 1996.13 Except where explicitly stated, CCQMS quality measures described in this report were based on 2005 NCCN guidelines, giving users the opportunity to use the CCQMS to look at cases over multiple years. All NCCN-based quality indicators used in this study are considered to have some evidence-base and have uniform NCCN guideline-panel consensus (at least NCCN evidence category 2A).14, 15 Table 1 lists the quality indicators.
In addition, the CCQMS data abstraction tool calculated the number of elapsed days for nine time intervals (e.g. number of days between a patient’s colorectal cancer diagnosis and the first treatment date). While there is limited current evidence that timeliness of care impacts outcomes of CRC treatment,16 there is a general consensus among oncology societies that timely cancer care is an important indicator of cancer care quality and patient-centeredness and timely access to colorectal cancer care has been positively associated with patient satisfaction.17, 18 In addition, increasing the efficiency of care is a significant focus of the VHA.19 Table 2 lists specific timeliness measures. It should be noted that the indicators and measures detailed here were intended for use in improving quality of care and are not currently part of an ongoing national VHA quality measurement program.
Inclusion in the CCQMS suggested to facilities participating in the collaborative was based on the following criteria: 1) histological or cytological confirmed diagnosis of colon or rectal cancer; 2) invasive cancer (in-situ diagnoses excluded); 3) primary tumor of the colon or rectum (metastases to colorectal sites from other primaries were excluded); 4) no prior invasive cancer of the colon or rectum (recurrences excluded); and 5) ≥ 18 years of age at diagnosis.
Facilities were instructed to exclude patients with a diagnosis of another type of cancer within two months of the diagnosis of colorectal cancer. Although the Surveillance Epidemiology and End Results [SEER] registry system defines simultaneous diagnoses as two separate primary tumors, of the same or different sites, diagnosed within a two-month period, there was no recommendation to exclude patients with more than one primary tumor of the same site (that is, two colon primaries) within two months.
Seventeen of the 27 facilities utilizing the CCQMS during the collaborative (1 facility entered only a very limited number of cases) also had local VA Central Cancer Registry processes for identifying patients with colorectal cancer. VA Central Cancer Registry case finding methods adhere to the standards established by the American College of Surgeons’ Commission on Cancer Facility Oncology Registry Data Standards (FORDS) Manual for data collection and definitions (the definitions can be found in the FORDS manual available at http://www.facs.org/cancer/coc/fordsmanual.html, accessed April 25, 2011).20 Other facilities utilized clinic lists to identify patients.
As part of the quality improvement process for the collaborative, C4 organizers surveyed facility abstractors and other collaborative participants at the conclusion of the collaborative. That survey asked individuals who collected data for the CCQMS whether they disagreed or agreed with a series of statements about the tool. The mean of all answers from a given facility was taken as a facility-level score. Means and standard deviations were calculated for these facility-level scores. The 40 responding abstractors represented 22 of the 27 CCCQMS-utilizing facilities (81%)trhat utilized the CCQMS. Based on the responses from non-abstractor collaborative team members, information on plans to continue tracking quality indicators is available for an additional 3 facilitates for a total of 25 facilities.
We report the percentage of patients who had guideline-concordant care as assessed (or measured) by the quality indicators and the mean number of days between major CRC treatment events. To be included in the denominator for a guideline concordance indicator or in results for a timeliness measure, patients had to meet inclusion criteria for that measure (see Tables 1–2). Unlike the reports generated for the facilities in the C4 collaborative, which required five or more eligible patients from a facility to have those patients included in the report, data from all eligible patients were included in analyses for the present paper. Further, results included in this reprot are based on a post-hoc analysis of data elements that were converted into a SAS data set and cleaned for inconsistent data using SAS version 9.2 (SAS Institute Inc., Cary, NC).
Between March 1, 2007 and May 15, 2008 data on approximately 1,373 patients with stage I-IV CRC (984 colon; 389 rectal) from 28 VA medical centers diagnosed primarily in 2006–2007 were entered into the C4 Cancer Care Quality Measurement System. The 28 VA facilities participating in the treatment improvement collaborative included at least one participating medical center from each of the 21 VHA regions. However, one of the 28 C4 facilities did not fully utilize the CCQMS [although data on a limited number of patients were entered]. The remaining 27 VA medical centers entered information on enough cases to utilize the reports feature of the CCQMS (at least five patients meeting inclusion criteria for one or more quality indicators). Men represented 98% of entered cases and the mean age was 69 years (standard deviation (SD) = 10.6). Race/ethnicity included 81% white, non-Hispanic, 15% black, non-Hispanic, 3% white, Hispanic, and 1% other race/ethnicity.
In addition to geographic variation, the 27 participating facilities that utilized the CCQMS included different degrees of complexity (VA measure based on types of services offered, organizational units, and size) [20 high complexity, 4 medium complexity, 3 low complexity],21 volume of CRC patients [6 facilities < 25 total cases each; 7 facilities = 25–50 total cases each; 7 facilities = 50–70 total cases each; 7 facilities = 71–110 total cases each], and cancer care accreditation [13 facilities, or approximately 48%, of participating facilities were accredited by the American College of Surgeons Commission on Cancer in 2005; an additional eight facilities (30%) were in the process of applying for accreditation].21 Characteristics of the 27 facilities utilizing CCQMS reports are detailed in Table 3.
Facility cancer registrars conducted data abstraction at 13 of the 27 utilizing sites. Additionally, cancer registrars were integrally involved with CCQMS at an additional three sites. Nurses conducted data abstraction at six sites. The role of the CCQMS abstractor at two of the facilities was unknown.
The survey of quality improvement team members from the participating facilities indicated that that at the end of the collaborative, 17 of 25 reporting sites (68%) hoped to continue to track half or more than half of the CCQMS measures. Only two of the reporting facilities indicated that they did not intend to track any of the CCQMS measures.
It appears that the CCQMS was generally well-received. Based on 22 reporting facilities, the mean facility-level score on a one (strongly disagree) to five (strongly agree) scale for surveyed CCQMS data abstractors indicated good agreement with the following statements: 1) I understand what needs to be entered into each data entry field [mean = 4.44 (SD = 0.51)]; 2) I know where in the medical record to find the information for each data entry field [mean = 4.40 (SD = 0.67)]; 3) The information provided by the tool makes the time spent entering data worthwhile [mean = 3.88 (SD = 1.01)] and 4) The data reports are easy to understand [mean = 3.71 (SD = 1.05)].
The C4 teams received significant technical support and tool alterations in response to team feedback. Agreement scores for related survey items included: 1) I received the information/support that I needed from the Durham Data Abstraction Group [mean = 4.38 (SD = 0.79)]; 2) The on-line instructions and supporting information are very clear [mean = 3.95 (SD = 0.94)]; and 3) The tool has substantially improved since the first time I used it [mean = 4.39 (SD = 0.68)]. There was, however, still room to improve the tool. Related agreement scores included: 1) It is easy to enter data into the tool [mean = 3.42 (SD = 1.14)] and 2) The tool is as good as it can get [mean = 2.70 (SD = 0.88)].
CCQMS chart abstraction data were collected for quality improvement and not research purposes. A primary goal was to use the measurement tool to teach facilities about individual NCCN guidelines. The greatest opportunities for improvement involved surveillance colonoscopy and other services following curative intent treatment. Consistent with other findings inside and outside the VHA,22 the proportion of patients receiving follow-up colonoscopy within one year of surgical resection was low. For example, only 22.6% of 328 stage II-III colon cancer patients without preoperative obstructing lesions and at least a year of follow-up at the time of data abstraction had a documented colonoscopy within one year of surgery. Among patients who had a follow-up colonoscopy, the median number of days between end of curative intent treatment and follow-up colonoscopy [stage I-III patients] was 288.0. Consistent with other studies, the proportion of patients receiving guideline-concordant care during the curative-intent treatment process was significantly higher.22, 23 The specific proportion of patients receiving concordant care and mean number of days between major treatment events can be found in Tables 1 and and22 respectively.
NCCN guidelines provided the basis for quality indicators that were integrated into a computerized data abstraction system (not part of the VA electronic health record) that was successfully used by more than 25 VA medical centers. Use of NCCN guidelines allowed us to create a system that could be used to examine quality across the spectrum of CRC care, from diagnosis through neoadjuvant therapy, surgery, adjuvant therapy, and surveillance or end-of-life care.
The tool had important features that made it useful for the facilities. The most significant was the ability to provide immediate feedback to a facility using instantly-generated reports. These reports provided the individual facility results and allowed for extensive comparison across participating facilities. The reports could be easily printed for distribution to leadership and clinicians.
In addition, the tool was enhanced with a number of features, many of which were suggested by abstractors. These included helpful hints for specific questions (e.g. where to find information in the electronic health record), the ability to search for specific patients included in a quality indicator or timeliness measure, an email help desk that was used during the C4 collaborative, and regularly scheduled abstractor conference calls during the collaborative. Responsiveness to the needs of facilities was critical in making the tool useful for quality improvement.
The majority of participating facilities utilized cancer registrars to collect data for the CCQMS. These individuals are employed as part of the system for contributing data to the VA Central Cancer Registry. As a result they have extensive training related to chart abstraction for monitoring patterns of cancer care. Other facilities utilized registered nurses with extensive related clinical knowledge for chart abstractions.
The CCQMS had important limitations, many of which are currently being addressed through other VHA projects. Because this was an abstraction tool, not a data extraction tool, significant time was required to enter data. Anecdotal reports indicated that it was not uncommon for abstractors to spend up to two hours abstracting a complex case. We hypothesize that this is the reason that abstractors reported that there was still an opportunity to improve the computerized tool at the end of the C4 collaborative. For example, the proportion of patients with guideline concordant surveillance may be underestimated because facilities did not always enter data not related to initial treatments into the CCQMS. Because of the burden of data abstraction, several projects are underway in the VHA to automate data extraction. These include use of natural language processing (NLP)24 to extract information from progress notes (e.g. number of lymph nodes resected), development of templated oncology notes, and extraction of data from existing data sources such as the VA Central Cancer Registry.
The tool was used for quality improvement purposes for specific facilities. While these data can be used to generate hypotheses about VHA system-wide quality challenges, facilities may have focused data collection on specific measures.
An important goal of the CCQMS measures was to teach facilities about NCCN guidelines. As a result, we categorized measures by those addressing colon versus those addressing rectal cancer care. When this is combined with the stage-specific nature of many guidelines, the number of patients is fewest for less common situations (e.g. rectal cancer patients of a specific stage). In addition, we noted that including multiple processes under one indicator (e.g. patients with documented history & physical examination (H&P) and postoperative visits every three months for two years, then every six months in years 2–5) greatly reduced the percentage of patients who appeared to be getting guideline concordant care, but did not provide details of the specific processes completed. Data collection may have been further complicated if patients received surveillance outside of the VA healthcare system. However, we included these indictors as a way of encouraging potential quality improvement efforts around the issue of survivorship.
Another quality improvement limitation was lack of measures related to patient experience of care. To address this limitation, the VA, Department of Defense, and the National Cancer Institute have developed and piloted a survey of patient-reported experiences with CRC care. This includes both a mailed patient and caregiver survey aimed at augmenting clinical data with information on patient experience.25, 26
NCCN guidelines provided the basis for efforts in the VHA that have evolved over time. Quality improvement activities began with manual data abstraction but are now focusing on development of templated notes and automated data extraction. The experience of the VHA provides an example for other large healthcare systems seeking to use quality measurements to inform initiatives for enhancing the quality of cancer care.
This paper represents the work of 28 VA Medical Centers that participated in the VA Colorectal Cancer Care Treatment Improvement Collaborative. We thank the abstractors and quality improvement team members who collected the data reported here. In addition, we thank Radhika Khwaja, MD for providing important clinical input into the development of CCQMS quality indicators. Bryan Paynter, formerly of the Durham VAMC HSR&D Center of Excellence, served as the lead computer programmer for the development of the CCQMS. Other individuals involved in the development of the CCQMS at the Durham VAMC HSR&D Center of Excellence include: Catherine Caprio, RN; Melissa Garrett, MD; Natia S. Hamilton, MA; Mike Harrelson; Katie Mitchell, RN; and Christopher B. Newlin, MPH. Preliminary results were presented at the Association of VA Hematology and Oncology Annual Meeting on September 12, 2008 in Nashville, TN.
Financial Support: This study was supported by the Veterans Affairs Health Services Research & Development Service (VA HSR&D grant CRT 05-338) and National Cancer Institute (NCI grants YI-PC-6039-01 and V246S-00054). Dr. Jackson was supported by a Merit Review Entry Program (MREP) award from the Department of Veterans Affairs HSR&D Service (VA HSR&D grant MRP 05-312). Ms. Zullig is supported by funding from the National Cancer Institute (5R25CA116339). During part of this work, Dr. Zafar was supported by a National Research Service Award-Agency for Healthcare Research and Quality Post-Doctoral Fellowship (institutional training grant to Duke University T32HS000079). During part of this work, Dr. Powell was supported by a Veterans Affairs Health Services Research & Development Career Development Award (VA HSR&D CDA 08-024VH). Dr. Gellad was funded in part by a National Research Service Award-National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) institutional training grant (T32 DK007568-17 to Duke University). Dr. Povenzale was funded in part by a K-24 career development award from NIDDK (5 K24 DK002926).
Disclaimer: The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States Government.