|Home | About | Journals | Submit | Contact Us | Français|
There is a growing need for students and practitioners of Traditional Chinese Medicine to gain experience with standardized data collection, patient outcomes measurement, and practice-based research. The purpose of this paper is to describe the development of a process for standardized data collection that could eventually be adopted for clinical, research, and quality assurance purposes.
The setting for this study was an acupuncture and Oriental medicine teaching clinic in Bloomington, Minnesota.
Four (4) aspects of data collection were assessed and improved, including intake and post-treatment questionnaires, follow-up with patients, integration of data collection into clinic flow, and commitment of resources to the project.
The outcomes measures were data collection and missing data rates, burden on patients and clinic staff, and efficiency of data entry.
Revision to the data collection process resulted in decreased burden to patients and staff, more detailed and aggressive follow-up protocols, enhanced training for clinic staff, and increased personnel and data-related resources.
The systematic collection of descriptive and clinical characteristics can be accomplished in a teaching clinic with thoughtful attention paid to data collection protocols, dedicated resources, and the involvement of all relevant personnel.
While health care consumers continue to seek out acupuncture and Traditional Chinese Medicine (TCM) for their health and wellness needs, public health groups and policymakers remain steadfast in their call for quality research in this arena.1–3 TCM, like other whole system approaches to health care, presents unique challenges to traditional research methodology.4–6 While there is considerable debate regarding how best to address this,7 it is generally accepted that the collection of data on trends of use and outcomes of care are timely and relevant.8–13 Also, there is value in engaging TCM providers in the data collection process; they can provide valuable perspective to research regarding cultural context and current practice norms.14–18 However, there are special challenges in doing so, specifically a lack of practitioner resources, interest, and skills.19 Subsequently, academic institutions have a unique opportunity to educate the next generation of TCM providers in these areas.20
TCM education in the United States has historically placed less emphasis on research skills and evidence-based medicine compared to other alternative health care programs such as chiropractic and naturopathy.21,22 This is problematic because educational institutions are graduating students who, without these skills, are at a disadvantage to utilize and participate in the growing body of research in their field. One way to address this is by expanding practical educational experiences in teaching clinics to include standardized data collection, patient outcomes measurement, and practice-based research.
While acupuncture data collection and outcomes measurement have been conducted in various settings,8,12,23–26 few studies have illustrated methods to engage a TCM academic community in describing and evaluating patients in their teaching clinics. In 2006, Xing et al. published one of the first studies of patient characteristics and outcomes collected specifically in an acupuncture teaching clinic. This retrospective survey collected descriptive and subjective outcome data from 110 patients, and found the patient characteristics to be similar to those of other CAM studies.27 The authors acknowledged the need for future prospective studies to better assess patient experience and outcome in a practice-based setting.
In 2004, our group implemented a pilot project to prospectively collect demographic and outcome data to describe patients seeking care at an acupuncture and Oriental medicine teaching clinic.28 The intent was to design processes for standardized data collection that could eventually be adopted for educational, clinical, research, and quality assurance purposes. A team of researchers, practitioners, educators, and students affiliated with the Minnesota College of Acupuncture and Oriental Medicine (MCAOM) collaborated to develop patient self-report questionnaires. Demographic and baseline data were collected on a self-report questionnaire prior to new patients' first clinic visit. Follow-up questionnaires were mailed at 2-week and 2-month intervals. As the number of participants increased, several limitations of the pilot study became apparent. Preliminary analysis showed a large amount of missing data, and follow-up response rates were consistently less than 50%. After 21 weeks of data collection on 200 patients, the pilot study ended and the results and processes of the project were further analyzed.
Despite the weaknesses of the pilot study, clinic and MCAOM leaders saw tremendous value in a data collection process that could systematically gather descriptive patient data and efficiently document treatment outcomes. A new project team, some of whom had contributed to the pilot study, was formed to improve upon the methods of the pilot project and facilitate the implementation of a full-scale data collection initiative.
The purpose of this paper is to describe the development of a full-scale data collection process based on knowledge gained from a pilot study, with a long-term goal of informing the successful development of other TCM practice-based research initiatives.
This project was conducted at the Edith Davis Teaching Clinic (EDTC), a public teaching clinic for students of MCAOM at Northwestern Health Sciences University in Bloomington, MN. There, students gain practical experience during their final years of education, working under the guidance of licensed TCM practitioners.
The project team consisted of research scientists, the Deans of the MCAOM and Research programs, a student intern, a TCM practitioner, and EDTC staff. This group assessed the methods and results of a previous pilot project, and identified four major areas in need of revision. These included the questionnaires, follow-up processes, integration of data collection into clinic flow, and commitment of resources to the project. The team analyzed each of these areas for strengths and weaknesses, and devised new protocols to be implemented in a full-scale data collection effort.
The first task of the project team was to assess the questionnaires that were implemented in the original pilot study. The pilot study questionnaires had been developed collaboratively by TCM practitioners and clinical researchers. The intent was to be comprehensive and inclusive, which resulted in a large number of variables representing a range of interests from Eastern and Western clinical and research perspectives. Although the questionnaire development process generated dialogue and a deeper understanding between TCM practitioners and researchers, it also resulted in a very lengthy data collection instrument. With 51 questions on 10 pages, and a considerable amount of missing data, the project team concluded that the pilot study questionnaire, while comprehensive, had created excess burden for both patients and staff. Furthermore, many of the questions were open-ended and elicited descriptive, write-in responses. These were time-consuming for data entry and required additional coding for analysis. Additionally, because there were so many questions in the questionnaire, the forms were visually busy and time consuming to complete.
The project team then proceeded to identify which data from the original questionnaires were most relevant to collect. A series of meetings were used to identify and discuss items that were of limited clinical utility. Furthermore, the team considered additional questions that clinicians routinely ask their patients during the clinical encounter that would be appropriate for assessment via questionnaire. Additionally, completed pilot study forms were reviewed to determine whether the questions effectively solicited the desired clinical information.
The team's analysis yielded changes to the questionnaires, including the elimination of many open-ended responses, and the creation of new multiple choice questions. On the intake form, the number of variables was reduced by over half, from 362 separate items in 51 questions to 152 items in 41 questions in the revised form. See Appendices 1 and 2 for final versions of the intake and follow-up forms.
The project team also assessed the methods employed in the pilot study for collecting the follow-up questionnaires. This included assessing the process of administering and collecting the questionnaires and the frequency of follow-up questionnaire administration.
During the pilot project, a student research assistant was hired to mail follow-up questionnaires and ensure they were returned in a timely fashion. This included tracking all questionnaire mailing and due dates, and making repeated telephone reminders to patients who did not return their questionnaires in a timely fashion. In addition, the research assistant was also responsible for data entry of questionnaires. Due to the large volume of data collected, the research assistant spent the majority of her time entering data, limiting the amount of time she could devote to ensuring complete data collection via timely telephone reminders. The project team concluded that the low response rate observed in the pilot study was due, at least in part, to an inadequate resource commitment to the follow-up process. Consequently, they provided more personnel resources to the full-scale data collection project, employing an additional person, increasing the number of hours dedicated to the project.
The project team also felt that the protocol for contacting patients for unreturned questionnaires needed to be more structured and assertive. The team developed a follow-up protocol for the full-scale data collection project that was intended to increase the follow-up data collection rates. Goals were established for follow-up rates and standards for missing data were set. A follow-up questionnaire, along with a postage-paid, self-addressed return envelope, was mailed to the patient 3 weeks after their initial visit. It was estimated that a completed questionnaire would then be returned 4 weeks after the patient's first visit. If no response was received by the 5th week, reminders were made via phone and e-mail for the following 3 weeks, or until the patient was contacted and the questionnaire was administered over the phone. In cases where the patient was unable to be reached in person, one message was left for each of those 3 weeks. At week 8, if there was no response, a second questionnaire was mailed out to the patient. The same reminder protocol was followed until week 12, at which time the patient was considered lost to follow-up.
The poor response rate during the pilot project also caused the project team to weigh the benefit and costs of administering two follow-up questionnaires. The team decided that it would be advantageous to reduce the burden to both patients and staff by reducing the number of follow-up administrations. Therefore, the group agreed to collect one follow-up questionnaire at a 1-month time-point, anticipating that this may result in greater collection rates.
The project team also assessed the extent to which the pilot study had been truly integrated into the clinic system. In the pilot study, the intake questionnaire was given to new patients by clinic front desk staff when they checked in for their first visit. The questionnaire was part of a new patient information packet, which also included a welcome letter explaining the purpose of the teaching clinic, the cancellation policy, and treatment consent and Health Insurance Portability and Accountability Act forms.
The front desk staff had sole responsibility for administering the questionnaire and collecting the data. However, they were never fully trained in administering the new pilot study forms; it was assumed that they could simply dispense them with the usual clinic paperwork. Furthermore, they had been provided little in the way of information on the purpose of the pilot project, the necessity of complete data collection, or the importance of their role. This left the staff disadvantaged in their ability to answer patients' questions, and placed full burden on the patient to understand all the information given to them, in addition to completing the questionnaire completely and accurately. When the patient submitted their paperwork to the staff, the questionnaire was directly inserted into the patient's file without review for missing data.
To remedy this problem, the project team paid particular attention to improving communication between front desk staff and researchers for the full-scale study.
A series of training meetings were held between the staff, student research assistant, and researchers to discuss the relevance and importance of data collection. The project was explained in detail and the importance of the staff role was emphasized. The staff reported they had not understood the educational and clinical value of standardized data collection, and came to recognize its utility for quality assurance purposes. With this new perspective, the front desk staff was instrumental in developing new protocols for the full-scale project.
The front desk staff supervisor was made part of the project team to provide a practical perspective in terms of clinic operations that may affect the project. Examples of this included how new data collection protocols might fit into the usual clinic flow, and how to facilitate student and clinician compliance with these protocols. Several new protocols and processes were developed. Standardized language was created for staff to explain the project to patients and to answer questions in a uniform manner. A staff signature line was also added to the consent form. This was intended to reinforce staff responsibility in the informed consent process to ensure that patients understood the follow-up procedures. Staff were trained to review questionnaires for completeness, and address missed information with patients prior to proceeding with the appointment.
The last area evaluated by the project team was resources, specifically whether or not the pilot project had sufficient personnel, training, and skills to meet its objectives. Like many practice-based studies, this data collection project was internally funded, and integrated, to the extent possible, with existing clinic activities. During the pilot study, in-kind contributions were provided by the research department and teaching clinic for the participating researchers and clinicians' time. Furthermore, additional responsibilities were given to front desk staff. One student research assistant managed the project, interfaced with participants, and did all the data entry and management in a part-time position. Although the parent institution appeared dedicated to the potential of the project, it was clear to the project team that the pilot project had been underresourced. A renewed commitment to resources was required for implementation of a full-scale data collection effort.
For the full-scale project, two student research assistants were employed for a total of 20–30 hours per week to implement the new follow-up data collection protocols described above. Responsibilities included daily tracking of completed questionnaires, mailing, and reminder calls for follow-up questionnaires, data entry, and close communication with clinic staff regarding adherence to data collection protocols and barriers to implementation. Further in-kind contributions were made by the research department, to enlist the efforts of a data manager to design a database with a user-friendly data-entry interface. This was intended to increase the ease of data entry, and decrease the amount of time spent processing questionnaires. Furthermore, monthly quality assurance checks were conducted to ensure data was being entered accurately and in a timely fashion. Regular reports were generated to monitor the quantity and quality of data being collected, and then presented to the project team during monthly meetings.
The project team's assessment of the pilot study resulted in a number of revisions to processes and protocols implemented in the full-scale data collection project. This project was implemented in 2005, and over the course of 1 year, a total of 485 new patients consented to participate in the study. A summary of patient demographics, clinical characteristics, and outcomes is reported separately.29
The project team focused on four main areas for the full-scale study, including improvements related to questionnaires, follow-up, clinic integration, and resources.
Overall, these changes resulted in decreased burden to patients and staff, more detailed and aggressive follow-up protocols, enhanced training for clinic staff, and increased personnel and data-related resources. Importantly, increased follow-up rates and decreased missing data rates were also observed. This process is summarized in Table 1.
The volume of information collected in the full-scale project was reduced substantially from what was collected in the pilot project. By reducing the number of variables to what was considered essential, questionnaire length was reduced substantially from 362 separate items in 51 questions in the pilot project, to 152 items in 41 questions in the revised form. Furthermore, instead of completing two follow-up questionnaires, patients were asked to complete only one, which contained 14 questions (compared to 20 in the pilot project). The decrease in questions allowed for a questionnaire that had more white space and a larger font size, making it easier to read, and ultimately less time-intensive to complete.
These changes not only minimized patients' burden, but also decreased personnel effort requirements. By reducing follow-up questionnaires to only one administration, the tracking and follow-up procedures for the research assistants were reduced by half. Furthermore, the time needed to enter data into the database was also substantially lessened. The average time for data entry went from 30–45 minutes per questionnaire in the pilot project to only 10 minutes in the full-scale study.
The new unambiguous and assertive protocols for questionnaire administration and data retrieval made it easier for the research assistants to complete their tasks. More attention was paid to training and involvement of the clinic staff; this appeared to result in a strong sense of ownership over the role of data collection once they had realized its importance and their responsibilities. This shifted the clinic staffs' roles from passive observers to active project team members, allowing them to contribute substantially to the successful integration of data collection into standard clinic operations.
The questionnaire response rate in the full-scale project doubled from what was observed in the pilot study. While the questionnaire response rate after 2 weeks and 2 months in the pilot was 41% and 25%, respectively, the follow-up collection rate in the full-scale project was 86%. Missing data rates also dramatically improved, with the majority in the full-scale project meeting a quality assurance standard of <5% missing variables. At intake, 387/485 (80%) of questionnaires were 100% complete, and only 42 (9%) did not meet the quality assurance standard. At follow-up, 392 (94%) of questionnaires collected were 100% complete, while 23 (6%) did not meet the quality assurance standard.
This paper describes a collaborative process by which a pilot project was used to design a full-scale systematic data collection study. While some methods may not be generalizable to all clinical settings, many of the experiences could be informative to the development and implementation of other practice-based research projects. Seeking input from participating clinicians, clinic staff, students, and researchers was essential to adequately address the challenges posed by this project from multiple points of view. Not only did this approach result in successful data collection, but it also served as a platform for developing relationships and ideas for new initiatives.
The range of data collected in this study was chosen by clinicians and researchers, with the goals of broadly describing this patient population and collecting patient-oriented outcome measures in the short term. The selection of these variables was informed by demographic and outcome data commonly used in clinical research.30,31 Furthermore, questions were also informed by previous qualitative studies conducted by this group of investigators that identified outcome measures most important to patients.32–34 While of interest, the assessment of treatment effectiveness or the comprehensive documentation of the patient experience were objectives outside the scope of this study. This would require different research methods. Results from this data collection project have been reported separately.
Reflecting on the success of this project, the Dean of the acupuncture and Oriental medicine program and members of its faculty feel this activity raised the awareness of research among students, faculty, and administration. Exposure to participation in practice-based data collection initiated interest in research that had not previously been evident in campus culture. For example, during the course of this project, students initiated changes in the curriculum that led to the introduction of research coursework earlier in the academic program. Although students and faculty share a broad spectrum of opinions regarding the role of research in TCM, they noted that this project generated discussion about the current body of research, how it contributes to the development of the profession, and the ways it impacts the professional status of its practitioners.
This data collection project has proved fruitful for the teaching clinic in several ways. First, the data collected has contributed to what is known about patients who seek acupuncture and Oriental medicine treatment, uniquely describing those presenting to a teaching clinic. This has helped clinic administrators more thoroughly assess their patient population as well as the clinical experience provided to patients. For example, although patients were highly satisfied with their care, they commonly did not return to the clinic because of lengthy appointments. This alerted administrators of the need to address time management within the clinic system. Second, initiation of a data collection system served as a catalyst for implementing additional best-practice initiatives in this clinic setting, including a standardized form for practitioners to document clinical information. Finally, operational protocols and collaborative relationships established by the data collection project have laid the groundwork for future practice-based research in the clinic.
Acupuncture and TCM have become an increasingly visible component of the U.S. health care system,35–38 which has piqued interest among other health care providers and researchers. Can TCM become accessible and integrated in such a way that its identity is preserved and the art of care uncompromised?20,39 Empowering practitioners to generate practice-based research could be a vehicle for communicating with Western biomedical culture, while maintaining TCM's unique approach to care. Now is a pivotal time for educational institutions to take responsibility for training future TCM practitioners in the skills necessary for this role.
The systematic collection of descriptive and clinical characteristics can be accomplished in a teaching clinic with thoughtful attention paid to data collection protocols, dedicated resources, and the involvement of all relevant personnel. It is our desire that sharing this experience will aid other clinics and educational institutions to successfully implement similar projects.
This project was funded by Northwestern Health Sciences University, Bloomington, MN. Eileen McKenzie was additionally supported by the Minnesota Consortium for CAM Clinical Research (1-T32-AT00487). The authors would like to acknowledge Paul Osterbauer, D.C., M.P.H., for his preliminary work on this project. We would also like to thank Susan Anderson, Sher Demeter, Lic.Ac., Lori Baldwin, M.Om., Lic.Ac., Corrie Vihstadt, and Senia Tuominen for their commitment to and assistance with this project.
No competing financial interests exist.