This project was conducted at the Edith Davis Teaching Clinic (EDTC), a public teaching clinic for students of MCAOM at Northwestern Health Sciences University in Bloomington, MN. There, students gain practical experience during their final years of education, working under the guidance of licensed TCM practitioners.
The project team consisted of research scientists, the Deans of the MCAOM and Research programs, a student intern, a TCM practitioner, and EDTC staff. This group assessed the methods and results of a previous pilot project, and identified four major areas in need of revision. These included the questionnaires, follow-up processes, integration of data collection into clinic flow, and commitment of resources to the project. The team analyzed each of these areas for strengths and weaknesses, and devised new protocols to be implemented in a full-scale data collection effort.
The first task of the project team was to assess the questionnaires that were implemented in the original pilot study. The pilot study questionnaires had been developed collaboratively by TCM practitioners and clinical researchers. The intent was to be comprehensive and inclusive, which resulted in a large number of variables representing a range of interests from Eastern and Western clinical and research perspectives. Although the questionnaire development process generated dialogue and a deeper understanding between TCM practitioners and researchers, it also resulted in a very lengthy data collection instrument. With 51 questions on 10 pages, and a considerable amount of missing data, the project team concluded that the pilot study questionnaire, while comprehensive, had created excess burden for both patients and staff. Furthermore, many of the questions were open-ended and elicited descriptive, write-in responses. These were time-consuming for data entry and required additional coding for analysis. Additionally, because there were so many questions in the questionnaire, the forms were visually busy and time consuming to complete.
The project team then proceeded to identify which data from the original questionnaires were most relevant to collect. A series of meetings were used to identify and discuss items that were of limited clinical utility. Furthermore, the team considered additional questions that clinicians routinely ask their patients during the clinical encounter that would be appropriate for assessment via questionnaire. Additionally, completed pilot study forms were reviewed to determine whether the questions effectively solicited the desired clinical information.
The team's analysis yielded changes to the questionnaires, including the elimination of many open-ended responses, and the creation of new multiple choice questions. On the intake form, the number of variables was reduced by over half, from 362 separate items in 51 questions to 152 items in 41 questions in the revised form. See Appendices 1
for final versions of the intake and follow-up forms.
The project team also assessed the methods employed in the pilot study for collecting the follow-up questionnaires. This included assessing the process of administering and collecting the questionnaires and the frequency of follow-up questionnaire administration.
During the pilot project, a student research assistant was hired to mail follow-up questionnaires and ensure they were returned in a timely fashion. This included tracking all questionnaire mailing and due dates, and making repeated telephone reminders to patients who did not return their questionnaires in a timely fashion. In addition, the research assistant was also responsible for data entry of questionnaires. Due to the large volume of data collected, the research assistant spent the majority of her time entering data, limiting the amount of time she could devote to ensuring complete data collection via timely telephone reminders. The project team concluded that the low response rate observed in the pilot study was due, at least in part, to an inadequate resource commitment to the follow-up process. Consequently, they provided more personnel resources to the full-scale data collection project, employing an additional person, increasing the number of hours dedicated to the project.
The project team also felt that the protocol for contacting patients for unreturned questionnaires needed to be more structured and assertive. The team developed a follow-up protocol for the full-scale data collection project that was intended to increase the follow-up data collection rates. Goals were established for follow-up rates and standards for missing data were set. A follow-up questionnaire, along with a postage-paid, self-addressed return envelope, was mailed to the patient 3 weeks after their initial visit. It was estimated that a completed questionnaire would then be returned 4 weeks after the patient's first visit. If no response was received by the 5th week, reminders were made via phone and e-mail for the following 3 weeks, or until the patient was contacted and the questionnaire was administered over the phone. In cases where the patient was unable to be reached in person, one message was left for each of those 3 weeks. At week 8, if there was no response, a second questionnaire was mailed out to the patient. The same reminder protocol was followed until week 12, at which time the patient was considered lost to follow-up.
The poor response rate during the pilot project also caused the project team to weigh the benefit and costs of administering two follow-up questionnaires. The team decided that it would be advantageous to reduce the burden to both patients and staff by reducing the number of follow-up administrations. Therefore, the group agreed to collect one follow-up questionnaire at a 1-month time-point, anticipating that this may result in greater collection rates.
Integration with clinic flow
The project team also assessed the extent to which the pilot study had been truly integrated into the clinic system. In the pilot study, the intake questionnaire was given to new patients by clinic front desk staff when they checked in for their first visit. The questionnaire was part of a new patient information packet, which also included a welcome letter explaining the purpose of the teaching clinic, the cancellation policy, and treatment consent and Health Insurance Portability and Accountability Act forms.
The front desk staff had sole responsibility for administering the questionnaire and collecting the data. However, they were never fully trained in administering the new pilot study forms; it was assumed that they could simply dispense them with the usual clinic paperwork. Furthermore, they had been provided little in the way of information on the purpose of the pilot project, the necessity of complete data collection, or the importance of their role. This left the staff disadvantaged in their ability to answer patients' questions, and placed full burden on the patient to understand all the information given to them, in addition to completing the questionnaire completely and accurately. When the patient submitted their paperwork to the staff, the questionnaire was directly inserted into the patient's file without review for missing data.
To remedy this problem, the project team paid particular attention to improving communication between front desk staff and researchers for the full-scale study.
A series of training meetings were held between the staff, student research assistant, and researchers to discuss the relevance and importance of data collection. The project was explained in detail and the importance of the staff role was emphasized. The staff reported they had not understood the educational and clinical value of standardized data collection, and came to recognize its utility for quality assurance purposes. With this new perspective, the front desk staff was instrumental in developing new protocols for the full-scale project.
The front desk staff supervisor was made part of the project team to provide a practical perspective in terms of clinic operations that may affect the project. Examples of this included how new data collection protocols might fit into the usual clinic flow, and how to facilitate student and clinician compliance with these protocols. Several new protocols and processes were developed. Standardized language was created for staff to explain the project to patients and to answer questions in a uniform manner. A staff signature line was also added to the consent form. This was intended to reinforce staff responsibility in the informed consent process to ensure that patients understood the follow-up procedures. Staff were trained to review questionnaires for completeness, and address missed information with patients prior to proceeding with the appointment.
Renewed commitment of resources
The last area evaluated by the project team was resources, specifically whether or not the pilot project had sufficient personnel, training, and skills to meet its objectives. Like many practice-based studies, this data collection project was internally funded, and integrated, to the extent possible, with existing clinic activities. During the pilot study, in-kind contributions were provided by the research department and teaching clinic for the participating researchers and clinicians' time. Furthermore, additional responsibilities were given to front desk staff. One student research assistant managed the project, interfaced with participants, and did all the data entry and management in a part-time position. Although the parent institution appeared dedicated to the potential of the project, it was clear to the project team that the pilot project had been underresourced. A renewed commitment to resources was required for implementation of a full-scale data collection effort.
For the full-scale project, two student research assistants were employed for a total of 20–30 hours per week to implement the new follow-up data collection protocols described above. Responsibilities included daily tracking of completed questionnaires, mailing, and reminder calls for follow-up questionnaires, data entry, and close communication with clinic staff regarding adherence to data collection protocols and barriers to implementation. Further in-kind contributions were made by the research department, to enlist the efforts of a data manager to design a database with a user-friendly data-entry interface. This was intended to increase the ease of data entry, and decrease the amount of time spent processing questionnaires. Furthermore, monthly quality assurance checks were conducted to ensure data was being entered accurately and in a timely fashion. Regular reports were generated to monitor the quantity and quality of data being collected, and then presented to the project team during monthly meetings.