In the preimplementation stage of a nationwide study of an interactive, Web-delivered system to increase provider and patient engagement in smoking cessation, we conducted a rigorous planning and evaluation of the system. The primary purpose of our preimplementation evaluation was to identify the strengths that might be used to promote the program, and weaknesses that might be mitigated prior to initiating the main study. We conceptualized and developed e-referral functions in Web-based form. We report the functions we developed and the results of our usability testing in the Results section. We evaluated the Web system and the implementation plan rigorously with community-based providers. Our approach involved four phases: (1) system conceptualization, (2) agile programming and think-aloud usability testing, (3) implementation planning (using the NGT), and (4) lessons learned from pilot implementation in 7 physician practices. summarizes the identified barriers and facilitators to practice implementation based on our evaluation work. In the section below, we focus on the implementation protocol changes that will be used in the main trial to address the four primary barriers we uncovered in the pilot testing.
Identified issues related to e-referral system implementation
The first barrier was the difficulty contacting the practice and lack of study champion. With no champion identified at each practice, we were constantly speaking with or leaving messages for different staff members, who had little sense of ownership of or urgency in the process. This breakdown in communication was made more complicated with staff turnover, a reality in most medical offices. In order to overcome this particular barrier, we modified the study protocol to include a request for each practice to identify two staff members to serve as implementation coordinators. These implementation coordinators will be the primary contacts for the practice and will work closely with our study personnel. Their responsibility will be to implement and promote the study intervention in the practice. Two implementation coordinators will allow for backup in the event that one individual is unavailable or leaves the practice. Our study personnel will communicate with these implementation coordinators to confirm practice information, hold training sessions, answer any questions, and provide feedback.
Second, we identified that successful implementation required training and assistance with registration in the system. Pilot practices reported that the system was easy to use, but with no one trained at the practice to complete the registration process and refer patients and to champion others through the process, the task went undone. Consequently, we increased study personnel and created a proactive helpdesk to provide training and help with registration. In the main study, our staff will initiate contact with each practice within 2 weeks of receiving the returned consent form. Study personnel will verify practice information and schedule a training and registration call for each of the implementation coordinators. During this call, our staff will walk the implementation coordinators through the actual registration process. The study personnel will be on hand to answer any questions. Following registration, study personnel will review the process for referring patients, getting the implementation coordinators to enter a dummy referral to have the full experience of the ease of referral. Each implementation coordinator will be encouraged and provided information to train others in the office to also register and refer patients. Following the training call, study personnel have planned a booster call to verify receipt of printed materials sent and answer any questions that may have arisen in the first few attempts to refer. If no referrals have been made yet, our staff will assess any reasons for no referrals and encourage implementation coordinators to use the system.
The third barrier was a lack of motivation and start-up incentives. It became abundantly clear that motivation to participate was low. We focused on increasing both extrinsic and intrinsic motivation. Pilot data immediately indicated that financial incentives would spur participation. Additional funds from the American Recovery and Reinvestment Act has enabled us to provide honoraria to participating practices and individual staff members who complete training and surveys.
Finally, we learned that clinicians within participating practices simply forgot about the study and the need to refer patients through the system. We believe that with convenient reminders we will be able to activate them to use the system. In addition to calling the implementation coordinators to aid them in the registration process and answer any questions, we will increase the work-flow support. We also improved the printed information prescription pads sent to practices for distribution to their patients simultaneously with their online referral. The “Information Rx” that was used in the pilot was small, about the size of a regular prescription pad, and simply provided an optional patient handout. The new and improved pad is spiral bound and has easy check-off boxes with duplicate pages. The bottom half of the first page will be given to the patient. The top half will be returned to study personnel, and the duplicate copy will be ready to place in a patient medical record file. Further, the information prescriptions for the intervention arm have a space for the providers to write the patient’s email address. Additionally, posters to serve as visual stimulation to use the system, posters to encourage patients to talk with their provider about quitting, and 1-page instruction sheets outlining the steps for referring patients will be sent to participating practices.
To increase intrinsic motivation and to maximize the brief phone contact with practices, study personnel will incorporate a concept called motivational interviewing into each interaction. Miller and Rollnick define motivational interviewing as a client-centered, directive method for enhancing intrinsic motivation to change by exploring and resolving ambivalence [43
]. Key concepts involved in motivational interviewing are accurate empathy, reflective listening, and overcoming ambivalence, with the ultimate goal of facilitating some changed behavior. From our pilot study, we learned that implementation coordinators must face an increased workload because of their participation in the study enrollment, training, and implementation, especially in system registration and patient referral. Using motivational interviewing techniques, our staff will be better able to communicate effectively with implementation coordinators by identifying and overcoming their ambivalence. As an example, if a study staff member has attempted to contact a particular implementation coordinator on multiple times and failed, it may be a natural inclination to reflect negatively on that statement, which can be highly detrimental to the relationship with that particular office. Focusing on a positive reflective statement might increase the likelihood of the practice either becoming or continuing to be a happy participant in the study and increase positive feedback, which will, in turn, facilitate study task completion.
Results of Implementation Protocol Changes
Because of our implementation protocol changes, we were successful in engaging practices and improving participation in the nationwide trial. We measured rates of referral and patient participation in the first 3 months of practice engagement. To date, we have analyzed data from the first 50 e-referral practices. Practices’ mean e-referral rate was 14 (SD 13.63). In the first 3 months, the maximum number of referrals by a practice was 62, and 3 practices did not refer. Per practice, the patients mean registration rate was 3.4 (SD 5.09). The maximum number of patients registered with a practice was 28, and 13 practices did not yet have any patients registering.
Strengths and Limitations
In preparation for a nationwide randomization trial testing an e-referral process for referring patients to a smoking-cessation system by providers, we detail the rigorous steps taken to develop the Web-based e-referral system. At each step of the development process, we applied user input to conceptualize and refine the system. Although the numbers of users are low, the multiple and comprehensive nature of the interactions and data collected provided significant information on which to improve the system’s usability. The results of the mini pilot study gave us critical insight into the recruitment and use barriers that our randomized trial must overcome to succeed.
Our how-to report demonstrates how a small, rigorously conducted, multistep preimplementation evaluation can affect the success of a larger study. To gain valuable information regarding potential improvements to an interactive, Web-delivered provider–patient system to increase engagement in smoking cessation, we used a multidimensional approach to conceptualize, develop, implement, and test the product and process. The results of this rigorous process led us to make significant changes to the practice implementation approach study, prior to its nationwide randomized, controlled trial. After refining our information system with usability testing, we further uncovered serious barriers to implementation: lack of study champions within the practice, lack of training and assistance in use of the system, and lack of motivation to participate. We identified several improvements to address and made changes to the main study protocol before trial implementation. Our preliminary analysis with the first 50 practices using the system for 3 months demonstrates the preimplementation evaluation was successful in overcoming the barriers to recruiting and retain study participants.