Search tips
Search criteria 


Logo of hsresearchLink to Publisher's site
Health Serv Res. 2002 August; 37(4): 1067–1077.
PMCID: PMC1464005

Issues in Conducting Randomized Controlled Trials of Health Services Research Interventions in Nonacademic Practice Settings: The Case of Retail Pharmacies



To describe unexpected challenges and strategies to overcome them when conducting randomized controlled trials (RCT) of health services research interventions in retail pharmacies.

Study Setting

Thirty-six retail drug stores in Indianapolis.

Study Design

We conducted an RCT to evaluate the effectiveness of an intervention to increase pharmacists’ involvement in caring for customers. We describe: (1) our RCT as originally designed, (2) unexpected challenges we faced; and (3) how we resolved those challenges.

Data Collection/Extraction Methods

Randomized controlled trial.

Principal Findings

Major modifications in research design were necessitated by factors such as corporate restructuring, heightened sensitivity to patient confidentiality, and difficulties altering employees’ behavior. We overcame these barriers by conducting research that is consistent with corporate goals, involving appropriate corporate administrators and technical personnel early in the process, and being flexible.


Health services researchers should conduct RCTs in a variety of nonacademic practice settings to increase generalizability and better reflect the true impact of interventions. Pragmatic problems, although significant, can be successfully overcome.

Keywords: Pharmaceutical care, randomized trial

Health services researchers who conduct randomized controlled trials (RCT) evaluate interventions to improve the organization, delivery, quality, and/or outcomes of care. Often, such trials are implemented in academic or other clinical settings in which the investigator has a reasonable degree of control over factors essential to conducting a well-designed RCT. Maximizing such control is important to enhancing the internal validity of an RCT. However, in so doing, findings may be less externally valid (i.e., generalizable) to nonacademic practice settings. This tension between internal and external validity is especially important to health services researchers.

We report our experience with conducting an RCT in retail pharmacies. After describing our RCT as originally designed, we present some unexpected situations that are unlikely to be experienced by health services researchers who conduct RCTs in more traditional venues. We then discuss how we resolved those challenges through sometimes-dramatic changes in the study protocol. Our experiences may be useful to other investigators who plan to evaluate health services interventions in nonacademic practice settings.


Study Overview

We evaluated the effectiveness of a pharmaceutical care program for patients with reactive airways disease (i.e., asthma or chronic obstructive pulmonary disease with a reversible component) who fill their prescriptions in retail drug stores. Such programs seek to increase pharmacists’ involvement in improving patients’ outcomes, rather than simply dispensing medications (Hepler and Strand 1990). Specifically, it was hypothesized that pharmacists can help optimize drug therapy by: educating patients; implementing strategies to foster medication compliance; coordinating drug therapy prescribed by multiple physicians; monitoring adverse drug events; being available to address patients’ questions and concerns; and ensuring that patients know when, who, and how to contact their physicians (Hepler and Strand 1990; Strand et al. 1991). Unfortunately, several recent literature reviews suggest that enthusiastic reports about the effectiveness of pharmaceutical care are often plagued by serious design flaws, particularly a lack of well-designed RCTs (Hatoum and Akhras 1993; Kennie, Schuster, and Einarson 1998; Singhal, Raisch, and Gupchup 1999; Tett, Higgins, and Armour 1993). Moreover, these reviews find no well-designed studies supporting the effectiveness of pharmaceutical care in retail pharmacies. We conducted the RCT in chain pharmacies because, as the largest provider of prescription services in the United States (Latner 2000), they provide a venue through which effective pharmaceutical care programs can have the greatest impact on patients’ lives. Moreover, effective programs that are successfully implemented in a few stores can efficiently be disseminated throughout a retail chain, thereby rapidly enhancing their impact upon patients’ lives. We targeted reactive airways disease because of its prevalence, morbidity, mortality, responsiveness to appropriate medication therapy, and associated health care costs (Ferguson and Cherniack 1993; Gourley, Gourley et al. 1998; Gourley, Portner et al. 1998; McFadden and Gilbert 1992; Murray, Stang, and Tierney 1997; Solomon et al. 1998; Stroupe, Gaskins, and Murray 1999; Weiss and Sullivan 1993).

Original Study Design

Study Sites

The study was to be conducted in Revco pharmacies. With 99 stores and more than 200 pharmacists, Revco pharmacies filled half of the prescriptions for 1.5 million persons living in central Indiana. Importantly, Revco was participating in the Indianapolis Network for Patient Care (INPC). Funded by the National Library of Medicine National Information Infrastructure Initiative, INPC was designed as a model integrated network that linked data from Indianapolis’ major hospitals, neighborhood health centers, and public health and homeless clinics, as well as Revco drugstores (Overhage, McDonald, and Tierney 1995). For each medication filled, Revco transmitted prescription information to INPC each night (approximately twenty thousand prescriptions per night). We planned to capitalize on the INPC infrastructure both to identify eligible patients (by prescriptions filled) and collect data on emergency department and hospital visits made during the study period.

Patient Eligibility

Patients would be eligible if they: (1) had reactive airways disease (i.e., filled a prescription for methylxanthines, inhaled corticosteroids, inhaled or oral sympathomimetics, inhaled parasympathetic antagonists, or inhaled cromolyn, and confirmed the diagnosis as an active problem during enrollment); (2) were ≥18 years of age; (3) received at least 70 percent of their medications from a single Revco store; (4) had no significant impairment in speech, vision, or hearing that prevented their being interviewed; (5) resided in the community; and (6) provided written informed consent. Initially, we intended to identify potential patients using Revco medication records stored in INPC. The remaining criteria would be assessed during telephone screening.

Study Groups

The core of the pharmaceutical care program involves pharmacists receiving clinically relevant, patient-specific data that allows them to use their skills and training to improve patient outcomes. Thirty-six Revco stores proximal to a major Indianapolis hospital contributing data to the INPC were divided into twelve clusters of three stores. The three stores within each cluster were matched on geographic location, percent of Medicaid-insured adults with reactive airways disease (a proxy for socioeconomic status), and the number of prescriptions filled (high versus low volume). Within each cluster, stores were randomly assigned to one of three study groups: (1) pharmaceutical care program group, (2) peak flow meter monitoring group, or (3) usual care control group.

Essential to our conception of pharmaceutical care was pharmacists’ having access to clinically relevant, patient-specific data. Thus, patients in the pharmaceutical care program group received a peak flow meter and personalized instruction about its use during the baseline interview. Then, during monthly telephone interviews conducted by research assistants, patients were asked to use the meter and report the peak flow result. The Revco-owned computer present in all stores to dispense medications was to alert pharmacists when a study patient filled any prescription. When this occurred, pharmacists were instructed to go to a separate study computer to review patient-specific information relevant to our pharmaceutical care program (i.e., peak flow results, recent emergency department visit, recent hospital admission, data on breathing medications).

Because persons in the pharmaceutical program group were called monthly to obtain a current peak flow result, we were concerned that these calls alone could constitute an intervention, that is, patients might increase their self-care due to greater monitoring of their disease activity. Thus, we included a peak flow meter only control group in which patients also received a peak flow meter and instructions about its use. However, to separate the effects of our program from increased monitoring, peak flow data in this group were not provided to the pharmacist. The usual care control group did not have peak flow rates assessed during monthly interviews.

Pilot Test

After completing and pilot testing the program (including recruitment procedures), all pharmacists received training in April 1997. Enrollment was projected to begin in May 1997.

Lessons from the Field: Modifications in the Research Design

Corporate changes affect the study

Literally within days of completing pharmacist training, CVS Pharmacy purchased Revco. The impact of this event was immediate and substantial:

  • The corporate executives who were vested in our project no longer worked for the company. As a result, we had to acquaint ourselves with, and gain commitments from, a new organization that had no prior knowledge of our project. We were fortunate in that CVS was enthusiastic about participating in the study, and remains supportive to date.
  • The CVS and Revco computer systems differed substantially. CVS lacked a centralized database for prescription information. Instead, each store had a separate database with unique patient identifiers. Thus, we had to develop entirely new strategies to: (a) transfer pharmacy data from CVS to the INPC; (b) merge these data into a unique record for each patient; and (c) transfer patient-specific data from the INPC to the individual CVS drug stores. The conversion from the Revco to the CVS computer system occurred between October 1997 and January 1998, after which we had to pilot test all data transfer programs.
  • Enrollment had to be delayed until we completed testing the datatransfer programs (February 1998). Given this substantial delay, wethought it was critical to retrain intervention pharmacists (completed in January 1998). Thus, enrollment was scheduled to begin in February 1998.

Patient confidentiality

Soon after acquiring Revco, CVS reviewed Revco's practices related to sharing data. This was clearly appropriate, given that academic and lay media brought national attention to many complex issues surrounding patient confidentiality. Thus, CVS halted data flow to INPC so that they could thoughtfully consider the implications of providing data to INPC. It is critical to note that Indiana law allows data transfer between two health care providers (e.g., CVS and INPC), with the stipulation that recipients of these data (i.e., INPC) are bound by the same rules of confidentiality as the health care provider transmitting the data. In the weeks that followed, CVS adopted a policy to safeguard patients’ confidentiality in a manner even more protective than that required by Indiana law. Specifically, CVS prohibited transfer of any prescription data to INPC (or released to us) without patients’ signed permission (“opt-in”). This made it impossible to use our original strategy to identify potential patients using prescription data from INPC. Over the next months, we worked closely with CVS to develop a recruitment strategy that, although more complex, balanced scientific and privacy concerns. The protocol, which was implemented in July 1998 after being approved by CVS and the Institutional Review Board at Indiana University-Purdue University at Indianapolis, was as follows (Figure 1):

Figure 1
Recruitment Protocol
  • CVS programmers queried their database to identify all customers ≥18 years old who had filled a prescription for a breathing medication at any study drug store within the previous four months (N=14,195).
  • CVS mailed letters to these potential participants stating that: (1) CVS was working with university investigators to develop programs to improve the health of its customers; (2) investigators were evaluating these programs by talking to customers; (3) customers would be paid up to $60 for participating in the evaluation; and (4) they could page an on-call investigator with any questions about the project. Subjects were asked to sign and return an “opt-in” form indicating that CVS had permission to release their names to investigators or that they were not interested in participating. We received forms from 3,019 of these individuals, 2,195 of whom expressed interest in participating.
  • A CVS employee attempted to telephone the 11,176 persons who failed to return a signed form to insure that they received the letter, determine their interest in participating, and offer to send them another form. CVS callers were unable to talk to 7,641 persons, despite at least two calling attempts and leaving messages on answering machines. Of the remaining 3,535 persons, 2,065 had an incorrect address or phone number (and lacked more accurate contact information), 682 refused, 32 were ineligible, and an additional 756 returned forms indicating their interest in participating, thus increasing the potential respondent pool by 34.4 percent.
  • Only after CVS received a signed opt-in form indicating a person's willingness to be contacted was the person's name released to the project manager.
  • The project manager conducted a screening interview by telephone during which she described the study, determined patient eligibility, and, for eligible patients, arranged a baseline interview at a time and location convenient for the patient. During that interview, patients signed an informed consent statement allowing investigators’ access to their CVS data. Only then did CVS transmit encrypted medication data to INPC.

Altering employees’ behavior

From the outset, we thought it was essential that investigators be in close contact with stores in the pharmaceutical care group. Thus, three of the investigators were each assigned 3–5 intervention stores to visit periodically to identify any problems in the field and encourage full participation in the program. Early on, it became apparent that some pharmacists were not compliant with the various components of the program. Thus, we initiated strategies to enhance compliance. We began by monitoring the percentage of patients for whom pharmacists: (a) viewed data in the study computer and (b) took actions consistent with pharmaceutical care (e.g., made comments in the field available, distributed handouts). These data comparing pharmacists to their colleagues in other stores were faxed to pharmacists weekly, with the hope that this would encourage pharmacists to use data in the study computer. This, alone, was not successful. As a consequence, their direct supervisors were also provided these data and asked to follow-up with pharmacists who did not participate in the program. This strategy also was unsuccessful. Our final strategy was to offer pharmacists incentives ($50 gift certificates each month) if they met or exceeded predefined levels of compliance (viewing data for 90 percent of patients and documenting pharmaceutical care for 75 percent of patients). Pharmacists in the two stores with the highest compliance rates over the study period would receive an additional incentive of $100. These incentives appeared to encourage pharmacists to implement our program.

While it is not possible to determine the precise barriers to implementation, we have several recommendations that may be useful to others. First, the program needs to be as convenient as possible. In our particular case, it was not possible to integrate our program into the regular CVS store computers. Thus, we provided a separate study computer that contained the patient-specific data. Advances in web-based applications (including firewalls) may now permit integration of such programs into the store computer. Second, we trained all intervention pharmacists to deliver our program. Soon after training, it became evident that pharmacists were not uniformly interested in implementing the program. A better strategy may have been to identify one person in each store who was both enthusiastic about delivering pharmaceutical care and responsible for implementing it at his or her store. Third, implementation of the program must be viewed by pharmacists as part of their jobs. Thus, with acceptance of responsibility for the program, corporations may consider providing incentives for performance using measures reflecting either implementation of the program or actual patient outcomes.


We believe that RCTs of health services research interventions conducted in nonacademic practice settings can have a major impact on our health care system. While our experience comes from retail pharmacies, we believe that many lessons we learned may be applicable to other nonacademic practice settings. Based on our experience, we believe we can offer other investigators some advice.

  • Conduct research that is consistent with corporate goals. If the research involves a corporation, investigators need to be prepared to explicitly state how the proposed intervention will fit into the corporation's long-term plans. Investigators should be familiar with a particular industry, as well as the goals and objectives of any corporation being considered as a research venue. If the corporation needs to provide resources, it must understand the potential value of its investment. Investigators must use language and incentives that differ from those used in academic settings. We had to explain the rationale for methodological decisions using nontechnical vocabulary. While we provided CVS management with a copy of the grant proposal, that alone was insufficient. We had multiple face-to-face meetings to ensure clear and explicit understandings and expectations. This should ideally be obtained prior to initiating the study. Fortunately, in our case, the corporate goals were similar within the two companies. Thus, the change in ownership did not alter CVS's fundamental interest in evaluating our pharmaceutical care program, allowing us to conduct our investigation.
  • Involve the appropriate corporate persons early in the process. While one certainly needs buy-in at a corporate decision-making level, it is essential to identify key individuals at all levels who are sufficiently empowered and capable of facilitating smooth implementation of the project. In our case, involving key regional and local persons (including pharmacists and pharmacy technicians in the stores) was as necessary as would be involving physicians, nurses, and other clinic personnel when conducting an RCT of a clinical guideline in their clinic.
  • Be flexible. Sometimes, investigators tend to be somewhat rigid about modifying their research design. Modifications that compromise the integrity of the study should not be initiated. However, careful thought may identify options that are scientifically sound. In our case, while the final patient recruitment protocol was substantially more cumbersome and labor intensive than originally designed, it ultimately proved successful. Furthermore, we were able to deal with substantial differences in the two corporations’ information systems.

In summary, conducting RCTs in a variety of practice settings should continue to be a goal for health services researchers. When hearing about our experiences, one might expect us to discourage investigators from pursuing this line of research. Nothing could be further from the truth. While we were delayed in starting our study, we completed recruitment in December 1999, having enrolled 1,113 patients. Moreover, our follow-up has been outstanding, as some 85 percent of patients have completed the 6-month follow-up. We had a similarly excellent rate for completing the 12-month closeout interviews. Moreover, we are already in the process of capitalizing on our established relationship with CVS to conduct additional studies related to pharmaceutical care for patients with heart failure.


The authors wish to acknowledge the contributions of Clement McDonald, Larry Lemmon, Robert Thompson, Birgit Erickson, Paul Traficante, Marina Weisburd, Faye Smith, Carol DeHaven, Rosalie DeMoss, Elizabeth Jobe, Michael Del Busto, David White, Randall Miller, Stacey Hart, Andy Arland, and Laura Garza.


This study was funded by a grant from the Agency for Healthcare Research in Quality (5 R01 HS09083). Dr. Weinberger was supported by the Health Services Research and Development Service, Department of Veterans Affairs, through a Research Career Scientist Award.


  • Ferguson GT, Cherniack RM. “Management of Chronic Obstructive Pulmonary Disease” New England Journal of Medicine. 1993;328(14):1017–22. [PubMed]
  • Gourley DR, Gourley GA, Solomon DK, Portner TS, Bass GE, Holt JM, Braden RL, Rawls N, Wicke WR, Ogden J, Lawrence B. “Development, Implementation, and Evaluation of a Multicenter Pharmaceutical Care Outcomes Study” Journal of the American Pharmaceutical Association. 1998;38(5):567–73. [PubMed]
  • Gourley GA, Portner TS, Gourley DR, Rigolosi EL, Holt JM, Solomon DK, Bass GE, Wicke WR, Braden RL. “Humanistic Outcomes in the Hypertension and COPD Arms of a Multicenter Outcomes Study” Journal of the American Pharmaceutical Association. 1998;38(5):586–97. [PubMed]
  • Hatoum HT, Akhras K. “A 32-year Literature Review on the value and Acceptance of Ambulatory Care Provided by Pharmacists” Annals of Pharmacotherapy. 1993;27(September):1106–19. [PubMed]
  • Hepler CD, Strand LM. “Opportunities and Responsibilities in Pharmaceutical Care” American Journal of Hospital Pharmacy. 1990;47(March):533–43. [PubMed]
  • Kennie NR, Schuster BG, Einarson TR. “Critical Analysis of the Pharmaceutical Care Research Literature” Annals of Pharmacotherapy. 1998;32(January):17–26. [PubMed]
  • Latner AW. “The Top 200 Drugs of 1999” Pharmacy Times. 2000;66(4):16–32.
  • McFadden ER, Gilbert IA. “Asthma” New England Journal of Medicine. 1992;327(27):1928–37. [PubMed]
  • Murray MD, Stang P, Tierney WM. “Health Care Use by Inner-City Patients with Asthma” Journal of Clinical Epidemiology. 1997;50(2):167–74. [PubMed]
  • Overhage JM, McDonald CJ, Tierney WM. “Design and Implementation of the Indianapolis Network for Patient Care and Research” Bulletin of the Medical Library Association. 1995;83(1):48–56. [PMC free article] [PubMed]
  • Singhal PK, Raisch DW, Gupchup GV. “The Impact of Pharmaceutical Services in Community and Ambulatory Care Setting: Evidence and Recommendations for Future Research” Annals of Pharmacotherapy. 1999;33(December):1336–55. [PubMed]
  • Solomon DK, Portner TS, Bass GE, Gourley DR, Gourley GA, Holt JM, Wicke WR, Braden RL, Eberle TN, Self TH, Lawrence BL. “Clinical and Economic Outcomes in the Hypertension and COPD Arms of a Multicenter Outcomes Study” Journal of the American Pharmaceutical Association. 1998;38(5):574–85. [PubMed]
  • Strand LM, Cipolle RJ, Morley PC, Perrier DG. “Levels of Pharmaceutical Care: A Needs-Based Approach” American Journal of Hospital Pharmacy. 1991;48(3):547–50. [PubMed]
  • Stroupe KT, Gaskins D, Murray MD. “Healthcare Costs of Inner-City Patients with Asthma” Journal of Asthma. 1999;36(8):645–55. [PubMed]
  • Tett SE, Higgins GM, Armour CL. “Impact of Pharmacist Interventions on Medication Management by the Elderly: A Review of the Literature” Annals of Pharmacotherapy. 1993;27(January):80–5. [PubMed]
  • Weiss KB, Sullivan SD. “The Economic Costs of Asthma: A Review and Conceptual Model” Pharmacoeconomics. 1993;4(1):14–30. [PubMed]

Articles from Health Services Research are provided here courtesy of Health Research & Educational Trust