Search tips
Search criteria 


Logo of amjpharmedLink to Publisher's site
Am J Pharm Educ. 2016 June 25; 80(5): 88.
PMCID: PMC4937983

A Systematic Review of the Effects of Continuing Education Programs on Providing Clinical Community Pharmacy Services


Objective. To summarize the effects of media methods used in continuing education (CE) programs on providing clinical community pharmacy services and the methods used to evaluate the effectiveness of these programs.

Methods. A systematic review was performed using Medline, SciELO, and Scopus databases. The timeline of the search was 1990 to 2013. Searches were conducted in English, Portuguese, and Spanish.

Results. Nineteen articles of 3990 were included. Fourteen studies used only one media method, and the live method (n=11) was the most frequent (alone or in combination). Only two studies found that the CE program was ineffective or partially effective; these studies used only the live method. Most studies used nonrobust, nonvalidated, and nonstandardized methods to measure effectiveness. The majority of studies focused on the effect of the CE program on modifying the knowledge and skills of the pharmacists. One study assessed the CE program’s benefits to patients or clients.

Conclusion. No evidence was obtained regarding which media methods are the most effective. Robust and validated methods, as well as assessment standardization, are required to clearly determine whether a particular media method is effective.

Keywords: continuing education, community pharmacy, pharmacy service


Community pharmacists are among the most accessible health professionals. They are in a unique position to help patients manage several health conditions. In recent decades, the role of community pharmacists has shifted from providing medications to providing clinical community pharmacy services (eg, drug therapy management, residential medication management review, and pharmacy-based minor ailment treatment regiments, among others).1-3 This shift has created a need to develop and maintain expertise and competence in new areas, including pharmacotherapy, interpersonal communication, and patient information documentation.4,5

However, pharmacy curricula in several countries do not focus on clinical community pharmacy services. Only a few disciplines focus on developing communication skills and pharmacotherapy knowledge.6-8 Studies report that community pharmacists in several countries have insufficient knowledge in these areas,9-12 and community pharmacists commonly provide clinical community pharmacy services at an unsatisfactory level of quality.13-15 Such a situation poses problems concerning appropriateness, effectiveness, safety, and adherence to patients’ drug therapy regimens.

Continuing education (CE) programs can play a vital role in expanding basic pharmacy education and enhancing therapeutic management skills, particularly in areas for which insufficient training has been received or achieved during undergraduate studies.16 Increasingly, more countries are implementing CE program for community pharmacists as obligatory lifelong learning programs in an attempt to improve clinical community pharmacy services. Nevertheless, much remains unclear about which media methods in CE programs provide the most effective educational approach to providing clinical community pharmacy services. “Media method” refers to the method by which the CE program activity is delivered. We adopted the definition of media methods proposed by the Johns Hopkins Evidence-based Practice Center (Table 1).17 To our knowledge, no published systematic review evaluates the effects of various media methods in CE programs on providing clinical community pharmacy services.

Table 1.
Definition of Media Methods17

Considering the importance of improving the quality of clinical community pharmacy services provided to patients, the primary aim of this study was to summarize scientific evidence of the effects of media methods in CE programs on providing clinical community pharmacy services. The secondary aim was to summarize methods used to evaluate the effectiveness of CE programs.


A systematic review was performed according to recommendations of Preferred Reporting Items of Systematic Reviews and Meta-Analyses (PRISMA).18 Because this was a systematic review of published data, no approval was needed from the university’s institutional review board. Studies were included if they were original articles published in English, Portuguese, or Spanish from 1990 to 2013 that evaluated the effects of CE programs on providing clinical community pharmacy services. We searched Medline, Scopus, and Scientific Electronic Library Online (SciELO) databases. Additional articles were identified by searching the references of articles identified in these searches.

The search strategy for the databases is described in Appendices 1, 2, and 3. Search terms included: Continuing education (Title/Abstract); Continuing Professional development (Title/Abstract); Life-long learning (Title/Abstract); Professional development (Title/Abstract); Training (Title/Abstract); Postgraduate education (Title/Abstract); Adult learning (Title/Abstract); Competence development (Title/Abstract); Competence (Title/Abstract); Training and education (Title/Abstract); Recurrent education (Title/Abstract); or Practical work training (Title/Abstract) Study selection.

Two independent reviewers scanned the titles of the articles in parallel. If both reviewers felt a title was potentially eligible, then it was promoted to abstract review; if there were disagreements between reviewers, a third reviewer determined if the manuscript was promoted to abstract review. The abstract review phase was performed similarly to the title scans by the same reviewers. After the abstract review, the reviewers read the full article and determined whether it met the inclusion criteria. The articles proceeded to the data collection process if the two reviewers determined that an article was potentially eligible; if there was disagreements between reviewers, a third reviewer determined if the article was potentially eligible.

A data extraction sheet was developed, pilot-tested, and refined via discussion and consensus by three authors. Two independent reviewers conducted the data extraction using the data extraction sheet in parallel. Disagreement between reviewers was resolved by consensus. If no agreement could be reached, then a third reviewer made the determination. The data extracted from the articles included the first author’s name, year of publication, country where the study was conducted, study design, number of participants, content area of the CE program, media method used in the CE program, CE program description, methods of evaluation used to assess the effectiveness of the CE program.

The methodological quality of each study was assessed according to criteria based on Reed et al19 and Best Evidence Medical Education (BEME).20 According to the BEME, the strength equates with critical appraisal and is a statement of one’s confidence that the results of the study are credible. The BEME proposes five levels of strength: (1) no clear conclusions can be drawn; not strong, (2) results ambiguous; there seems to be a trend, (3) conclusions can probably be based on the results, (4) results are clear and very likely to be true, and (5) results are unequivocal. Strength was evaluated independently by three reviewers, and differences between these reviewers about the strength assessment was resolved by discussion; interrater agreement (determined on the basis of the ratings before consensus was reached) was high (kappa 0.89; 95 percent confidence interval 0.82 to 0.99).

The BEME recommends evaluating how each study reports the outcomes of an intervention and then collecting these into a common format. The BEME defines this evaluation as “importance,” which was evaluated independently by three reviewers, and differences between their judgments was resolved by discussion; interrter agreement (determined on the basis of the ratings before consensus was reached) was high (kappa 0.88; 95 percent confidence interval 0.81 to 0.99). The BEME levels of importance are level 1: participation – covers learners’ views on the learning experience, its organization, presentation, content, teaching methods, aspects of the instructional organization, materials, and quality of instruction; level 2a: modification of attitudes or perceptions – outcomes at this level relate to changes in the reciprocal attitudes or perceptions between participant groups toward intervention or simulation; level 2b: modification of knowledge and skills – for knowledge, this relates to the acquisition of concepts, procedures, and principles and for skills, this relates to the acquisition of thinking and problem solving and psychomotor and social skills; level 3: behavioral change – documents the transfer of learning to the workplace or willingness of learners to apply new knowledge and skills; level 4a: change in organizational practice – wider changes in the organization or delivery of care, attributable to an educational program; and level 4b: benefits to patient or clients – any improvement in the health and well-being of patients and clients as a direct result of an educational program.


The initial search identified 3974 articles (988 from PubMed, 2986 from Scopus, 0 from SciELO). An additional 16 articles were added by reviewing the reference lists of these articles, yielding a total of 3990 articles. After eliminating duplicates, 3951 remained. A total of 111 articles remained after title scans. After abstract review, 31 articles remained. Finally, full-text articles were retrieved for review, and 19 articles met the eligibility criteria and were included in this systematic review (Figure 1).21-39

Figure 1.
Flowchart of search strategy and study selection.

Most the articles were published from 2000 to 2013 (n=18). One study was published in 1997.39 The studies were performed in countries on the following continents: four in North America,27,28,36,30 six in Europe,22-24,27,30,36 four in Asia,21,25,26,34 and five in Australia/Oceania.32,33,35,38,39 The sample sizes of these studies were generally small. Nine studies had a community pharmacists sample size of less than 50,23,27,31,33,34,36-39 seven studies had a community pharmacists sample size of 50 to 100,22,24,25,28-30,35 two studies had a community pharmacists sample size of 101 to 200,26,32 one study had a community pharmacists sample size of more than 200.21 With regard to study design, 12 studies were before-and-after studies,21,24-27,31-37 and seven studies were controlled trials.22,23,28-30,38,39 The features of these studies are summarized in Tables 2, 3, and 4.

Seventeen CE program content areas were identified (Tables 2-4). The content areas of the CE programs varied, from specific disease management (eg, asthma, childhood diarrhea, and depression) to general issues related to providing pharmaceutical services (eg, providing written drug information and information needed to identify and resolve drug-related problems). Most of the CE programs (n=13) had their content areas chosen by the researchers who conducted the studies without assessing gaps in knowledge or the self-reported educational needs of the community pharmacists.22,23,25,26,28-30,32-37

Most of the studies (n=14)21-34 used just one media method in the CE program. The live method was the most frequent (n=11),21-31 followed by the Internet (nonreal time; n=2),32,33 and print (n=1)34 (Table 2). Four studies used two media methods in the CE programs.35-38 Live methods were used in each of these combinations. The combinations included live computer-based and offline (n=1),35 live Internet and nonreal time (n=1),36 and live and print (n=2)37,38 (Table 3). One study used a combination of three media methods in the CE program: live, video, and print39 (Table 4).

Various outcome measures were used to assess the effectiveness of the CE programs. Most of the studies used nonvalidated instruments (n=11) to measure outcomes.21,28-30,32,33,35-39 Fifteen studies used more than one different outcome measure concomitantly (one study used four outcome measures;39 seven studies used three outcome measures;21,28-30,32,33,38 seven studies used two outcome measures22,25-27,34-36). The majority of the outcome measures focused on the effect of the CE program on modifying the knowledge and skills of the community pharmacists (n=14),21,23,25-30,32-35,38,39 modifying the attitudes or perceptions of the community pharmacists (n=14),21,22,25-28,30-34,36,38,39 and the learning experience of the community pharmacists (n=8).22,28,32,33,35,36,38,39 Four studies assessed the effects of the CE program on changes in community pharmacists behavior.21,24,29,30 One study assessed the effects of the CE program on changes in organizational practice.29 One study assessed the CE program’s benefits to patients or clients.39

Almost every study reported that the CE program was effective with regard to the outcome measures used. Only the study by Watson et al found that the CE program was ineffective for the sale of over-the-counter antifungals.30 Rouleau et al found that the CE program focused on asthma patient care was ineffective on two (interventions described by the community pharmacists in the study log and appropriateness of asthma medications that were used by patients during the study period) of three outcome measures.29 These two studies that found that the CE program was ineffective on one or more outcome measures used the live media method as the sole educational approach.


The systematic review evaluated the effects of media methods in CE programs on providing clinical community pharmacy services. Awareness of the effectiveness of each media method may help governments, pharmacist associations, and CE providers develop higher-quality CE programs. The results provided no evidence regarding which media methods were the most effective. Most of the studies reported that their media methods improved the outcome measures chosen by the researchers.

The results of this systematic review suggest the need to standardize outcome measures used to evaluate CE programs. Without such standardization, it is impossible to compare the effectiveness of the different media methods and the strategies used in the different studies. Kirkpatrick et al recommended that evaluation of CE programs should consist of four levels: learner satisfaction – reaction (level 1), learner outcomes – learning (level 2), performance improvement – behavior (level 3), and patient or health outcomes – results (level 4).40 Kirkpatrick’s model recommends that evaluations should begin with level 1 and then sequentially move through the other levels.40 Each successive level represents a different measure of the effectiveness of CE programs and increases the clinical significance. We strongly recommend that future studies use Kirkpatrick’s model, which will allow comparisons of different CE programs and identification of which strategies are the most effective.

Although our systematic review and the studies’ results do not provide definitive answers regarding which media method is most effective, the findings may be useful for indicating which strategies may be worth pursuing in larger, longer-term trials with more standardized outcomes. The most thoroughly evaluated media method was live media. This method has been shown to improve community pharmacists´ participation, attitudes, knowledge, behavior (based on questionnaire surveys), simulated patient visits, and community pharmacy services based on direct observation.21-31,35-38 However, the unique study evaluating the effects of CE programs that used live media on organizational practice (n=1)29 and benefits to patients or clients (n=1)39 found that the CE program was ineffective in these outcome measures.

Another important point in the evaluation of CE programs is the extent to which the knowledge, confidence, and skills obtained can be sustained. Patient benefits must be continuous and not limited to certain periods of time. Follow-up evaluations are needed to determine the duration of effectiveness of the educational interventions, strategies that can reduce the decline of the effectiveness of CE programs, and the time intervals at which CE programs should be implemented.41,42 The studies in our sample did not report follow-up evaluations, so we were unable to assess such variables.

This systematic review has some limitations that must be mentioned. The restriction of languages in the inclusion criteria likely reduced our sample. We also only included studies that were indexed in the PubMed, Scopus, and SciELO databases.


We found no evidence regarding which media method is the most effective. The majority of the CE programs were reported to be effective based on the studies’ outcome measures. Standardizing outcome measures would make it possible to evaluate what media method is the most effective as would evaluating outcomes rather than satisfaction and knowledge of learners. The use of the Kirkpatrick’s model (evaluating every proposed level) could be a viable alternative to standardizing outcome measures and augmenting the outcomes evaluated.


The authors would like to thank to Marília Silveira de Almeida Campos for her support in the manuscript preparation.

Appendix 1. Search Strategy on Medline

An external file that holds a picture, illustration, etc.
Object name is ajpe80588-t5.jpg

Appendix 2. Search Strategy on Scopus

An external file that holds a picture, illustration, etc.
Object name is ajpe80588-t6.jpg

Appendix 3. Search Strategy on SciELO

An external file that holds a picture, illustration, etc.
Object name is ajpe80588-t7.jpg


1. Paudyal V, Hansford D, Cunningham S, Stewart D. Pharmacy assisted patient self care of minor ailments: a chronological review of UK health policy documents and key events 1997-2010. Health Policy. 2011;101(3):253–259. [PubMed]
2. Berbatis CG, Sunderland VB, Joyce A, Bulsara M, Mills C. Enhanced pharmacy services, barriers and facilitators in Australia’s community pharmacies: Australia’s national pharmacy database project. Int J Pharm Pract. 2007;15(3):185–191.
3. Christensen DB, Farris KB. Pharmaceutical care in community pharmacies: practice and research in the US. Ann Pharmacother. 2006;40(7-8):1400–1406. [PubMed]
4. Cipolle RJ, Strand LM, Morley PC. Pharmaceutical Care Practice: The Patient-Centered Approach to Medication Management. 2nd edition. New York, NY: McGraw-Hill Medical; 2012.
5. Storpirts S. Perspectives and challenges in pharmaceutical care. Braz J Pharm Sci. 2012;48(4)
6. de Castro MS, Correr CJ. Pharmaceutical care in community pharmacies: practice and research in Brazil. Ann Pharmacother. 2007;41(9):1486–1493. [PubMed]
7. Ghilzai NM, Dutta AP. India to introduce five-year doctor of pharmacy program. Am J Pharm Educ. 2007;71(2):Article 38. [PMC free article] [PubMed]
8. Volmer D, Vendla K, Vetka A, Bell JS, Hamilton D. Pharmaceutical care in community pharmacies: practice and research in Estonia. Ann Pharmacother. 2008;42(7):1104–1111. [PubMed]
9. Obreli-Neto PR, Pereira LRL, Guidoni CM, et al. Survey of Brazilian community pharmacist knowledge about combined oral contraceptives. Afr J Pharm Pharmacol. 2013;7(31):2214–2219.
10. Hellerstedt WL, Van Riper KK. Emergency contraceptive pills: dispensing practices, knowledge and attitudes of South Dakota pharmacists. Perspect Sex Reprod Health. 2005;37(1):19–24. [PubMed]
11. Ahmed SI, Hassali MA, Aziz NA. An assessment of the knowledge, attitudes, and risk perceptions of pharmacy students regarding HIV/AIDS. Am J Pharm Educ. 2009;73(1):Article 15. [PMC free article] [PubMed]
12. Liekens S, Smits T, Laekeman G, Foulon V. Pharmaceutical care for people with depression: Belgian pharmacists’ attitudes and perceived barriers. Int J Clin Pharm. 2012;34(3):452–459. [PubMed]
13. Smith F. The quality of private pharmacy services in low and middle-income countries: a systematic review. Pharm World Sci. 2009;31(3):351–361. [PubMed]
14. Obreli-Neto PR, Pereira LR, Guidoni CM, et al. Use of simulated patients to evaluate combined oral contraceptive dispensing practices of community pharmacists. PLoS One. 2013;8(12):e79875. [PMC free article] [PubMed]
15. Norris PT. Purchasing restricted medicines in New Zealand pharmacies: results from a ‘‘mystery shopper’’ study. Pharm World Sci. 2002;24(4):149–153. [PubMed]
16. International Pharmaceutical Federation: Appendix 5: Summary of CPD and CE systems by country, Global Pharmacy Workforce and Migration Report; 2006. Accessed March 25, 2014.
17. Marinopoulos SS, Dorman T, Ratanawongsa N, et al. Effectiveness of continuing medical education. Evid Rep Technol Assess. 2007;149:1–69. [PubMed]
18. Moher D, Liberati A, Tetzlaff J, Altman DG. PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264–269. [PubMed]
19. Reed D, Price EG, Windish DM, et al. Challenges in systematic reviews of educational intervention studies. Ann Intern Med. 2005;142(12, Part 2):1080–1089. [PubMed]
20. BEME Collaboration. Best Evidence Medical Education homepage on the Internet. Dundee: BEME Collaboration, 2003. Accessed November 10, 2013.
21. Minh PD, Huong DT, Byrkit R, Murray M. Strengthening pharmacy practice in vietnam: findings of a training intervention study. Trop Med Int Health. 2013;18(4):426–434. [PubMed]
22. Liekens S, Smits T, Laekeman G, Foulon V. A depression training session with consumer educators to reduce stigmatizing views and improve pharmacists’ depression care attitudes and practices. Am J Pharm Educ. 2013;77(6):Article 120. [PMC free article] [PubMed]
23. Liekens S, Vandael E, Roter D, et al. Impact of training on pharmacists’ counseling of patients starting antidepressant therapy. Patient Educ Couns. 2014;94(1):110–115. [PubMed]
24. Meštrović A, Staničić Z, Hadžiabdić MO, et al. Individualized education and competency development of Croatian community pharmacists using the general level framework. Am J Pharm Educ. 2012;76(2):Article 23. [PMC free article] [PubMed]
25. Chen HY, Lee TY, Huang WT, Chang CJ, Chen CM. The short-term impact of a continuing education program on pharmacists’ knowledge and attitudes toward diabetes. Am J Pharm Educ. 2004;68(5):Article 121.
26. Chiang YC, Lee CN, Lin YM, Yen YH, Chen HY. Impact of a continuing education program on pharmacists’ knowledge and attitudes toward asthma patient care. Med Princ Pract. 2010;19(4):305–311. [PubMed]
27. Fitzgerald N, Watson H, McCaig D, Stewart D. Developing and evaluating training for community pharmacists to deliver interventions on alcohol issues. Pharm World Sci. 2009;31(2):149–153. [PubMed]
28. Lalonde L, Normandeau M, Lamarre D, et al. Evaluation of a training and communication-network nephrology program for community pharmacists. Pharm World Sci. 2008;30(6):924–933. [PubMed]
29. Rouleau R, Beauchesne MF, Laurier C. Impact of a continuing education program on community pharmacists’ interventions and asthma medication use: a pilot study. Ann Pharmacother. 2007;41(4):574–580. [PubMed]
30. Watson MC, Bond CM, Grimshaw JM, Mollison J, Ludbrook A, Walker AE. Educational strategies to promote evidence-based community pharmacy practice: a cluster randomized controlled trial (RCT) Fam Pract. 2002;19(5):529–536. [PubMed]
31. Currie JD, Chrischilles EA, Kuehl AK, Buser RA. Effect of a training program on community pharmacists’ detection of and intervention in drug-related problems. J Am Pharm Assoc. 1997;37(2):182–191. [PubMed]
32. Walters C, Raymont A, Galea S, Wheeler A. Evaluation of online training for the provision of opioid substitution treatment by community pharmacists in New Zealand. Drug Alcohol Rev. 2012;31(7):903–910. [PubMed]
33. Zolezzi M, Blake A. Principles-based learning design for an online postgraduate psychiatric pharmacy course. Am J Pharm Educ. 2008;72(5):Article 107. [PMC free article] [PubMed]
34. Adepu R, Shariff A. Development, validation and implementation of continuous professional development programmes for community pharmacists. Indian J Pharm Sci. 2010;72(5):557–563. [PMC free article] [PubMed]
35. Stafford L, Peterson GM, Bereznicki LR, Jackson SL, van Tienen EC. Training Australian pharmacists for participation in a collaborative, home-based post-discharge warfarin management service. Pharm World Sci. 2010;32(5):637–642. [PubMed]
36. Laaksonen R, Duggan C, Bates I, Mackie C. Performance in clinical therapeutics training of community pharmacists. Pharm Educ. 2007;7(2):167–175.
37. Tibbs K, Swensgard K, Conrad W, Heaton PC. Implementation of pharmacist training and counseling program on vitamins and minerals. J Am Pharm Assoc. 2007;47(4):502–504. [PubMed]
38. Aslani P, Benrimoj SI, Krass I. Development and evaluation of a training program to foster the use of written drug information in community pharmacies: part 2 – evaluation. Pharm Educ. 2006;7(2):141–149.
39. Saini B, Smith L, Armour C, Krass I. An educational intervention to train community pharmacists in providing specialized asthma care. Am J Pharm Educ. 2006;70(5):Article 118. [PMC free article] [PubMed]
40. Kirkpatrick DL. Evaluating Training Programs: The Four Levels. San Francisco, CA: Berrett-Koehler; 1994.
41. Curtis P, Carey TS, Evans P, Rowane MP, Mills GJ, Jackman A. Training primary care physicians to give limited manual therapy for low back pain: patient outcomes. Spine. 2000;25(22):2954–2960. [PubMed]
42. Thompson C, Kinmonth AL, Stevens L, et al. Effects of a clinical-practice guideline and practice-based education on detection and outcome of depression in primary care: Hampshire Depression Project randomized controlled trial. Lancet. 2000;355(9199):185–191. [PubMed]

Articles from American Journal of Pharmaceutical Education are provided here courtesy of American Association of Colleges of Pharmacy