PMCCPMCCPMCC

Search tips
Search criteria 

Advanced


 
Formats
Article sections
Authors
Related links
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptNIH Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
J Am Board Fam Med. Author manuscript; available in PMC 2012 January 26.
Published in final edited form as:
PMCID: PMC3266837
NIHMSID: NIHMS347454
Copyright notice and Disclaimer
A logic model framework for evaluation and planning in a primary care practice-based research network (PBRN)
Holly Hayes, MSPH,1 Michael L. Parchman, MD, MPH,1,2 and Ray Howard, MBA3
1Department of Family and Community Medicine, University of Texas Health Science Center San Antonio
2VERDICT Health Services Research Program, South Texas Veterans Health Care System
3Academic Center for Excellence in Teaching
Corresponding Author: Holly Hayes, MSPH, 7703 Floyd Curl Drive, Mail Code: 7728, San Antonio, TX 78229-, Phone: 210-567-7863, Fax: 210-567-7868, hayesh/at/uthscsa.edu
Small right arrow pointing to: The publisher's final edited version of this article is available free at J Am Board Fam Med
Small right arrow pointing to: See other articles in PMC that cite the published article.
Abstract
Evaluating effective growth and development of a Practice-Based Research Network (PBRN) can be challenging. The purpose of this article is to describe the development of a logic model and how the framework has been used for planning and evaluation in a primary care PBRN.
An evaluation team was formed consisting of the PBRN directors, staff and its board members. After the mission and the target audience were determined, facilitated meetings and discussions were held with stakeholders to identify the assumptions, inputs, activities, outputs, outcomes and outcome indicators.
The long-term outcomes outlined in the final logic model are two-fold: 1.) Improved health outcomes of patients served by PBRN community clinicians; and 2.) Community clinicians are recognized leaders of quality research projects. The Logic Model proved useful in identifying stakeholder interests and dissemination activities as an area that required more attention in the PBRN. The logic model approach is a useful planning tool and project management resource that increases the probability that the PBRN mission will be successfully implemented.
With the heightened emphasis on translational and comparative effectiveness research to improve patient outcomes, Practice-Based Research Networks (PBRNs) have an unprecedented opportunity to become effective laboratories to address high priority research questions. As PBRNs engage in more funded research, these research dollars come with increased accountability to demonstrate the effectiveness of the work conducted in PBRNs. Despite a significant growth in the number of PBRNs over the past 15 years, little is known about effective and useful methods of evaluating PBRNs (1). One method with significant potential for PBRN evaluation and planning is a logic model.
What is a logic model?
The logic model has proven to be a successful tool for program planning as well as implementation and performance management in numerous fields, including primary care (2–14). A logic model (see Figure One) is defined as a graphical/textual representation of how a program is intended to work and links outcomes with processes and the theoretical assumptions of the program (6). It is a depiction of a program or project showing what the program or project will do and what it is to accomplish. It is a series of “if then” relationships that, if implemented as intended lead to the desired outcomes. Stated another way, it is a framework for describing the relationships between resources, activities and results as they related to a specific program or project goal. The logic model also helps to make underlying assumptions about the program or project explicit. It provides a common approach to integrating planning, implementation and evaluation. Figure One below defines the key components of a logic model and what variables are included for each section
Figure 1
Figure 1
Program/Project Logic Model Framework
Why use a logic model?
A logic model is an efficient tool that requires little resources other than personnel time. Since evaluation dollars are not usually set aside in PBRN budgets, the cost-efficiency of this framework is attractive. In addition, the process of developing the logic model requires PBRN team members to work together in a manner that has a side benefit of improving team relationships and focus. A logic model can also provide much needed detail about how resources and activities can be connected with the desired results which helps with project management, resource allocation and strategic planning (2–14). The process of developing the logic model also facilitates critical thinking through the process of planning and communicating network objectives and outcomes. According to the Kellogg Foundation, the development of a logic model is a “conscious process that creates an explicit understanding of the challenges ahead, the resources available, and the timetable in which to hit the target” (6). For more detailed information regarding logic models, refer to the W.K. Kellogg Foundation Logic Model Development Guide (6).
To date, there are no publications demonstrating how a logic model framework can be used for evaluation and program planning in a primary care PBRN. The purpose of this article is to describe the development of a logic model and how the framework has been used in a primary care PBRN, the South Texas Ambulatory Research Network (STARNet).
STARNet was founded in 1992 “to conduct & disseminate practice-based research that results in new knowledge and improves the health of patients in South Texas.” STARNet has 165 practitioners in 108 primary care practices. These are primarily small group practices or solo practitioners located throughout south Texas – spanning a territory from the southernmost Mexico/Texas border to north central Austin, Texas. Over the years, STARNet has published over 20 peer-reviewed manuscripts of research findings from studies conducted in member primary care practice settings (15–34).
Development of a Logic Model
Step One: Agree on the Mission and Target Audience
The STARNet Board of Directors had previously agreed that the primary goal of all STARNet projects is to improve the health of primary care patients in South Texas. The Board believed that to achieve this goal, STARNet clinicians and academic investigators (Target Audiences) were both equally critical for the success of the network. Investigators facilitate the research process and pursue grant opportunities for the overall sustainability of the network and STARNet clinicians are needed to frame and define the research questions that are relevant to their daily practice and assist in the interpretation of results.
Step Two: Identify and describe assumptions, inputs and activities
After defining the mission and the target audience, the STARNet coordinator and evaluation specialist facilitated ten meetings and discussions with key stakeholders over a six month period. Stakeholders at the meetings included: STARNet Board of Directors who are full-time primary care clinicians in family and internal medicine, practice facilitators who visit clinics regularly and assist with change processes, two STARNet directors with over 10 years of experience with the Network and STARNet partners including the School of Public Health and the South Texas Area Health Education Center. This group was tasked with identifying the assumptions, inputs and activities for the STARNet logic model. Assumptions are identified elements that you assume are in place and necessary to carrying out your strategies. For example, one assumption for PBRN research is that clinicians have time to participate in PBRN research and that investigators have funded grants that will contribute to network support. Once assumptions are identified, inputs are defined. Inputs include a list of identified resources (e.g. Network directors with clinical expertise and connections with the community) as well as constraints (e.g. lack of discretionary funds for relationship building – food, small gifts).
After assumptions and inputs are defined, the activities are described for the program which meets the needs of the target audience. Since the network has existed for over 18 years, it took a concerted effort on all members to think beyond current and past activities and initiatives. The coordinator encouraged the team to place equal attention to thinking about STARNet’s past and current activities and what activities need to take place in order to fulfill its mission. Well-designed activities are an essential element for logic model development. For STARNet, if activities could not be linked directly or deemed relevant to the two long-term outcomes (improved health outcomes of patients and clinician-led research projects), they would not be included in the logic model.
Step Three: Identify Outputs, Outcomes, and Outcome Indicators
To demonstrate STARNet’s growth and development, it was necessary to identify the specific outputs and outcomes necessary to fulfill its mission. Outputs are the actual deliverables or the units of service specific to STARNet– what occurs as a result of the planned activities. For example, the specific output for recruiting STARNet clinicians to the network is the number of new network members. The outcome is the actual impact and change associated with each output and is typically broken down into short-term (1–3 years), intermediate (3–5 years) and long-term (5–10 years). For example, an outcome that would apply to most PBRNs would be the development of the research and resource capacity of the STARNet clinicians (short-term) would lead to an increase in the number/quality of research projects in which STARNet clinicians participate (intermediate) which would in turn result in STARNet clinicians becoming recognized leaders of quality research projects (long-term). Once the outcomes were identified, we created the outcome indicators.
The outcome indicators are the milestones that can be observed and measured toward meeting the program’s mission. These measures are an indicator of how successful your program is in making progress towards the identified goals.
The most time-consuming component of the logic model process was identifying the activities, outputs and outcomes, especially ensuring that linkages existed between these three components. Developing meaningful outcomes that would be useful for grants, reports, publications and that informed members was the most difficult exercise during the logic model development process. The evaluation specialist was extremely helpful in assisting the logic model team in determining what outcomes were important enough to measure. The initial model was circulated to the group several times through e-mail and monthly meetings and further refined in an iterative process.
Final Logic Model
As a result of the above activities, the logic model in Figure 2 was agreed upon by all members. [Insert Figure 2] The logic model begins with the target population and underlying assumptions and leads into the inputs, activities, outputs and outcomes (short-term, intermediate, and long-term). The long-term outcomes of STARNet are two-fold: 1.) Improved health outcomes of patients served by STARNet clinicians; and 2.) STARNet clinicians are recognized leaders of quality research projects. Every input, activity, and outcome in STARNet’s logic model can be linked back to these two long-term outcomes – our mission’s “bull’s eye”.
Figure 2
Figure 2
Figure 2
STARNET LOGIC MODEL
Application of the Logic Model to PBRN Activities
Development of the logic model was considered only an initial phase in the process of evaluating, planning and developing the network. It remained clear throughout the process that an ongoing review and refinement of the logic model would be necessary to ensure that the PBRN implementation activities remained consistent with established outcomes. The group agreed that the first step in using our logic model would be to track the key indicators outlined in the outcomes.
Collecting Outcome Data
The group agreed that the first step in using our logic model would be to track the key indicators outlined in the outcomes. The group created detailed “to-do” lists based on the logic model, quarterly reports and updated Board member job descriptions. STARNet staff made a concerted effort to collect data on all of the outputs in an excel spreadsheet. Thus, the logic model informed and focused staff on what specific data needed to be recorded. The STARNet coordinator is charged with collecting all of the quantitative process and qualitative data each year. Detailed minutes and recordings are now being kept for the following meetings: Network staff, all membership, Board of Directors, and one-on-one site visits with STARNet clinicians. Qualitative data has proven to be very important in documenting the extent of involvement of members in network activities (output 9), not just the number involved, and network contextual changes.
Assessing PBRN Progress
The team meets monthly to assess progress and perform an internal evaluation based on logic model activities, outputs and outcomes. One example of this use of the Logic Model occurred when we discussed our progress towards conducting the activities outlined in the logic model framework during one of our monthly meetings. It was obvious that no efforts had been made to “disseminate research findings” to the network members and the broader community (Activity 9 and Output #6). The Board of Directors and STARNet leadership considered this to be a major process gap if the ultimate outcome is to improve patient health. As a result, STARNet is currently working with the University of Texas School Of Public Health and the South Central Area Health Education Center (AHEC) to create two comprehensive social marketing plans regarding research findings of studies conducted in STARNet for clinicians and their staff as well as patients. STARNet Directors and members will participate in focus groups in Summer, 2011 to develop a strategic communication and dissemination plan. This exemplifies how the logic model can also be used for problem identification and reallocation of resources in order to meet a pre-determined outcome.
Another example of the value of our logic model is when the STARNet Board of Directors decided to take a more proactive role in the financial status of the network. Board agendas now include a financial report at every meeting. STARNet recently became an incorporated non-profit and has updated by-laws and elected officers to Executive Committee. The Board of Directors considers these as crucial steps in meeting the mission of the network and is now developing a business plan to assist with future planning.
Subsequent to initiating our work on a logic model, Bleeker (35) and colleagues from the Netherlands identified only two existing PBRN evaluation tools. These tools were developed by Clement (36) and Fenton (37) to evaluate the overall effectiveness of PBRNs. Clement (36) proposed a conceptual framework to evaluate primary care PBRNs based on seven primary objectives with specific process and outcome indicators. The objectives could be categorized into network infrastructure, activity and dissemination efforts. Based on our review of the evaluation framework proposed by Clement (36), it appeared to be a very usable and feasible tool for implementation. However, Bleeker (35) questioned the validity of these indicators and the feasibility of Clement’s framework for conducting an overall evaluation.
Fenton (37) and colleagues developed the second identified evaluation tool, a Primary Care Research Network Toolkit, which includes a contextualized case study of five networks in the United Kingdom. This toolkit described eight primary dimensions of networks – each one with associated sub-dimensions. Networks could score themselves over time and even conduct comparisons across networks. Although the Primary Care Research Network Toolkit may be a useful in conducting formal evaluations, it lacked sufficient information regarding resources and time needed to successfully replicate the process in the United States.
Considering the relative limited resources of PBRNs, it is not surprising that a majority of PBRNs have not conducted a thorough evaluation of their efforts. Although evaluating a network takes time and requires the involvement of various individuals throughout the process, outcome evaluation efforts are a worthwhile investment. Unfortunately we realized early on, that our budget would not allow us to complete all of the activities outlined in the logic model. It became important to prioritize activities within the logic model due to budget constraints. The logic model should be modified regularly based on the changing capacity and resources of the network. It is yet to be proven whether our logic model framework will meet the planning and evaluation needs of STARNet.
In addition, logic models can be a tremendous tool in determining what is working well and what is not. The Board of Directors continually reminded the staff that all of the activities need to be centered on the mission -- improving patient care. As a result, all of the activities – planned and not planned - are viewed critically from that perspective. It is important to note, however, that every activity cannot be linked directly to long-term outcomes. Based on the logic model framework, the Co-Directors turned away investigators wanting to initiate projects in the network that did not meet the current priorities of the members. This was one of the first times in the history of the Network that it appropriately said “no” to an incompatible research interest. The logic model, in essence, united and empowered the efforts of members in advancing the STARNet mission.
Finally, the logic model reminded the PBRN team that a balance has to be maintained between the hard/traditional measures such as number of studies and publications and the more subjective measures such as easy access to PBRN member offices by PBRN coordinators and researchers. In addition, the core tenet of successful PBRNs is developing and maintaining respectful and trusting long-term relationships that continue beyond research studies (38). The complexity of the relationships and communication within a network is difficult to capture in evaluation efforts. The logic model helped us realize that it’s not just about the quantitative outcomes. In order to share a comprehensive story of STARNet, we also began to collect qualitative data (e.g. rich stories from the members). The logic model helped us realize that in the future, we need to collect this data more systematically from members and patients following the completion of research studies.
Conclusion
In conclusion, we found the logic model to be an effective planning and evaluation tool and a useful project management resource that greatly increases the probability that PBRN goals will be reached consistent with its mission. The logic model framework not only helped facilitate the Network evaluation process, but equally important, it engaged the leadership and members in a meaningful way. As a result, the Board of Directors, community clinician members, academic investigators and staff all have taken a more proactive role working together to advance the STARNet mission.
Acknowledgments
Funding for this study was provided by Clinical Translational Science Award # UL1RR025767 from NCRR/NIH to the University of Texas Health Sciences Center at San Antonio. The authors would like to thank the members of the South Texas Ambulatory Research Network for their support and contribution to this study.
Footnotes
None of the authors have a conflict of interest.
References
1. AHRQ PBRN Resource Center. PBRN networks listed by acronym. Available at http://pbrn.ahrq.gov/portal/server.pt/community/practice_based_research_networks.
2. Frechtling JA. Logic modeling methods in program evaluation. San Francisco: Jossey-Bass; 2007.
3. Knowlton LW, Phillips CC. The logic model guidebook: Better strategies for great results. Los Angeles: Sage; 2009.
4. McLaughlin JA, Jordan GB. Using logic models. In: Wholey JS, Hatny HP, Newcomer KE, editors. Handbook of practical program evaluation. San Francisco, CA: Jossey-Bass Inc; 2004. pp. 7–32.
5. University of Wisconsin Extension. Program Development and Evaluation. Logic Model. Available at http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html.
6. W.K. Kellogg Foundation. Using logic models to bring together planning, evaluation and action: logic model development guide. 2004.
7. Innovation Network. Logic Model Workbook. Available at http://www.innonet.org/client_docs/File/logic_model_workbook.pdf.
8. Kaplan S, Garrett K. The use of logic models by community based initiatives. Evaluation and Program Planning. 2005;28:167–172.
9. McLaughlin J, Jordan G. Logic models: A tool for testing your program’s performance story. Evaluation and Program Planning. 1999;22:65–72.
10. Taylor-Powell E. Building capacity in evaluating outcomes. Madison, WI: University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation; 2008.
11. Taylor-Powell E, Jones AL, Henert E. Enhancing program performance with logic models. 2009 Available at http://www.uwex.edu/ces/lmcourse.
12. Ribeiro I, Torres A, Parra D, Reis R, Hoehner C, Schmid TL, Pratt M, Ramos L, Simões EJ, Brownson RC. Using logic models as iterative tools for planning and evaluating physical activity promotion programs in Curitiba, Brazil. Journal of Physical Activity and Health. 2010:S155–S162. [PubMed]
13. Stinson S, Wilkinson C. Creating a successful clinical extern program using a program planning logic model. Journal for Nurses in Staff Development. 2004:140–144. [PubMed]
14. Burge SK, Schneider FD, Ivy L, Catala S. Patients’ advice to physicians about intervening in family conflict. Ann Fam Med. 2005;3(3):248–254. [PubMed]
15. Dergance JM, Calmbach WL, Dhanda R, Miles TP, Hazuda HP, Mouton CP. Barriers to and benefits of leisure time physical activity in the elderly: differences across cultures. J Am Geriatr Soc. 2003;51(6):863–868. [PubMed]
16. Katerndahl DA, Trammell C. Prevalence and recognition of panic states in STARNET patients presenting with chest pain. J Fam Pract. 1997;45(1):54–63. [PubMed]
17. Katerndahl DA. Diagnosing panic attacks in chest pain patients. Strategic Medicine. 1998;2(1):12–17.
18. Mouton CP, Calmbach WL, Dhanda R, Espino DV, Hazuda H. Barriers and benefits to leisure-time physical activity among older Mexican Americans. Arch Fam Med. 2000;9(9):892–897. [PubMed]
19. Parchman ML, Burge SK. Continuity and quality of care in type 2 diabetes: A Residency Research Network of South Texas study. J Fam Pract. 2002;51(7):619–624. [PubMed]
20. Parchman ML, Romero RL, Pugh JA. Encounters by patients with type 2 diabetes--complex and demanding: an observational study. Ann Fam Med. 2006;4(1):40–45. [PubMed]
21. Wen L, Parchman ML, Linn WD, Lee S. Association between self-monitoring of blood glucose and glycemic control in patients with type 2 diabetes mellitus. Am J Health Syst Pharm. 2004;61(22):2401–2405. [PubMed]
22. Wen LK, Parchman ML, Shepherd MD. Family support and diet barriers among older Hispanic adults with type 2 diabetes. Fam Med. 2004;36(6):423–430. [PubMed]
23. Parchman ML, Pugh JA, Romero RL, Bowers KW. The case of elevated glycosylated hemoglobin. Ann Fam Med. 2007;5:196–201. [PubMed]
24. Parchman ML, Pugh JA, Wang CP, Romero RL. Glucose control, self-care behaviors and the presence of the chronic care model in primary care clinics. Diabetes Care. 2007;30:2849–2854. [PubMed]
25. Parchman ML, Zeber JE, Romero RL, Pugh JA. Risk of Coronary artery disease in type 2 diabetes and delivery of care consistent with the chronic care model in primary care settings: A STARNet Study. Med Care. 2007;45:1129–34. [PubMed]
26. Kaissi AA, Parchman ML. Assessing chronic illness care for diabetes in primary care clinics. Joint Commission Journal on Quality and Patient Safety. 2006;32(6):318–323. [PubMed]
27. Parchman ML, Pugh JA, Culler SD, Noel PH, Arar NH, Romero RL, Palmer RF. A group randomized trial of a complexity-based organizational intervention to improve risk factors for diabetes complications in primary care settings: study protocol. Implement Sci. 2008;3:15. [PMC free article] [PubMed]
28. Parchman M, Kassi AA. Are elements of the chronic care model associated with cardiovascular risk factor control in type 2 diabetes? Jt Comm J Qual Patient Saf. 2009;35(3):133–8. [PubMed]
29. Parchman ML, Munoz A. Risk factors for methicillin-resistant Staphylococcal aureus skin and soft tissue infections presenting to primary care clinics: A STARNet study. J Am Board Fam Med. 2009;22(4):375–9. [PubMed]
30. Parchman ML, Flannagan D, Ferrer RL, Matamoras M. Communication competence, self-care behaviors and glucose control in patients with type 2 diabetes. Patient Educ Couns. 2009;77:50–59. [PubMed]
31. Kaissi AA, Parchman M. Organizational factors associated with self-management behaviors in diabetes in primary care clinics. Diabetes Educator. 2009;35:843–50. [PubMed]
32. Katerndahl D, Parchman M. Dynamical differences in patient encounters involving uncontrolled diabetes: an orbital decomposition analysis. J Eval Clin Pract. 2010:211–219. [PubMed]
33. Nagykaldi Z, Calmbach W, Dealleaume L, Temte J, Mold J, Ryan JG. Facilitating patient–self management through telephony and web technologies in seasonal influenza. Inform Prim Care. 2010;18:9–16. [PubMed]
34. Bleeker J, Stalman W, van der Horst H. Evaluating primary care research networks: a review of currently available tools. JABFM. 2010:465–475. [PubMed]
35. Clement S, Pickering A, Rowlands G, Thiru K, Candy B, De Lusignan S. Towards a conceptual framework for evaluating primary care research networks. British Journal of General Practice. 2000:651–652. [PMC free article] [PubMed]
36. Fenton E, Harvey J, Sturt J. Evaluating primary care research networks. Health Services Management Research. 2007:162–173. [PubMed]
37. Westfall J, Fagnan LJ, Handley M, Salsberg J, McGinning P, Zittleman L, Macaulay A. Practice-based research is community engagement. JABFM. 2009:423–427. [PubMed]