Subsequent to initiating our work on a logic model, Bleeker (35
) and colleagues from the Netherlands identified only two existing PBRN evaluation tools. These tools were developed by Clement (36
) and Fenton (37
) to evaluate the overall effectiveness of PBRNs. Clement (36
) proposed a conceptual framework to evaluate primary care PBRNs based on seven primary objectives with specific process and outcome indicators. The objectives could be categorized into network infrastructure, activity and dissemination efforts. Based on our review of the evaluation framework proposed by Clement (36
), it appeared to be a very usable and feasible tool for implementation. However, Bleeker (35
) questioned the validity of these indicators and the feasibility of Clement’s framework for conducting an overall evaluation.
) and colleagues developed the second identified evaluation tool, a Primary Care Research Network Toolkit, which includes a contextualized case study of five networks in the United Kingdom. This toolkit described eight primary dimensions of networks – each one with associated sub-dimensions. Networks could score themselves over time and even conduct comparisons across networks. Although the Primary Care Research Network Toolkit may be a useful in conducting formal evaluations, it lacked sufficient information regarding resources and time needed to successfully replicate the process in the United States.
Considering the relative limited resources of PBRNs, it is not surprising that a majority of PBRNs have not conducted a thorough evaluation of their efforts. Although evaluating a network takes time and requires the involvement of various individuals throughout the process, outcome evaluation efforts are a worthwhile investment. Unfortunately we realized early on, that our budget would not allow us to complete all of the activities outlined in the logic model. It became important to prioritize activities within the logic model due to budget constraints. The logic model should be modified regularly based on the changing capacity and resources of the network. It is yet to be proven whether our logic model framework will meet the planning and evaluation needs of STARNet.
In addition, logic models can be a tremendous tool in determining what is working well and what is not. The Board of Directors continually reminded the staff that all of the activities need to be centered on the mission -- improving patient care. As a result, all of the activities – planned and not planned - are viewed critically from that perspective. It is important to note, however, that every activity cannot be linked directly to long-term outcomes. Based on the logic model framework, the Co-Directors turned away investigators wanting to initiate projects in the network that did not meet the current priorities of the members. This was one of the first times in the history of the Network that it appropriately said “no” to an incompatible research interest. The logic model, in essence, united and empowered the efforts of members in advancing the STARNet mission.
Finally, the logic model reminded the PBRN team that a balance has to be maintained between the hard/traditional measures such as number of studies and publications and the more subjective measures such as easy access to PBRN member offices by PBRN coordinators and researchers. In addition, the core tenet of successful PBRNs is developing and maintaining respectful and trusting long-term relationships that continue beyond research studies (38). The complexity of the relationships and communication within a network is difficult to capture in evaluation efforts. The logic model helped us realize that it’s not just about the quantitative outcomes. In order to share a comprehensive story of STARNet, we also began to collect qualitative data (e.g. rich stories from the members). The logic model helped us realize that in the future, we need to collect this data more systematically from members and patients following the completion of research studies.