We report the results in four sections: attendance at the learning modules, satisfaction ratings, overall practice specific impacts, and module-specific impacts.
Attendance at the learning modules by topic area
The percentages shown in Table
indicate that the relative interest in the various topic areas changed across the first three years of the PSP. For both GPs and MOAs, the Advanced Access module was the most highly attended module in the first year, with 49.8% of all GPs and 50.3% of all MOAs choosing to attend this module. When the Adult Mental Health module became available in the fall of 2009, it saw a strong uptake by the GPs, both in terms of actual numbers of attendees as well as the relative percentage. As a consequence, the chronic disease management, and especially the patient self management and group medical visits topic areas, saw a general decrease in attendance. (As noted in the introduction, CDM was combined with the PSM and/or GMV topic areas for a period of time as the PSP evolved. To streamline our findings, we included the responses of participants who completed these combination learning modules in each of the constituent topic areas).
Satisfaction with the learning modules
For the GP respondents, a one-way ANOVA on the total Satisfaction Scale scores indicated a statistically significant effect of type of module, F(4, 913) = 27.907, p<.001, MSE = 84.2, η2 = .109. Post-hoc tests indicated that the average satisfaction rating for the AMH module was statistically higher (M = 93.2, SEM = 0.44) than the average satisfaction ratings for the other four modules which did not differ statistically from each other (means ranged from 86.2 to 87.5, and SEMs ranged from 0.65 to 0.96).
In assessing whether any of the socio-demographic or practice based variables were related to the GPs’ satisfaction ratings, we found statistically significant differences in satisfaction only by geographic region in the province, F(4, 894) = 10.358, p<.001, MSE = 81.3. This result was found across all five module topic areas, and did not interact with the topic (F (15, 894) = 1.232, p = .241). None of the GP characteristics (gender, age group) nor work-related variables (time worked in family practice, type of practice [solo, two-physician, small group or large group practices], full- versus part-time work schedule) were related to the satisfaction ratings (all p-values > .012 and greater than our Bonferroni-corrected cut-off of .006).
Ratings on the individual Satisfaction Scale items were consistently high, although slight variations across the different learning modules were observed. Examples of four key items (of 22 items) are shown in Figure
. Of particular note is the almost unanimous agreement by GP respondents that the facilitators of all learning modules (but especially CDM) were well informed and knowledgeable (over 94% agreement across all modules). Satisfaction varied somewhat across module topics areas, although all modules were rated highly. For example, whether the GPs learned something new that they incorporated into their practice during the action period was lower for those attending Advanced Access (87.7%) compared with those attending the Adult Mental Health (97.0%). Generally, however, responses on most items were similar and consistently high, indicating satisfaction with all modules.
GPs’ Ratings of Overall Satisfaction With the Five PSP Learning Modules.
The satisfaction ratings of the MOA respondents generally paralleled those of the GPs. There were no statistically significant group differences in the MOAs’ ratings by type of learning module or any of the socio-demographic variables. As was the case with the GP respondents, large percentages of the MOA respondents rated their satisfaction with the learning modules highly. Figure
shows the MOAs’ ratings on the same sample of items as were shown for the GPs. The percentages of MOAs who agreed or strongly agreed with these four key items (of 16 items) on the Satisfaction Scale are even higher than those of the GPs. The one notable exception is the item asking whether the respondents learned something new that they incorporated into their own practice: Although still high at 87.1%, the percentage of MOAs in the Adult Mental Health learning module was about 10% lower than for the GPs.
MOAs’ Ratings of Overall Satisfaction With the Five PSP Learning Modules.
Perceived impact on the GPs’ practices
A one-way ANOVA on the GPs’ Perceived Impact scale scores also indicated a statistically significant effect of type of module, F(4, 822) = 61.372, p<.001, MSE = 89.1. Post-hoc tests found that average perceived impact of the Adult Mental Health module was statistically higher (M = 68.4, SEM = 0.49) than that of the other modules.
As we observed with the Satisfaction scale scores for the GPs, Perceived Impact scores varied by geographic region, F(4, 822) = 61.37, p<.001, MSE = 89.12, η2 = .230. In addition, perceived impact also varied slightly but statistically significantly by the GPs’ gender (F(1,813) = 9.26, p=.002, MSE = 88.84), with male GPs rating the perceived impact higher (M = 63.3) than female GPs (M = 61.3). The perceived impact ratings did not differ by GPs’ age group, time in family practice, nature of the practice, nor full- versus part-time work schedules (all p-values >.056).
Five individual items of particular interest (of 19 items on the Perceived Impact Scale) are shown in Figure
. Over 80% of GPs agreed or strongly agreed that attending the module improved their practice. Different levels of agreement of the GPs were observed across the five different module topics. For example, 91.5% of GPs who attended the Adult Mental Health module agreed or strongly agreed that attending the module had improved patient care, compared with 69.1% of GPs who attended the Advanced Access module. In contrast, 86.3% of GPs who attended Advanced Access agreed or strongly agreed that attending had provided them with insights about practice based quality management, compared with 58.6% of GPs who attended Adult Mental Health.
GPs’ Ratings of Perceived Impact of the Five PSP Learning Modules.
The perceived impact ratings by the MOAs are shown in Figure
for a sample of five items (of 8 items) on the Perceived Impact Scale. These items were different from those on the GP survey. But as was observed with the GPs, the different learning modules had differential impacts on different aspects of the MOAs’ work environments. For example, fewer MOAs (44.3%) were asked to take on more responsibilities following their attendance of the Advanced Access module, one of the earliest modules offered, compared with 80.8% of those who attended the Adult Mental Health, with parallel ratings about how they felt about taking on these additional responsibilities. In fact, some 40.3% to 80.8% of MOAs noted that they had been asked to take on more responsibilities, and 67.9% to 86.4% said that they felt good after taking on additional responsibilities (see Figure
). For large percentages of MOAs (86% to 96% across the five modules) attending the learning module was perceived as an overall positive experience. Furthermore, for most MOAs (over 70% overall), attending the module with their GP had improved their relationship with their GP.
MOAs’ Ratings of Perceived Impact of the Five PSP Learning Modules.
Module-specific impacts on GP practices
The evaluation has evolved along with the evolution of the learning modules over time. For the first four modules an end of module survey was used (this practice has continued over time). For the Adult Mental Health Module, a supplement was added to the end of module survey to obtain additional information. The supplement survey was also sent out 3–6 months after the completion of the learning module to access the sustainability of the learnings, and the impact on GP practices over time. For the newly implemented end of life module, a baseline survey was added which is to be administered at the beginning of the first learning session. For future learning modules we shall also be adding a patient experience survey to obtain the patients’ perspectives’.
It was not possible to compare the participant characteristics of module attendees to all GPs or to compare the characteristics of module attendees who did, and did not, complete the surveys, as comparable data were not collected. However, it is quite clear that the satisfaction with, and the impact of, the learning modules were, on average, quite positive across all GPs who attended all modules.
This module offered office management tools to improve office efficiencies. Table
shows the changes in wait times (in days) on urgent, regular and third next available appointments, based on estimates given by all GP respondents. The estimates show that, on average, wait times for all respondents decreased by 1.2 days for urgent appointments (t(142) = 5.18, p<.001), 3.3 days for regular appointments (t(140) = 8.09, p<.001), and by 3.4 days for third next available appointments (t(127) = 36.68, p<.001). Not all GPs, however, reported a decrease in their wait times following their attendance at the learning module. Specifically, for urgent appointments, only 51.0% of GPs indicated a decrease, whereas for regular and third next available appointments, 70.2% and 75.8% of GPs indicated a decrease. For these subsets of GPs, the average decreases were 2.4 days, 4.9 days and 4.6 days for urgent, regular and third next available appointments, respectively (and all p-values < .001). In addition, of all GP respondents: 67.3% reported having reduced their backlog; 66.9% were able to start and end their work days on time; 24.2% were able to take more time off; and 37.1% had increased their panel size as a result of attending this module.
Impact of attending the advanced access learning module on three types of patient appointments
Chronic disease management (CDM)
A large percentage of GP respondents agreed or strongly agreed that having attended this module: had allowed them to take better care of their patients (87.6%); had helped them identify patients with chronic disease (84.2%); prompted the development of a CDM patient registry for their practice (88.6%); and prompted them to actively consider CDM guidelines in care delivery (84.7%). Slightly lower percentages, but still the majority of GPs, agreed that attending the module made patients seem satisfied with, and/or engaged in, their care (65.0%) and increased the GP’s satisfaction with work (61.3%).
Patient self-management (PSM)
Large percentages of GPs agreed or strongly agreed that: they felt comfortable with helping their patients adopt self-managed care (87.6%); they would make PSM an ongoing part of their practice (86.5%); and they were partners with their patients in their patients’ care (92.0%). Moderate levels of agreement were obtained from GPs about their patients seeming to like self management (69.3%), being more involved in self-management (69.7%) and being satisfied with their care (61.8%). Also, 65.2% of GPs agreed or strongly agreed that attending this module had increased their work satisfaction; however, a few GPs (23.6%) noted that offering self-managed care was too time consuming.
Group medical visits (GMV)
Some 73.9% of GPs were comfortable conducting group visits after completing this module. Most GPs agreed that their patients liked the peer learning (78.8%), liked the patient self-management support provided by the group visits (79.8%), and were satisfied with their care (76.2%). Some 77.4% of the GPs agreed that group visits allowed them to use a team-based approach to care, although 45.9% agreed that scheduling them was difficult. Some 60.0% indicated that they planned to make group visits an ongoing part of their practice.
Adult mental health (AMH)
On completing the AMH module, 94.1% of GPs agreed that they felt more comfortable helping patients who required mental health care. Attending the module had enhanced their skills (86.6% of GPs agreed) and their confidence (82.1%) in diagnosing mental health conditions, and their skills (88.8%) and confidence (87.0%) in treating mental health conditions. Attendance increased work satisfaction for 68.2% of GP respondents.
GPs agreed or strongly agreed that attending the module enhanced their skills in: conducting a diagnostic assessment interview (86.8%); offering and coaching the Antidepressant Skills Workbook
] (81.4%); offering and encouraging the use of the Bounce Back Program
] (91.4%); and coaching patients on a variety of cognitive behavioural and interpersonal skills (82.7%). Attending the module had improved their ability to develop appropriate care plans (79.2% of GPs agreed), increased care partnerships of GPs with their patients (91.8%), and increased patients’ engagement in self-management of mental health concerns (60.9%). Furthermore, 41.0% of GPs indicated that they had decreased their frequency of prescribing medications, if appropriate, while only 20.8% indicated they had increased their prescribing.
Furthermore, data from the Supplement Survey indicate that 60.2% of the GPs reported high or very high success in implementing the newly learned tools and skills into their practices, and 73.2% reported they will continue to use them. Some 22.9% of the GPs reported a high degree of MOA involvement, with an additional 40.0% indicating a moderate involvement. The impact of the AMH module on their patients was rated as positive or very positive by 94.5% of GP respondents, with the remaining GPs indicating “no impact”. Specifically with regard to helping their patients remain at work or return to work following cognitive behavioural interventions (as compared to without): 89.2% of GPs reported their patients were better or much better able to continue to work; and 78.4% of GPs reported their patients were better or much better able to return to work (the remaining GPs reported “no change”).
Findings from the 3–6 month follow-up supplement survey indicated that the new learning was sustained. Comparisons of self-ratings of confidence, at the completion of the module and at follow-up, are shown in Table
. Confidence remained at similar levels or increased slightly over time on many dimensions. Of particular note is item (m) that asks for their overall assessment of their confidence in providing quality mental health care to their patients: At the end of their training, the average rating was 1.71 (on a four-point scale where 1 was very confident, 2 was somewhat confident, 3 was not very confident, and 4 was not at all confident); this dropped (indicating an increase in the confidence rating) to an average confidence rating of 1.46. In addition, almost all responding GPs (99.1%) indicated that they were confident in their ability to provide quality mental health care to their patients. (A caveat to this finding is that this result may be, at least in part, due to a selection bias, as it is possible that primarily those GPs who were successful in using the tools and skills responded on the 3–6 month follow-up survey.)
Long-term maintenance of GPs’ confidence in providing mental health care