Using qualitative feedback from learners as a needs assessment to guide curricular revision and website design is associated with greater learner satisfaction and larger gains in knowledge. Qualitative learner feedback on desired improvements centers mostly on the general quality of the module rather than any specific component of the content or website function. When making specific comments, the greatest number of comments is on the educational content of the curriculum, particularly its factual elements. A small proportion of learners request reorganization of content, typically to include more tables, charts, figures, and images. Those who comment on practice questions typically want more of them. A small number of learners comment on website function, usually to request additional features.
While others have commented that taking bad educational content and putting it online doesn’t improve learner satisfaction, we found that responding to learner feedback on content that is generally rated as good and reorganizing that content (i.e., adding tables, figures, and images) and changing website design/function in response to specific feedback improved learner satisfaction [7
]. What we found demonstrates the value of recurrent needs assessment in curricular revision. Needs assessment should be done when a curriculum is first developed, but since learners might not fully know their needs until exposed to subject matter, and learner needs evolve, recurrent needs assessment should be built into online curricular design [1
]. Over time, our curricular content became more visual (through the addition of tables, figures, and images) and more interactive (through the addition of more questions and greater opportunity for feedback). With these changes, satisfaction among our learners improved. Getting learners to repeatedly access an online curriculum requires that they be satisfied with it [8
]. Our website design, including a strong evaluation component, and our process of using this information as a recurrent needs assessment, is one way to increase the chances of success of an online curriculum.
While placing a curriculum online offers additional tools to enhance content (i.e., interactivity, hyperlinks, audio and video) our results confirm what others have shown: the quality of educational content is what matters most to learners [7
]. The majority of learner comments on the curriculum were related to the overall quality of the modules and their educational content, suggesting that the educational quality of the content was most important to learners. Learner satisfaction is also strongly associated with knowledge gains. In a meta-analysis of studies of predictors of learner satisfaction done before the Internet was widely used, learner satisfaction with a curriculum was determined most by how much learners felt they got out of the educational content (i.e., “teaching effectiveness”) [19
]. In our study, we used the average change in scores on a module (i.e., post test score minus pretest score) as a measure of teaching effectiveness. We found that by responding to learner feedback, the teaching effectiveness of our modules improved. Since teaching effectiveness is such a driver of learner satisfaction, it is not surprising that our results showed similar findings in teaching effectiveness and learner satisfaction. Although we are unable to determine which changes (i.e., website functionality, content organization etc.) contributed to these improvements, it may not matter. The more important finding is that the process of incorporating learner feedback into curricular revision is associated with improved outcomes in learner satisfaction and knowledge outcomes.
It is our opinion that online curricula should be designed to include powerful evaluation tools (including learner satisfaction as described here) to assess the reliability and validity of education outcomes. We have continually leveraged the capabilities of being online to increase the evaluation component of our curriculum. Assessment tools (i.e., pretests and post tests) were expanded, and item discrimination and Cronbach’s alpha calculations were added to these instruments. Group performance measures, subgroup (i.e., training year; training program) performance measures, and individual performance measures (including standard scores) were added or expanded. This evaluation component allowed us not only to perform a continuous needs assessment of our learners, but also to serve the needs of the residency program directors that chose to implement our curriculum. The Accreditation Council of Graduate Medical Education (ACGME) requires that program directors demonstrate the educational outcomes of their curricula, which is provided at a group and individual level by our curriculum [20
]. Use of our curriculum has grown from 24 internal medicine residency programs in 2002 to over 160 programs in 2011. As others have pointed out, tracking evaluation outcomes and learner feedback consumes resources, as does the resultant curricular revision and implementation of website improvements, but these costs are minimized by sharing resources among users [4
]. Developing a strong evaluation component to online curricula allows educators to advance the science of education and answer the call to increase evaluative research on education outcomes and those features that improve them [15
Our study has several limitations. Individual learners changed from year to year, and so changes in learner satisfaction and knowledge outcomes may have been due to learner characteristics (e.g., post-graduate year, program type) rather than the curriculum. Since several changes to curricular content and website design were made in any given year, we were unable to measure the impact of any single change on learner satisfaction or knowledge outcomes. Over the years of study, online education resources expanded greatly, and learners may in general have become more satisfied with online didactics independent of our changes. Also over the years of study, clinician educators who wrote the educational content may have improved their writing skills, contributing to improvement in the educational content of the curriculum.