|Home | About | Journals | Submit | Contact Us | Français|
The research evaluated participant satisfaction with the content and format of the “Web 2.0 101: Introduction to Second Generation Web Tools” course and measured the impact of the course on participants' self-evaluated knowledge of Web 2.0 tools.
The “Web 2.0 101” online course was based loosely on the Learning 2.0 model. Content was provided through a course blog and covered a wide range of Web 2.0 tools. All Medical Library Association members were invited to participate. Participants were asked to complete a post-course survey. Respondents who completed the entire course or who completed part of the course self-evaluated their knowledge of nine social software tools and concepts prior to and after the course using a Likert scale. Additional qualitative information about course strengths and weaknesses was also gathered.
Respondents' self-ratings showed a significant change in perceived knowledge for each tool, using a matched pair Wilcoxon signed rank analysis (P<0.0001 for each tool/concept). Overall satisfaction with the course appeared high. Hands-on exercises were the most frequently identified strength of the course; the length and time-consuming nature of the course were considered weaknesses by some.
Learning 2.0-style courses, though demanding time and self-motivation from participants, can increase knowledge of Web 2.0 tools.
The Medical Library Association's (MLA's) Task Force on Social Networking Software was created by President Mark E. Funk, AHIP, in May 2007 during MLA '07, MLA's annual meeting. The task force was charged with investigating issues relating to MLA's implementation of blogs, wikis, really simple syndication (RSS) feeds, and other social networking tools in order to accomplish President Funk's top presidential priority, upgrading the association's use of technology [1,2]. The immediate goal was to improve communication and facilitate networking; the long-term goal was to evaluate individual social networking tools and make recommendations that could be used by association members, sections, committees, and task forces.
The task force quickly organized and as a first step sought to gauge the extent that social networking tools were being incorporated into medical librarians' daily work with an open survey issued to all members in July–August 2007. The survey received 495 responses. There was a strong indication that social networking technologies were important to MLA members, but only to a certain extent. According to the task force report, “many social networking technologies may not yet have a practical purpose for professional activities … technology for its own sake is not inherently useful, especially in professional lives that are already busy and full” . This response was not a surprise to the task force. With any new technology, there are early adopters who immediately seek to integrate new tools into their daily work, while others move at a slower pace and need to see the value before committing time and resources. Members of MLA were looking to the association for information and guidance.
In conjunction with the 2008 National Program Committee's Geek Squad, the task force developed an eight-week, online course, “Web 2.0 101: Introduction to Second Generation Web Tools,” which was offered March 10–April 27, 2008 . This time frame was chosen to introduce and familiarize MLA members with Web 2.0 technologies prior to MLA '08's Plenary IV session, “Web 2.0 Tools for Librarians: Description, Demonstration, Discussion, and Debate” . The course was conceptually based on the Learning 2.0 program started at the Public Library of Charlotte and Mecklenburg County, “a self-discovery program which encourages staff to take control of their own learning and to utilize their lifelong learning skills through exploration and play” . In other words, MLA members would become totally immersed in Web 2.0 technologies by simultaneously learning about and using the tools. The course was open to all members of MLA at no cost.
In addition to preparing members for the annual meeting, the task force hoped that by improving members' knowledge of social software tools, members would translate that knowledge into practical use to benefit both MLA and the members' own workplaces and careers. Other Learning 2.0 programs have produced such an effect. For example, Rethlefsen and Farrell reported that Mayo Clinic Libraries staff began using blogs, wikis, LibraryThing, Facebook, and Delicious to communicate with each other and patrons in the few months immediately following such a course . Similarly, Perry and Scott found that University of California, Santa Cruz, staff implemented blogs, wikis, instant messaging, a shared online calendar, and more after their Learning 2.0 course , and Larsen indicated that 75% of library supervisors in the Multnomah County Library had either implemented or were planning to implement use of Web 2.0 tools with their staff  after a similar course experience. Gross and Leslie noted that after the 2.0 program at the Edith Cowan University, RSS feeds and tagging were integrated into the library online public access catalog (OPAC) . Also, participants have been shown to continue personal use of Web 2.0 tools: Rethlefsen and Farrell found that 79% of program completers were using 1 or more tools immediately after their course and that 4 months post-course, participants were continuing to use many tools, some even daily .
This paper examines research questions related specifically to the “Web 2.0 101: Introduction to Second Generation Web Tools” course: Did the course increase the knowledge of participants? What were participants' opinions of the course? What were the strengths of the course? What could improve the course? The paper also discusses implications of creating a Learning 2.0 environment for an association.
Although much has been related regarding the experience of creating and administering a Learning 2.0 program [11–13], systematic assessments of Learning 2.0 programs, in general, are rare. The typical measure of success of any given program is its completion rate , and evaluations tend to be based on qualitative feedback or focus groups and are often used to suggest best practices for those building or administering their own programs [8,10,14]. Some exceptions do exist. Larsen administered a brief survey to get feedback from supervisors about planned use of Web 2.0 tools after Learning 2.0 program participation by their staff , and Sjoblom  administered a short survey for pilot group participants, both in public library settings. Rethlefsen and Farrell reported that participants felt significantly more knowledgeable about Web 2.0 tools at the end of their program . This outcomes-based form of assessment—whether or not desired learning goals have been reached, on average, for a cohort—is an important proof of viability for others interested in pursuing a Learning 2.0–type program but is generally missing from the literature.
Eight modules were developed by two or more volunteer experts from the task force, and each team constructed their content independently. Content creators pulled from a range of previously published content from other Creative Commons–licensed Learning 2.0 programs as well as created unique content based on their knowledge of medical libraries and the technologies. Instructors used a WetPaint wiki to collaboratively create and edit course content prior to submitting it to the MLA Continuing Education Committee to qualify as a course with continuing education (CE) credit. All MLA members were invited to participate in the MLA “Web 2.0 101” CE course through announcements in MLA FOCUS and on the Task Force on Social Networking Software blog. The course was provided online over an eight-week period using a WordPress blog hosted on MLANET .
Each week introduced a new module in a blog post describing a Web 2.0 technology and giving participants a hands-on assignment using the technology. In addition, each week, course instructors posted a discussion question for the group on the course blog. In addition to the course blog and associated RSS feeds, course instructors used an email discussion list for announcements (i.e., when a new module was available). Help was available through a Meebo chat widget , a weekly blog post where participants could pose questions, and direct contact with course instructors.
The course covered blogs and RSS (week one), wikis (week two), social networking tools (week three), social bookmarking (week four), web office tools (week five), online photo sharing (week six), podcasting and online hosted video (week seven), and mashups and application programming interfaces (APIs) (week eight). Table 1 provides details about the content in each module. The team of course instructors who developed module content also provided assistance to students during that week via the blog, email, and the Meebo chat widget. Though modules were released each week and participants were encouraged to complete them according to the time frame to enhance group communication, participants had the flexibility to work on their assignments as time was available to them, as long as all modules were completed by the final cutoff date. Each module was designed to take one to two hours, though it was expected that this would vary depending on participants' levels of familiarity and skill with web technology. Participants tracked their progress using a progress form created in Zoomerang .
After the program finished, all participants, including those who did not complete all eight modules, were asked to complete an evaluation of the course. The survey was created in Zoomerang and promoted through the course blog and via email . Receiving CE credit did not depend on completing the evaluation. Depending on level of course completion, the survey used branching to provide different questions. For those completing the course, the survey used a posttest/retrospective pretest design to assess knowledge of the technologies covered in the course.
When evaluating programs using a traditional pretest/posttest design that asks participants to self-report their knowledge, pretest self-ratings may be overly inflated. People tend to overestimate their knowledge, not realizing what they do not know, which introduces a confounding factor into results, response-shift bias. Response-shift bias occurs when a change takes place in participants' frame of reference toward their knowledge, based on the effect of the instructional program. By using a retrospective pretest, participants can reflect on what their knowledge was prior to the program, based on their new frame of reference. The posttest/retrospective pretest design can thereby help eliminate response-shift bias [19–21]. The evaluation was based on the survey used by Rethlefsen and Farrell to assess an internal Learning 2.0 program at the Mayo Clinic Libraries .
The primary measure, self-reported knowledge, was evaluated using a 5-point Likert scale. The scale asked participants to rate their knowledge of each of 9 tools individually, using the scale: 1=no knowledge; 2=aware of, but never tried; 3=some experience or knowledge; 4=familiar with, but not an expert; and 5=expert-level knowledge. Participants were asked to give the course an overall grade (A–D). Other questions assessed participants' opinions of the course's strengths and weaknesses using a combination of Likert scales, open-ended questions, multiple-choice questions, and check boxes. Participants' library type was also assessed for demographic purposes.
The quantitative survey results were analyzed using JMP 7 . The posttest/retrospective pretest questions assessing knowledge were examined using the Wilcoxon signed rank test. The chi square test was used to assess significant differences in overall course satisfaction, course completion, and survey completion by library type. Each qualitative question's full set of responses were evaluated and coded for common themes by a single reviewer using Weft QDA, an open source software tool designed for qualitative textual analysis . As each set of responses was analyzed, a continually evolving codebook was developed for each set of responses indentifying themes and subthemes. After coding each set, coding validity was assured by reviewing the text for miscoding and errors. Coded sections were sorted by theme and counted for occurrence.
Six hundred seventy-one MLA members registered for the course, and 359 completed it by the deadline, for a 53.5% completion rate. Three hundred seventy-four participants completed the survey, including 309 who completed the entire course, a 55.7% response rate overall and an 86.1% response rate among course completers. Of the remaining respondents, 36 (9.6%) completed only part of the class and 29 (7.8%) registered for the class but never participated. There was no a significant difference in course completion rate (P>0.07) or survey completion rate (P>0.13) between participants from academic, hospital, or other library settings (Table 2).
Survey respondents who completed the entire course or who completed part of the course (n=345) self-evaluated their knowledge of 9 social software tools and concepts prior to and after the course, using a Likert scale ranging from 1 (no knowledge) to 5 (expert-level knowledge). Perceived knowledge was lowest for web office tools (mean=2.23) and mashups (mean=1.67) prior to the course and highest for blogs (mean=3.13) and online photo sharing (mean=2.89). After the course, perceived knowledge increased for each tool; the greatest mean changes were for mashups (mean change=1.57), web office tools (mean change=1.55), and social bookmarking (mean change=1.54) (Table 3). These respondents' self-ratings showed a significant change in perceived knowledge for each of the 9 tools, using a matched pair Wilcoxon signed rank analysis (P<0.0001 for each tool/concept). The authors were able to reject the null hypothesis that the program would have no effect on knowledge. The respondents' self-ratings reflected the increase in knowledge shown by the posttest/retrospective pretest: 304 (88.1%) respondents named information gained as a course strength, and 332 (96.2%) agreed or somewhat agreed that the course provided information or skills they can use.
Because the online, blog-based course format was new for the association, participants were asked for their opinion of the various technologies used for communicating with course participants. The most favorably reviewed technology was use of the course blog to give assignments: 173 respondents (50.1%) thought this tool was excellent. Very few respondents rated any of the technologies as poor or not very good, though 2 modes of communication, getting assistance via course blog comments and using the Meebo chat widget, were not evaluated by many participants, either because they were neutral to or did not use the tool. Two hundred forty-two (70.1%) respondents either did not use or had no opinion of the Meebo chat widget, and 103 (29.9%) either did not use or had no opinion of using the course blog comments for assistance (Table 4).
To elicit more specific feedback, participants were given the opportunity to comment on the parts of the class they felt most helpful and least helpful. Participants gave a wide range of responses to both of these questions. For the most helpful part of the class, responses grouped into a few general themes: course format, course content, specific topics covered by the course, and the instructors. Individual themes and supporting quotations are listed in Tables 5 and 6 (online only). The most commonly identified helpful part of the course was the hands-on exercises (n=79), followed by general exposure to Web 2.0 technologies (n=56). Blogs (n=33), wikis (n=29), and social bookmarking (n=28) were the technologies most frequently singled out as helpful.
Though fewer respondents chose to single out the part of the course that was least helpful to them, the most frequently mentioned were the mashup (n=37) and social networking (n=37) portions of the course. One of the most persistent themes both of the question asking for the least helpful aspect of the course and of the general comments, which also queried the non-completer group, was that the course took too much time or that there was not enough time to complete the assignments. Many respondents also commented on problems related to the course organization and layout, course instructors, course content, course logistics and format, and behavior of other students. Tables 7 and 8 (online) give a more comprehensive list of identified themes and supporting quotes.
Overall satisfaction with the course appeared high: 64.7% (n=218) respondents gave the course an A grade; an additional 31.5% (n=106) gave the course a B grade. There was not a significant difference in course grades between participants from academic, hospital, or other library settings (P>0.2695).
As initially developed, Learning 2.0 programs require a substantial amount of learner self-motivation and drive, particularly for those students who are new to Web 2.0 technologies or those who may be uncomfortable with technology. The time and commitment needed for completing the entirety of what is generally a multi-week endeavor is usually cited as the cause for the low completion rates reported in the literature [7–9]. Reported completion rates have varied from 30%–64% [7,8,12,15], similar to the 53.5% completion rate reported here.
Other reports of Learning 2.0 programs have evaluated single-location or single-library-system courses [7–10, 12–15]. “Web 2.0 101,” however, included participants from academic, hospital, and other library settings, where staff sizes range from solo librarians to dozens of employees. Though it has been reported that MLA members from hospital libraries and smaller libraries were less inclined to use social software tools like blogs or believe they were important to the association [24,25], interestingly, there were no significant differences between participants from different library types in course completion rates, survey completion rates, or overall grades for the course. In fact, although total lack of or restricted access to social networking tools at work was a common refrain early in the course and has been widely discussed as a reason for hospital librarians' less frequent use of social software tools , only four survey respondents who did not complete the course cited workplace restrictions as their reason for not completing the course. Highly motivated participants also worked from home or public libraries when their workplace blocked access to course tools. Because of the small number of non-completers responding to the survey, it is likely an understatement of the impact workplace technology restrictions have on library staff, particularly in hospital and corporate environments.
Because “Web 2.0 101” was offered to the entire MLA membership and not hosted in a single library or library system, most participants did not have an opportunity to work together in person, nor were there any in-person jumpstart sessions or courses. All other reported Learning 2.0 courses have included an in-person component, and nearly all have reported that in-person sessions—whether formal training, drop-in sessions, or casual workgroup get-togethers—have increased completion rates and the sense of community [8–10]. As Larsen stated, “When [staff] take traditional training classes, simply having a scheduled class meeting time once or twice a week helps to keep people on track. Without the requirement for physical presence, online learning becomes too easy to put off” . Other Learning 2.0 evaluations have pointed to the strong sense of community built by the course [7,12]: both Larsen and Rethlefsen and Farrell reported that higher rates of success or completion were found in workgroups who scheduled time to work on assignments in groups [7,9]. Although the absence of in-person meetings was a concern for “Web 2.0 101,” it appears that the online-only course format was successful for MLA members, though perhaps “Web 2.0 101” did not foster as great a sense of community and networking as in other reported Learning 2.0 programs. Less than a third of survey respondents listed networking as a course strength, and in fact there were far more complaints about fellow participants than there were positive comments about working with a community.
The great benefit of the Learning 2.0 course design is that it uses the very tools it teaches, both for presenting the course and for hands-on assignments for the students . For instance, because the course required students to blog about every module, participants had to become very familiar with the processes and tools used in blogging, which likely accounted for blogs having the highest level of mean perceived knowledge post-course. It was the only tool, in fact, to have a mean rating above 4, indicating participants were nearing expert-level knowledge. Though the tools used to teach the course were deployed with varying amounts of success—Meebo, for example, was used by only a handful of students—overall, the participants' reactions to the course design and structure was overwhelmingly positive. The hands-on activities and self-pacing were particularly well received, making the course a great success. In the comments section of the evaluation, many asked MLA to offer more online courses and to repeat the “Web 2.0 101” course. Many also gave suggestions for creating shorter or advanced courses on specific Web 2.0 topics. The Task Force on Social Networking Software created their online short course series, “Dig Deeper with Social Media,” in response to these requests .
The significant increase in perceived knowledge (P<0.0001 for each topic) shown for “Web 2.0 101” course participants resembles the significant change in knowledge shown by Mayo Clinic Libraries staff . Interestingly, the highest increases in mean self-rated knowledge did not correspond with perceived utility: Mashups, for example, showed the greatest mean increase in perceived knowledge but also were the most frequently mentioned “least helpful” module. The increase in knowledge, while a positive finding simply in terms of program evaluation, also has implications for the long-term effects of the course. With such a widespread group of participants, it is difficult to ascertain how much the “Web 2.0 101” program inspired participants to make changes in technology use personally or professionally. The survey did show that nearly all participants felt the course provided tools and skills they could or would use. As discussed above, evaluations of other Learning 2.0 programs have shown that once a Learning 2.0 program is run at a library or library system, staff begin thinking about how to incorporate these technologies into their work.
“Web 2.0 101” benefited the association. Because the course was offered free to MLA members and granted 8 CE credits upon completion, several librarians joined MLA simply to participate in the class. MLA headquarters staff believe the association gained between 50 and 100 new members and renewals from the course . Additionally, the course also prepared many MLA members for MLA '08, themed “Connections: Bridging the Gaps” and largely centered on Web 2.0 applications and their use. The course blog remains online for anyone to read and work through on one's own. Because the content is Creative Commons–licensed, it may be adapted for use elsewhere and, indeed, has already been so. The class has been replicated by the Tucker Medical Library for the staff of the National Jewish Medical and Research Center.
This study has several limitations. The main limitation stems from the voluntary nature of the evaluation survey. Though a very high percentage of participants who completed the course also completed the survey (86.1%), very few individuals who had registered for the course, but did not finish, responded. The small number of such respondents is not enough to give a clear picture of reasons for non-completion, very likely grossly underestimating the impact of blocked technologies on hospital librarians, for example. Furthermore, more criticism of the course might have emerged had more individuals who dropped out responded to the evaluation survey. The design of the course, in particular the fact that different instructors created modules, might have contributed to lack of cohesiveness of content and complexity. The quantitative analysis did not show an effect—all modules produced a significant increase in perceived knowledge—but the qualitative analysis did show some dissatisfaction with uneven difficulty and instruction.
The evaluation questions did not allow for distinguishing between finding the instruction about each tool or the actual tools most or least helpful, making it difficult to determine whether a module should be merely rewritten or completely removed if taught in the future. A survey edited to distinguish these differences would be beneficial for evaluations of other online courses. Using merely this program evaluation, it is also difficult to assess the long-term impact of the course on members' workplaces and on the association. Future studies might be warranted to uncover any long-term effects.
Learning 2.0–style courses, though demanding time and self-motivation from participants, can increase knowledge of Web 2.0 tools. “Web 2.0 101: Introduction to Second Generation Web Tools” provided an online, hands-on, and free learning experience for MLA members, a first for the association. This course was positively received by participants and increased their perceived knowledge. The Task Force on Social Networking Software, members of which created and managed the course, has taken the lessons learned from “Web 2.0 101” and created new, online, hands-on, topical courses designed to give participants an advanced look at several of the Web 2.0 tools. “The Dig Deeper with Social Media” series has been directly based on feedback from the “Web 2.0 101” course evaluations. For instance, its short course format allows members to pick and choose topics of interest, instead of being forced to complete a whole series of modules before CE credits are granted. The courses are also spread apart to give participants more breathing room between new concepts and tasks, while they still continue the popular hands-on format . This type of online course could help the association keep members, gain new members, and help those members who cannot attend the annual meeting or other MLA CE courses.
The authors gratefully acknowledge the past and present members of and liaisons to the Task Force on Social Networking Software—Bart Ragon, Sue Ben-Dor, AHIP, Melissa DeSantis, AHIP, Mark E. Funk, AHIP, Marie Kennedy, Maureen (Molly) Knapp, Michelle Kraft, AHIP, Rikke Ogawa, AHIP, Gabriel Rios, and James Shedlock, AHIP, FMLA—and most particularly Kate Corcoran, without whom this article could not have been written, for her help assembling any and all data we needed.