This study failed to demonstrate a learning advantage provided by readily accessible educational materials on a hospital's information system. Although it seems obvious that ready access to learning resources during times of patient contact would improve students' knowledge base, this apparent advantage was not reflected in their performance on examinations. In spite of investment of a great amount of time and energy on the part of the investigators, this is a negative study.
An immediate concern is the statistical validity of the study; specifically, were there enough subjects and what was the likelihood of a type II error? For sample size analysis, the following assumptions were made: expected effect size, 10 percent; expected standard deviation on mean for both groups combined, 10 percent; significance level (alpha), 0.05; power (beta), 0.80; and proportion of participants in each group, 0.50. Using these figures, the minimum number of subjects needed for the study would be 32 (16 in each group). The actual study had 45 participants, so sample size was adequate. With each group achieving an identical score of 65 percent on the post-rotation examination, the chance of a type II error (accepting the null hypothesis when it is, in fact, false) was small.
Another potential flaw in the study design was the lack of randomization of the subjects. Perhaps the students in the control group who chose the hospital without an information system were brighter or more independent learners. Also, prior computer experience was not measured between the two groups and may have been a source of bias.
Another concern was the reliability and validity of the instrumentation—in this study, the multiple-choice examinations. The two examinations used to measure the students' knowledge of surgery were developed a year prior to the study and tested on a different group of final-year medical students. These students had scores similar to those of the students participating in the study; the few questions that were determined to be too easy or too difficult were discarded to provide a more reliable examination.
Despite these precautions, the multiple-choice examinations may not have been an appropriate method of determining differences in knowledge. Book learning and examination performance do not always correlate with achievement in real-life situations, and the students in the treatment group may have developed into superior clinicians. However, this skill may not have been captured in the examination format used to assess student knowledge.
Perhaps the primary learning environment at the Foothills Hospital, even when supplemented by computer information, was inadequate to enhance learning over that of the control group. For example, it is suggested that educators should consider four critical tasks to enhance learning.13
To be effective, educators must first select situations that will engage the learner in complex, realistic, problem-centered activities that will support the desired knowledge to be acquired and applied. Perhaps the students did not have an opportunity to actually apply the newly acquired information, thus reducing its utility.
Second, educators must provide a scaffold for learners; that is, the educator must know the type and intensity of guidance necessary to help the learner master the knowledge and skills. Perhaps follow-up and challenge to the newly acquired knowledge were inadequate.
Third, as the computer was the main vehicle for the dissemination of this information to the treatment group, the educators needed to recast their roles from content transmitters to facilitators of learning by tracking progress, assessing students, providing appropriate challenges, and encouraging reflection. The last task is to model the appropriate behaviors and provide adequate support for the intellectual growth of the learner. Students should have an opportunity to observe how instructors solve problems and use the information in the appropriate treatment and care of their patients. In summary, the tools for enhancing student learning include discussion, reflection, evaluation, and validation from a medical learning perspective. Any one or all of these steps could have been insufficient at the Foothills Hospital to foster the desired growth.
As a comforting afterthought, we speculate that, although performance on the end-of-rotation examination was the same for the two groups, the online educational materials may have facilitated learning. The Foothills group may have been able to accumulate the same knowledge as the control group with the expenditure of far less time or energy. Students expressed a great deal of enthusiasm during the project, and the screens were left in place after completion of the study. Five years later, the same order sets and educational materials are still used by new groups of medical students.
Most hospital computers have humble roots in financial and accounting systems. Only relatively recently have clinical applications been developed to aid physicians in the care of patients. The next step in the evolution of the hospital information system will be to utilize the system for education of members of the health care team. The appeal of placing instructional materials on the hospital's computer system seems logical—clinicians can hone their craft and educate themselves with minimal effort during the execution of their daily duties.
Three trends in computing are converging. Personal computers and larger systems continue to offer improved performance at less cost, health organizations are constructing huge databases to serve as rich repositories of patient information, and medical education is becoming increasingly digitalized. Medical educators, who have the responsibility for training the next generation of physicians, must recognize the changing role of the hospital information system as it emerges from its historical financial roots and matures into sophisticated clinical applications. The ready availability of such powerful information systems in our hospitals invites the creative educator to exploit their full educational potential. The failure of this study should not deter educators and informaticians from pursuing the inevitable merger of educational software and patient care systems.