Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Inf Serv Use. Author manuscript; available in PMC 2010 December 14.
Published in final edited form as:
Inf Serv Use. 2010 January 1; 30(1-2): 17–30.
doi:  10.3233/ISU-2010-0608
PMCID: PMC3001620

Defining the next generation journal: The NLM–Elsevier interactive publications experiment*



A unique collaborative project to identify interactive enhancements to conventional-print journal articles, and to evaluate their contribution to readers’ learning and satisfaction.


It was hypothesized that (a) the enhanced article would yield more knowledge acquisition than the original article; (b) the interactivity aspects of the enhanced article would measurably contribute to the acquisition of knowledge; and (c) the enhancements to the original article would increase reader acceptance.


Fifteen SNMA medical students, assumed to have a greater generational familiarity and comfort level with interactive electronic media, reviewed 12 articles published in three Elsevier clinical and basic science journals. They used the Student National Medical Association’s asynchronous online discussion forum over a four month period to suggest desired enhancements to improve learning. “Prognostic Factors in Stage T1 Bladder Cancer”, published in the journal Urology was selected by the investigators as presenting the best opportunity to incorporate many of the students’ suggested interactive and presentational enhancements in the limited timeframe available prior to the established test date. Educational, statistical, and medical consultants assisted in designing a test protocol in which 51 second to fourth year medical students were randomly assigned to experimental and control conditions, and were administered either the original or enhanced interactive version of the article on individual computer workstations. Test subjects consisted of 23 participants in the control group (8 males, 15 females) and 28 participants in the experimental group (9 males, 19 females). All subjects completed pre- and post-test instruments which measured their knowledge gain on 30 true-false and multiple-choice questions, along with 7 Likert-type questions measuring acceptance of the articles’ format. Time to completion was recorded with the experimental group taking 22 min on average compared to 18 min for the controls; pre- and post-test times were 6 and 7 min, respectively. Statistical comparisons were based on change scores using either the Student t-test or the Two Way Analysis of Variance or Covariance. Significance was set at α = 0.05 or better.


on the dependent measure of knowledge acquisition showed no difference overall on the 30 questions, but learning gain was statistically significant for the subset of 10 questions that measured gain on content that was accessible by the user-invoked interactive features of the enhanced article. Further analyses revealed significant interactions by student year and gender. Second year students (11 in the control group, 8 in the experimental group) were the best performers in terms of knowledge acquisition from both articles. The female medical students received a larger learning gain from journal enhancements and interactivity components than their male counterparts. Acceptance overall was greater for the experimental group who rated the experience more favorably than the controls.


Failure to consider human factors such as gender and learning style may obscure underlying differences and their impact on the interactive aspects of scientific publications. Preliminary findings suggest the need for further study to include a heavier focus on interactivity apart from presentational enhancements; a more rigorous treatment of time as a specific variable; and an expanded experimental design that evaluates acquisition, understanding, integration and acceptance as dependent measures.

Keywords: Electronic publishing, multimedia, graphics, computer, learning, information theory, medical students

1. Introduction

It is estimated that interactive publications containing, for example, data sets and images that are manipulatable by the reader, account for only approximately 2% of the published basic science and clinical journal articles indexed by the US National Library of Medicine (NLM) in 2007 [7]. It is expected that this number will grow dramatically in the years ahead. Some place this number considerably higher now, with the Journal of Clinical Investigation estimating upwards of 87% of published research articles in 2009, if all forms of ‘supplemental’ journal materials are included [10]. In any case, this will entail new technical challenges for NLM and other libraries in providing effective storage and accessibility, while improving the potential for journal article readers to experience and benefit from enhanced learning and understanding.

NLM is presently pursuing three complementary initiatives that provide platforms and tools for experimenting with interactive technologies and assessing its impact on users. They differ in strategy by initially emphasizing, variously, the perspective of the end-user as studied in the present preliminary experiment carried out in collaboration with Elsevier and the Student National Medical Association (SNMA); the author/editor/reviewer as pursued in a separate collaborative experiment with the Optical Society of America, and reported elsewhere in this special issue [1]; and [the perspective of the technologist/engineer. In this latter initiative, an in-house research group at NLM developed several dynamic interactive publications containing a range of medical images, video, Flash anatomy, large tables and graphs [11]. A visualization and analysis tool called Panorama was developed and tested; and was judged a semi-finalist in the Elsevier Grand Challenge contest conducted in 2008–2009.

The utility of interactive multimedia tools for scientific knowledge acquisition has not been fully realized, nor has the overhead been fully documented and the remedies identified. For example, publishers are concerned with the need to innovate and improve the way scientific information is communicated and used. They also recognize that in an age of online publishing with a rapid expansion of supplemental materials, including those that are interactive, there may be drawbacks beyond the additional monetary costs involved. For authors, they may feel compelled to include large amounts of data that exceed traditional restrictions of the printed article. Similarly, reviewers may feel responsible for evaluating the equivalent of two papers, devoting the same attention they would to the main body of the article. Readers may find it difficult to navigate through large supplements, and question the quality of the review process [9].

Communication amongst all stakeholders is essential. To that end, the International Council for Scientific and Technical Information (ICSTI) brought together publishers, library representatives, technologists and students of the scientific communication process to explore the emerging forms of interactive publications, and to speculate on the future evolution of the scholarly journal in the web era [8]. Going forward, there exists also a need for a consensus on standardized bibliographic and publishing policies for improved management, access, and discoverability of interactive and other forms of supplemental materials. An initial step in that direction has been taken by at a roundtable organized by the National Information Standards Organization (NISO) and the National Federation of Advanced Information Services (NFAIS), and attended by journal publishers, scholarly organizations and libraries [10].

User-centered approaches are commonly accepted to be the best way to create more effective learning tools, and take advantage of the increasingly utilized human–computer interaction [2]. The research objective of the present experiment was to determine whether users of an interactive journal program as operationally defined and implemented by Elsevier have better comprehension of the content embodied in a scientific journal article than readers of the conventional-print version. A randomized, time-controlled, pre- and post-test experiment was conducted to make this determination.

Medical students were selected as the users/subjects in the belief that as ‘digital natives’ this age cohort would have greater comfort and familiarity than older readers with electronic interactive technologies and, therefore, would be in the best position to suggest desirable interactive enhancements to a conventional-print journal article from the users’ perspective, and also serve as experimental test subjects comparing original with enhanced versions.

The project was conducted in two phases. The first phase solicited recommendations from student members of the SNMA regarding a candidate pool of basic science and clinical journal articles published by Elsevier, from which a test article(s) would be selected. The students subsequently reviewed the articles and provided informed judgments regarding the addition of desirable interactive enhancements to each. Elsevier’s e-Journals department developed an interactive test platform that enabled a second group of SNMA students to experience the enhancements that were made to the article that was ultimately selected by Elsevier for comparative testing in the second phase.

2. Methods: Phase 1

Fifteen students were selected by the SNMA governing board from amongst medical schools throughout the US, with a primary focus on identifying individuals who are matriculating in joint degree programs and who have an interest in research. Each received an honorarium of $1,000 for their participation in the study. The SNMA’s MedConnect online forum was used to support asynchronous communication between the students during the four month period of December 2007–March 2008. Elsevier nominated three journals whose editors expressed a willingness to participate in an interactive journal program. The journals were Urology, the Journal of Oral and Maxillofacial Surgery and the Annals of Emergency Medicine. The students identified 12 articles from these three journals as candidates for experimentation, and each article was posted to MedConnect for review and comment in the forum discussions.

Suggested enhancements

The students were asked to keep the following top-level questions in mind when suggesting interactive enhancements: What information are you looking for in this article? What are the important points you need to know after reading this article? How could the article help you reach that information more quickly? How could the article help you better understand that information?

The students were also asked to provide specific suggestions in three potential areas for developing the interactive components: Supporting Data such as tables and graphs that could provide a more complete picture, or enable manipulation of mathematical/statistical models to view and run further data analysis; Visual Illustrations allowing further manipulation and exploration (zooming, rotation, color variance, time lapse) of images and videos to improve understanding; and Other Learning Aides consisting of materials not typically part of a traditional journal article (e.g., software simulations, audio content, author notes) and that could enhance understanding of the subject. A total of 175 postings were made by the students addressing 32 topics. For example, this is a commentary offered by one student on the suggestions of another “… I agree that an actual ultrasound would be a great addition to this article … color-coded with voice over explaining what was going on … and a chart in the results section would make this material easier to understand”.

Elsevier processed the students’ suggestions and used the following criteria to select four candidate articles for development in the limited time available prior to the pre-scheduled Phase 2 experimental test date: Degree and variety of interactivity possible; degree of difference between the interactive and plain text versions of the article; and the Length of the article. Table 1 provides a general overview conceived by Elsevier of how the test article(s) could be made interactive.

Table 1
Interactive/Instructional design enhancements

The selected article

Elsevier ultimately selected as the test article “Prognostic Factors in State T1 Bladder Cancer: Tumor Pattern (Solid or Papillary) and Vascular Invasion More Important Than Depth of Invasion”. See Fig. 1 for the title page as published in Urology, October 2007 (Vol. 69, Issue 4, pp. 758–762).

Fig. 1
Conventionally printed article.

See Fig. 2 for the enhanced version that incorporated Presentation Improvements such as use of multiple colors, artistic backgrounds, textual reorganization, sidebars, bullets and informative summary statements. While these enhancements would be expected to influence the acquisition of the knowledge embodied in the article, they are not “interactive” per se. Interactivity is operationally defined here as a “purposeful physical action by the user which causes a directional change or some meaningful response by the system with regard to the article”. In essence, the interactive features would require the user to actively intervene in the process, while the presentation improvements would allow the user to more effectively engage the material, passively.

Fig. 2
Experimental version.

3. Methods: Phase 2

Experimental design

A randomized, controlled experiment was designed in which a cohort of medical students read a scientific article presented on a computer screen in either the original, conventionally printed format – control group, or an enhanced format that included interactivity – experimental group. All subjects completed pre- and post-test questionnaire instruments which measured knowledge gain and acceptance of the method used.

The research questions were: (a) Do the subjects who read the enhanced article acquire more knowledge than those who read the original article? (b) Does interactivity measurably contribute to the acquisition of knowledge about the subject of bladder cancer? (c) Do the presentation and interactivity enhancements to the original article increase reader acceptance?


A knowledge test was developed consisting of 30 true-false and multiple-choice items. In order to determine the value of adding interactivity to the original article, it was necessary to isolate the interactive aspects from the presentation improvements. To do this, two types of questions were included in the pre-test and post-test. They were identical except that the items were randomized in the post-test. Type 1 questions were 20 test items on information provided in the original article. For example:

Regarding Stage T1 bladder cancer, which of the following are not associated with increased risk of progression and death? (a) Transurethral resection; (b) vascular invasion; (c) large tumor size; (d) deep invasion of the lamina propria; (e) I’m not sure.

Type 2 questions were 10 test items on information accessible through links in the enhanced article. For example:

The lamina propria: (a) Is found under simple columnar cells only; (b) may be found under simple cubordal cells; (c) occurs under simple squamous cells only; (d) is under the basement membrane; (e) I’m not sure.


The subjects in this experiment were second, third or fourth year medical students who were attending the SNMA regional conference at the Johns Hopkins University Schoold of Medicine, Baltimore, MD, USA in June 2008. Subjects were recruited through a letter mailed from the SNMA prior to the conference, notifying them of the experiment and inviting them to participate in the study. An honorarium of $50 was offered for their participation. Sixty (60) students appeared for the meeting; 51 students participated in the experiment. Participants were randomly assigned to each, 23 to the control group and 28 to the experimental group.

Test site

A room in the University of Maryland at Baltimore School of Medicine’s resource center containing 20 Dell desktop computers was used for the experiment. Eight computer systems on one side of the room were used for the control group. Each computer was loaded with the original scientific article in PDF format, along with the evaluation software. The 12 systems on the other side of the room were reserved for the treatment group that was presented with the enhanced version of the article. Each was loaded with the evaluation software and a link to the enhanced article on the Elsevier website containing the test platform.

Experimental sessions

There were four scheduled test sessions, two in the morning and two in the afternoon. The test subjects assembled at the meeting site and were escorted as a group to the test site by an SNMA representative. The experiment was designed for random and equal division of the subjects into control and treatment groups. Near the end of the day, the unexpected arrival of several unscheduled students resulted in an uneven division of subjects.

All subjects completed the pre-test, finished the article in his/her own time frame, completed the post-test, and provided comments in writing on a handout sheet. Upon completing the session, each participant received the honorarium.

The control group averaged 18 min to finish the article; the average for the experimental group was 22 min. Approximately 6 min were required to complete the pre-test and 7 min for the post-test. The entire session lasted approximately 50 min.

Statistical comparisons were based on change scores using either the Student t-test or the Two Way Analysis of Variance or Covariance.

4. Results

Do the subjects who read the enhanced article acquire more knowledge than those who read the original article?

Pre- and post-test scores for the total content (30 questions) were calculated and plotted for both groups. The difference in the average number of points gained by the two groups on the pre- and post-test was 7.8 ± 4.0% (p = 0.0564), which was very close and not quite significant at the α = 0.05 level. The pre- and post-test scores for all subjects are presented in Figs 3 and and44.

Fig. 3
Control group (N = 23).
Fig. 4
Experimental group (N = 28).

Post-test scores were analyzed separately to address the first research question of whether the treatment group acquired more knowledge than the control group. The data plotted in Fig. 5 clearly shows the added learning benefit and the calculated size of the effect of the presentation/interactivity enhancements in the article.

Fig. 5
Post scores – all sessions.

The effect size based on the post-test scores was 1.35, which is considered to be very large. (Note: the effect size is the mean difference divided by the pooled standard deviation of the data.) According to the Cohen Standard, 0.20 = “small”, 0.50 = “medium” and 0.80 = “large” [4].

Does interactivity measurably contribute to the acquisition of knowledge about the subject of bladder cancer?

Learning gain for each subject was computed by dividing the difference between the pre-and post-test score by the pre-test score and converting that number to a percentage. The average learning gain for the control group was 33.9% (±2.7%); for the experimental group, it was 48.9% (±2.7%). The effect size based on average learning gain was 1.04, which means that the distribution of the experimental group moved a full standard deviation in the positive direction from the control group. The Fig. 6 depicts the separation of the groups.

Fig. 6
Learning gain – all questions.

Interactivity and knowledge acquisition

As previously described, the 30-question pre-/post-test covered content from two sources: 20 questions pertained to content in the original article and 10 questions pertained to content from external sources available through links in the enhanced article. The 20 questions about original article content were used to measure the effect of the presentation improvements made to the original paper. The 10 questions about external source content were used to measure the effect of the active links in the enhanced paper, which defined interactivity in this experiment.

Presentation effects

The 20-question portion of the exam was designed to measure differences in learning gain that might occur as a result of the presentation improvements made to the original article. These enhancements included aesthetic improvements, reorganization of text, and summarization tables that were automatically available to the reader. The researchers expected that these presentation improvements of the original would result in better acquisition of knowledge by the experimental group. However, this was not the case. Calculations showed that the experimental group scored higher than the control group, but not significantly. The statistical result regarding the effect of the various presentation enhancements to the original paper was 4.1 ± 4.9% (p = 0.4006).

Interactivity effects

The links to external material available in the enhanced article were used as the measure of interactivity in this experiment. The 10-question portion of the exam pertained to material that was available only to the experimental group through interactive links. This treatment was not available to the control group. The researchers expected that the experimental group would take advantage of this capability and, as a result, learn the content available through the links. This proved to be the case. The difference in learning gain between the two groups with respect to the content in the enhanced article through active links was 15.0 ± 4.9% (p = 0.0035).

While there was not a significant learning gain from reading the enhanced article, the experimental group experienced a statistically significant learning benefit from interacting with the available links. A subsequent analysis involving covariates showed this difference to be statistically significant at the α = 0.05 level.

Differences by gender and class year

The ANCOVAs by group and gender showed significant interactions between group and gender for the responses to questions on the content covered in the paper and for the overall results. The females showed positive increases while the males showed negative differences. It is also interesting to note that the females in the control group, who viewed the conventional form of the publication, improved their pre-test score by 38%, while the males in that group improved by 57%. The females in the experimental group seemed to perform quite well with the enhanced article, while the males in the experimental group scored lower with the enhanced article than their control group colleagues, who read the original article. Independent of gender, second year medical students (11 in the control group, 8 in the experimental group), produced the highest scores on the exam.

Do the presentation and interactivity enhancements to the original article increase reader acceptance?

Table 2 displays the 5-point Likert-scale evaluations used to measure the “acceptance” of the experience. A rating of “3” on the questions would indicate a neutral position. Consequently, any scale score average above three indicates a positive response; below three, a negative response. The experimental group rated each item more favorably than did the control group.

Table 2
Evaluation of the experience by group

5. Summary and conclusions

The enhanced version of the article proved to be superior to the original article. Specifically, the enhancements to the original article involving presentation improvements and interactive aspects, made the material more desirable for readers and more effective with regard to acquisition of the information. The experimental group, which read and interacted with the enhanced article, showed significantly higher learning gains than the control group that read the original article. The interactivity aspects available in the enhanced version seemed to be primarily responsible for the increased knowledge acquisition demonstrated by the experimental group, as evidenced by the large effect size and the learning gain for the experimental group.

The students’ acceptance of the two modalities was measured with seven Likert-type scales. The data revealed that the experimental group felt significantly more positive about their treatment experience than the control group felt about their experience. The enhanced article was more appealing as a modality for acquiring this scientific information than the conventional article. Our findings are similar to other published reports demonstrating a statistically significant gain from the use of interactive multimedia enhancements in other educational arenas [12].

Second year medical students produced the highest scores on the exam in both the control and experimental groups. This surprise finding might be due to their having been in medicine for a shorter period of time and being more compliant with the instructions of authorities. It is interesting to note that male students performed better with the conventional form of the scientific article than with the enhanced version. The female medical students received a larger learning gain from journal enhancements and interactivity components than their male counterparts. Taken together, these results suggest possible learning differences specific to gender that could be exploited using the interactive format. Ford and colleagues demonstrated individual variations in learning effect attributable to gender when multimedia tools were used to enhance scientific knowledge acquisition [5,6]. Further studies are needed to elucidate this question using our format.

It is useful to note that user characteristics, including gender and level of education, may interact significantly with treatment effects. If evaluated in isolation, the failure to consider such human factors may obscure underlying differences and their impact on the interactive aspects of scientific publications. It may also prove valuable to examine the influence of learning style differences. Recent studies in neuroscience applied to the classroom setting give reason to believe that how we are ‘wired’ may enable some individuals to benefit more from interactive technologies than others [3].

Clearly, this was not a definitive study regarding the value of an interactive journal article. However, there were many interesting characteristics embodied in the data which have not been reported elsewhere in this context, and beg further exploration. Therefore, the researchers recommend that follow-up study include the following ideas:

  1. The experimental material should have a heavy focus on interactivity in order to further explore the finding in this experiment that interactivity was the most significant aspect of knowledge acquisition. Three types of interactivity could be incorporated in the experimental article: (a) interactive links that allow the user to drill down on the specific content of the article, such as active interaction with tabular data; (b) interactive links that allow the user to access material outside the article, such as those used in the current experiment; and (c) interactivity that personalizes the article and connects the user with the author.
  2. The experimental design should include a more rigorous treatment of time as a specific variable being studied. The current experiment set 30 min as the treatment period, but few if any of the students studied the material that long. Consequently, there is little known about time other than the amount of time each group studied their material. There is reason to believe that the self-direction gained through active links and other types of interactivity might require more time and might interfere with a reader’s continuity and focus. A study designed as time-dependent would answer some of these questions.
  3. The evaluation aspects of the experimental design should include acquisition, understanding, integration and acceptance measures. This more elaborate design would require the active involvement of the author and other experts who are intimately familiar with the content.


We wish to thank the following individuals for their contributions to this project. Study Planning: Donald West King, MD; Sheldon Kotzin, MLS; Michael Ackerman, PhD; George Thoma, PhD, all at the National Library of Medicine, National Institutes of Health, Bethesda, Maryland. Phase 1 Article Critiques: Wallace Davenport II, PhD, Student National Medical Association, Washington, DC. Phase 1 Article Prototyping: Diane Bartoli, e-Journals, Elsevier, New York. Phase 2 Experimental Testing: Marcia A. Zier, MA; Glenn Harless; John Connors, all at Interactive Drama, Inc., Maryland. Experimental Design and Statistics: Robert C. Duncan, PhD, University of Miami Miller School of Medicine; William R. Ayers, MD, Georgetown University School of Medicine. Test Site Arrangements: M.J. Tooey, MLS, Health Sciences and Human Services Library, University of Maryland at Baltimore. Manuscript Reviews: David Nichols, MD; Nancy Roderer, MLS, Welch Medical Library, both at the Johns Hopkins University School of Medicine; Fred Wood, DBA, National Library of Medicine.


*Paper presented at the ICSTI Interactive Publications Conference 2010.


1. Ackerman MJ, Siegel E, Wood F. Interactive Science Publishing: A joint OSA–NLM project. Information Service & Use. 2010;30(1, 2):39–50. (this issue)
2. Adamczyk C, Holzer M, Putz R, Fischer M. Student learning preferences and the impact of a multimedia learning tool in the dissection course at the University of Munich. Annals of Anatomy. 2009;191:339–348. [PubMed]
3. Christensen C, Horn M, Johnson C. Disrupting Class. McGraw-Hill; New York: 2008.
4. Cohen J. Statistical Power for the Behavioral Sciences. Erlbaum; Mahwah, NJ: 1988.
5. Ford N, Chen SY. Individual differences, hypermedia navigation, and learning: an empirical study. Journal of Educational Multimedia and Hypermedia. 2000;9:281–311.
6. Ford N, Chen SY. Matching/mismatching revisited: an empirical study of learning and teaching styles. British Journal of Education. 2001;32:5–22.
7. Marcetich J. Interactive Media: Implications for Content and Preservation, Professional Scholarly Publishing and NLM Pre-Conference. Association of American Publishers; Washington, DC: Feb, 2007. Presentation of NLM data on interactive journals.
8. McMahon B. Interactive publications and the record of science. Proceedings of the Winter Workshop of the International Council for Scientific and Technical Information; Paris. February 8, 2010; available at:
9. Marcus E. Taming supplemental material, editorial. Cell. 2009;130(1):11. [PubMed]
10. Roundtable on Best Practices for Supplemental Journal Article Materials, co-sponsored by the National Federation of Advanced Information Services and the National Information Standards Organization, Washington, DC, January 22, 2010, available at:
11. Thoma G, Ford G, Chung M, Vasudevan K, Antani S. Proceedings SPIE Electronic Imaging. Vol. 6076. Digital Publishing; 2006. Interactive publications: creation and usage; pp. 607603-1–607603-8. available at:
12. Wong G, Greenhalgh T, Pawson R. Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Medical Education. 2010 Feb 2;10(12) Review. [PMC free article] [PubMed]