|Home | About | Journals | Submit | Contact Us | Français|
Many shoulder and elbow abstracts presented at the American Academy of Orthopaedic Surgeons (AAOS) annual meeting are cited in the orthopaedic literature or are used to guide orthopaedic practice, but not all of these abstracts are submitted, survive peer review, or eventually are published. Presuming unpublished works have not been scientifically confirmed, one could question whether it is academically responsible to cite abstracts presented at the AAOS before they are peer-reviewed and published. To partly address this issue we determined the peer-reviewed publication rate for 558 abstracts (233 papers and 325 posters) presented at the shoulder and elbow sessions of the AAOS from 1999 to 2004. In April 2007, we searched the computerized database MEDLINE® and PubMed® for published articles based on these abstracts. We examined the published articles to assess publication rate, time to publication, change in contents, change in authors, and change in conclusions of abstracts. The overall publication rate in peer-reviewed journals was 58% (321 of 558), similar to other orthopaedic meetings and medical disciplines. We believe it is unacceptable to cite shoulder and elbow abstracts submitted to the AAOS because only slightly more than ½ (58%) of them are authenticated scientifically.
Specialty meetings allow new information to be presented quickly, stimulate discussion, and encourage research by young investigators. In some circumstances, patient care may be altered based solely on findings presented via papers or posters at these meetings. The need for accurate abstract information is highlighted by the fact that 50% of chapters in major orthopaedic textbooks contain references to abstracts presented at meetings . However, abstracts do not always contain the information necessary for readers to assess their validity [6, 18, 24]. The peer-review process helps ensure information accepted in the literature is based on data that have been collected and meticulously analyzed. Rates of publication of complete articles after presentation of abstracts at international meetings for various medical and surgical specialties have ranged from 11% to 78% [2–4, 9, 10, 12, 14, 19, 21, 23, 25].
As in other specialty areas, the program selection committee has the difficult task of selecting abstracts for presentation at the AAOS based on limited information (fewer than 200 words). These accepted abstracts then are published in the AAOS proceedings and read by thousands of individuals in orthopaedic surgery and orthopaedic industries, and, they frequently are cited in journals that allow citation of abstracts. Presuming unpublished works have not been confirmed scientifically, however, one could question whether it is academically responsible to cite abstracts presented at the AAOS before they are peer-reviewed and published.
To address the concern regarding publication of abstracts, we had two goals: (1) calculate publication rates and characterize changes between abstract and respective article before they are peer-reviewed and published and (2) identify predictors for eventual peer-reviewed publications by noting similarities among the abstracts that have been published as complete articles across the categories of year published, type of presentation (paper or poster), paper topic, sample size, location of study, number of authors, and methodologic characteristics.
We retrospectively reviewed the programs of the shoulder and elbow sessions (papers and posters) for the AAOS meetings from 1999 to 2004. This allowed between 3 and 8 years for full-text publication of the presented abstracts. We were likely to find most, if not all, subsequent publications given that a previous report identified a mean of 1.5 years’ publication delay from presentation to full-text publication . During the shoulder and elbow sessions of the AAOS annual meetings from 1999 to 2004, 558 abstracts were presented. Of these 558 abstracts, 325 (58%) were posters and 233 (42%) were papers.
All of the abstracts presented at these meetings were searched on the MEDLINE® database, National Library of Medicine® (via Ovid 3.0 Search Software from CD Plus Technologies, Inc, New York, NY), by one of three reviewers (PMD, JAA, KR) in April 2007. The database contains 3188 medical journals and is updated monthly for journals with wide circulation and every 1 to 3 months for journals with smaller circulation. We chose to restrict our study to publications listed in the MedlinePlus® database (Index Medicus) of peer-reviewed journals. This methodology has been described previously and allows a comparison with the rates of publication in other disciplines [5, 7]. Searches were conducted by author with the papers initially searched by one of us (PMD) and the first broad keyword appearing in the title. If not successful, this was followed by a search of the subsequent authors using the same search format. All authors’ names were searched before declaring an abstract unpublished. Publication rates were determined for shoulder and elbow sessions, and for individual presentations (poster or paper). To ensure consistency in the abstraction of data from the abstracts and the final publications, all three reviewers (PMD, JAA, KR) completed data forms on the same 50 studies. Intraclass correlation coefficients with 95% confidence intervals (CIs) provided estimates of interobserver agreement. All discrepancies were resolved by consensus.
In addition to determining publication rates, the published articles were examined by the three reviewers (PMD, JAA, KR). Date of publication was noted. Any changes in the number of authors, order of authors, number of study subjects, or conclusion of the studies also were noted. In addition, the percentage of published abstracts was calculated in conjunction with the frequency with which the title or authorship changed.
To standardize the searching process, only published papers that were similar (ie, similar hypothesis, study design, protocol, number of specimens, and results) to presented posters or papers were included as a match. In addition, articles that contained all of the data presented in the abstract, but also included additional data, were regarded as a match. We excluded publications that provided dissimilar data despite similar techniques or study design. When we identified differences in the title or authors of a final publication, the content of the original abstract compared with that of the final publication provided the basis for a consensus regarding eligibility. These criteria have been used in prior studies [8–10, 13–17, 23, 27]. No effort was made to check for multiple papers stemming from a single abstract; once a suitable match was found, the search for that presentation was concluded. All data were entered into a database with date of publication. Time to publication was rounded to the nearest month. Articles published in journals where only the year was available were given the date of February of the year they were published. Articles published in peer-reviewed journals before the appropriate AAOS annual meeting were given a zero value for time of publication.
Two of the authors (JAA, KR) recorded the following study design issues: study objective and/or hypothesis, study method (randomized trial, observational study, review, case report, survey, or basic science), sample size, statistical methods, primary outcome measure (main outcome of interest), actual study result (numerical value), and measure of precision around the study result (standard deviation and CI). The outcome measures were defined as the actual measures used (ie, mortality, infection, nonunion, etc), and the study result was defined as the actual numerical result for a specific outcome (ie, a percentage or a mean). When abstracting data from randomized trials, we noted the method of randomization, concealment of randomization, blinding, completeness of followup, active followup (direct contact with patients), intention-to-treat analysis, withdrawals, or dropouts. When abstracting data from observational studies, we noted whether there was a statement regarding followup and whether the authors had adjusted for potential confounding variables (either through matching or statistical analysis). Analysis of reviewer agreement in the data abstraction yielded intraclass correlation coefficient = 0.80, 95% CI = 0.74–0.86).
For each presentation format (paper or poster), we calculated the proportion subsequently published in full, and the number of months between the meeting and publication. We used logistic regression to test for differences in the publication rates between the presentation formats (posters and papers). We performed linear regression to determine whether the time to publication was related to an abstract’s presentation format. The year of the meeting was included as a covariate in all regressions to control for potential differences in the selection process between years. Differences between presentation formats were tested using the test for the equality of coefficients. Data analysis then was performed using SPSS® statistical software (Version 10; SPSS Inc, Chicago, IL). The results were recorded as percentages and a 95% CI was applied to each value.
Our search suggested 321 of 558 abstracts were published in peer-reviewed journals yielding a publication rate of 58%.
Small changes in author, cohort, title, or number of participants did not appear to influence the final conclusions of the published articles compared with the abstracts. Conclusions were altered in 4% of the abstracts published as papers. An author was added 44% of the time and an author was deleted 29% of the time from the original AAOS abstract submission to the published article. Lead authors were changed in 19% of the published papers. Changes to the cohort and title occurred in approximately 1/3 of the published articles compared with their respective abstracts. The number of study participants was changed in 89 of 321 papers (28%). The changes were generally small, often involving the omission of one or two patients from the final paper. Presumably, these patients were dropped from the study after closer scrutiny of the data. In most cases, no explanation was given.
The three possible predictors for successful publication of an abstract were location of the study, number of participants, and clarity of objective. Abstracts followed by publication of a complete paper had a greater (p < 0.01) sample size and were more often (p < 0.05) of North American origin than those not published. The abstracts followed by publication of a complete paper had a mean of four authors and were single-center initiatives. Sample sizes in these published abstracts ranged from two to 8141, with a median of 49. Most (84%) of the published studies were performed in North America. A clear objective or hypothesis was identified in 86.0% of the abstracts followed by publication of a paper, whereas statistical methods were described in only 11%. There were also a few other weak correlations among the abstracts obtaining publication. Abstracts that had the highest rate of publication (highest to lowest) included topics on rotator cuff tears, shoulder arthroplasty, and shoulder instability, whereas those with the lowest rate of publication dealt with tumor, pectoralis ruptures, pulmonary embolism, and practice management (Table 1). Of the published papers, 74% were clinical and 26% were experimental. Characteristics that did not affect the rate of publication included year published, type of presentation (paper or poster), and methodology. The rate of publication was similar (p = 0.362) for all 5 years examined (Fig. 1). The breakdown of abstracts presented that became published according to AAOS meeting year yielded 48.4% in 1999 (the lowest rate of publication), 59.6% in 2000, 65.5% in 2001 (the highest rate of publication), 56.5% in 2002, 59.4% in 2003, and 57.5% in 2004 (Fig. 1). The average time between presentation at the meeting and publication was 31 months (± 28 months). Of the 558 total abstracts presented, 1.2% were published on or before presentation, 11% were published by 1 year after presentation, 23.5% by 2 years, and 30% by 3 years. An increase in the delay until publication increased (p < 0.03) the likelihood of an inconsistency. Podium versus poster presentation did not influence likelihood of publication: 66% of the podium presentations were published in peer-reviewed journals whereas 51% of the posters presentations were published. We found a striking lack of difference in methodologic characteristics between the abstracts followed by publication of a complete paper and those not followed by publication of a complete paper. Only 6% of the abstracts followed by publication of a paper pertained to randomized trials.
Many of the shoulder and elbow abstracts presented at the AAOS annual meeting are cited in the orthopaedic literature or are used to guide orthopaedic practice, but not all of these abstracts are submitted, survive peer-review, and eventually are published. First, we calculated publication rates and characterized changes between abstract and respective article to determine if it is academically responsible to cite shoulder and elbow abstracts presented at the AAOS before they are peer-reviewed and published. Our second goal was to identify predictors for subsequent peer-reviewed publications by noting similar characteristics among the abstracts published as complete articles across the categories of year published, type of presentation (paper/poster), paper topic, sample size, location of study, number of authors, and methodologic characteristics.
Our study had several limitations. Articles published in journals not accessed by PubMed® (MEDLINE®) were not identified in the search. Our search results would have failed to identify peer-reviewed publications in journals indexed only in EMBASE or Cochrane. Although this is a clear limitation, our methods allow for comparison to other studies as the majority of published articles pertaining to this topic have used similar methodologies to ours. Studies may have undergone such major changes to study design and authorship between abstract stage and publication stage that they may have been unrecognizable in a PubMed® search. We assumed all peer-reviewed publications resulted from a single abstract; however, single publications may have arisen from multiple abstracts or data from a single abstract may have led to multiple publications. This may have changed the publication rates. It is also possible some abstracts could still be published. However, 90% or more of studies are published within 4 years of the meeting [21–23]. Although some may question the validity of peer review as the gold standard of research quality , scientific peer review remains the standard by which the quality of science is assessed. There is also the danger that the abstract peer-review process is actually leading to underreporting of valid research. One study of abstracts not published at a scientific meeting reported researchers whose abstracts were rejected by a scientific meeting were more likely to believe a journal would reject their full manuscript . Other research has suggested the major reason for nonpublication of research is nonsubmission . Researchers, and principal investigators in particular, have a responsibility to their patients and to their specialty to push their projects to completion.
We believe abstracts presented at the shoulder and elbow sessions of the AAOS should not be cited in the literature as legitimate sources because only 58% of them survive the peer-review publication process. We also found greater sample size and location of the study in North America increased the likelihood of the article being published. The criteria chosen to define a poster or paper as published were liberal, recognizing publication in any peer-reviewed journals. Applying more stringent criteria, such as publication in the most influential orthopaedic and subspecialty journals (ie, impact factor), would have lowered the publication rate substantially. The International Committee of Medical Journal Editors in its “Uniform Requirements for Manuscripts Submitted to Biomedical Journals” advises authors to avoid using abstracts as references [11, 24]. Our study supports this advice as 42% of the abstracts presented in the shoulder and elbow section of the AAOS failed to be published in peer-reviewed indexed journals.
Failure to publish has been attributed to lack of time, low priority, and rejection by peer-reviewed journals. Some of the works presented at national meetings are performed by students, residents, and research fellows who have changing interests or insufficient time, preventing the production of a complete manuscript . We made no attempt to contact the authors of abstracts who failed to publish. However, other studies have examined this point. Riordan  surveyed 88 of 89 authors whose abstracts, presented at two international pediatric meetings, were not followed by publication of a paper. The major reason for nonpublication was failure of the investigators to submit papers to peer-reviewed journals. Weber et al.  conducted a review to identify all abstracts presented at a medical meeting in 1991 not followed by publication of a paper within 5 years. Only 44 (20%) of the 223 respondents had submitted their manuscripts to journals. Manuscript submission was not associated with abstract quality, positive results, or other study characteristics.
Although the presentation of a paper at a national meeting is a well-accepted and timely mode for conveying the conclusions of research, the publication of such data in a peer-reviewed journal represents a more permanent, more accessible, and validated culmination of the scientific process [5, 7, 9, 10]. An orthopaedist who misses the presentation of a paper or poster at the shoulder and elbow section during the AAOS annual meeting has an approximately 60% chance of retrieving that information from a peer-reviewed journal within the next 3 years. However, approximately ½ of the material presented at the annual meetings will never be permanently recorded in the MEDLINE®-indexed scientific literature.
Each author certifies that he or she has no commercial associations (eg, consultancies, stock ownership, equity interest, patent/licensing arrangements, etc) that might pose a conflict of interest in connection with the submitted article.