PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of mjiriLink to Publisher's site
 
Med J Islam Repub Iran. 2017; 31: 18.
Published online 2017 March 18. doi:  10.18869/mjiri.31.18
PMCID: PMC5609332

Assessing abstracts of Iranian systematic reviews and metaanalysis indexed in WOS and Scopus using PRISMA

Abstract

Background: Systematic reviews and meta-analysis have significant advantages over conventional reviews in that all available data should be presented. This study aimed to evaluate Iranian systematic reviews and meta-analysis abstracts indexed in WOS and Scopus during 2003-2012 based on PRISMA checklist.

Methods: This is an analytical study. We evaluated 46 article abstracts indexed in WOS, 89 article abstracts indexed in Scopus and 158 article abstracts indexed in WOS and Scopus both (overlapped group). The quality of the abstracts was evaluated according to the PRISMA checklist for abstracts. Some indicators including distribution per year, total citation, average citations per year, average citations per documents and average citations per year in each article were determined through searching the WOS and Scopus Databases’ analytical section. Then, the correlations between the abstract's PRISMA scores, average citations per year, and publication year were calculated.

Results: The abstract’s quality is not desirable as far as the PRISMA criteria are concerned. In other words, none of the articles’ abstracts is in line with the PRISMA items. The average of scores of the current study was 5.9 while the maximum score was 12. The PRISMA criteria showed the highest compliance with "Objectives" (98.6%), the second highest with "Synthesis of result" (85%) and "Title" (80.2%) and the lowest compliance with "Registration" (2%). There was a positive correlation between the compliance of PRISMA score and the average citations per year while there was a negative correlation between PRISMA score and the publication year.

Conclusion: It seems that the suggested criteria for reporting Iranian systematic reviews and meta-analysis are not considered adequately by the writers and even scientific journal editors.

Keywords: Systematic Review, Meta-analysis, Iran, Article Abstracts, PRISMA, WOS, Scopus

↑ What is “already known” in this topic:

Systematic reviews and specifically meta-analyses are important in that they can bring a synthesis of knowledge in one article while the information originally appeared in several articles. Iranian researchers are increasingly developing such reviews.

→ What this article adds:

As the quantity of Iranian meta-analyses is increasing during the past 10 years, their quality is decreasing each year. This was apparent from the received number of citations.

Introduction

The past few years have witnessed a large increase in the use of systematic reviews in both medicine and the social sciences. A systematic review attempts to compare all empirical eligible evidence to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimize the bias. Key characteristics of a systematic review are:

• A clearly stated set of objectives with pre-defined eligibility criteria for academic studies;

• An explicit, reproducible methodology;

• A systematic search that attempts to identify all studies that would meet the eligibility criteria;

• An assessment of the validity of the findings of the included studies, for example, through the assessment of the risk of bias; and a systematic presentation, and synthesis of the characteristics and findings of the included studies (1).

Thus, a systematic review attempts to collate all relevant evidence that fit pre-specified eligibility criteria to answer a specific research question. It uses explicit, systematic methods to minimize bias in the identification, selection, synthesis, and summary of studies.

When done well, this provides reliable findings from which conclusions can be drawn and decisions made (2-4). The method is intended to identify the studies related to a common subject that aims to get unbiased knowledge in a comprehensive, systematic and replicable review of the scientific literature (5). Indeed, a systematic review is the reference standard for synthesizing evidence in health care and attempts to collate all relevant evidence that fit pre-specified eligibility criteria to answer a specific research question (4).

One of the subsets of a systematic review is a meta-analysis. As aptly defined by many authors, meta-analysis is a quantitative approach for systematically assessing the result of previous research in order to arrive at conclusions about the body of research (6-9) or it is the use of statistical techniques to combine and summarize the results of multiple studies; they may present a systematic review. By combining data from several studies, meta-analyses can provide more precise estimates of the effects of health care than those derived from the individual studies (4).

PRISMA is an evidence-based minimum set of items for reporting systematic reviews and meta-analyses. PRISMA not only focuses on the reporting of reviews evaluating randomized trials but also can be used as a basis for reporting systematic reviews of other types of research, particularly evaluations of interventions. This guideline prepares 27 preferred reporting items for systematic reviews and meta-analyses items (4). One item in this checklist describes the main criteria for evaluating abstract as one of the most important parts of each article. A well-written abstract provides a description of a clinical problem or research question; the methods used to address it and the significant results and implications (10). In other words, the main function of an abstract of a systematic review should be to signal its systematic methodology. For most readers, the findings described in the abstract will also be of special significance, either as the sole part of an article that will be read and will determine whether to read the full text. Abstracts of systematic reviews are very important, as some readers cannot access the full paper, then abstracts may be the only option for gleaning research results (11).

Nowadays, we see a significant growth in the systematic reviews and meta-analysis studies published by researchers. If systematic reviews and meta-analysis articles do not structurally comply with internationally accepted protocols, they cannot effectively meet the objectives of these studies. Thus, in this study first we intended to describe the Iranian systematic reviews and meta-analysis productions indexed in WOS and Scopus and to evaluate their abstracts, based on the PRISMA checklist. Then, some scientometric indicators including distribution per year, total citation, average citations per year, average citations per documents and average citations per year in one article will be identified by searching in the WOS and Scopus Databases. In the end, the correlation between the abstract's PRISMA Scores, average citations per year, and publication year was calculated.

Methods

This is an analytical study. We did some “advanced search” in WOS and “document search” in Scopus during 2003-2012. The search strategies in WOS were as follows:

Title=("meta-analysis") OR Title=("meta analysis") OR Title=("meta_analysis") OR Title=("metaanalysis") OR Topic=("meta-analysis") OR Topic=("meta-analysis") OR Topic=("meta_analysis") OR Topic=("metaanalysis") OR Title=("systematic review") OR Title=("systematic reviews") OR Topic=("systematic review") OR Topic =("systematic reviews") Refined by: Countries /Territories=(Iran) Timespan=2003-2012. Databases =SCI-EXPANDED.

The search strategies in Scopus was:

(TITLE-ABS-KEY(systematic review) OR TITLE-ABS-KEY (systematic reviews) OR TITLE-ABS-KEY(meta-analysis) OR TITLE-ABS-KEY(meta-anlysis) OR TITLE-ABS-KEY(meta_anlysis) AND AFFILCOUNTRY(Iran)) AND SUBJAREA (mult OR agri OR bioc OR immu OR neur OR phar OR mult OR medi OR nurs OR vete OR dent OR heal) AND PUBYEAR >2002 AND PUBYEAR <2013 AND (LIMIT-TO (EXACTKEYWORD, "Systematic review") OR LIMIT-TO (EXACT KEYWORD, "meta-analysis")) AND PUBYEAR >2002 AND PUBYEAR <2013

Total number of retrieved records were 677 (Scopus (261) and WOS (416)) of which 158 were overlapped; So, with the overlaps dropped, 519 remained. Then, the 158 overlapped articles, which had been indexed in both the WOS & Scopus were also studied and labeled as “overlapped group”. Furthermore, there were some irrelevant articles indexed in WOS and Scopus. They included different issues like letters, editorials, congresses, incorrect affiliations, inaccessible abstracts, wrong research methodologies, and wrong titles. As such, after eliminating the irrelevant records 293 remained.

We evaluated 46 article abstracts indexed in WOS, 89 indexed in Scopus and 158 indexed in WOS and Scopus both (overlapped group). Quality of the abstracts were evaluated according to the PRISMA checklist for abstracts as its bibliographic information comes in the 6th references.

According to Figure 1 , we assigned one score for each item defined in Figure 2. Thus, the maximum score for article abstracts evaluated by PRISMA checklist is 12 and the minimum is zero.

The average of received citations per year, the number of authors, the publication year and the organization affiliation were determined through searching in WOS and Scopus Databases analytical section. These indicators in “overlapped group” were obtained from Scopus database because this database covers more articles than ISI.

ANOVA and Duncan tests were used to compare the databases in point of the difference between PRISMA score, and article's average of received citation and the Pearson correlation coefficient was used to calculate the correlation between the abstract's PRISMA score with the average of received citations per year, the number of authors and the publication year.

Results

The results indicated that the number of Iranian systematic review and meta-analysis articles had an increasing rate during 2003 to 2012 (Table 1).

Table 1
Distribution of Iranian systematic review and meta-analysis articles publication year indexed in WOS, Scopus and Overlapped group

According to Figure 2, there was 1 article published in 2003 while this rate was 105 in 2012.

Figure 3 shows that the most of compliance with PRISMA criteria is observed in “Objectives” (98.6%) then “Synthesis of result” (85%) and “Title” (80.2%). The less compliance with PRISMA is observed in “Registration” (2%), ”Strengths & limitation of evidence” (3%), “Eligibility criteria” (3.4%) and “Risk of bias” (4.4%), respectively.

Table 2 shows that the average score of Iranian systematic review and meta-analysis articles indexed in WOS and Scopus is 5.9, while the highest score could be 12. The highest compliance with PRISMA criteria is observed in 2003 while the lowest was in 2004. None of the abstracts in this study obtained the complete score which is 12.

Table 2
Average score of PRISMA for the article abstracts indexed in WOS, Scopus and overlapped group during 2003-2012

Table 3 represents that the average rate of total citations in Iranian systematic review and meta-analysis articles publication year indexed in WOS during 2003 to 2012 was 16 (STD=24.67). The average rate of citation/year was 2 in 2008 in the minimum rate while it was 31.7 in 2007 in the maximum rate. The average rate of citation /documents was 3.4 in 2012 in the minimum rate while it was 46.7 in 2006 in the maximum rate. The average citation/year in one article was 3.4 (STD=3.39). Since there was no publication during 2003 - 2005, we were unable to determine other indicators.

Table 3
Average citation per year and per document of Iranian systematic reviews and meta-analysis articles in WOS

Table 4 represents that the average rate of total citation in Iranian systematic review and meta-analysis articles publication year indexed in Scopus during 2003 to 2012 was 4.6 (STD=10.93). The average rate of citation/year was 0.1 in 2004 in the minimum rate while it was 40.5 in 2012 at the maximum rate. The average rate of Citation /documents was 1 in 2004 in the minimum rate while it was 17.7 in 2008 at the maximum rate. The average citation/ year in one article was 1.2 (STD=2.29). Since there was no publication during 2003 - 2005, we were unable to determine other indicators.

Table 4
Average citation per year and per document of Iranian systematic reviews and meta-analysis articles in Scopus

Table 5 represents that the average rate of total citation in Iranian systematic review and meta-analysis articles publication year indexed in overlapped group during 2003 to 2012 was 19.9 (25.78 STD). The average rate of citation/year was 1.5 in 2003 in the minimum rate while it was 157.8 in 2009 at the maximum rate. The average rate of Citation /documents was 5 in 2012 in the minimum rate while it was 116 in 2004 at the maximum rate. The average citation/year in one article was 4.3 (STD=4.17). Since there was no publication during 2003 - 2005, we were unable to determine other indicators.

Table 5
The Status of Iranian systematic review and meta-analysis articles publication year indexed in Overlapped group*

Table 6 represents that the average rate of total citation in Iranian systematic review and meta-analysis articles publication year indexed in all databases during 2003 to 2012 was 14.6 (STD=23.04). The average rate of citation /year was 1.5 in 2003 in the minimum rate while it was 213.4 in 2009 at the maximum rate. The average rate of citation /documents was 3.8 in 2012 in the minimum rate while it was 58.5 in 2004 at the maximum rate. The average citation / year in one article was 3.2 (STD=3.90).

Table 6
Average citation per year and per document of Iranian systematic reviews and meta-analysis articles in WOS, Scopus & Overlapped group.

Table 7 shows that if we consider the results of Tables 3--7,7, there is a positive correlation between PRISMA score and average citation/year (p<0.001, r=0.309). It means that the more PRISMA score rate, the more citation/ year.

Table 7
Pearson correlation coefficient test result for determining the relation between PRISMA score and average citation / year

Table 8 shows that if we consider the results of Table 3, (As we mentioned above, the average score of Iranian systematic review and meta-analysis articles indexed in WOS and Scopus is 5.9, while the highest score could be 12. The highest compliance with PRISMA criteria is observed in 2003 while the less of them is in 2004. None of the abstracts in this study could obtain the complete score which is 12), there is a negative correlation between PRISMA score and publication year (P=0.021, r=-0.135).

Table 8
Pearson correlation coefficient test result for determining the relation between PRISMA score and publication year

It means that from 2003 to 2012, the quality of abstracts according to PRISMA checklist is decreased.

Discussion

Nowadays, evidence-based studies have become an important concept in public health and community health promotion. In most evidence paradigms, systematic reviews, along with meta-analyses, are most highly valued (12). As the importance of systematic review conclusions relies upon the scientific rigor of methods and the currency of evidence, we aimed to evaluate Iranian systematic reviews and meta-analysis abstracts and clarify the real situation of articles including their strengths and weaknesses in an attempt to offer some recommendation to enhance the quality of the articles.

According to the results, PRISMA checklist complied differently with various sections of the abstracts as follows: “Registration”(2%), “Strengths & limitation of evidence”(3%), “Eligibility criteria” (3.4%), “Risk of bias” (4.4%), “Included study” (5.8%), “Funding” (25.3%) , “Information sources” (40.6%), “Description of the effect” (44%); on the other hand, the PRISMA showed the highest compliance with “Objectives” (98.6%) , “Synthesis of result” (85%) , “Title” (80.2%) and “Interpretation” (70%), respectively. Ideal systematic review and meta-analysis should be conducted comprehensively, by maximizing the precision level, minimizing the bias and narrating the concepts clearly. In other words, out of 12 main items of the PRISMA checklist considered for systematic reviews & meta-analysis abstracts evaluation; only three items (25%) were reported higher than 50%. These results are consistent with the findings of Le et. Al (13), Xiao et al. (14) and Willis et al. (15) results who reported low quality for a set of articles published in China, according to PRISMA checklist. They reported that “compliance with the PRISMA statement was generally poor: none of the reviews completely adhered to all checklist items”. This may be the result of lack of awareness of authors of forms and standard checklists. Similar problems have been indicated in other related studies as well (e.g. Hróbjartsson, 2009). According to Hróbjartsson (2009) research authorities are required to consider to the development of international guidelines for the reporting of trial protocols and public access to the protocols (16). Added to this is the important role that can be accomplished by the journal editors. They can consider a specific framework, based on accepted criteria (e.g PRISMA), for the reviewing process. This framework may then lead to the publication of scientifically more qualified articles. As Soori (2002) said, most scientific journals in Iran and many other countries worldwide do not have proper specific instruction for writing and reviewing epidemiologic articles (17).

This study shows that the quantity of Iranian meta-analysis and systematic reviews’ articles were increased during 2003-2012 while their citations have decreased. Greater attention should be paid to the quality of the articles rather than just the number of publications. According to Valayi et al. there is a sharp increase in the number of peer-reviewed articles in Iran in recent years. The quality of the articles, however, was not satisfactory (18).

Our results showed that there is a positive correlation between PRISMA score and average citation per year, which means greater PRISMA score, can result in the more citations/years. So, the low quality of Iranian systematic review and meta-analysis articles could have a direct impact on general aspect of rating based on citation levels. This result, as a predictable outcome, is approved by Panic et al. who found that the endorsement of PRISMA can improve both the methodology of the study and the way it is reported (19-21). Therefore, it is better to pay attention to “reporting guidelines” which have been developed to improve the quality of the manuscript as well.

Conclusion

Compliance with the PRISMA statement in abstracts of Iranian systematic reviews and meta-analysis was poor. None of the reviews completely observed all the PRISMA checklist items. It can be concluded that the more compliance with the criteria, the more citations to systematic reviews and meta-analysis studies. Thus, by paying more attention to the accepted publication criteria, the authors can obtain more citation impact.

Acknowledgements

This study is based on the a thesis, entitled " Assessment of Iranian systematic reviews and meta-analysis indexed in Web of Sciences & Scopus with PRISMA checklist and determine the correlation between the obtained results with scientometric indicators" which was presented to the Department of Medical Library and Information Science, Paramedical School, Shahid Beheshti University of Medical Sciences.

Conflict of Interests: None declared.

Notes

Cite this article as: M Kazerani, Davoudian A, Zayeri F, Soori H. Assessing abstracts of Iranian systematic reviews and meta-analysis indexed in WOS and Scopus using PRISMA. Med J Islam Repub Iran. 2017 (8 Mar);31:18. https://doi.org/10.18869/mjiri.31.18

References

1. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org.
2. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC:. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts: treatments for myocardial infarction. JAMA. 1992;268(2):240–248. [PubMed]
3. Oxman AD, Guyatt GH: The science of reviewing research. Ann N Y Acad Sci 1993; 703:125,33. discussion 133–134. [PubMed]
4. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M. et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4:1. [PMC free article] [PubMed]
5. Silva VM, Júnior JIS, Prata DN, Letouze P. The importance of systematic review as a scientific research method for computer science: a quantitative systematic review. Int Proc Econ Dev Res. 2014;78:44.
6. Pettiti DB.Meta-Analysis, Decision analysis and cost effectivenessin analysis:methods for quantitative synthesis in medicine ,2nd ed.Newyork:Oxford university press; 2000; 320 p.
7. Haidich AB. Meta-analysis in medical research. Hippokratia. 2010 Dec;14(Suppl 1):29–37. [PMC free article] [PubMed]
8. Moher D, Altman DG. Four proposals to help improve the medical research literature. PLoS Med. 2015 Sep 22;12(9):e1001864. [PMC free article] [PubMed]
9. Moher D. Moher DOptimal strategies to consider when peer reviewing a systematic review and meta-analysisBMC medicine. 2015 Nov. 2;13(1):274. [PMC free article] [PubMed]
10. Bingham R, O’Neal D. Developing Great Abstracts and Posters. Nurs Womens Health. 2013 Apr 1;17(2):131–8. [PubMed]
11. Beller E, Glasziou PP, Altman DG, Hopewell S, Bastian H, Chalmers I. et al. PRISMA for abstracts: Reporting systematic reviews in journal and conference abstracts. PLoS Med. 2013;10(4):e1001419. [PMC free article] [PubMed]
12. Anderson MJ. A systematic evaluation of search methods, search reporting, and selection methods of reviews of physical activity interventions to prevent obesity [Internet] [M.P.H.]. [United States -- Texas]: The University of Texas School of Public Health; 2009 [cited 2016 May 10]. Available from: http://search.proquest.com.proxy.iknito.com/docview/305004728/abstract/87AAF73F79FD4104PQ/7.
13. Li L, Tian J, Tian H, Sun R, Liu Y, Yang K. Quality and transparency of overviews of systematic reviews. J Evid-Based Med. 2012 Aug 1;5(3):166–73. [PubMed]
14. Xiao Z, Zhang Y, Wang Y, Xu F. Quality Assessment for Systematic Review /Meta-Analysis on Antidepressant Therapy Published in Chinese Journals. Int J Pharmacol. 2012 Jul 1;8(7):614–20.
15. Willis BH, Quigley M. The assessment of the quality of reporting of meta-analyses in diagnostic research: a systematic review. BMC Med Res Methodol. 2011;11:163. [PMC free article] [PubMed]
16. Hróbjartsson A, Pildal J, Chan A-W, Haahr MT, Altman DG, Gøtzsche PC. Reporting on blinding in trial protocols and corresponding publications was often inadequate but rarely contradictory. J Clin Epidemiol. 2009 Sep 1;62(9):967–73. [PubMed]
17. Soori H. Review of Epidemiology articles. Gilan University of Medical Science Journal. 2002;11(41):1–5 [Article in Persian].
18. Valayi N, Mousavi F, Nasiri E, Kowsarian M. Quantitative and qualitative evaluation of journal’s articles Mazandaran university. Mazandaran University of Medical Science. 2005;16(52):131–9[Article in Persian].
19. Panic N, Leoncini E, Belvis G de, Ricciardi W, Boccia S. Evaluation of the Endorsement of the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) Statement on the Quality of Published Systematic Review and Meta-Analyses: e83138. PLoS One. 2013 Dec;8(12):e83138. [PMC free article] [PubMed]
20. Cobo E, Cortés J, Ribera JM, Cardellach F, Selva-O’Callaghan A, Kostov B. et al. Effect of sing reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomised trial. British Medical Journal, International edition. 2011 Nov 26;343(7833):1084. [PMC free article] [PubMed]
21. Tao K, Li X, Zhou Q, Moher D, Ling C, Yu W. From QUOROM to PRISMA: A Survey of High-Impact Medical Journals’ Instructions to Authors and a Review of Systematic Reviews in Anesthesia Literature: e27611. PLoS One [Internet]. 2011 Nov [cited 2016 Jan23];6(11). Available online from:. [PMC free article] [PubMed]

Articles from Medical Journal of the Islamic Republic of Iran are provided here courtesy of Iran University of Medical Sciences