1. Ziman J. Real science. What it is, and what it means. Cambridge, UK: Cambridge University Press; 2000.
2. British Academy. Peer Review: the challenges for the humanities and social sciences. London, UK: The British Academy; 2007.
3. Hemlin S, Rasmussen SB. The shift in academic quality control. Science Technology & Human Values. 2006;31:173–198.
4. Marsh HW, Bond NW, Jayasinghe UW. Peer review process: assessments by applicant-nominated referees are biased, inflated, unreliable and invalid. Australian Psychologist. 2007;42:33–38.
5. Cicchetti DV. The reliability of peer review for manuscript and grant submissions: a cross-disciplinary investigation. Behavioral and Brain Sciences. 1991;14:119–135.
6. Lindsey D. Assessing precision in the manuscript review process - a little better than a dice roll. Scientometrics. 1988;14:75–82.
7. Weller AC. Editorial peer review: its strengths and weaknesses. Medford, NJ, USA: Information Today, Inc; 2002.
8. Campanario JM. Peer review for journals as it stands today - part 1. Science Communication. 1998;19:181–211.
9. Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. Boston, MA, USA: Houghton Mifflin Company; 2002.
10. Glass GV. Primary, secondary, and meta-analysis. Review of Research in Education. 1976;5:351–379.
11. Marsh HW, Jayasinghe UW, Bond NW. Improving the peer-review process for grant applications - reliability, validity, bias, and generalizability. American Psychologist. 2008;63:160–168. [PubMed]
12. Bornmann L, Mutz R, Daniel H-D. Gender differences in grant peer review: a meta-analysis. Journal of Informetrics. 2007;1:226–238.
13. Marsh HW, Bornmann L, Mutz R, Daniel HD, O'Mara A. Gender effects in the peer reviews of grant proposals: a comprehensive meta-analysis comparing traditional and multilevel approaches. Review of Educational Research. 2009;79:1290–1326.
14. White HD. On extending informetrics: an opinion paper. In: Ingwersen P, Larsen B, editors. Proceedings of the 10th International Conference of the International Society for Scientometrics and Informetrics. Stockholm, Sweden: Karolinska University Press; 2005. pp. 442–449.
15. LeBreton JM, Senter JL. Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods. 2008;11:815–852.
16. *Hargens LL, Herting JR. A new approach to referees assessments of manuscripts. Social Science Research. 1990;19:1–16.
17. Bakanic V, McPhail C, Simon RJ. The manuscript review and decision-making process. American Sociological Review. 1987;52:631–642.
18. *Beyer JM, Chanove RG, Fox WB. Review process and the fates of manuscripts submitted to AMJ. Academy of Management Journal. 1995;38:1219–1260.
19. *Bhandari M, Templeman D, Tornetta P. Interrater reliability in grading abstracts for the Orthopaedic Trauma Association. Clinical Orthopaedics and Related Research. 2004;(423):217–221. [PubMed] 20. Blackburn JL, Hakel MD. An examination of sources of peer-review bias. Psychological Science. 2006;17:378–382. [PubMed] 21. *Bohannon RW. Agreement among reviewers. Physical Therapy. 1986;66:1431–1432. [PubMed] 22. *Bornmann L, Daniel H-D. The effectiveness of the peer review process: inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie-International Edition. 2008;47:7173–7178. [PubMed] 23. *Callaham ML, Baxt WG, Waeckerie JF, Wears RL. Reliability of editors' subjective quality ratings of peer reviews of manuscripts. Journal of the American Medical Association. 1998;280:229–231. [PubMed] 24. Cicchetti DV, Conn HO. A statistical analysis of reviewer agreement and bias in evaluating medical abstracts. Yale Journal of Biology and Medicine. 1976;49:373–383. [PMC free article] [PubMed]
25. *Cicchetti DV, Conn HO. Reviewer evaluation of manuscripts submitted to medical journals. Biometrics. 1978;34:728–728.
26. *Cicchetti DV, Eron LD. The realiability of manuscript reviewing for the Journal of Abnormal Psychology. Proceedings of the American Statistical Association (Social Statistics Section) 1979;22:596–600.
27. *Cicchetti DV. Reliability of reviews for the American Psychologist - a biostatistical assessment of the data. American Psychologist. 1980;35:300–303.
28. *Cohen IT, Patel K. Peer review interrater reliability of scientific abstracts: a study of an anesthesia subspecialty society. Journal of Education in Perioperative Medicine. 2005;7
29. *Cohen IT, Patel K. Peer review interrater concordance of scientific abstracts: a study of anesthesiology subspecialty and component societies. Anesthesia and Analgesia. 2006;102:1501–1503. [PubMed]
30. *Conn HO. An experiment in blind program selection. Clinical Research. 1974;22:128–134.
31. *Daniel H-D. An evaluation of the peer-review process at Angewandte Chemie. Angewandte Chemie - International Edition in English. 1993;32:234–238.
32. Glidewell JC. Reflections on thirteen years of editing AJCP. American Journal of Community Psychology. 1988;16:759–770.
33. Gottfredson SD. Evaluating psychological research reports - dimensions, reliability, and correlates of quality judgments. American Psychologist. 1978;33:920–934.
34. Gupta P, Kaur G, Sharma B, Shah D, Choudhury P. What is submitted and what gets accepted in Indian Pediatrics: analysis of submissions, review process, decision making, and criteria for rejection. Indian Pediatrics. 2006;43:479–489. [PubMed]
35. *Hendrick C. Editorial comment. Personality and Social Psychology Bulletin. 1976;2:207–208.
36. *Hendrick C. Editorial comment. Personality and Social Psychology Bulletin. 1977;3:1–2.
37. Herzog HA, Podberscek AL, Docherty A. The reliability of peer review in anthrozoology. Anthrozoos. 2005;18:175–182.
38. Howard L, Wilkinson G. Peer review and editorial decision-making. British Journal of Psychiatry. 1998;173:110–113. [PubMed] 39. *Justice AC, Berlin JA, Fletcher SW, Fletcher RH, Goodman SN. Do readers and peer reviewers agree on manuscript quality? Journal of the American Medical Association. 1994;272:117–119. [PubMed]
40. Kemp S. Editorial Comment: agreement between reviewers of Journal of Economic Psychology submissions. Journal of Economic Psychology. 2005;26:779–784.
41. *Kemper KJ, McCarthy PL, Cicchetti DV. Improving participation and interrater agreement in scoring ambulatory pediatric association abstracts: how well have we succeeded? Archives of Pediatrics & Adolescent Medicine. 1996;150:380–383. [PubMed]
42. *Kirk SA, Franke TM. Agreeing to disagree: a study of the reliability of manuscript reviews. Social Work Research. 1997;21:121–126.
43. *Lempert RO. From the editor. Law and Society Review. 1985;19:529–536.
44. *Linden W, Craig KD, Wen FK. Contributions of reviewer judgements to editorial decision-making for the Canadian Journal of Behavioural Science: 1985–1986. Canadian Journal of Behavioural Science. 1992;24:433–441.
45. Marsh HW, Ball S. Interjudgmental reliability of reviews for the Journal of Educational Psychology. Journal of Educational Psychology. 1981;73:872–880.
46. *Marusic A, Lukic IK, Marusic M, McNamee D, Sharp D, et al. Peer review in a small and a big medical journal: case study of the Croatian Medical Journal and The Lancet. Croatian Medical Journal. 2002;43:286–289. [PubMed]
47. *McReynolds P. Reliability of ratings of research papers. American Psychologist. 1971;26:400–401.
48. *Montgomery AA, Graham A, Evans PH, Fahey T. Inter-rater agreement in the scoring of abstracts submitted to a primary care research conference. BMC Health Services Research. 2002;2 [PMC free article] [PubMed] 49. *Morrow JR, Bray MS, Fulton JE, Thomas JR. Interrater Reliability of 1987–1991 Research Quarterly for Exercise and Sport reviews. Research Quarterly for Exercise and Sport. 1992;63:200–204. [PubMed]
50. Munley PH, Sharkin B, Gelso CJ. Reviewer ratings and agreement on manuscripts reviewed for the Journal of Counseling Psychology. Journal of Counseling Psychology. 1988;35:198–202.
51. *Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchison BG, et al. Agreement among reviewers of review articles. Journal of Clinical Epidemiology. 1991;44:91–98. [PubMed]
52. Petty RE, Fleming MA. The review process at PSPB: correlates of interreviewer agreement and manuscript acceptance. Personality and Social Psychology Bulletin. 1999;25:188–203.
53. *Plug C. The reliability of manuscript evaluation for the South African Journal of Psychology. South African Journal of Psychology. 1993;23:43–48.
54. *Rothwell PM, Martyn CN. Reproducibility of peer review in clinical neuroscience: is agreement between reviewers any greater than would be expected by chance alone? Brain. 2000;123:1964–1969. [PubMed] 55. *Rubin HR, Redelmeier DA, Wu AW, Steinberg EP. How reliable is peer review of scientific abstracts? Looking back at the 1991 Annual Meeting of the Society of General Internal Medicine. Clinical Research. 1992;40:A604. [PubMed] 56. *Rubin HR, Redelmeier DA, Wu AW, Steinberg EP. How reliable is peer review of scientific abstracts? Looking back at the 1991 Annual Meeting of the Society of General Internal Medicine. Journal of General Internal Medicine. 1993;8:255–258. [PubMed]
57. *Scarr S, Weber BLR. The reliability of reviews for the American Psychologist. American Psychologist. 1978;33:935.
58. *Scott WA. Interreferee agreement on some characteristics of manuscripts submitted to Journal of Personality and Social Psychology. American Psychologist. 1974;29:698–702.
59. *Scott JR, Martin S, Burmeister L. Consistency between reviewers and editors about which papers should be published. 2005. Fifth International Congress on Peer Review and Biomedical Publication. September 16–18, 2005. Chicago, Illinois.
60. Strayhorn J, McDermott JF, Tanguay P. An intervention to improve the reliability of manuscript reviews for the Journal of the American Academy of Child and Adolescent Psychiatry. American Journal of Psychiatry. 1993;150:947–952. [PubMed] 61. *Timmer A, Sutherland L, Hilsden R. Development and evaluation of a quality score for abstracts. BMC Medical Research Methodology. 2003;3:2. [PMC free article] [PubMed] 62. *van der Steen LPE, Hage JJ, Kon M, Mazzola R. Reliability of a structured method of selecting abstracts for a plastic surgical scientific meeting. Plastic and Reconstructive Surgery. 2003;111:2215–2222. [PubMed]
63. *Whitehurst GJ. Interrater agreement for reviews for Developmental Review. Developmental Review. 1983;3:73–78.
64. Wood M, Roberts M, Howell B. The reliability of peer reviews of papers on information systems. Journal of Information Science. 2004;30:2–11.
65. *Yadollahie M, Roshanipoor M, Habibzadeh F. The agreement in reports of peer reviews in the Iranian Journal of Medical Sciences. Saudi Medical Journal. 2004;25(Supplement):S44.
66. Vacha-Haase T. Reliability generalization: exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement. 1998;58:6–20.
67. Beretvas SN, Pastor DA. Using mixed-effects models in reliability generalization studies. Educational and Psychological Measurement. 2003;63:75–95.
68. Hedges LV. Fixed effects models. In: Cooper HM, Hedges LV, editors. The handbook of research synthesis. New York, NY, USA: Russell Sage Foundation; 1994. pp. 285–299.
69. Hedges LV, Vevea JL. Fixed and random effects models in meta-analysis. Psychological Methods. 1998;3:486–504.
70. Field AP. Meta-analysis of correlation coefficients: a Monte Carlo comparison of fixed- and random-effects methods. Psychological Methods. 2001;6:161–180. [PubMed]
71. Bateman IJ, Jones AP. Contrasting conventional with multi-level modeling approaches to meta-analysis: expectation consistency in UK woodland recreation values. Land Economics. 2003;79:235–258.
72. DerSimonian R, Laird NM. Evaluating the effect of coaching on SAT scores: a meta-analysis. Havard Educational Review. 1983;53:1–15.
73. DerSimonian R, Laird NM. Meta-analysis in clinical trials. Controlled Clinical Trials. 1986;7:177–188. [PubMed] 74. DerSimonian R, Kacker R. Random-effects model for meta-analysis of clinical trials: an update. Contemporary Clinical Trials. 2007;28:105–144. [PubMed]
75. Goldstein H, Yang M, Omar R, Turner R, Thompson S. Meta-analysis using multilevel models with an application to the study of class effect size effects. Applied Statistics. 2000;49:399–412.
76. van den Noortgate W, Onghena P. Multilevel meta-analysis: a comparison with traditional meta-analytic procedures. Educational and Psychological Measurement. 2003;63:765–790.
77. van Houwelingen HC, Arends LR, Stijnen T. Advanced methods in meta-analysis: multivariate approach and meta-regression. Statistics in Medicine. 2002;21:589–624. [PubMed] 78. Egger M, Ebrahim S, Smith GD. Where now for meta-analysis? International Journal of Epidemiology. 2002;31:1–5. [PubMed] 79. Baker WL, White CM, Cappelleri JC, Kluger J, Coleman CI. Understanding heterogeneity in meta-analysis: the role of meta-regression. International Journal of Clinical Practice. 2009;63:1426–1434. [PubMed]
80. Hox JJ. Multilevel analysis. London, UK: Lawrence Erlbaum; 2002.
81. Dickersin K. The existence of publication bias and risk-factors for its occurrence. Journal of the American Medical Association. 1990;263:1385–1389. [PubMed] 82. Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database of Systematic Reviews; 2009. [PubMed]
83. Ebel RL. Estimation of the reliability of ratings. Psychometrika. 1951;16:407–424.
84. Thompson B, Vacha-Haase T. Psychometrics is datametrics: the test is not reliable. Educational and Psychological Measurement. 2000;60:174–195.
85. Campanario JM. Peer review for journals as it stands today - part 2. Science Communication. 1998;19:277–306.
86. Little RC, Milliken GA, Stroup WW, Wolfinger RD, Schabenberger O. SAS for mixed models. Cary, NC, USA: SAS Institute Inc; 2007.
87. Landis JR, Koch GG. Measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174. [PubMed]
88. Daniel H-D. Guardians of science. Fairness and reliability of peer review. Weinheim, Germany: Wiley-VCH; 1993.
89. Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to meta-analysis. Chichester, UK: Wiley; 2009.
90. Goldman RL. The reliability of peer assessments - a meta-analysis. Evaluation & the Health Professions. 1994;17:3–21. [PubMed]
91. Neff BD, Olden JD. Is peer review a game of chance? BioScience. 2006;56:333–340.
92. Jayasinghe UW, Marsh HW, Bond N. A new reader trial approach to peer review in funding research grants: an Australian experiment. Scientometrics. 2006;69:591–606.
93. Bailar JC. Reliability, fairness, objectivity, and other inappropriate goals in peer review. Behavioral and Brain Sciences. 1991;14:137–138.
94. Langfeldt L. The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social Studies of Science. 2001;31:820–841.
95. Good CD, Parente ST. A worldwide assessment of medical journal editors' practices and needs - results of a survey by the World Association of Medical Editors. South African Medical Journal. 1999;4:397–401. [PubMed] 96. Smith R. Peer review: a flawed process at the heart of science and journals. Journal of the Royal Society of Medicine. 2006;99:178–182. [PMC free article] [PubMed] 97. Cho MK, Justice AC, Winker MA, Berlin JA, Waeckerle JF, et al. Masking author identity in peer review - What factors influence masking success? Journal of the American Medical Association. 1998;280:243–245. [PubMed] 98. Godlee F. Making reviewers visible - Openness, accountability, and credit. Journal of the American Medical Association. 2002;287:2762–2765. [PubMed]
99. Snodgrass R. Single- versus double-blind reviewing: an analysis of the literature. Sigmod Record. 2006;35:8–21.
100. Falagas ME, Zouglakis GM, Kavvadia PK. How masked is the “masked peer review” of abstracts submitted to international medical conferences? Mayo Clinic Proceedings. 2006;81:705. [PubMed]
101. Cole JR. The role of journals in the growth of scientific knowledge. In: Cronin B, Atkins HB, editors. The web of knowledge A festschrift in honor of Eugene Garfield. Medford, NJ, USA: Information Today; 2000. pp. 109–142.
102. Siegelman SS. Assassins and zealots - variations in peer review - special report. Radiology. 1991;178:637–642. [PubMed]
103. Fiske DW, Fogg L. But the reviewers are making different criticisms of my paper - diversity and uniqueness in reviewer comments. American Psychologist. 1990;45:591–598.
104. LaFollette MC. Stealing into print: fraud, plagiarism and misconduct in scientific publishing. Berkeley, CA, USA: University of California Press; 1992.
105. Lienert GA. Schulnoten-Evaluation. Frankfurt am Main, Germany: Athenäum; 1987.
106. Eckes T. Rater agreement and rater severity: a many-faceted Rasch analysis of performance assessments in the “Test Deutsch als Fremdsprache” (TestDaF). Diagnostica. 2004;50:65–77.
107. Hunter JE, Schmidt FL. Methods of meta-analysis: correcting error and bias in research findings. Newbury Park, CA, USA: Sage; 1990.
108. Hunter JE, Schmidt FL. Fixed effects vs. random effects meta-analysis models: implications for cumulative research knowledge. International Journal of Selection and Assessment. 2000;8:275–292.
109. Hunter JE, Schmidt FL. Methods of meta-analysis: correcting error and bias in research findings. Thousand Oaks, CA, USA: Sage; 2004.
110. Muthén LK, Muthén BO. Mplus User's Guide. Los Angeles, CA, USA: Muthén & Muthén; 1998–2007.