One argument for excluding LGS1 studies is that the specific nature and shorter time frame make them less likely to undergo the usual conventions of scientific peer review. Peer review can provide a check on science that is poorly designed, does not conform to established conventions, adopts arbitrary methods, or is poorly written. Such science is less likely to be accepted in peer-reviewed journals. Peer review is also important in encouraging scientists to refine underlying models and arguments. Still, we know that peer reviewers frequently disagree about whether to accept papers (Rothwell and Martyn 2000
). We also know that poorly designed and analyzed studies can easily receive favorable reviews (Curfman et al. 2006
; Smith 2006
). Indeed, many published articles undergo only the most cursory peer review (Jasanoff 2008
). Either via poor judgment or conscious intention, scientists choose study topics, opt for study designs, do analyses, and interpret results in ways that bias conclusions in one direction or another (Melnick et al. 2008
). Sometimes, peer review will reject such studies, but often it will not. In addition, unlike the idealized image of peer review, the decisions of peer reviewers often do not determine publication. Editors always have the final say, so well-reviewed articles may not be published, whereas at the same time, editors approve publication of poorly reviewed studies (Jasanoff 1990
). Moreover, although peer review may expose weak study design or lapses in a scientist’s understanding or logic, it is unlikely to detect any but the most blatant fraud or scientific misconduct (Smith 2006
). Here we differ with the idealized picture of peer review presented by Henry and Conrad (2008)
. Peer review is not a remedy for Judge Kosinski’s concerns [see also Jasanoff (1990)
Legal cases sometimes engender research to fill a void in scientific knowledge or to answer questions specific to a given setting. In such cases, the peer review process may be too slow and cumbersome to provide timely information to the legal system. Alternatively, research appropriate to answer factual questions critical to a legal case may be too narrow to warrant peer-reviewed publication, even if its methods are impeccable. In other cases, innovative methods may be exactly the type needed to answer questions of fact raised in litigation, although they may fare poorly in peer review that rewards “inside the box” thinking and penalizes the new idea or method. Many classic articles in economics, including seminal articles that eventually led to the awarding of Nobel Prizes, had great difficulty being accepted for publication (Gans and Shepherd 1994
). These points are recognized in the Daubert
opinion (Daubert v. Merrell Dow Pharmaceuticals, Inc. 1993
In some instances well-grounded but innovative theories will not have been published. . . . Some propositions, moreover, are too particular, too new or of too limited interest to be published.
Finally, peer-reviewed publication can be manipulated by the parties to litigation. It can “fall apart if lawyers and litigation experts invade the realm of scientific research and manipulate the medical and scientific publication system to achieve their litigation ends” (Anderson et al. 2001
Few, if any, journal peer-review processes are as stringent or as probing as the usual cross-examination performed in an adversarial setting. It is simply not true that LGS1 studies do not undergo peer review. Lawyers routinely hire consultants to go over the minutiae of any study offered up by the other side. Alleged study flaws are then used in cross-examination to devalue or deconstruct the study in the eyes of the jury.
We do not claim that peer review and cross-examination are interchangeable. Each is based on different assumptions, and each operates in a different manner. Peer review typically assumes that the submitted article is an accurate representation of the underlying research and that the research itself is honest and not intentionally biased or misleading. Peer review is a filter, designed to let through research that is original and significant, as well as based on good data and a valid research design. It also serves the functions of improving analysis and clarifying exposition. Overall, peer review is designed to improve the quality of the scientific literature. Conversely, cross-examination assumes that research is designed to buttress the opponent’s position and is both biased and misleading and probably poorly designed as well. Cross-examination is designed to deconstruct and undermine the credibility of an adversary’s expert testimony or research, not to improve its quality (Jasanoff 1992
). By taking an adversarial stance, cross-examination may reveal hidden assumptions and errors not uncovered by peer review (Jasanoff 1996
Given these differences, cross-examination may be a better tool than peer review to expose purposefully misleading research. A competent attorney, aided by competent experts, should be in a better position to expose the flaws in such research than is the peer reviewer, who often takes less time than the expert in a legal case and has more limited resources to probe than does the cross-examining attorney (Jasanoff 1996
Perhaps the strongest argument related to peer review derives from the fact that LGS1 is typically unpublished and therefore not exposed to the scrutiny of the scientific community. One aspect of such scrutiny is that publication may lead to new research that contradicts the original findings. Perhaps more important is that publication allows scientists’ work to be read by their scientific peers. Scientists do not want their peers to read their badly flawed or, even worse, dishonest research. So, it may be possible that they are willing to engage in such research within the confines of a trial but would be unwilling to submit it for publication. This is an argument against all unpublished research, not just LGS1, and it is also an argument against all adversarial experts, not just scientists testifying about LGS1.
This then comes back to the question of the effectiveness of cross-examination in revealing to the jury the nature of poorly done and deceptive research. Cross-examination by attorneys who have been briefed by their own experts can accomplish the task and is potentially more useful than conventional publication peer review.