Take for instance the workhorse of human genetics, the simple genome scan. I will illustrate with genome-wide association studies (GWAS), but the same principles apply to linkage and many other types of analyses as well. We collect cases and controls, and genotype them at large numbers of SNPs spanning the genome. We then perform a statistical analysis at each SNP, looking for the SNPs with the strongest evidence of being associated with disease. Then to be sure that any findings to emerge are not false positives, we attempt to replicate the salient results in an independent sample.

The accumulation of evidence with increasing amounts of data follows certain patterns that we can all agree upon. For instance, suppose that we flip a coin to see whether it is ‘fair’ (lands heads 50% of the time) or biased, and we do so in two stages, with data set D₁ comprising n₁ tosses and having evidence E₁, and D₂ comprising n₂ tosses with evidence E₂. Let E be the total evidence, considering both D₁ and D₂. Given what we mean by evidence, I believe that with a little thought everyone will agree that if D₁ and D₂ each individually support ‘bias’, then the total evidence E must be greater in favor of bias than the larger of E₁ and E₂, each considered on its own. Similarly, if D₁ and D₂ each support ‘no bias’ then the total evidence E must be less (more strongly in support of ‘no bias’) than the smaller of E₁ and E₂. That's how evidence behaves. But it is easily shown that p values do not necessarily follow this pattern [2]. They will under many circumstances tend to behave more like averages of E₁ and E₂. (The same is true of the maximum likelihood ratio (LR) or its logarithm, the maximum LOD score [3].) We can think of the overall evidence E as resulting from some particular concatenation operation performed on E₁ and E₂. We may not have a precise understanding of what this concatenation operation is, but we can see that whatever it is, it is not the operation that occurs when we combine the data sets and recompute the p value. This means that the p value is in some very fundamental way simply not tracking with the evidence at all. The point is not that the p value never tracks with the evidence, but rather that there is no intrinsic connection between the p value and the evidence such that we can rely upon it to do so.

In fact, the widely accepted ‘winner's curse’ – a form of regression to the mean – is simply a manifestation of this problem. It has been shown that it is reasonable to expect, when following up on the smallest p value under fairly ordinary circumstances and assuming a true association, that increasing the sample size will tend to lead to larger, not smaller, p values [2]. We can express this by saying that the p value can indicate less evidence for association even while the evidence itself is going up. Thus the smallest p values do not necessarily replicate, even if they represent true positives, even setting aside all other complications [4, 5]. This of course means that failure to replicate in no way establishes the falsehood of a prior finding, as is well known. What is less appreciated is that the winner's curse is symptomatic of an underlying measurement problem, viz., the failure of the p value to consistently track with the evidence as data accrue.

One important additional feature of our current practice is that we can only accumulate evidence (no matter how badly measured) in favor of genetic effects, but we have no mechanism for accumulating evidence against genetic effects. We can of course use failure to replicate for this purpose, but this begs the question of how we will know when a replication sample weakly supports an initial finding versus when it actually gives evidence against this initial finding, bringing us back to the problem of measuring evidence against a hypothesis. Once a result is reported in the literature, subsequent literature reviews will find the result without any systematic way of assessing whether the preponderance of evidence actually weighs against it. The consequence of this is that ‘findings’ are seldom removed from our (real or virtual) databases of genomic results, leading to a monotonically increasing proportion of the genome becoming implicated for any given disorder over time. In view of this, it should come as no surprise that in many cases the more literature we accumulate on a given topic, the murkier is the picture that emerges of the evidence in its entirety. If we do science properly, however, the more data we accumulate the clearer things should become. This is after all the purpose of collecting data.

In statistical circles, problems with using p values to measure evidence have been widely discussed (e.g. [6, 7, 8, 9]) and the very practical pitfalls of relying on them in biomedical research are drawing increasing attention even in the popular press [10], thanks in part to recent work by Ioannidis [11]. Arguably, many, though not all, of the problems that this work is uncovering stem from the reliance on p values and significance testing in lieu of proper evidence measures. But in any case, it is hard to see the justification for continuing to use p values for a purpose for which they are clearly unsuited.

I have also thus far skipped over one standard feature of GWAS analysis that provides a singularly telling clue that what we have on our hands is an underlying measurement problem. Before one uses the p value as a measure of evidence, one ‘corrects’ it for multiple tests. This seems on the face of it innocuous. The p value is the probability of seeing a result as extreme as the observed result under the hypothesis of ‘no association’, but clearly seeing an event with probability of, say, 0.001 means something quite different in the context of a single SNP than in the context of evaluating a million SNPs. Thus we use that number −0.001 – as a measure of evidence, but only after correcting it for the total number of tests.

Throughout the 18th and 19th centuries, the topic of temperature permeated a great deal of work in physics, ranging from the very practical problem of designing efficient steam engines to reconciliation of the various conceptions of caloric, a substance postulated as the medium of heat transfer that we now know to be nonexistent. But physicists of the day had no recourse to thermometers in their modern form to assist them in these pursuits. Instead, they relied on instruments called thermoscopes, which resembled ordinary thermometers but for one key feature: they were, in essence, uncalibrated.

For example, some thermoscopes contained alcohol, while others contained mercury. In each case, when heated, the liquid expanded and the amount of expansion could be used as an indication of increased temperature. But because different liquids expand at different rates when heated, the unit of measurement – the degree – did not mean the same thing across the different devices. Furthermore, within each thermoscope the rate of expansion was a function of the range in which the temperature was varied, among other factors, further complicating comparisons across devices and even across the scale for a single device.

This posed several problems. First, the measured temperature could be both 50 and 60° at the same time and place, if two different thermoscopes were used, or it could be 50° at each of two places, while the actual temperatures were quite different. Second, it was unclear whether an increase in temperature of 1° at the low end of the scale meant the same thing as an increase of 1° at the high end, even for measurements made with a single thermoscope. And third, above all, physicists lacked an objective procedure for resolving these difficulties. This represented both a significant practical challenge and a deep conceptual obstacle: It is easy to verify that a particular thermoscope is accurately measuring the temperature if there exists an established thermometer for comparison, but how does one go about calibrating the first thermometer?

What makes the calibration of thermoscopes so difficult is that it poses a problem of what Chang [12] calls nomic measurement, or measurement of a quantity that is itself not directly observable, based on application of an underlying law. In this case, we want to measure a quantity X (temperature) on the basis of some observable phenomenon Y (expansion of a liquid when heated), through a law f(Y) = X, expressing X as a function of Y. But such a law cannot be empirically discovered or verified, because this would require knowing the true value of X, which is obtained only subsequent to application of the law. The hallmark of nomic measurement is thus its apparent circularity. This circularity led many early physicists to eschew talk of the temperature as something with unique physical existence, advocating instead a purely relative calibration in which one thermoscope was selected and others were simply calibrated against it. Of course, ‘the temperature’ presents no such dissonance to the modern ear. This represents a fundamental triumph of physics over metaphysics. And interestingly, it was in fact relative – rather than absolute – calibration that proved intractable, due to the difficulty of establishing the degree as a constant unit in the absence of a deeper theoretical understanding of the nature of temperature. As Chang [12] notes, the derivation of the absolute measurement scale gave rise to the rigorous thermodynamic definition of temperature, and not the other way around.

This leads to fundamental difficulties. For example, one may have p values of both, say, 0.01 and 0.06 for the same data, if one uses two different methods (e.g., variance components vs. regression) for calculating the p value. Or one may obtain 0.01 and 0.06 for two different sets of data, based on two types of analysis, with no way to say which represents the stronger evidence. Additionally, different evidence measures (e.g., LRs and p values) are on incomparable scales. Moreover, it is unclear whether a change in LR from 1 to 2 as new data are added means the same thing as a change from 10 to 11, or perhaps 10 to 20 (similarly for p values). Even assuming that one experiment could succeed in rank-ordering its own results with respect to the strength of the evidence, across experiments, results would remain incomparable and the units in which evidence is measured obscure.

This ought to bother us deeply. Think again about thermoscopes. Would we be satisfied if the measured temperature at a particular time and place was both 50 and 60°? Or if a measure of 50° at a second location gave us no information as to which was the warmer locale? Science is in large part the art of careful measurement, and a fixed measurement scale is the sine qua non of this art. It is obvious to us that thermal measurement devices lacking fixed units and constancy of scale across applications are problematic, yet we seem oddly laissez faire in our approach to evidence.

Of course, if we are interested merely in establishing the warmest room in a building, we can carry a single thermoscope from room to room, waiting in each location for it to reach equilibrium with its environment and recording the results. Imagine, however, trying to integrate large numbers of temperature readings into a unified model for purposes of weather prediction, with those readings coming from reporting stations around the globe and based on a multitude of different thermoscopes. Such data would be useless. Just so, in simple biological settings, any measure that can reasonably rank-order data with respect to relative evidence may be sufficient (though how we would verify that the measure was correctly rank-ordering by the evidence is itself a conundrum), but any attempt to simultaneously interpret measures derived from many distinct approaches, or to layer multiple levels of analysis into unified experimental outcomes, requires a system for calibrating the various and sundry evidence measures against one another on an absolute basis, i.e., independent of context.

But the critical point here is to recognize that evidence, like temperature, represents a nomic measurement problem. We wish to measure the strength of the evidence based on some observable, or computable, quantity, for instance, the p value or LR (or perhaps a non-stochastic measure). Here the LR, say, is analogous to the volume of liquid within a thermoscope, in the sense that, at least in principle, as the underlying evidence goes up or down, the LR should decrease or increase accordingly. We seek a definition of the degree of evidence that has the same meaning across measurement ‘devices’ (alternative statistical models), across different experimental domains (multiple biological levels of analysis), and across the range of the evidential scale. But how will we know whether we have correctly defined the law mapping the evidence onto these computable quantities if we can only infer what is the true evidence from the application of that same law? We are confronted with the same circularity that stymied thermometry for so long.

To give the flavor of how this might go, consider first the fundamental physical construct known as the equation of state. An equation of state gives a complete description of a system in terms of a (generally small) number of interrelated parameters describing the system at a macroscopic level. For instance, the behavior of m moles of an ideal gas can be described in terms of the absolute temperature T (that is, temperature measured on the absolute scale), volume V, and pressure P through the equation of state (with R = gas constant). This equation represents a complete description of the system in the sense that knowing the value of any two of the three parameters (T, V, P) determines the value of the third, while any additional macroscopic properties of the system (e.g., entropy) can be derived once this equation is known. Equation 1 is therefore a particular instance of an equation of state, which can be represented in the more general form where the subscript IG (for ideal gas) indicates that the function f will take on a particular form specific to ideal gases.

A great deal of fundamental theory can be worked out without an explicit definition of T, based only on the assumption that an equation of state in the form of equation 2 exists. This theory can then be related to the behavior of actual gases provided only that we have access to a device for measuring empirical temperature t (that is, temperature as it would be measured by a thermoscope), such that T equals some function g of t. Here the specific law g(t) = T need not be known, but there is a presumption that a device for measuring t exists such that this law could be established in principle. In practice, this simply means that the empirical temperature t must ‘track’ with the absolute temperature, going up when T goes up and down when T goes down.

Statistical models can also be viewed as equations of state, similarly framed in terms of an empirical (‘thermoscopic’) measure of evidence e, even before we have a definition of what would be the absolute evidence E. All we need is an empirical measure that behaves like a thermoscope, that is, one that goes up or down as the evidence goes up or down, at least in simple settings and under normal circumstances (see also [2] for further discussion of how we know when an evidence measure is behaving like a thermoscope). For instance, think of binomial data (N coin tosses of which X land heads) and the simple hypotheses P[heads] = q = 0.05 versus q = 0.5. Let us assume at least for the moment that the LR itself behaves like a thermoscope for this simple system, that is, that it correctly tracks with the evidence, or in other words, that e = LR. (This is probably a safe assumption for comparisons between simple hypotheses [3, 7], but does not necessarily assist us in measuring evidence for compound hypotheses, where unknown parameters can take multiple values.) In this case we have

Now if we hold e constant and increase N, X will have to change by a compensatory amount; similarly for other relationships among e, N, and X. Indeed, if we fix any two of the three variables in this equation, the value of the third is known and simple to calculate. Thus equation 3 is an equation of state in e, N, and X.

But what can we do with this equation of state? In physics, the equation of state allows construction of a Carnot engine, a device for quantifying the transformation of heat into work in the absence of any prior operational definition of heat [18]. Note that the Carnot engine is itself a purely mathematical device, based on certain assumptions – e.g., perfect reversibility – which cannot be realized by physical systems. Kelvin and Joule defined the absolute temperature scale by deriving it from mathematical features of the Carnot engine, rather than defining it ab initio [18]. It is hard to overstate the ingeniousness of this maneuver. Without knowing what heat was (this was still the era of caloric, after all) or what precisely was meant by T, mathematical insights based on the Carnot engine yielded both a definition of T and an absolute scale for its measurement. The details are beyond the scope of this essay, but the interested reader is referred to Chang [12] for a lucid presentation that is quite accessible to non-physicists.

Returning to the evidential problem, for a given set of data, the simple equation of state given above (equation 3) for the binomial case represents a ‘system’, which can be plotted as a curve with q on the x-axis and e = LR on the y-axis. Insofar as equation 3 is an equation of state, this plot conveys all of the information in the data regarding the strength of the evidence for or against q = 0.05 for given data. The plot itself has some very real, physical properties, including for instance the area A under the curve and the maximizing value Q of q. We can, therefore, reformulate equation 3, which is written in terms of the data (N, X) to being about properties of the graph itself, such as A and Q. This yields an equation of state in the form

[Here the subscript BIN (for binomial) is a reminder that the particular form of f will be dictated by the behavior of binomial systems.] Equation 4 is a simple reparameterization of equation 3, which also encapsulates the property that if, say, we hold the evidence e constant and increase A by a specific amount, Q will have to shift by a compensatory amount (and similarly for other relationships among e, A, and Q).

In the physical system, T, V, and P constitute macroscopic properties of the system that are affected by changes in energy, in particular by the influx or outflux of heat. We can similarly think of the influx of new information – like the influx of heat – as performing ‘work’ on the graph associated with equation 4, that is, changing its physical properties A and Q. Viewing the statistical system in this way suggests that we should be able to run ‘evidential’ Carnot cycles and to study their properties. This opens the door to derivation of an absolute scale for the measurement of evidence, following the Kelvin-Joule template. There is an enormous amount of mathematical detail remaining to be worked out, and, of course, the devil may well reside in the details. However, if this basic framework is even approximately correct, this means that evidential equations of state can be derived prior to defining what precisely is meant by the evidence, and deployed in ‘evidentialism’ just as they are in thermodynamics, to provide the basis for (and definition of) absolute measurement.

This requires of course that we accept evidential analogues of the laws of thermodynamics, but this is not so farfetched as it might at first appear. While there may not be a comparable physical basis for an evidential version of the 1st law (which stipulates the conservation of energy), certain elementary principles come to mind that could stand in nicely: for instance, the law of total probability, or perhaps, a comparable law constraining total evidential information. The analogue of the 2nd law (which, in one form or another, describes the tendency of systems to equilibrate in their maximum entropy configurations) is perhaps more obscure. However, given the close connection between thermodynamics, statistical mechanics, and the entropy-based information theoretic framework of Shannon [15], it seems reasonable to expect that an entropy-based formulation will also be forthcoming in the evidential case, and this would in turn allow concise formulation of an evidential 2nd law.

Not only would all of this give us a framework for solving our nomic measurement problem, but application of the theory to specific subject-matter domains could produce additional results paralleling those of physics. For example, the amount of heat required to raise the temperature of a liquid by a given amount depends on the liquid, or in other words, different liquids have different specific heats. This means that different liquids have different equations of state, so that the thermal meaning of a given change in volume within a thermoscope is not fixed. Just so, the quantity of data required to change the evidence by a fixed amount will depend upon the hypotheses of interest, among other things. We can think of this as a matter of different ‘specific heats’ for different statistical applications. We will therefore need to derive empirical adjustments to equations of state for particular applications, to ensure that a change of 1° in our measure of the evidence always means the same thing. The Carnot engine also yielded a theoretical upper bound on the efficiency with which heat could be converted to work, generating new metrics for investigating the relative efficiencies of various real engines. Just so, our evidential analogue of the Carnot engine could yield a theoretical upper bound on the efficiency with which data, or the information conveyed by the data, can be converted to evidence, in turn leading to new ways to evaluate mathematical modeling methods by comparison with this upper bound.