The results show that the median rule is a useful tool for measuring onset latencies of ERs. Simulations indicate that this rule is not overly sensitive to outliers as are methods based on SD thresholding. The median rule can easily be applied to real MEG data, and at group level produces results comparable to those obtained with SD thresholding and t-tests.
Previously widely used methods for measuring onset latencies are suboptimal. The t
-test assumes that the population from which the baseline is sampled is normally distributed. EEG/MEG data sets are non-stationary and heavy-tailed, which is inconsistent with an assumption of normality (Chau et al., 2004
; Palus, 1996
). Outliers and heavy tails significantly reduce statistical power of the t
-test (Benjamini, 1983
; Wilcox, 2005
). In the context of onset latencies this translates to poor detection sensitivity. The t
-test is also based on standard deviation, which is a non-robust measure of variability that is overly sensitive to outliers as demonstrated by the simulations. Most importantly, the t
-test can only be used to measure onset latency from an ER averaged across subjects and gives no estimate of the variance in onset latencies across subjects. Without variances, it is impossible to make statistical comparisons. Some studies have adapted the t
-test method to individual subjects by windowing the data to create the pointwise distribution rather than forming the distribution across subjects (Rodriguez-Fornells et al., 2002
; van Schie et al., 2004
). However, windowing further reduces sensitivity and requires an arbitrary selection of window size.
SD thresholds have been used extensively for outlier detection, despite having been shown to be ineffective due to their sensitivity to outliers (Wilcox and Keselman, 2003
). Our simulations confirmed that the SD method is overly sensitive to outliers. With normally distributed noise, the onset latencies measured with the median rule and the 3.1 SD threshold were identical, as they theoretically should be. Replacing only two percent of the baseline data points with outliers delayed the SD onset such that it was significantly later than the median rule onset by 4 ms. The median rule produces a statistically more correct representation of the data, and is not affected as strongly by a relatively small number of outliers in the data.
All measures of onset latency are sensitive to skewness in data with high noise levels. When attempting to detect a positive deflection, positive skewness increases the probability of a random noise sample crossing threshold thereby decreasing the measured onset latency. Correspondingly, a negative skewness increases the observed onset. While in this study we analyzed responses that can only have positive values, typical evoked EEG/MEG responses have both positive and negative deflections. For negative deflections, the relations between direction of skewness and latency change are reversed: positive skewness increases and negative skewness decreases the observed onset latency. Skewness is often changed by elementary operations, for example if the noise in two planar gradiometers is normally distributed (unskewed) then the vector magnitude will follow a Rayleigh distribution (positively skewed). One should therefore exercise caution when comparing onset latencies between studies that have clearly different noise levels or use dissimilar computational methods that differentially influence noise level and skewness. However, the results of the simulation with N=4 were a worst-case scenario with high noise and few averages. The effects of skewness on onset latency disappear with a large number of subjects.
Statistically robust methods would be particularly important for applications in which only a few trials (segments of raw data) are used for calculating ERs and noise levels are very high (Leonowicz et al., 2005
). However, the median rule can be useful in all data sets, even in those with low noise levels. Using robust measures in future onset latency studies does not invalidate previously published results. In our real data from 8 subjects the difference between boxplot and SD onset latencies was not significant. The t
-test latencies fell within the confidence intervals of both the other methods, although they could not be compared statistically because of the limitations of the t
-test method. The lack of a consistent method for measuring onset latencies makes comparisons of different studies difficult. An ideal method should not make assumptions about the shape of the ER or the distribution of the baseline. For outlier detection, boxplot rules such as the median rule are widely used and have been proven versatile enough for a wide range of applications. The median rule is both statistically robust and easy to implement. Code for implementing the median rule is available in Wilcox (2005)
A requirement that the data stay above threshold for a certain period of time is arbitrary and therefore poorly justified unless done in a statistically rigorous way (Achim, 1995
). A statistically rigorous requirement has not been developed for the median rule or SD method. Therefore we propose that when constraints are necessary, they should be based on the physiology of the response, which may vary across applications. For instance, in intracranial recordings it takes about 15 ms for activations to reach the auditory cortex (Celesia, 1976
). Thus, when measuring the evoked response over auditory cortex, a physiologically relevant constraint would be to require that onset can occur only after 15 ms, as was done in this study. When interested in the onset of only the main (which is not necessarily the earliest) component of a response, a relevant constraint would be to choose the onset that stays above threshold longest (Maris and Oostenveld, 2007
). Ideally, such constraints should be chosen when designing the experiment and not applied post-hoc.
We have shown through simulations and with real data that the median rule, a statistically robust outlier detection method, is simple and effective for measuring onset latencies of evoked responses. As opposed to previously used methods, the median rule makes no assumptions about the distribution of the data, is not sensitive to outliers in the pre-stimulus baseline, does not require additional constraints, and can be applied to finding the onset of individual ERs. Simulations show that the median rule performs well even for noisy and/or skewed data. The median rule can be easily applied to real data and produces results similar to other methods across subjects. This encourages comparison of results obtained with the median rule and previous studies using less robust methods, and may encourage use of the median rule in future studies.