In order to have true millisecond-level temporal resolution on standard PC with off-the-shelf video display and data acquisition hardware, and running a modern multi-tasking operating system, several potential obstacles must be overcome or at least managed appropriately: 1) Video displays are updated (refreshed) with a relatively slow periodicity that is not in any way synchronized to a subject’s time-varying behavioral output; 2) Most data acquisition hardware have on-board data buffers that temporarily store acquired analog samples before transferring them to motherboard memory at relatively infrequent intervals, and so these data are not available for behavioral monitoring until the transfers have taken place; 3) High-level programming environments, such as Matlab, trade ease of coding for high costs in execution speed, and furthermore run atop operating systems which are not are not “hard” real-time capable. Therefore, they may be interrupted by competing processes and applications, potentially resulting in unexpected delays in behavioral monitoring and control. Here we examine each of these three issues in turn, and describe our solutions in each case.
Video Timing Issues
The first timing issue is a property of standard video display hardware. Strictly periodic video refreshes result in relatively poor predictability of video stimulus timing relative to a subject's time-varying behavior. In other words, if a visual stimulus is to be turned on after some specified behavioral event (e.g., the subject fixates), and that behavioral event occurs 2 milliseconds after the latest refresh, the stimulus will not appear for another 8 milliseconds (on a monitor running at 100 Hz, and disregarding the vertical blank interval). If, on the other hand, the subject had fixated slightly later, say 7 milliseconds after the latest refresh, that stimulus will appear only ~3 milliseconds later. (This does not take into account the position of that stimulus on the screen when using a CRT: an image in the center of the screen will require another n milliseconds for the raster beam to reach that point, where n is the refresh period minus the vertical blank duration, multiplied by the fraction of the total raster travel distance to the object, here 0.5). One way around this limitation (still used by many labs) is to employ an LED array. However, many experiments designed to examine higher level sensori-motor or cognitive behaviors require the ability to display more complex stimuli. In such cases, the timing of the refresh (i.e., vertical blank) can be time-stamped to give an accurate indication of when a visual stimulus was presented, even though this specific instant itself cannot be pre-determined. We found that such a time-stamp generated from within Matlab had an accuracy that was within one-millisecond of the actual appearance of the stimulus on the video display, as determined using a photoresistor (standard deviation = 0.3 ms).
Furthermore, in order to minimize the lag between the software call to activate a stimulus and its appearance, the relatively slower steps of creating a video memory buffer and copying image data into that buffer can be performed in advance (e.g., during the inter-trial interval). Then, during trial execution, the appropriate buffers can be “blitted” to the video back-plane (a very fast operation on modern graphics hardware). Once this back-plane has been fully populated with those stimuli that are intended to appear simultaneously, this back-plane can be “flipped” into the active (visible) position. While this preemptive strategy may necessitate copying images into video memory that, depending on the subject’s behavior, may not actually be used, the large amount of video RAM available on modern graphics cards allows more than sufficient memory for hundreds, if not thousands, of images, depending on the size of each.
Data Acquisition Issues
The second timing limitation results from the manner in which analog data is transferred from a data acquisition device (DAQ) to the PC. If the DAQ is set to acquire and store a continuous stream of data (e.g., eye- or joystick-position data), this data is stored initially in a buffer on the device itself and then uploaded to the PC in chunks at regular intervals. If one simultaneously attempts to sample this analog input stream for on-line behavioral control, only the last uploaded data point will be available. Therefore, if these uploads occur every 50 milliseconds (a value we found to be typical, using default settings in Matlab), the on-line sampled data will lag the actual signal by up to that amount of time. Nevertheless, in Matlab, data is returned to the user at a much faster rate (at least 1500 samples per second can be retrieved, and often many more, depending on the system configuration); however, the retrieved data values will simply be copies of the last transferred data point until the next upload (). Thus, the raw number of samples retrieved is deceiving, and may have confounded previously published attempts to achieve high-level behavioral control with millisecond temporal resolution (
Meyer and Constantinidis, 2005). We found that even if the buffer size is shrunk so that these transfers are forced to occur more frequently, there was a ceiling to the benefit provided by this tactic. Specifically, even after setting a buffer size that should allow only one data point, gaps between transfers of around 15 milliseconds were still common (), corresponding to a sampling rate of under 70 Hz (far less than is ideal for tracking eye movements or other analog behavioral data).
One way around this problem would be simply to avoid logging analog input data to memory. In this situation, one can sample data from many types of data acquisition devices even when they are in a “free-running” state (i.e., not set to log data to memory or disk), and the values returned will reflect the most recent state of the analog-to-digital converters. However, the inability to store acquired data for post-hoc analysis is severely limiting. In such a case, no continuous, regularly-sampled record of the behavioral signals will be available for post-hoc, off-line analysis; this is because the data points sampled on-line (i.e., under software control) will not be strictly clocked, but will instead occur at irregular, sub-millisecond intervals. Therefore, we devised a simple solution to this problem: we split the analog input signal into two identical data acquisition boards. One is set to log data to memory whereas the other is left in a free-running state for on-line sampling. With this configuration, we found that unique data samples could be retrieved at a rate well above 1 kHz (). In contrast, the rate of unique sample retrieval with only one acquisition board was limited to 20–70 Hz ().
Under our conditions, and qualitatively agreeing with a previous report (
Meyer and Constantinidis, 2005), we found no temporal cost associated with sampling one vs. eight analog channels. Thus, with two data acquisition boards, it is possible to sample and simultaneously to store multiple behavioral signals with millisecond precision.
Importantly, we found that accessing data samples from USB DAQ devices took significantly longer than from PCI devices. Specifically, using two USB-6008 devices from National Instruments, the maximum number of unique samples retrievable per second was only 200 to 400 (compared with at least 1–2 kHz for PCI devices, above), despite the 10 kHz maximum sampling rate of these devices. So, although these devices tend to be several hundred dollars cheaper than the PCI devices, their slower performance may make them less suitable for real-time behavioral control.
As an aside, we found that for sending digital event markers to separate neural data acquisition systems, parallel ports were generally faster than the digital lines on the DAQ device. On our test system, this difference was large: The per-operation time was 0.4 ms for the parallel port versus 4 milliseconds for the DAQ digital subsystem. However, newer machines can achieve significantly better performance: ~0.1 ms for the parallel port versus ~0.2 ms for the DAQ (David Freedman, personal communication). Thus, because it takes two operations to send each number (one to set the value bits, another to trigger the strobe bit), it is preferable on older systems to use a parallel port for these operations.
Software Issues
Many behavioral researchers have significant concerns regarding the use of a high-level programming environment running on a non-real-time operating system (e.g., Matlab on Microsoft Windows) to control behavior with reliable millisecond temporal resolution. These concerns may be classified broadly into three categories: 1) concerns about the adequacy of the average cycle-rate performance of such a system (that is, its ability to perform the basic steps required for behavioral monitoring and control sufficiently rapidly to be able to repeat these steps about every millisecond); 2) concern that there is simply too much temporal jitter in a high-level application such as Matlab to provide accurate time-stamps; and 3) concerns about the possibility of rare, unpredictable highlatencies introduced by software events external to the experimental environment (e.g., the stealing of CPU time by background applications). These are serious issues that must be resolved in order to have confidence that behavior is being sampled with sufficient speed and without unexpected delays so that critical measures such as movement and reaction times – in which milliseconds matter – are not distorted.
The first concern, regarding the average speed of execution within a high-level programming environment is easily allayed. Matlab running in an empty-loop (whose only function is to time-stamp each successive cycle) can execute several hundred thousand cycles per second on a modern PC running Windows (and over one million cycles per second have been measured on the newest machines). Even when code is added to check analog inputs, transform these into calibrated x-and-y coordinates, and check these coordinates against multiple possible targets, the average cycle rate still approaches or surpasses one kilohertz on modern, multi-core PCs (). Thus, despite the use of a high-level, interpreted programming environment such as Matlab, more than adequate average performance can be achieved using standard computer hardware.
The second issue, concerning the degree of temporal jitter observed in time-stamps generated by a high-level application such as Matlab, also turns out not to be significant. Using a separate data acquisition system running at 40 kHz, we tested the jitter in the arrival times of 1000 event markers sent one-at-a-time every 100 milliseconds. We found that 99.7 percent of time-stamps arrived within 0.1 millisecond of their intended times (). The largest error observed was only 1.2 milliseconds.
The third concern, regarding the possibility of rare, unpredictable delays resulting from non-experiment-related software events competing for CPU time, can also be shown to be inconsequential in practical application. Specifically, we allowed Matlab to run an empty loop continuously for one hour, time-stamping each cycle. We performed this test three times at each of three process priorities allowed by Windows: “Normal,” “High,” or “Real Time” (). Setting the priority for matlab.exe as “High” or “Real Time” resulted in zero latencies above one millisecond. Even at the lowest priority setting tested, no latencies were measured to be above 1.3 ms over the entire hour this test was run. At the highest priority setting, the longest observed latency was only half of this value, 0.6 milliseconds. These longer latencies would occur relatively rarely: For example, at the highest priority setting, latencies greater than 0.2 ms would be encountered only once every 8.2 seconds, on average. In practice, we use software to increase the priority setting during the execution of a trial and decrease it during the intertrial-interval. This is to allow background processes to use CPU time preferentially during the inter-trial-interval, thereby hopefully lessening competing demands on processor time during the trials themselves (this is borne out by a slight increase in the number of cycles executed per second when a 2-second, low-priority “pause” is inserted between 10 second epochs running at the highest priority).
While these three concerns are the ones most commonly raised regarding software timing issues, there are at least a couple of other software-related timing matters that are must be appreciated: 1) a slightly slower speed is consistently measured for the first versus all subsequent trials; 2) there is a temporal cost of accessing the experimenter’s display to update the behavior trace (e.g., a moving dot corresponding to eye or joystick position).
First, an added temporal cost is associated with the initial execution of a software function in an environment such as Matlab. This cost comes as a result of the time it takes to load the function into memory, parse the commands in its script, and compile these commands into a machine-executable format. Subsequent executions of the same function can rely in some part on these pre-compiled sections of code. The practical result of these events is that the execution speed of events within the first trial is somewhat slower than in subsequent trials. The exact cost will vary greatly from task to task, depending on the type and number of sub-functions called. To minimize this effect, one can load the function into memory and initialize its sub-functions by running a “dummy” trial (executing a trial with null stimuli and subsequently discarding any behavioral signals acquired). For example, in a task in which we were able to initialize all top-level sub-functions, we found that the first trial was 3.6 +/− 2.5 (mean +/− standard deviation) percent slower than the subsequent ones, whereas the same task showed a 7.3 +/− 1.6 percent first-trial cost when those functions were not initialized (t-test comparing the means of the percent differences across the two cases: p < 0.01).
Second, because most users will want some sort of graphical feedback about the subject’s behavior in near-real-time, there could be a temporal cost associated with these video activities, even if this feedback takes the form of something as simple as a moving dot corresponding to instantaneous eye-or joystick position. To assess the cost of this added functionality, we tested the time required for periodic updates of the position of a dot in a Matlab figure window (updated by issuing a “drawnow” command every 50 or 100 milliseconds), reflecting a varying analog signal. We found that there is a fixed, one-time cost associated with accessing this window (). Subsequent updates do not result in similar time gaps. While the time lost is relatively large (23 milliseconds, on our test machine), it occurred at a pre-determined latency: that of the first update. Therefore, to minimize the impact of this phenomenon, it is possible to perform the first graphical update upon the first cycle of the behavioral monitoring loop, and thereby be confident that subsequent updates are not interfering with the millisecond-by-millisecond sampling of behavior. Importantly, recognize that the “lost” time here results simply in the lack of behavioral data sampling during that interval, not in the slippage of temporal measurements or in erroneous time-stamps.
Note that drawing to the experimenter’s display window was accomplished by calls to Matlab’s built-in graphics routines rather than through calls to the low-level graphics functions that controlled the subject’s display in full-screen mode. Thus, a screen update resulting from the “drawnow” command would appear at some later time, as allowed by OpenGL (the graphics library used by Matlab) and the screen refresh rate. This delay was acceptable because updating the experimenter’s (not subject’s) display had a relatively low-priority; all that was required was the subjective impression that the eyeor joystick-position trace was moving smoothly. The advantages of using Matlab’s high-level graphics functions included the ability to construct, very simply, an information-rich display to aide the experimenter’s interpretation of behavioral events in real-time, during task performance.
The publisher's final edited version of this article is available at 
