|Home | About | Journals | Submit | Contact Us | Français|
Distinct modules of the neural circuitry interact with each other and (through the motor-sensory loop) with the environment, forming a complex dynamic system. Neuro-prosthetic devices seeking to modulate or restore CNS function need to interact with the information flow at the level of neural modules electrically, bi-directionally and in real-time. A set of freely available generic tools is presented that allow computationally demanding multi-channel short-latency bi-directional interactions to be realized in in vivo and in vitro preparations using standard PC data acquisition and processing hardware and software (Mathworks Matlab and Simulink). A commercially available 60-channel extracellular multi-electrode recording and stimulation set-up connected to an ex vivo developing cortical neuronal culture is used as a model system to validate the method. We demonstrate how complex high-bandwidth (>10 MBit/s) neural recording data can be analyzed in real-time while simultaneously generating specific complex electrical stimulation feedback with deterministically timed responses at sub-millisecond resolution.
The experimental method of placing a biological system within a low-latency closed loop control system is well established in the “dynamic clamping” of single nerve cells (Sharp et al., 1993; Robinson, 1994; Prinz et al., 2004). Taking a similar approach to the network level, however, currently requires a highly specialized software and hardware set-up to achieve a feedback system with sufficiently deterministic latencies and narrow response-time windows for realistic motor-sensory loops (Rutten, 2002; Wolpawa et al., 2002; Ahissar and Kleinfeld, 2003; Karniel et al., 2005; Novellino et al., 2007). This constitutes both a significant investment in terms of time and cost, but also limits the scope of the methods in being applicable only for the specific set of experiments they were designed for. In recent years, however, the increasing computational power in low-cost desktop computers has made it possible to perform complex calculations on high-bandwidth signals in real-time on a number of versatile standard platforms.
Here, we demonstrate how The MathWorks Matlab and Simulink programming framework can be adapted in a generic way (1) to acquire multi-channel electrophysiological signals from a neuronal experimental preparation, (2) to perform complex data analysis, (3) to control almost any kind of telemetry and sensor system and, (4) to provide contingent multi-channel feedback stimulation with sub-millisecond latency. Our software programs are freely available for download at ModelDB (http://senselab.med.yale.edu/modeldb/ShowModel.asp?model=128068) and can be adapted to a wide range of closed-loop experimental paradigms. Advanced programming is not required as the real-time stage consists of a high-level visually developed Simulink signal processing model and simple control scripts.
The feedback system consists of two standard desktop computers: (1) The self-contained “target” PC, running a dedicated real-time operating system kernel (Mathworks Real-Time Workshop xPC Target), is connected to all the instruments relevant for experimental measurement and control. Visually programmed signal based Mathworks Simulink models are compiled to run independently on the real-time target PC to acquire data, analyze spike-trains and generate feedback signals at synchronous iteration steps paced by an interrupt at an interval of 50–500 μs (depending on the number of channels, required sampling rate and feedback processing complexity, 10MBit/s data throughput represents an approximate upper limit). (2) The “host” PC, runs Mathworks Matlab and an asynchronous Simulink interface providing the experimenter with control of parameters, as well as on-line visualization and data archiving. The two parts of the system communicate through a 100-MBit TCP/IP (UDP for data streaming) network connection (see Figure Figure11).
A multi-channel electrode array (Stett et al., 2003) interface with a primary neuronal culture was chosen as a well-established experimental model system. Such cultured nerve cells exhibit considerable potential for plasticity (Shahaf and Marom, 2001; Eytan et al., 2003; Van Pelt et al., 2005), and information can be transferred to and from the network in a well-defined manner. The model implementation is thus sufficiently complex to demonstrate the capacity of our approach but remains generic enough to serve as a straightforward starting-point for the development of custom in vitro or in vivo applications.
Cortical tissue was taken from newborn rats and plated (after dissociation) on commercially available substrate integrated 60-electrode array dishes (Multi Channel Systems MCS GmbH, electrode diameter: 30 μs, electrode spacing: 200 or 500μm, material: TiN) using standard procedures described elsewhere (Eytan et al., 2004) and maintained at 37°C in an atmosphere of 5% CO2 and 95% air. Experiments were performed after a 21-day incubation period, during which time a mature densely inter-connected network forms that spontaneously exhibits a characteristic bursting activity pattern (Maeda et al., 1995; Beggs and Plenz, 2003; Eytan and Marom, 2006; Plenz and Thiagarajan, 2007). The dissociated neuronal cultures develop to exhibit a rich repertoire of spontaneous spatio-temporal activity (Wagenaar et al., 2006) that may be classified by various methods, such as an order-based representation (Shahaf et al., 2008). Electrical stimulation at different electrodes, or using different amplitudes and frequencies has distinguishable effects (Marom and Shahaf, 2002).
Action potentials are recorded from the electrode array using a head-stage and a filter-amplifier (Multi Channel Systems MCS GmbH MEA1060-BC and FA60) and acquired by a standard desktop PC (running the Mathworks xPC Target operating system) through a supported data acquisition card (see xPC Target documentation). Every 500 μs, an interrupt service routine is executed to process the acquired “frame” of data (8×60 12 bit samples) while the card is continuously acquiring the next frame of data in the background. Action potentials are detected and aligned by a threshold criterion and further processed by the feedback generation system. More complex detection algorithms for either individual events (e.g., spikes) or spatio-temporal composites can be implemented without compromising the performance of the system. A tunable refractory period (2.5ms) prevents multiple detection of single events that cross the threshold several times.
Interactive feedback can be provided through a physical telemetry motor and sensor system or directly through an electric stimulus. The analog output of the data acquisition card could be used in principle to synthesize a stimulus waveform (through a stimulus isolator circuit), however, in our example, we use the programmable stimulus generator supplied with the MEA set-up (Multi Channel Systems MCS GmbH STG1002), which is connected to the head-stage at the electrode array. The MEA1060-BC head-stage is programmable, so that a set of any of the 60 recording electrodes can be dynamically selected for stimulation. During stimulation, a TTL trigger and blanking signal lasting 1ms is applied to the head-stage, electronically switching the designated stimulus electrodes to the stimulus generator lines and all other electrodes to ground. In this way, stimulation artifacts are almost completely suppressed allowing action potentials to be recorded again within 2ms after stimulation.
The STG1002 stimulus generator and the MEA1060-BC head-stage are two independent instruments, controlled by the real-time model through independent serial RS232 programming lines and triggered by two of the digital output lines of the data acquisition card (Simulink implementation of the programming protocols are included in the example software). All experimental control is thus performed by the real-time target PC, the drivers and control software provided by MCS for Windows are not used. The subset of stimulating electrodes is dynamically selected using a serial command string of 40–60 bytes length (depending on the complexity of the desired waveform). Likewise, the applied waveform is dynamically controlled using a serial command string of 54 bytes length (biphasic symmetric unipolar voltage stimulation, amplitude: 100–900mV, duration: 200μs negative pulse, 200 μs positive pulse).
It therefore takes less than 10ms to transmit the commands for both selecting a new set of stimulating electrodes and changing the stimulus waveform by reprogramming the respective instruments through the two independent serial lines. Integration of the control software within in the real-time model allows a high bandwidth feedback signal to be delivered as a function of the ongoing activity at a rate of up to 100 Hz (the stimulus generator STG1002 is limited to a maximum triggering frequency of 50Hz). This means that a stimulus of dynamically varying form can be delivered through a variable set of electrodes every 10ms (this can be improved by using different apparatus and/or firmware if necessary).
The above feedback procedure may be adapted to other modalities, e.g., activity-dependent application of neuromodulators, sound, light, motion, and so forth.
The Mathworks Simulink platform provides a wide range of commercially available real-time signal processing and analysis block sets, including adaptive filters, statistical operations, look-up tables and Fourier and wavelet analysis. Complex multidimensional feedback loops can be designed very easily and visually by “dragging and dropping” the appropriate blocks inside the model and connecting them with signal lines. For operations that are not supported out-of-the-box, standard procedural code (Matlab M code or C code) can be embedded inside function blocks.
In this example application, we analyze spontaneous spatio-temporal activity patterns (monitored by a 60-channel electrode array) as they occur. The specific realization described here is aimed at detecting the relative timing of the first-spike of each electrode in relation to the onset of a Network Spike (NS) (Eytan and Marom, 2006; Shahaf et al., 2008). Network spikes are detected by a threshold criterion, and the recruitment order is extracted in real-time from the delays to first spikes. The resulting recruitment order is compared to pre-defined templates for on-line classification, and a corresponding output is generated using relevant actuators or stimulus generators. The Simulink model shown in Figure Figure22 shows an example for real-time detection of electrode onset order.
A limited amount of visualization can be done on the target (Figure (Figure3),3), but it is the function of the host PC running Matlab to receive and archive the data streams from the real-time system. Spike shapes are extracted from the raw data and stored separately for post-processing. If a feedback signal is generated dynamically on the target, the parameters of the feedback signal (electrode, time, amplitude) are also transmitted and logged as they occur. In this way, the host PC serves as an independent visualization, archiving and post-processing station allowing the experimenter to observe the behavior of the neuronal preparation and state of the feedback system without disturbing its operation.
The host PC is also used to modify the parameters of the real-time system and to execute different experimental conditions consecutively, making experimental control fully scriptable. The real-time target PC can run continuously for days and weeks interacting with a chronic experimental preparation while the host PC documents relevant aspects of the information flow between preparation and experimental set-up.
In what follows, we describe some of our experiences with issues during configuration of this set-up. Note that the real-time model is executed at a fixed rate, which requires a fixed-rate solver to be used for the Simulink model. Each step must execute within the time available, if the step-size is smaller than the maximum task execution time (TET), the model will crash. Power save options and USB controllers in the BIOS should be turned off as they can cause TET spikes. A good approach is to start with a large step size of tens of milliseconds and to reduce it gradually, observing the TET log.
A large number of I/O devices are supported by xPC Target natively (see Mathworks documentation). In our example set-up, we are using the United Electronic Industries PD2-MF-64 for analog data acquisition and for digital output. We have also successfully used National Instruments NI6071E and the Measurement Computing DAS6402 in other versions of this set-up (the AD card supplied by MCS is not supported). In order to synchronize data acquisition with the execution steps of the real-time model, model execution should be paced by an interrupted generated by the data acquisition card to execute the next step when the next batch of samples are available for processing.
If required I/O hardware is not supported natively, it is possible to write custom drivers for this in C code. Likewise, if a desired data processing step is inefficient to implement with Simulink blocks, custom C code can be incorporated inside the model (e.g., for signal processing and spike detection). Note that we don't use any of the software supplied by MCS, but have implemented all steps of experimental control, data acquisition, processing, archiving, and visualization in a generic and customizable fashion using Matlab/Simulink. It is however possible to run the MCS data acquisition system and software (MCRack) on a separate PC in parallel to the set-up described here through a signal divider (SD-MEA).
In what follows we demonstrate a number of experimental paradigms that can be realized with the method described. Firstly, we show how spontaneously active multi-channel spike-trains can be monitored while detecting network events in real-time. Raw data in the region of interest before and after the time of threshold crossing are stored for on-line or off-line analysis and categorization. Secondly, we demonstrate how an electric stimulus can be used to interact and interfere with the ongoing activity, both as a “single-shot” and as a modulated stimulus train. These applications were chosen because they are challenging to implement in a generic fashion, and relevant for experimental paradigms aimed at characterization and modulation of complex interactions in neural systems leading to the development of neuro-prosthetic devices.
The spontaneous activity exhibited by a dissociated neuronal culture after 3weeks in vitro consists of intermittent bursts of activity (“network spikes”) lasting around 100ms each that are separated by several seconds of relatively sparse activity. As shown in Figure Figure44 (middle and right panel), the spontaneously occurring multi-channel spike-train patterns of individual network spikes are highly variable and short-lasting, properties that are well suited for demonstrating real-time monitoring and classification capacities.
The real-time target processes 60-channel data at a rate of 11.5MBit/s, generates a trigger if a network event is detected and holds the preceding multi-channel data for analysis. During each 500 μs execution step, a sliding window of variable size, containing spike data history, is available for inclusion in online the analysis. It is important to note that the system is capable of interacting with the ongoing neuronal activity at any point during a network spike using the available preceding data (the build-up phase of the activity) to determine the properties of the feedback signal at a response time resolution of 500μs.
It is a frequent requirement in biomedicine applications to suppress or otherwise counteract an undesired event, such as synchronized brain activity or re-entrant circular cardiac muscle activity. Here also, a dissociated neuronal culture is used as a model system to demonstrate the efficacy of the method in selectively interfering with neuronal activity using selective electric feedback stimulation.
The feedback generating model consists of two distinct stages: in the first stage, the 60 channel signal is continuously monitored, spikes are detected and summed across channels over a sliding window of 25ms. Following an earlier study (Eytan and Marom, 2006), network spike onset was determined by using a threshold of 15 channels that are active within that 25ms time window. A second stage then analyses the spatio-temporal pattern within a variable period of time that begins 50ms before the threshold crossing, and extends into the time envelope of the network spike as required. If a predefined property of the spike-train is detected (in the example of Figure Figure5,5, left panel, the order “1-2-3” was matched) a stimulus is provided at a predefined time in relation to any given temporal reference point (e.g., the network spike threshold crossing or the matching of the spike pattern). This experiment therefore demonstrates the possibility of interfering specifically with a complex activity pattern. The post-stimulus time histogram (Figure (Figure5,5, right panel, inset) shows one particular channel, in which the feedback stimulus reliably evokes actual spikes. This is thus an example for a short latency closed loop paradigm where a particular activity pattern leads to an interference stimulus which in turn evokes specific activity.
In this example, we consider the case where both network activity as well as feedback signals are continuous rather than discrete. Network activity is expressed in terms of number of individual action potentials recorded throughout the electrode array within a 25-ms sliding time window. The amplitude of the feedback stimulus was matched to the level of network activity following an arbitrarily pre-defined function. In providing a non-linear delayed feedback signal we implement an algorithm based on the principles described by Popovych et al. (2006): (1) the instantaneous overall activity of the network is estimated by summing events inside a sliding window, (2) an imaginary part is generated for this signal through a Hilbert transform, (3) the result is squared and multiplied with the delayed complex conjugate of the same signal, (4) the real part of the resulting complex signal is taken to determine the feedback amplitude. To approximate a continuous varying feedback signal, a stimulation of variable magnitude between 100 and 900mV is applied every 20ms.
Since the Simulink programming environment used in the approach presented here is designed for control systems, it is a trivial matter to implement such mathematical signal processing algorithms efficiently (blocks shown in Figure Figure66 left panel, bottom) which would ordinarily require many lines of procedural code. In this example, feedback is generated by setting the stimulus amplitude of a programmable stimulus generator in real-time through a serial interface and a TTL pulse trigger line.
The rapid advances in computer technology over the past years have made it feasible to handle a considerable amount of multi-channel long-term recording data. Comprehensive software toolkits are available for the specific task of analyzing large matrices of spike data post hoc using Mathworks Matlab (Egert et al., 2002). Here, we present a framework that complements this approach with the ability to continuously interact with the experimental preparation while at the same time recording all relevant data for subsequent evaluation. Beyond the ease of integration with post-processing and control scripts, we consider the main strengths of our approach to be its generic applicability, low cost and relative ease of implementation. In this, it stands out among the many alternative closed-loop approaches, ranging from purpose built simulators of nerve cell network components (Marder et al., 1994) through embedded programmable digital signal processor (DSP) architectures such as RT-NEURON all the way to a real-time operating system on standard PC hardware [e.g., real-time Linux (Wagenaar et al., 2005), RT-LAB, and others].
Whereas commercial packages (such as National Instruments LabVIEW) can be easy to get started with, because of the considerable choice and frequently expensive hardware and software requirements it can become a frustrating undertaking when it turns out that the particular approach has inherent limitations regarding its processing capabilities, communication with peripherals, integration with post-processing software, or the availability of interfaces for experimental control scripts. At the other end of the spectrum, custom software for real-time Linux can represent a low-cost and highly flexible alternative. Nevertheless, developing and maintaining low-level real-time code requires a significant investment of time even for experienced engineers.
Here, we are proposing a third, “middle ground”, option, whose high-level real-time software models maintain the flexibility of custom-made solutions while leveraging the capabilities of a well-supported commercial rapid prototyping environment. We have chosen an architecture whereby a self-contained real-time processing and interfacing stage is completely separated from an independent visualization, archiving and experimental control flow and automation stage. Data is continuously streamed from the real-time target to the host system so that spike trains and any events can be logged while the experiment is run continuously for many weeks without any gaps in the data and control stream.
In terms of performance, our approach improves on previous methods in several aspects: (1) The feedback latency and jitter is reduced to a deterministic sub-millisecond resolution while allowing for considerable processing at 10MBit/s data throughput. If necessary, computationally intense operations can be executed in a concurrent task that is paced at a slower rate. (2) Use of the high-level Mathworks Simulink visual signal processing and programming language makes this approach broadly accessible and allows rapid development of complex real-time feedback paradigms. (3) Direct integration with Mathworks Matlab enables full scriptability of experimental control, on-line visualization, archiving and immediate analysis of streamed data.
The comparatively complex example model that we use for demonstration purposes requires approximately 300μs (out of the 500 μs available) to execute one step on a standard PC processor. The data throughput in our example is 60 channels×16kHz acquisition rate which corresponds to 11.5 Mbit/s (at a signal resolution of 12bit). At a reduced number of channels, the model can be paced at a higher rate achieving the same data throughput: 12 channels at a data acquisition rate of 80kHz/channel can be executed at a fixed-rate of 100μs. The performance limits are determined by the data throughput and processing capacity of the target computer hardware (PCI bus and CPU). If a bandwidth beyond that of standard PC hardware is required, one host system can be configured to control multiple targets (industrial rack PC hardware), each processing up to 20 MBit of data per second in sub-millisecond execution steps.
We hope that this demonstration of the relative ease by which versatile dynamic feedback interactions can be realized using standard experimental tools, may lead to the “closed loop” experimental approach being considered by a wide range of scientists in basic and clinical neuroscience.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
The authors wish to thank Eleanora Lyakhov, Vladmir Lyakhov, Volker Gauck, Thomas Ott, Thomas Henneck, Ute Großhennig, Karl-Heinz Boven and Gordon Weast. The work received funding from the Max-Planck Society Minerva Foundation (Christoph Zrenner), the Böhringer-Ingelheim Foundation (Christoph Zrenner), the MFG Baden-Württemberg Foundation (Christoph Zrenner), the German Research Foundation Collaborative Research Centre SFB550 (Peter Thier), the Israel Science Foundation (Shimon Marom), the Technion Lokey Center (Danny Eytan, Shimon Marom) and the Etai Sharon Rambam Atidim Program for Excellence in Research (Danny Eytan).