The behavioral arena used initially to test and develop our system consisted of a 24.5 cm diameter platform with an overhead FireWire camera and infrared lighting (). The software component consists of a tracker for computing fly trajectories from captured digital video (), and a behavior detector, which may be trained from examples (). The system is accurate: the x
position of a fly is estimated with a median error of 0.03 mm (2% of body length), orientation with a median error of 4° (, Supplementary Figs. 1
). Identity errors are absent with minimal user supervision, and occur every 1.5 h·fly-1
in fully automatic mode (see the Methods section, Supplementary Table 1
Figure 1 Walking arena with sample trajectories. (a) Schematic diagram of the walking arena. A 24.5 cm tall printed paper cylinder is backlit by an array of 8 halogen lights (only one shown). At the top is a 1280×1024-pixel camera with 8 mm lens and infrared (more ...)
Figure 2 Tracking algorithm and evaluation. (a) Example frame with the foreground/background classification for pixels within a subwindow. (b) Detection of individual flies We show the connected components of foreground pixels. The purple component corresponds (more ...)
Figure 3 Ethograms of eight automatically-detected behaviors. (a) Examples of behaviors detected (from trajectory in (b)). Triangles indicate the fly’s positions in every frame. A cyan/red triangle is plotted at the start/end of the behavior. For touching (more ...)
To illustrate the potential of using multiple fly trajectories for automated behavior analysis, we carried out three proofs-of-concept. First, we defined automatic detectors for several individual and social behaviors exhibited by flies walking in a circular arena. These detectors were then used to produce ethograms for flies in different gender groupings. To demonstrate that these ethograms are useful descriptions of the flies’ behavior, we used them to accurately classify flies according to gender (male vs. female) and genotype (wild type vs. fruitless
). The Fruitless protein is a transcription factor that plays a role in the sex determination pathway in flies. Male fruitless
mutants exhibit several behavioral abnormalities, including inter-male courtship chains. Second, we quantified differences in the behavior of individuals within a population, and show that those differences are stable throughout each trial. Third, we examined the spatial distributions of the relative positions of flies during social interactions. We compared the distributions for pairs of flies of the same and different sex, as well as for male fruitless
mutants. All analyses described below were derived from 17 30 minute trials, each containing 20 flies, for a total of 170 fly-hours. Four trials used only females, six only male, five were half-male and half-female, and two used fru1
male flies. Examples of each of the four trial types are provided in Supplementary Videos 1
We created automatic detectors for eight behaviors with a wide range of sequence durations, velocities, and accelerations (, Supplementary Video 5
, Supplementary Table 2
). These behaviors represented the majority of the flies’ actions in our circular arena. Most detectors were trained from a few labeled examples as described in the Methods section. The software is user-friendly, and detectors for new behaviors can be created without additional programming. Six of the behaviors involve basic locomotor actions, and two of the behaviors relate to social interactions between flies. Most of the time the flies either walked at a relatively constant velocity (walk
) or stopped in place (stop
). The next-most common behavior was the sharp turn
, in which a fly made a large, rapid change in orientation. Other locomotor classifications included crabwalks
, in which the fly walked with a substantial sideways component, and backups
, in which the flies’ translational velocity was negative. Jumps
consisted of rapid translations within the arena. A touch
occurred when the head of one fly came in contact with another fly. Chases
were cases in which one fly (always a male) followed another across the arena. An automatic detector for a given behavior (e.g. the walk detector) inputs the trajectory for an individual fly () (or pair of flies, for social behaviors), derives per-frame statistics such as the translational speed, angular speed, or distance to the second fly (for social behaviors), then segments the trajectory into bouts in which the fly is and is not performing the given behavior ().
By collecting the statistics of these eight behaviors into a vector, we created ethograms: rich, quantitative descriptions of each individual fly’s behavior. For each fly, we computed one such description, consisting of the frequency with which each individual fly performed each behavior (we explore other descriptions, the fraction of time a fly performs a behavior and mean behavior duration in Supplementary Fig. 3
). To visualize differences among female, male, and male fru1
flies, we grouped the flies by type, and displayed frequency in pseudocolor (). Inspection of this ‘behavioral microarray’ suggests that the behavioral vectors of female, male, and fru1
male flies differ in a consistent way. We quantified these differences by computing the mean and standard error behavior vectors for each type of fly (Supplementary Fig. 4
To demonstrate that these ethograms are powerful descriptors of behavior, we tested whether we could predict the sex of a fly (male vs. female) and its genotype (wild type males vs. fru1/fru1 male), based solely on components of the automatically-generated behavioral vector (). We found that predictors based on the statistics of each of the eight behaviors independently distinguished sex with accuracies all better than chance, with touch frequency performing best (96.8% accuracy), and sharp turn frequency performing best of the locomotor behaviors (83.9% accuracy). A predictor based on the combination of all behaviors had an accuracy of 96.9%. Even a predictor based solely on locomotor behaviors (excluding touches and chases) predicted sex with an accuracy of 95.5%. We emphasize that we are not advocating using behavioral statistics for sexing flies. Our mixed-sex trials (Figs. and ) used a fly’s median image area for determining sex, a technique that achieves 96.2% accuracy. Instead, these behavior prediction accuracies are evidence that the ethograms are strongly correlated with gender.
Figure 4 Differences within and among individual flies. (a) The first and second halves of trajectories for three male and three female flies from the same trial. (b) Scatter plots of walking statistics from each individual fly in the first 15 minutes of its trajectory (more ...)
Figure 5 Spatial analysis of social interactions. (a) Normalized histogram of inter-fly distances. We show a histogram of the distance to the nearest fly for each fly in each frame. Each line corresponds to a different condition, as indicated. For example, the (more ...)
Predictors of genotype (wild type vs. fru1
males) were even more robust (). Frequency of backups achieved the best performance (99.3% accuracy). Using all behaviors or all locomotor behaviors, fruitless
males could be classified with 100% accuracy. This technique of behavioral profiling could easily be extended to include more behaviors or more features of each behavior (see Supplementary Note
Behavioral variation between and within individuals
We observed that the trajectories of individual flies look qualitatively different (). For example, some flies traveled more than others, and some spent a larger fraction of time near the arena wall. Because our algorithm keeps track of each fly’s trajectory, we can easily gather data on a large number of flies and explore statistical differences in behavior across individuals. To this end, we computed behavioral statistics separately for the first 15 minutes and the second 15 minutes of each 30 minute trial and calculated the correlation between the two halves. We considered three statistics of locomotor behavior: the mean speed during walking episodes, the fraction of frames the fly was classified as walking, and the mean duration of walking episodes (). The correlation between the first- and second-half statistics was significant and positive for all three walking metrics, indicating that individuals maintained behavioral tendencies throughout the 30 minutes trials. Thus, although within the tested strain of wild type flies we found large and significant differences in walking behavior, each individual walks consistently over time.
We also investigated whether there were consistent differences in chasing behavior across individual flies during a 30 minutes trial. For the first- and second-half of each trial, we computed the frequency with which a fly begins chasing another fly, the frequency with which other flies begin chasing a given fly, and the mean time duration of chase sequences initiated by a given fly (). As with the walking experiments, we computed the correlation between behavioral statistics gathered during the first and second half of each trial. We found small, but significant, positive correlations for frequency of chasing and frequency of being chased, but no significant correlation for duration of chase sequences.
Gender differences and fly-fly interactions
Because our data consisted of the location and orientation of all individuals at all times, we could examine the spatial distributions of the relative positions of flies during social interactions. We compared the distributions of inter-fly distances for different gender pairings in single- and mixed-sex trials (e.g. male-to-male distance in mixed-sex trial) (). For a control, we created a semi-synthetic data set by artificially staggering in time all 20 trajectories relative to one-another (the first fly’s trajectory was left unchanged, but the second fly’s trajectory was shifted in time so that it started at t = 1.5 minutes, with the last 1.5 minutes of its original trajectory wrapped around to fill the time from t = 0 to t = 1.5 minutes. The third flies’ trajectory was then shifted by 3 minutes, the fourth by 4.5 minutes, etc.). These data approximate trajectories in which the flies do not interact.
The peaks in the male-to-male and male-to-female distributions compared to the synthetic data indicate that males actively approach other flies to a distance of 2.5 - 3.5 mm. In addition, the relatively low frequency of close interactions (< 4 mm) between females suggest that they maintain a larger buffer between themselves. These findings are robust across trial type (e.g. males approach other males as closely in mixed-sex arenas as in single-sex arenas). We also observed that the flies’ centroids never move within 1.5 mm of each other, which is expected given this distance roughly corresponds to a fly’s body width.
To further explore spatial differences during social interactions, we created a new behavioral classification termed ‘encounter’ describing those trajectory intervals in which the distance between a pair of flies was less than 10 mm. For each encounter, we computed the relative location of one fly in the coordinate system of the other at the time when the distance between them was minimal. We computed histograms of these relative locations over all encounters of each gender pairing and trial type (). These histograms are consistent with our qualitative knowledge of courtship behavior. For interactions involving males, the majority of the encounters occur very near the other fly, when the flies are almost in direct contact. In contrast, the relative locations of the female-female encounters are more diffuse. It is apparent from the forward hot spots in that males often take a position so that another fly is right in front of them, an orientation that is consistent with their chasing behavior. Conversely, a hot spot is visible directly behind females in mixed-sex trials, indicating that they are being chased by males. Interestingly, two hotspots are apparent in the encounter histograms of fru1
males (), indicating a social phenotype that is intermediate between that of males and females. The data in this figure represent a quantitative and reproducible measure of the chaining phenotype that is characteristic of many male fruitless