|Home | About | Journals | Submit | Contact Us | Français|
A bioptic telescope is a visual aid used by people with impaired vision when driving in many US states, though bioptic driving remains controversial. Objective data on how and when bioptic drivers use the telescope and what they look at with it are crucial to understanding the bioptic telescope's effect on driving. A video-based technique to track the telescope's aiming point is presented in this paper. With three infrared retro-reflective markers pasted on the bioptic spectacles' frame, its movement is recorded using an infrared camera unit with infrared LED illuminators. The angles formed by the three markers are used to calculate the telescope's aiming points, which are registered with road scene images recorded by another camera. The calculation is based on a novel one-time calibration method, in which the light spot from a head-mounted laser pointer projected on a wall while the head is scanning is recorded by the scene camera, in synchronization with the infrared camera. Interpolation is performed within small local regions where no samples were taken. Thus, non-linear interpolation error can be minimized, even for wide-range tracking. Experiments demonstrated that the average error over a 70°×48° field was only 0.86°, with lateral head movement allowed.
With increasing longevity, age-related vision loss is rising. Based on the 2000 US Census, an estimated 2.4 million American adults have low vision (best corrected visual acuity worse than 20/40), typically caused by conditions such as age-related macular degeneration, optic atrophy and diabetic retinopathy . Because of failure to meet the vision screening requirements for licensing, many people lose their driving privileges, which results in restricted mobility and negative impact on the quality of life. Spectacle-mounted bioptic telescopes (Fig.1a) were introduced in the early 1950s as a visual aid for people with impaired vision, and have been used as driving aid since the 1960s. The magnification provided by the telescope compensates for reduced visual acuity. Currently, people with moderately impaired vision can legally drive with a bioptic telescope in 39 US states  and as of 2009 also in the Netherlands. Most of the time, the users view and scan the environment through the carrier spectacle lenses, with an unrestricted field of vision (Fig.1b, left). With a slight downward head tilt, they spot intermittently through the telescope (Fig.1b right and and1c)1c) to read road signs, determine the status of traffic lights, or scan ahead for road hazards . A brief glance (recommended to be 1 to 2 sec.) through the telescope provides the user with the required high-resolution information needed to recognize details.
Although it is a legal option, whether the use of bioptic telescopes results in better (or worse) driving performance remains controversial [4, 5]. Both proponents and opponents agree the bioptic has optical limitations; one of which is an optically-induced ring scotoma. When viewing through the telescope, the magnified field occupies a larger retinal area than the non-magnified field by the factor of the telescope magnification. This creates a blind area or ring scotoma around the field of view through the telescope. It is not known whether accidents involving bioptic drivers occurred when looking through the narrow field of the telescope (which might obscure relevant traffic) (Fig.1c), or occurred because of failure to look through the telescope (which might result in poor perception of important traffic signs or other detailed information). Collecting data on when and what bioptic drivers view through the telescope in actual daily driving activities is crucial to addressing these issues [6-8].
We built an in-car surveillance system to record bioptic drivers' routine driving activities. The system continuously records video of the position of the driver's bioptic telescope and video of the road scene. In addition, the system records “black box” car data such as acceleration, speed, use of brakes and turn signals, and GPS coordinates. Recordings are captured by a trunk-mounted digital video recorder (DVR) for later interpretation. The system can operate unattended for months. Recordings are processed semi-automatically to cull the thousands of hours of recordings and identify segments to be shown to expert driving evaluators to assess the use of the bioptic telescopes and the performance of the drivers. The video recordings can be used to identify head tilt actions, and thus flag the segments incorporating bioptic telescope use. The ability to automatically estimate the telescope's aiming point allows us to show the evaluators the targets the drivers viewed through the telescope. We define the aiming point as the intersection point between the telescope axis and the scene-camera image plane. That is, the telescope aiming point is the image at the center of the magnified field of view of the telescope (Fig.1c). We needed a robust way to determine aiming point based on a calibration procedure that could be performed as simply as possible for each driver's telescope and car, without any need for the driver to take further action each time the car is driven. This paper presents the development and validation of our calibration method for deriving the aiming point.
Our determination of the telescope aiming point on the scene images needs to be accurate enough for a driving evaluator to judge what the driver was viewing through the telescope. Given the recording resolution of our system, features subtending as little as 2° of visual angle can generally be distinguished in the scene camera image, so an aiming point accuracy on the order of 1° would place the calculated aiming point within most features. It is possible, however, for drivers to spot targets anywhere within the telescope's field, typically about 10°-15° in diameter, and if there is more than one target within the field there is no way we can determine which is the intended target. However, image quality falls off with distance from the telescope aiming point, so it is likely that once a target is spotted, the wearer will center the aiming point close to it.
Because studies of normal daily driving usually require minimal attachment of tracking apparatus, video-based tracking methods [9-13] are usually more suitable for head tracking than those based on worn sensors. Feature detection is the first step in video-based techniques. Feature detection can be based on natural facial features [11, 14, 15] or fiducial markers placed on the face or head. Precise tracking of facial features is difficult in general and in bioptic driving situations in particular, and their locations can easily be altered by changing expressions, such as laughing and talking, as well as changes in facial hair and eyewear. We found that most existing face-based approaches either added considerable complexity to our application or were not accurate enough for our bioptic driving study.
Fiducial marker tracking is a simple, robust, and accurate approach that is widely used in many practical applications , for example, NaturalPoint Track IR (NaturalPoint, Inc., Corvallis, OR) for video games. Fiducial markers must be rigidly associated with head position (or, in our case, telescope position) to avoid the same limitations as face-based features. Markers mounted directly on the spectacle frames do not have that problem, and are easily detected automatically. That is our chosen method.
We pasted infrared retro-reflective markers onto the user's spectacle frame. Used together with an infrared (IR) band-pass filter on the camera, the markers' visibility remains stable in the real world driving environment, where illumination and shading may dramatically change. Once markers are detected, it is necessary to determine the relationship between the markers and their position and orientation in space, and from that, determine the aiming point of the telescope. Typically, the user needs to aim his gaze at some indicated calibration points, generally arrayed in a grid across the scene, while at the same time the image of the markers is recorded. Usually, calibration is performed indoors and the calibration points are on a screen. Carefully aiming at numerous points can become tedious and is error prone. A lengthy and cumbersome calibration procedure may be especially difficult for people with impaired vision and advanced age, as in our target population. The number of points needed can be reduced by using multiple cameras and IR illumination, but that increases system cost and complexity . The solution we developed overcomes these limitations.
In our application, it may be difficult to track the driver's eye movements, because the telescope often occludes the camera's view of the eye (Fig.1a). Fortunately, that is not necessary, as we know that when the user is looking through the telescope, the telescope's aiming point is along the telescope axis, and the driver's field of view is constrained to be within the telescope's field. Therefore, in order to determine what the driver is looking at through telescope, it is sufficient to just track and calibrate the aiming of the telescope. Even changes in position of spectacle frame on the face do not affect the tracking, as the users always looks where the telescope points. Because of this, we argue that the system can be calibrated by an experimenter wearing the subject's bioptic telescope, and the calibration will still be valid for the subject bioptic driver. This simplifies the process for the visually impaired subjects and ensures that the calibration is performed properly. We have verified this, as described below.
For most types of bioptic telescopes, the telescope is rigidly mounted in the frame. Thus, tracking the spectacle frame is sufficient. For a frame-mounted type of telescope (such as the Ocutech VHS, lower-right in Fig 1a), there could be some position variability between the frame and telescope after fitting. However, we do not expect the variability to be a problem, as a strong locking mechanism is available, which we can use to secure the telescope if it is used by some subjects in our studies. In addition, in some states this type of telescope is not permitted for driving as it blocks the view of the fellow eye when the user is looking through the telescope.
We developed a novel calibration method that is robust, flexible and trivially easy for the visually-impaired subject, as it can be accomplished without the subject by an experimenter in the subject's car, wearing the subject's telescopic spectacles. Ours is a wide-field calibration method in which a large number of calibration points located across scene field can be easily obtained and applied to estimate the aiming point.
We have developed an in-car recording system  that is comprised of a mobile DVR system that records multiple video channels, including images of the road scene and the driver's head (Fig. 2). A wide-angle camera is mounted forward of the rearview mirror to capture the road and traffic scene. A band-pass IR camera is mounted on the windshield on the driver's side to capture IR-reflective fiducial markers on the driver's spectacle frame, to track the telescope aiming point. A third camera (behavior camera) is mounted on the far right side of the windshield to capture the driver's head and body movements, to aid experimenters in assessing driver actions and confirm the bioptic use (by checking if the iris is aligned with the telescope tube).
Tracking the telescope aiming point is achieved by tracking the frame of the driver's bioptic spectacles. The frame tracking provides information about head movement (to supplement the detection of telescope usage by noting head tilt) as well as the aiming point through the telescope. Actual position of the spectacles on the driver's face is not important, since the driver must be looking through the telescope when using it. Shifting frame position and slips in its location on the nose can be tolerated, as they negligibly affect determining the location of a distant aiming point. Therefore, once the spectacle frame is calibrated in the car by one person, such as the experimenter, the calibration is valid no matter who wears the frame in driving, as long as the camera position remains stable. This means that our tracking method may be calibration-free for the visually impaired subjects in our study, and is not sensitive to minor changes such as driver seat adjustment or postural changes.
In order to acquire reliable video images in actual driving situations, where ambient light levels change dramatically between night and direct sunlight and sharp shadows can appear across the face, we used a near-IR camera, with an 850–900 nm band-pass filter, aimed at the driver's head to capture images. Three retro-reflective markers were pasted on the front of the driver's bioptic spectacle frame (Fig. 3a). These reflectors are illuminated by 890 nm IR LEDs mounted with the camera. This imaging system suppresses changes in the ambient light. The IR illumination is not visible to the driver, and therefore does not interfere with driving. A captured image of a bioptic driver's head and the reflective markers is shown in Fig. 3b. The three markers in these IR images can be detected robustly even under wide variations in lighting conditions. Direct bright sunlight can swamp the marker images, but drivers do not tolerate direct sunlight in their eyes, and use the visors to keep their eyes, and thus the spectacle frame, in shadow. The telescope aim is rigidly linked to the triangle formed by the three markers (Fig. 3c). The shape of the triangle, formed by the three markers, changes with head yaw (left-right) and pitch (up-down), but it is less sensitive to lateral head movement, as the results below show. The size of the triangle can change slightly with distance of the head from the camera, but this does not materially affect pointing direction. The telescope aiming point therefore can be calculated using two angles of the triangle to represent the triangle's shape. With the IR camera located to the left-front of the driver, the captured triangle shape changes with head rotation. Thus, the three markers are used as feature points to track the telescope's aiming point.
In our application, the system in the car has to be simple, yet many calibration points are needed to ensure accuracy over the wide range of head movement common in driving, and where the wide scene image is highly distorted by the wide-angle lens of the scene camera, and the distance from the scene camera to observed objects in real driving situations is highly variable. Because we only need to associate head yaw and pitch values with points in the 2-D scene images, in this situation direct interpolation based on scene images provides a simple solution. For calibration, we mounted a laser pointer on the spectacle temple to project a light spot on a distant wall in view of the scene camera. The light spot's location is arbitrary, but it can easily be located in the scene image by means of image processing. Moving the head to scan over the scene while recording the head and scene images provides a long trail of calibration points. Thus, we easily record a copious number of calibration points across the scene, without involving the actual driver and without requiring repetitive and careful aiming by the experimenter.
As illustrated in Fig. 4, calibration is performed in front of a wall (the “calibration plane”, typically about 5 meters away). The laser pointer is adjusted on the spectacle frame temple so that the laser spot coincides with the telescope aiming point on the wall, thus almost completely compensating for the parallax between the telescope and laser at that distance. During the calibration, the bioptic wearer freely moves his head to scan the scene field with the laser spot. The videos of head and scene are recorded in synchronization. Thus, the light spot position on the scene image and reflective marker positions are paired for each video frame.
Fig.5 illustrates the different shapes of the marker triangle at 9 (out of thousands) different telescope aiming points in the scene image. We used non-linear surface fitting to establish the relationship between aiming points and the triangle shapes determined from the angle pairs (A, C). The inverse of that relationship can then be used, via interpolation, to identify the aiming point from any given angle pair within the field.
Calibration is performed using a calibration plane (wall) at a fixed distance, typically about 5m. The estimate of aiming point location derived for this distance will deviate for objects at different distances due to the lateral parallax between the telescope and scene camera. Because the distance to observed objects in real driving situations is usually unknown, this is a systematic error that cannot be corrected easily. The magnitude and impact of this scene camera parallax error is analyzed in this section.
In the car, the scene camera is mounted forward of the rearview mirror so it does not obstruct the driver's view of the roadway (Fig.2). This installation creates parallax between the bioptic telescope and the scene camera. As illustrated in Fig. 6, for two targets at different distances along the telescope's line of sight, the telescope's aiming direction is the same but their positions in the scene camera's image are different. For example, target B and target T have the same telescope direction (and thus the same recorded marker triangle shape), but their images in the scene camera (Bi and Ti) are separated. Using the mapping relationship between marker angles and scene camera image coordinates that was established with points lying on the calibration plane, this telescope direction is mapped correctly to target B's location in the scene image (Bi). When target is at position T, however, its image in the scene camera (Ti) is displaced from the predicted scene image coordinates. Thus there is an angular error in the identified position of any targets not on the calibration plane. This parallax error can be computed (for targets along a line of sight perpendicular to the calibration plane) as
where s is the lateral distance from the scene camera to the telescope; m is the calibration distance (calibration plane from telescope); n is the distance from telescope to targets that are not on the calibration plane. The forward distance between the telescope and camera (a few tens of cm) is negligible compared to the distance to the targets (many meters), and so it is not included in this approximation.
Fig.7 shows the computed errors for targets at different distances, assuming the calibration distance, m, is 5m and s is 0.3, 0.49, 0.8, or 1m. A scene camera parallax of 0.49m is approximately the case for many vehicles, including the car we have presently instrumented. For that offset, when the calibration distance is 5m and the target distance is 50m, scene camera parallax error is about 4.9°, which would not be acceptable. Since most telescope-relevant targets in driving are at much larger distances than 5m, the impact of parallax error may be reduced by performing the calibration at a farther distance. The calibration plane, however, may not be practically set very far. It is restricted by the availability of a large enough wall to span the wide field of view needed while driving (wider than a highway) and by the need to reflect a laser spot bright enough to be picked up by the scene camera. Although the actual target distance is usually unknown, the trend of the systematic error is known, and thus the estimated aiming point can be biased to reduce the parallax error for the most common target distance. See the “shifted parallax” curve in Fig 7, which is biased for a 20m target distance. It can be seen that after shifting, the error at 50m target distances is only 0.85°.
In addition, the variability in the distance between the head and the scene camera s, which may be due to unpredictable lateral body movement of the driver, will cause random errors in aiming point estimation. Purely lateral movement is supposed to be associated with aiming point shift, but our method can not detect the change because the triangle remains unchanged. However, the error can be ignored in our application, given the confined lateral movement range in cars and the relatively large target distances. For instance, 10 cm lateral movement (most actual movements are likely smaller than that) causes an error smaller than 0.11° for a 50m target distance.
Experiments were conducted in a large meeting hall, where a large calibration wall was available and target distances could be measured easily. Following a calibration procedure performed by one subject at a distance of 5 m from the wall, four experiments were conducted: (a) verification at the calibration distance without lateral head movements (only yaw and pitch); (b) verification at the calibration distance with lateral head movements; (c) estimation of aiming points for targets at distances from 5m to 20m (where the expected errors are largest); (d) estimation of aiming point for other users (using the calibration obtained by the first subject) for targets distances from 5m to 20m. In these experiments, the head and scene video images were 352×240 pixels, and the field of view of the scene camera was 70°×48°.
In addition, we conducted an on-road experiment in a car with a prototype system installed.
The laser pointer was adjusted to bring the laser spot to the middle of the telescope field of view on the calibration wall and was locked in that position. During calibration, the subject moved his head to perform raster scans across the full field of the scene camera horizontally and vertically. The trajectory of the laser light spot derived from the scene image is shown in Fig.8a. The angles A and C of the marker triangle obtained for each of those light spot positions derived from the IR camera signal are shown in Fig.8b.
Based on the collected calibration data, the angles A and C for all image pixels in the scene camera were estimated by linear interpolation. As with any interpolation, this assumes that the mapping between the shape of the triangle formed by the markers and the telescope aiming point in the scene image behaves smoothly locally. Fig.9a shows the results of the interpolation. To calculate the telescope aiming points from angle pairs, a reversed interpolation from the angles (A, C) to scene coordinates (X, Y) is used (Fig.9b). This assumes the inverse is unique, which is evident by inspection of the data.
During this verification, the calibrating subject wore the spectacles with the laser pointer positioned as during calibration, and rotated his head, without lateral shifts, to project the laser spot at different arbitrary regions on the wall. The distance between the coordinates of the laser spot in the scene camera image and the aiming point, calculated from the calibration analysis of the triangle markers in the corresponding IR camera frame, was computed for each frame. Fig. 10 plots the laser spots and the corresponding calculated aiming points in scene camera coordinates. As can be seen, the subject moved his head to 19 arbitrary locations, and he did not hold his head absolutely still at any location. In total, we collected 310 measurement points. The largest error was 2.56°, the smallest error was 0.06°, and the average error was 1.22°.
Lateral body movements can't be avoided in driving. To assess the magnitude of the effect of lateral body movements on the tracking method, the prior experiment was repeated with intentional body movements. For each arbitrary head rotation, the subject attempted to keep looking through the telescope at the same position while he laterally shifted his body left to right, back and forth. The movement range was about 16cm backward and forward and 20cm left and right.
The tracking results are shown in Fig. 11. The largest error was 3.03°, the smallest error was 0.07°, and the average error was 0.86°.
Comparing Fig. 10 and Fig. 11, it is evident that when the bioptic wearer gazed at different targets with lateral body movements a smaller field area was covered with the gaze points. That is likely the reason the average error was slightly smaller in that experiment.
In this experiment, the subject looked through the bioptic telescope at targets at various distances (without a laser pointer). Tracking results are shown in Fig. 12. The estimated aiming point was computationally shifted to compensate for the average scene camera parallax error (as explained in section II D and Fig. 7). Three subjects performed the same experiment, with average errors of 1.29°, 0.75°, and 0.93°. Note that such a compensation shift can make parallax errors for larger distances (expected to be relevant in driving situations) even smaller.
To test that the calibration can be performed by an experimenter for a visually impaired driver, the experiment was conducted with subjects 2 and 3 wearing the same telescope and using the calibration data established by subject 1 (the experimenter). These errors include random alignment error between subjects, as subjects may judge the center of the telescope slightly differently. As anticipated, the calibration data from the experimenter was valid for other subjects, since the same bioptic spectacle frame was tracked, and the camera placements were unchanged. So after calibration is completed by anyone who wears the subject's bioptic spectacles when the recording system is installed in a car, it is a calibration free tracking system for all subsequent use of those spectacles in that car.
We installed the tracking system in a car, using two side-by-side scene cameras to provide a horizontally wide view (88°×33°) while maintaining high image resolution. One day after one of the authors (GL) performed a calibration procedure, he drove through downtown Boston wearing the bioptic telescope, without recalibrating. He spotted through the telescope on several occasions, some of which were while the car was moving. The objects spotted through the telescope were noted. The recorded videos were processed to show the telescope aiming points over the video. All other experimenters were able to determine correctly what the driver looked at in all the noted occasions. Fig. 13 illustrates one of the bioptic use events, when the car stopped at an intersection. The telescope aiming point track is superimposed on the scene image. It clearly shows that the driver was checking the status of the two traffic lights using the telescope.
We have successfully developed and implemented a video surveillance system which records usage of a bioptic telescope by vision-impaired drivers in their own cars and can provide a driving evaluator with the estimated aiming point of the telescope superimposed on a view of the road and traffic scene. A novel calibration procedure was developed that needs to be performed only once with the driver's bioptic spectacles in the driver's car. The procedure is simple enough for low-vision drivers to perform, and we have shown that an experimenter can perform the calibration for the patients. Although the method still leaves some parallax related errors uncorrected, tests and computations showed that the system can achieve its required accuracy within a normal range of driver's head and body movements.
In planned studies, the system will provide valuable objective data to help us better understand how bioptic telescopes are used in normal driving. We expect to establish associations between certain patterns of bioptic use and driving performance (as judged by specialized driving instructors reviewing the recorded data) and, based on those data and analyses, to develop evidence-based training programs. All of this would not be possible if we do not know what bioptic drivers look at through their telescopes and when they use telescopes. In a pilot study using an analog VCR system (with no ability to determine bioptic aiming point) we already found that such data can be valuable. We were able to determine two bioptic drivers' instances and length of time of using the telescope in driving . We found that their actual time viewing through the telescope was 60 times and 4 times less, respectively, than that reported by them in a prior survey study . Such large errors in self-reporting underscore the importance of the objective data collection techniques that our telescope tracking system will facilitate.
Henry Apfelbaum provided valuable help.
This work was supported by NIH grants AG034553 (GL), EY12890 (EP), EY05957 (EP), and the Massachusetts Lions Foundation.
XianPing Fu, Information Science and Technology College, Dalian Maritime University, Dalian, China, and was a Postdoctoral Fellow at the Schepens Eye Research Institute, Harvard Medical School, Boston, MA USA. (phone: 86-0411-84724510; fax: 86-0411-84743077)
Gang Luo, Schepens Eye Research Institute, Harvard Medical School, Boston, MA 02114-2500 USA.
Eli Peli, Schepens Eye Research Institute, Harvard Medical School, Boston, MA 02114-2500 USA.