|Home | About | Journals | Submit | Contact Us | Français|
To determine if remote administration of the Unified Batten Disease Rating Scale (UBDRS) Physical Impairment subscale by telemedicine is reliable and feasible across a broad range of disease severity.
For the majority (n = 10) of subjects, the examination was performed by a nonphysician who had been trained to perform the examination but not to score the subjects. A trained rater scored the subjects via live video; a second trained rater performed a separate examination in person and scored that examination. For 3 telemedicine evaluations, examinations were performed and scored by a trained rater while a second trained rater simultaneously scored the subjects via live video. Reliability was determined by intraclass correlation coefficient (ICC).
Subjects (n = 13) represented a wide range of disease severity. Remote administration of the UBDRS Physical Impairment subscale had high interrater reliability across all subjects (ICC = 0.94). When only the subjects (n = 10) who had been examined by the nonphysician and scored remotely were included in the analysis, the reliability was unchanged (ICC = 0.95).
The UBDRS Physical Impairment subscale is reliable and feasible for remote administration. Telemedicine has the potential to be a useful tool in rare neurologic disease research and clinical assessment.
Juvenile neuronal ceroid lipofuscinosis (JNCL; Batten disease; CLN3 disease) is an autosomal recessive neurodegenerative disorder of childhood. The disease course is characterized by vision loss, seizures, dementia, behavioral difficulties, and motor impairment. Symptoms typically begin around age 5 years and the disease progresses to severe disability and then death in the late second or third decade of life.1–3
We have characterized the clinical and neurobehavioral aspects of JNCL and designed the Unified Batten Disease Rating Scale (UBDRS) to assess multiple aspects of JNCL and its progression.4–6 The UBDRS encompasses 4 subscales: physical impairment, seizures, behavior, and functional capability. For physical impairment, the UBDRS has required in-person assessment. Because JNCL is rare (0.7–7 per 100,000 live births)7 and affected individuals are spread throughout the United States, participating families often travel long distances for evaluation. This travel imposes substantial burdens on families, including lost time at work, travel with a physically disabled blind child, and cost. A feasible and reliable method for remote physical assessment could reduce these burdens.
We hypothesized that telemedicine administration of the UBDRS Physical Impairment subscale would be reliable across a broad range of disease severity. We further hypothesized that remote sites could employ a nonphysician to perform the examination with a trained rater observing and scoring without loss of reliability.
Subjects were recruited sequentially over a 5-month period at the University of Rochester Batten Center (URBC) or at the 2010 annual meeting of the Batten Disease Support and Research Association (BDSRA). All subjects had clinical characteristics of JNCL1 and had mutations in both alleles of CLN3.8
The study was approved by the University of Rochester Research Subjects Review Board; informed consent was obtained for each subject.
Of the 4 UBDRS subscales, only the Physical Impairment subscale relies on direct physical examination. Therefore, remote assessment was limited to this subscale. The Physical Impairment subscale includes the following items, each scored from 0 (normal) to 4 (severe): speech clarity, abnormal repetitive speech sounds, tongue protrusion, visual acuity, neck and extremity tone, extremity strength, hand taps, upper extremity dystonia, normal spontaneous movements, gait, retropulsion pull test, heel stomps, motor tics or stereotypies, myoclonus, rest tremor, action tremor, finger-to-nose dysmetria, and appendicular chorea. The total score ranges from 0 to 84.
A Logitech camera was used to record these sessions. Either Polycom PVX or video chat software was used via wireless Internet for real-time video. Ratings were performed by 2 physician raters (E.F.A. and J.W.M.) who were previously trained to an interrater reliability on the UBDRS Physical Impairment subscale of greater than 0.90. A medical student (J.C.) was trained to perform the examination but not to score the UBDRS. Standardized training was provided by the developers of the UBDRS.4 For 10 subjects, J.C. performed the examination while E.F.A. scored the subjects via live video. Within 30 minutes of the video evaluation, J.W.M. performed an in-person examination and scored the subjects. For 3 evaluations, J.W.M. performed the examination and scored the subjects while E.F.A. independently scored the subjects simultaneously via live video.
To determine interrater reliability, total scores for the Physical Impairment subscale were treated as continuous variables. Interrater reliability was then determined by intraclass correlation coefficient (ICC) (Mater Medical Research Institute). We expected one element of the Physical Impairment subscale, muscle tone, to be difficult to score remotely because it often relies on the examiner actively moving the subject. Therefore, a separate analysis was performed with muscle tone eliminated from the total score.
Thirteen total subjects completed the UBDRS Physical Impairment assessment using telemedicine. Subject demographics are reported in the table.
The subjects represented a wide spectrum of disease severity (figure). Remote administration of the UBDRS Physical Impairment subscale had high interrater reliability across all subjects (ICC = 0.94). When only the subjects (n = 10) who had been examined by the nonphysician and scored remotely were included in the analysis, the reliability was unchanged (ICC = 0.95). When tone scores were excluded from the analysis, interrater agreement was less (ICC = 0.86). Thus, including tone in the assessment did not decrease the reliability of remote administration.
Telemedicine administration of the UBDRS was feasible. Remote assessment took approximately 20 minutes, which is similar to the length of time required for in-person evaluation. Extra time was needed for camera adjustment and occasional repetition of tests. However, this was well-tolerated by subjects and their families.
The UBDRS Physical Impairment subscale can be administered remotely and reliably even when the examination is performed by a nonphysician. Thus, the physical presence of a trained rater at the examination site is not required. The results demonstrate reliability across a broad disease severity spectrum. We anticipate the ability to employ telemedicine for UBDRS evaluation in all children with JNCL.
The reliability and feasibility of UBDRS Physical Impairment subscale administration via telemedicine has important implications. First, it allows for formal evaluation of disease severity in children with JNCL by expert physicians without requiring travel. Second, it provides a convenient method for outcomes assessment in clinical trials. Third, it suggests that a telemedicine approach to quantitative clinical assessment may be useful for research in other rare neurologic diseases. Reliable use of telemedicine has been demonstrated previously in the clinical evaluation of patients with Parkinson disease using the Unified Parkinson's Disease Rating Scale.9 Our data show that telemedicine is reliable and feasible in a rare disease clinical research setting as well.
The burden of administration was minimal. The time commitment was approximately that required for in-person ratings. Training the nonphysician examiner required minimal time investment, consisting of the examiner observing an evaluation and being provided with a training manual to use for subsequent evaluations. Furthermore, the technology was inexpensive. The only cost incurred was for the camera, although many computers now come with installed Webcams. There is a variety of inexpensive or free services that can be used to transmit live video via Internet.
One potential limitation of this study was the small sample size, which is to be expected given the rarity of this disease. However, even when considering only the 10 subjects examined by the nonphysician, the study had remarkably high power. The observed correlation had 99% power at the p < 0.05 level. Another potential limitation is that we only evaluated subjects who were able to travel, so we have not proven that remote administration is reliable in subjects with very advanced disease precluding travel. Nonetheless, one subject had a score representing 71% of the maximum severity UBDRS physical impairment, suggesting that inclusion of severely affected individuals is not likely to reduce reliability substantially. Finally, our data are limited to the UBDRS Physical Impairment subscale. The other subscales do not require direct interaction with the subject, so we did not specifically test the reliability of remote administration of the Seizure, Behavior, or Functional Capability subscales.
Incorporation of telemedicine into JNCL research will allow the expansion of current subject registries by adding quantitative clinical data. This will enhance our understanding of the disease, especially at its end stage. It also provides the potential to enhance enrollment in future clinical trials because the need for frequent travel to a distant research site would be reduced. The implications of these results extend beyond Batten disease to other rare neurodegenerative diseases, whether of childhood or adult onset. Telemedicine technology can increase access of patients with rare diseases to expert clinicians and to future clinical trials.
The authors thank Paul Rothberg, PhD, for his contributions to genetic data; and the patients and families.
J. Cialone: drafting/revising the manuscript for content, including medical writing for content; study concept or design; analysis or interpretation of data; acquisition of data; statistical analysis. E.F. Augustine: drafting/revising the manuscript for content, including medical writing for content; study concept or design; acquisition of data. N. Newhouse: drafting/revising the manuscript for content, including medical writing for content; study concept or design; acquisition of data; study supervision or coordination. A. Vierhile: drafting/revising the manuscript for content, including medical writing for content; study supervision or coordination. F.J. Marshall: drafting/revising the manuscript for content, including medical writing for content; study concept or design. J.W. Mink: drafting/revising the manuscript for content, including medical writing for content; study concept or design; analysis or interpretation of data; acquisition of data; statistical analysis; study supervision or coordination.
J. Cialone has received research support from the NIH/NCRR. Dr. Augustine has received funding from the NIH/NINDS, the FDA, and the Tourette Syndrome Association. N. Newhouse and A. Vierhile report no disclosures. Dr. Marshall serves on a data safety monitoring board (DSMB) for and has received funding for travel from Toyama Chemical Co., Ltd.; serves on the editorial board of the European Journal of Neurology; and receives research support from Medivation, Inc., Teva Pharmaceutical Industries Ltd., St. Jude Medical, Inc./Advanced Neuromodulation Systems, Inc., the NIH/NINDS, the U.S. Veterans Administration (DSMB), the BDSRA, the High-Q Foundation, and the Michael J Fox Foundation for Parkinson's Research. Dr. Mink serves as an Associate Editor of Neurology® and on the editorial boards of Journal of Child Neurology and Pediatric Neurology; and receives research support from the NIH, the CDC, and the Batten Disease Support and Research Association.