|Home | About | Journals | Submit | Contact Us | Français|
To evaluate the impact of a computer kiosk intervention on parents’ self-reported safety knowledge and observed child safety seat, smoke alarm use and safe poison storage. To compare self-reported vs. observed behaviors.
A randomized controlled trial with n=720 parents of young children (4mos–5yrs) was conducted in the pediatric emergency department (PED) of a level 1 pediatric trauma center. Enrolled parents received tailored safety information (intervention) or generic information (control) from a computer kiosk after completing a safety assessment. Parents were telephoned 4–6 months after the intervention to assess self-reported safety knowledge and behaviors; in-home observations were made one week after the phone interview for a subset of n=100 randomly selected participants. Positive and negative predictive values (PPVs and NPVs) were compared between the intervention and control groups.
The intervention group had significantly higher smoke alarm (82% vs. 78%) and poison storage (83% vs. 78%) knowledge scores. The intervention group was more likely to report correct child safety seat use (OR=1.36; 95% CI=1.05, 1.77 p=.02). Observed safety behaviors were lower than self-reported use for both groups. No differences were found between groups for PPVs or NPVs.
These results add to the limited literature on the impact of computer tailoring home safety information. Knowledge gains were evident four months post intervention. Discrepancies between observed and self-reported behavior are concerning, as the quality of a tailored intervention depends on the accuracy of participant self-reporting. Improved measures should be developed to encourage accurate reporting of safety behaviors.
In the United States, unintentional injuries are the leading cause of morbidity and mortality for children in all age groups after the age of one, and low-income children in particular are at increased risk.1 Motor vehicle collisions, unintentional poisonings and injuries from house fires are among the leading causes of these injuries and deaths.1 Widely recommended counter measures including use of child safety seats, smoke alarms, and cabinet locks effectively reduce injuries. Unfortunately many families, especially those with low income, do not consistently or properly practice these safety behaviors. 2,3,4,5
Both the American College of Emergency Physicians and the American Academy of Pediatrics have long recognized the importance of injury prevention counseling for parents of young children. 6,7 The pediatric emergency department (PED) provides an excellent venue for counseling while families wait for the child to be treated. 8,9,10,11 Several studies of brief interventions in EDs have yielded promising results from injury prevention counseling. In addition, computer technology can provide patients with highly relevant, real-time specific information without placing additional time burden on health care providers. 12,13,14,15
Computer tailoring has been effective at increasing self-reported outcomes for a variety of health behaviors.16,17,18,19,20,21,22 Previous research has demonstrated the superiority of tailored messages as compared to generic health information in terms of individuals’ engagement with the material, recollection of the information, and behavioral changes 16,23,24,25,26,27
Computer tailored interventions have been used effectively for counseling about the prevention of childhood injuries in both primary care and PED settings. 10,13,23 While this small body of work on computer tailoring in PEDs is promising, two limitations require further study. First, studies of tailoring to improve home safety generally evaluate patient outcomes shortly after the interventions, so little is known about whether they are long lasting. Secondly, such studies rarely address the fundamental concern that tailored feedback is wholly dependent on the validity of responses to the assessment questions used for the tailoring. Therefore, the validity of self-reported behaviors is critically important for both delivery of the intervention and evaluation of the outcomes. Epidemiological studies validating self-reported safety behaviors have mixed results. 27,28,29,30,31 We could find no published studies that examined this issue in the context of a computer-tailored intervention. We had an opportunity to address this gap in the literature by evaluating the effect of a tailored report on parents’ self-reported safety behaviors four months after the intervention was delivered, and to validate this reported behavior by conducting home observations on a random subset of study participants. The behaviors of interest were: use of child safety seats, having working smoke alarms, and safe storage of poisons. We hypothesized that the intervention group would have higher safety knowledge and self-reported safety behaviors than the control group. We further hypothesized that the “over reporting,” that is, the tendency to report safer behaviors would be higher in the intervention group as compared to the control group.
A randomized controlled trial of a computer tailored injury prevention program was conducted in the waiting area of a level 1 pediatric trauma center. A computer kiosk was used to randomly assign participants to study groups, collect baseline data and generate tailored or generic reports based on responses to safety assessment items. The kiosk was free-standing in a corner of the pediatric emergency room. Participants self-administered the assessment via a touch screen. No participants had difficulty working the computer. On average the assessment took 12 minutes to complete. The assessment was written at the 6th grade reading level so as to accommodate participants with low literacy skills. The intervention was planned and implemented to minimize the interruption of flow of the ED. Participants generally completed the assessment prior to being called back from the waiting room. In instances when participants were called back they were able to complete the assessment upon completion of their visit. The intervention group received a personalized and stage-tailored safety report and the control group received a personalized, but otherwise generic report on other child health topics. Telephone follow-up interviews were conducted 2–4 weeks and 4–6 months after enrollment. Home visits were completed for a randomly selected subset (n=100) of parents who completed the 4–6 month follow-up interview (due to cost constraints, home visits were not possible for all participants). Details of the study methods and results from the two-week outcome analysis have been previously reported.10 Here we provide a summary of the study methods, the outcomes at the 4–6 month follow-up, and the results of the validation subset. The study was approved by the Johns Hopkins Bloomberg School of Public Health’s Institutional Review Board.
Participants were recruited from September 2004 through December 2005 during the PED’s 12 busiest weekly shifts. Triage sheets were used to identify age-eligible children and to avoid approaching the parent of any child whose visit was noted with suspicion of child abuse or neglect. Eligible parents or guardians had to be English-speaking; have a child between 4 months and 66 months of age seeking treatment for any injury or medical complaint, or report having an age-appropriate sibling of the child being seen; live in Baltimore City; and live with the child at least most of the time. Only one child from each family was included. Parents of critical patients were not included in this sample because they bypass they emergency department waiting area were recruitment and assessment took place. Parents of children whose visit was noted with suspicion of child abuse or neglect were not approached for participation.
Following confirmation of eligibility and informed consent, a study recruiter escorted participants to the computer kiosk, where a random number generation program in FileMaker Pro® assigned them to the intervention or control group. All participants then completed an assessment of their current practices and beliefs (focused on safety for the intervention group and general child health for the control group), then received parent reports that were printed at the kiosk. Participants received $10 at the time of enrollment, a $20 gift card by mail after the 4–6 mos telephone interviews, and another $20 gift card at the completion of the home observation.
Estimates for sample size calculations were taken from our previous intervention work, which demonstrated rates of safe poison storage at 10% and working smoke alarms at 80%.2 32 The desired sample size of 375 per study group was based on type I error α at .05 and power of 0.80. For 0.10 ≤ p1 ≤ 0.80 (p1 is the proportion of safe practices in the control group), we can detect differences of ≥10% in p2 (proportion of safe practices in the intervention group) with sufficient protection from type I and type II error.33
The intervention condition -- Safety in Seconds™ program-- drew primarily on the Precaution Adoption Process Model (PAPM), a stage based behavior change theory asserting that individuals move from being unaware of a problem to planning to change before ultimately adopting and maintaining a new behavior.34,35,36 Details of the intervention’s theoretical underpinnings have been described previously;10 here, we summarize its key elements. At the computer kiosk, parents completed a 10–12 minute PAPM stage-based assessment of the three safety behaviors of interest along with other questions used for message tailoring (e.g., sociodemographic characteristics, prevention beliefs, cultural values). Based on responses to the assessment items and the information in the message library created for this project, the computer program printed a personalized, PAPM stage-tailored, four-page safety report immediately at the kiosk.
Control group participants completed an assessment at the kiosk, the same length as that completed by the intervention group. Items included sociodemographic characteristics and questions about four child health-related topics: development, sleep, neighborhood safety, and dog bites. The control group then received a personalized kiosk-generated report, which used the same four-page template and contained generic information on the four child health-related topics. We did not assess the child safety seat, smoke alarm, and poison storage behaviors for the control group at baseline for two reasons. First, the use of a randomized design allowed us to assume equivalence of the two groups. Second, the safety behavior assessments were an inseparable component of the intervention itself (i.e., the assessments determined the message content) and were not a baseline measure in the traditional sense of a pretest/posttest design. Therefore, we concluded that it was neither necessary nor appropriate to assess the safety behaviors for the control group.
At enrollment, both groups were asked their child’s age and gender, relationship to the child, ethnicity, education, yearly income, and marital status. Per capita income was calculated as total yearly household income divided by total number of individuals supported on that income.
To test knowledge of the information provided in the parent safety report, the 4–6 month follow-up interview included 10 multiple choice and true/false items that were developed and pilot tested with families in the same PED (3 child safety seat items, 3 smoke alarm items, 4 poison storage items). Safety knowledge measures were not collected from either group at enrollment.
At enrollment for the intervention group and at 4–6 month follow-up for both intervention and control groups, a series of PAPM stage-based items asked parents about their awareness and adoption of each of the three safety behaviors. Responses used a staging algorithm to categorize participants into PAPM stages. Based on the distributions of these ordinal variables in the total combined sample, the data were collapsed into the following variables for analysis:
During the home visits, the location and functionality of all smoke alarms were recorded. Data collectors asked to see any locked places the family used to store medications and household items. Reported storage places were observed and locking status was recorded. Data collectors also asked to see the index child’s car safety seat and noted the presence and type of car seat. Observations were coded as safe following the same definitions as provided above for the reported measures of smoke alarms and poison storage. We were unable to observe enough child safety seats to complete the analysis for this safety behavior because families often reported that the child safety seat was in a vehicle that was not available for inspection at the time of the observation.
To check randomization and equivalence of the groups at the 4–6 month follow-up, we compared sociodemographic characteristics between the intervention and control groups at enrollment and at the follow up interview and between the home observation sub-sample and the follow-up sample. Knowledge outcomes were compared between study groups using t-tests of the total mean percent correct scores for each safety topic area and for all ten items. For the self-reported behavioral outcome analyses, we used ordinal regression for the quartiles of child safety seat use and logistic regression for the dichotomous poison storage and smoke alarm use variables. For the comparisons between reported and observed behaviors, we calculated positive predictive value (PPV) and negative predictive values (NPV) for smoke alarms and poison storage. PPV was calculated as the proportion of those who reported the safe behavior that were observed to be safe and the NPV was calculated as the proportion of those who reported the unsafe behavior that were observed to be unsafe. Fisher’s exact test was used to determine whether these values differed between the intervention and control groups.
Figure 1 describes recruitment, enrollment, and 4–6 month follow-up results. Of the 1,412 parents with age-eligible children who were approached, 239 (17%) were ineligible, 201 (14%) refused to participate, and 69 (5%) were missed by the recruiters due to limited time or multiple patients presenting at the same time. No significant differences were found between those who enrolled and those who refused in child’s age or reason for visit, which were the only data available for comparison. A total of 901 parents were enrolled (n=448 intervention group, n=453 control group). Follow-up rates at 4–6 months were 80% in both the intervention (n=359) and the control group (n=361).
No differences were observed at study enrollment between intervention and control groups on child’s age and sex, reason for ED visit, respondent’s relationship to the study child, ethnicity/race, marital status, employment, mother’s age, and education; nor were there any differences in rates of completing the follow-up interview. (data not shown). Regardless of study group, participants who reported higher income (per capita income >$5000) (37%) at enrollment were more likely to complete the follow-up compared with those with lower income (income <$5000) (63%) (data not shown).
At the follow-up interview, no differences were found between the study groups on sociodemographic variables or reason for visit, and no differences were found between families who received home visits and those who did not on these variables (data not shown). Since there were no differences between study groups, Table 1 presents the sample characteristics for the group that completed follow –up. Children were typically between 1 and 2 years old (42%) with slightly more boys (51%) than girls. The majority of respondents were African-American (93%), mothers (90%), not married (69%), between 20–29 years of age (55%), with a high school degree (74%) and an annual per capita income of <$5,000 (63%). Most PED visits were for medical complaints (72%) rather than an injury.
The intervention group scored significantly higher than the control group on knowledge related to smoke alarms, poison storage, and on the total knowledge score (Table 2). Percent correct scores ranged from a low of 50% for car seat knowledge to a high of 83% for poison storage. The total knowledge scores, while significantly different between groups were less than 75% correct.
The intervention group was significantly more likely to be in a higher PAPM stage for using child safety seats compared with the control group (OR=1.36; 95% CI=1.05, 1.77 p=.02). Although not statistically significant, the odds ratios were in the positive direction for smoke alarms (OR=1.17; 95% CI=.76, 1.79) and poison storage (OR=1.12; 95% CI=.80, 1.57).
Comparison between observed and self -reported behaviors are available for n=98 smoke alarm observation and n=85 poison storage observations. Those not reporting having any poison at the 4–6 month follow-up were not asked if they had a locked place. Table 3 displays the PPV and NPV for smoke alarm and poison storage by study group and for the total sub-sample with home visits. Overall, PPVs ranged from 15% for those who reported having a locked place to store poisons to 44% for those who accurately reported having a working smoke alarm on each level. NPVs ranged from 32% for those who reported not having a working smoke alarm on all levels to 100% for those who reported not having a locked place to store poisons. No statistically significant differences (Fischer’s exact test, p>.05) were found between study groups for PPV and NPV for either smoke alarms or poison storage.
The need for injury prevention counseling in the delivery of health care is well recognized,6,7 and the PED is a promising venue to provide that counseling. 8–11 Claudius et al11 demonstrated improvements to home safety practices via a brief counseling intervention in the PED with 39% of their sample reporting a positive change in the home environment. Their measurement was limited to self-report and success was gauged across multiple behaviors, so it is unknown to what extent our outcomes are similar. Our study adds to the growing body of knowledge that the PED provides a suitable venue for the delivery of prevention education as intervention parents demonstrated knowledge and self-reported behavioral gains 4–6 months post intervention. Statistically significant differences favoring the intervention group were demonstrated for knowledge scores. Although we are encouraged by these knowledge gains, concern remains as substantial percentages of intervention parents remained unaware of important information that was communicated in the parent report including the state’s age requirement for car seat use, and that storing poisons “on a high shelf” was unsafe. Self-reported outcomes were encouraging with significant differences favoring the intervention group demonstrated for child safety seat behavior, and although not statistically significant, the odds ratios were also in the positive direction for smoke alarm and poison storage behaviors.
A strength of our study was the home observations of safety behaviors. PPVs were of concern because only 15% of reported safe poison storage and 44% of reported working smoke alarms were confirmed by observation. However, the intervention group was no more likely than the control group to over-report their safety behaviors. Our findings are similar to those reported by Chen 28 who reported a PPV of 26% for those reporting having a working smoke alarm on every level.28 However, Chen reported a much higher NPV with 85% correctly reporting not having a working smoke alarm on each level.28 The contrast between these findings is interesting as both studies were conducted in East Baltimore. We previously reported on data 37 from another East Baltimore sample that sheds light on the issue of over-reporting. In that study, respondents explained their reasons for over-reporting, which included assuming that alarms were working because they were mounted or because they were “beeping”, and believing that alarms were not needed in a basement or attic. Discrepancies between observed and reported behavior are concerning particularly because parents’ self-reported behaviors were used to create the original intervention report given to parents. The quality of a computer tailored intervention is dependent on accurate reporting. Further research is needed to validate the self-report of behaviors used to create a tailored report.
Moreover, anticipatory guidance and other health care provider counseling is typically driven by similarly self-reported patient information, and our data suggest that providers should not rely on this information alone to decide what they will communicate. Unfortunately, this may lead to delivering more generic advice to everyone which, according to tailoring theory and patient-provider communication literature, 34,38 will undermine the potential impact of the information on behavior change. Clearly, better measures are urgently needed to encourage accurate reporting of safety behaviors.
The strengths of this study include its randomized design, high follow-up rate and direct observation of safety behaviors. Sampling and enrollment were done in a busy PED over a 14-month period to insure a reasonable cross sample of conditions typically seen over many seasons. Follow-up rates were high (80%) in a traditionally difficult to reach, low-income, urban population. We were able to visit a subsample of 100 participant homes to directly observe selected safety behaviors. The generalizability of our results should be restricted to similar types of PEDs serving low-income, urban families.
A potential limitation for the current study is reliance on self-report as a means to determine behavioral outcomes. Although a subsample of 100 homes were visited to directly observe selected safety behaviors, funding constraints did not allow us to directly observe behaviors in all homes. Further, car seat observations were not possible because too few families had car seats available at the time of the follow up visit. The analysis for the subsample with observed behaviors indicated that there was no differential over-reporting between study groups. Thus, while our actual rates may be inflated, the differential between the two study groups may be accurate. Further research is needed to develop measures to increase accurate reporting of behaviors. Without accurate reporting, computer tailoring’s ability to appropriately council users is limited.
Despite these limitations, our results are encouraging for clinical settings given that they were achieved without burdening the health providers. Our findings should generate enthusiasm for using computer technology to provide prevention education in the PED. Further research is needed to examine the cost-benefit ratio of providing a computer tailored interventions in the PED.
This research was funded by a grant to Johns Hopkins Bloomberg School of Public Health from the National Institute of Child Health and Human Development, Grant No. 5RO1 HD042777-03 and a subcontract to the Health Communication Research Laboratory at Saint Louis University. The authors acknowledge the following individuals who contributed to this project: Nancy Weaver, Heather Jacobsen, Matthew Kreuter and Eric Wheetley at St. Louis University; Kari Burgess, Arnell Carr, Akisha Price and Nathan James at Johns Hopkins Bloomberg School of Public Health; and Neil Weinstein at Rutgers University. The authors also gratefully acknowledge the emergency department staff and patients’ parents who participated in this study.
Wendy Shields, Research Associate, Center for Injury Research and Policy, Johns Hopkins Bloomberg School of Public Health, 624 N. Broadway, Room 519, Baltimore, MD 21205.
Eileen McDonald, Johns Hopkins Bloomberg School of Public Health, Center for Injury Research and Policy, Baltimore, Maryland USA.
Lara McKenzie, Center for Injury Research and Policy, The Research Institute at Nationwide Children’s Hospital, 700 Children’s Drive, Columbus, OH 43205 USA.
Mei-Cheng Wang, Johns Hopkins Bloomberg School of Public Health, Department of Biostatistics, Baltimore, Maryland USA.
Allen R. Walker, Johns Hopkins University School of Medicine, Department of Pediatrics, Baltimore, Maryland USA.
Andrea Gielen, Johns Hopkins Bloomberg School of Public Health, Center for Injury Research and Policy, Baltimore, Maryland USA.