Three prototypes of the asthma kiosk were tested over eight months. In-depth, semistructured qualitative interviews were conducted one-on-one with parents of children with asthma at the beginning of the development process to review word choice regarding symptom description, medication use, and delivery devices to ensure that semantic content of the computer-based interview would match parents' mental models. Content that had not been validated previously in Spanish was subject to translation and retranslation to ensure accuracy. The final version of the system underwent formative evaluation in the pediatric ED setting of Children's Hospital Boston.
Formative evaluation of the asthma kiosk included multiple methods of data collection and analysis: (1) evaluation of parents' time to complete kiosk, (2) parents' completion of written questionnaire to elicit summary judgments of usability, (3) analysis of parents' kiosk entries and comparison to physicians' electronic medical records to describe the potential value of electronically entered data, (4) use of the NASA task load inventory to ascertain what domains of task load impacted parents' use of the system, (5) semistructured qualitative interviews with parents to explore their perceptions of the automated health message produced by the kiosk.
Subject Recruitment and Study Protocol
A bilingual research assistant approached parents of children with asthma between the ages of 1 year and 11 years of age who reported verbally that their child took at least one medication for asthma. Parents were recruited in the pediatric ED of an urban tertiary care Children's Hospital. A total of 66 parents from an eligible sample pool of 105 parents (62.8%) were recruited to test the system. One parent did not use the computer after enrollment. A system update performed midtrial induced an inadvertent, catastrophic data loss on 16 subjects, leaving 49 parents' kiosk entries available for analysis. details the demographic characteristics of the 49 parents with complete data.
Subjects' Demographic Characteristics and Technology Background
Parents independently used the asthma kiosk while in the ED, either at nursing triage or in a treatment room. Data entered onto the asthma kiosk was not shared with ED physicians and nurses during this phase of evaluation. All parents completed a written questionnaire that included items on demographics, self-efficacy regarding asthma care, previous technology experience, and judgments as to the usefulness and ease of use of the asthma kiosk.
Time to Completion of Kiosk
Forty-six of 49 parents (93.9%) were able to complete the kiosk interview during their ED visit. Mean time for completion was 11.8 minutes with an SD of 5.2 minutes. The range of time to completion was 4.7 minutes up to 25.9 minutes. Time was recorded beginning with the parents viewing the first introductory screen and ending with the parent completing the last question screen on the kiosk. This temporal data may overestimate the actual time on task spent by the parent as the total time could include periods wherein the parent may have ignored the computer to attend to other tasks.
The number of medications reported ranged from 1 to 5 with a median and mode of 2. Total time using the kiosk was independent of number of medications reported (analysis of variance [ANOVA] test using 4 DF, F value of 0.08, p = 0.98).
Parents' Summary Evaluation of the Kiosk
The 65 parents who used the kiosk system to enter information about their child were asked to endorse or reject several value statements specific to the technology. Of the 65 parents, 62 (95.3%) agreed that the entering data on the kiosk “was a good use of their time.” Nearly all subjects, 61 of 65 parents (93.8%), agreed that entering data on the kiosk “was a good use of their knowledge.” Fifty-seven of 65 parents (87.7%) agreed that a parent could enter information using the kiosk “as well as a doctor or nurse could.”
Parents were asked to compare their experience with the asthma kiosk to two other activities: (1) using a bank ATM and (2) writing down information on a piece of paper. Each question allowed for responses across a 5-point Likert scale from “much harder” to “much easier.” Forty-four of 61 parents (72.1%) rated the asthma kiosk as easier/much easier to use compared with a bank ATM. Only 4 of 61 (6.6%) parents reported the asthma kiosk as harder to use than a bank ATM. Fifty-six of 63 parents (88.0%) rated using the asthma kiosk as easier/much easier than writing down the same information. Only two parents of the 63 (3.3%) rated using the asthma kiosk as more difficult than the act of writing.
Task Load Assessment
A consecutive series of 12 parents from the study sample completed the NASA Task Load Index. The NASA Task Load Index informs users' experience across five relevant factors: (1) mental demand, (2) physical demand, (3) time demand, (4) effort, and (5) frustration. A factor's weight is determined by responses to pairwise comparisons asking which one was more important during the subject's experience with the technology. Numerical ratings for each factor are generated through a visual analog scale anchored by bipolar descriptors (high/low). An overall workload score per subject was created by multiplying each numerical rating by the weight given to that factor by that subject and then dividing the total by the sum of the weights. Weights for the factors comprising task load were summarized across all subjects to determine those factors that affected users the most.
The five factors in rank order of importance were (crude summary score in parentheses): 1. mental demand (37), 2. effort (31), 3. time demand (28), 4. physical demand (18), and 5. frustration (6). Overall workload demonstrated a wide range of perceived burden (summary scores ranged from 4.5 to 90.1 [mean, 51; SD, 24.2]). The two parents who reported the highest task load burden had both used an ATM in the last month and reported being “very comfortable” with using an ATM.
System's Ability to Collect and Analyze Parents' Data in Context of Guidelines
outlines how parents' responses regarding children's chronic disease severity mapped to the standard terminology used in the National Asthma Education and Prevention Program (NAEPP) guidelines.5
We accounted for potential uncertainty in parents' report within the rule-based schema. The summary equation that produces a severity rank assigns a higher weight to nighttime symptoms given the importance that the NAEPP guidelines place on this topic. We conservatively interpreted the response categories of “few” and “some” by assuming that “few” indicated a frequency less than 2, and that “some” indicated a frequency greater than 2.
Summary of how parents' report of symptoms maps to terminology of NAEPP guideline.
We examined data entered by the 46 parents who completed the kiosk tasks to determine the system's ability to produce meaningful and “actionable” output. The asthma kiosk was able to generate a disease severity classification for 45 of 46 children (97.8%). The parents' report of current controller medications was compared with the classification assigned to their children by the system's rules. We considered medications to be controllers if they matched a specific medication defined as such within the NAEPP guidelines. Twelve of 46 children (26.1%) were noted to have a disease classification of mild persistent disease or higher but did not currently take a controller medication as reported by the parent. By guideline criteria, this represents a population of children whose care requires action, a “step-up” to better control the disease.
The electronic medical records for the study cohort were abstracted and evaluated for the presence of documentation regarding chronic severity. Forty-three of 46 records were available for review. Only 2 of 43 records (4.6%) documented the level of chronic severity. Of the 12 patients identified by the asthma kiosk to be in need of a “step-up” in chronic control, only 1 of 12 medical records (8.3%) noted the degree of chronic severity. Documentation for this one subject noted a chronic severity of “mild intermittent asthma,” which contrasts with the kiosk-endorsed chronic severity of moderate persistent disease.
The existence and use of a written asthma plan has demonstrated a protective effect for asthma children for the outcome of hospital admission.19
The Care Needs and Home Assessment
module for the asthma kiosk included questions specific to this issue. Eighteen of 46 parents (39.1%) stated that they had not received a written asthma plan from the primary care doctor. Of the 27 parents who reported having such a plan in the past, only 16 (34.8% of the total sample of 46 parents) reported having a current copy.
Peak flow meters are recommended for adjunctive monitoring of obstructive patterns of respiration for children older than 7 years with persistent asthma symptoms.5
The Care Needs and Home Assessment
module queried parents on whether they had a peak flow meter at home for use in disease monitoring. Twenty-one children older than 7 years were part of the study cohort. Eleven of 21 parents (52.4%) reported the absence of this device despite their child experiencing persistent symptoms of asthma.
Modifiable environmental triggers for asthma include tobacco smoke, dust mites, pet dander, and mold. The Care Needs and Home Assessment module asked parents about these triggers and the potential applicability to their home environments. Of the 46 parents who completed the kiosk, 22 of 46 (47.8%) reported at least one modifiable environmental trigger that could be a target for educational interventions. Physician awareness of environmental triggers was only documented in 3 of 22 charts (13.6%) for patients in whom the issue was a present-tense issue for disease management.
Parents' Comprehension and Reactions to Output Messages Created by the Kiosk
A consecutive series of 14 parents within the study sample completed a semistructured qualitative interview with a bilingual research assistant immediately after their completion of the kiosk. This interview elicited parents' perceptions of the content within the automated health message. The parent was first asked to review the paper output and then momentarily set the paper aside. The parent then was asked to share their overall understanding—“Tell me what you think the message is.” Subsequently, the parent was invited to pick up the paper and use it to respond to three other questions: (1) What topics within the message did the parent perceive as important? (2) Were any words or sentences within the message confusing? (3) Did the parent disagree with any aspect of the message?
Parents' summary perceptions endorsed the system's overall goal of sharing a positive, action-oriented message. Ten of 14 parents reported that the message encouraged action, “how to” do something about their child's disease. Three of 14 parents focused on medication-specific aspects of the message. Eight of 14 parents noted topics of control and/or prevention as part of what they perceived the “take home” message to be.
No parent reported difficulty with words or sentences included as part of the message. One parent voiced disagreement with the tailored output for her child, taking issue with the message that her child was not on any controller medications. As this parent reported albuterol, a rescue B2 agonist drug, as the only medication, the output message was correct but perceived as wrong by the parent who thought that albuterol was intended to control asthma. This case highlights a “teachable moment” that arises from the collaborative technology-based approach, as this parent's review of output exposed a misperception in how a medication actually works.