Using scenario-based testing with participants unfamiliar with the system, we found numerous problems with terminology, portal navigation, task completion, satisfaction, and ease of use. These had not been uncovered in our previous heuristic usability testing or in focus groups and questionnaire feedback from parents who were portal users. Despite the difficulties they encountered, participants had many positive perceptions about the system.
Medical terminology was a significant obstacle for participants, which could be because participants are not used to having to decipher medical terms by themselves. Our findings are consistent with past research. In studies by Pyper et al. 16
and De Clercq et al., 10
participants also had problems with jargon even when they had experience with medical terminology. Similar to our experienced parents, the research by De Clercq et al. revealed that even patients who are regularly exposed to medical terms in office visits with their doctor did not comprehend terminology when they encountered it in their health records. Tran et al. also reported difficulties with jargon in their initial evaluation of a prototype portal. They redeveloped the PHR to include a “user-centered vocabulary. 22
” We attempted to reduce jargon and improve explanations during development, although our final testing demonstrated some ongoing problems. In contrast, participants in a study by Cimino et al. 4
did not have difficulty understanding the medical information. However, the participants all had a college level education or higher and used a computer daily. Therefore, they might have more knowledge of what the medical terms mean or be able to find the definitions of these terms more quickly on the Internet.
Adding landing pages and drop down tabs helped address the issues raised. Other studies have demonstrated that patients who use electronic health records have specific preferences for layout and navigation, 22,23
favoring structures of sites that were most clearly organized even if they were less “visually appealing.”
Similar to our methodology, Tran et al. asked participants unfamiliar with the test system to complete tasks and provide feedback, although not in the “think-aloud” method we employed. His participants preferred a navigation scheme that resembled the organization of physical file folders, 22
which is similar to the drop down tabs we added to make navigation easier.
Participants also encountered problems viewing test results and graphing. Similar to problems with medical jargon, participants were not prepared to interpret this information alone. Based on feedback in Round 1, we converted to bar charts for laboratory results, with some improvements in subsequent testing rounds. These findings mirror those of Marchionini et al., 21
whose participants also performed better with bar charts than tables. Difficulty with graphs was also reported by Cimino et al. 4,5
and Earnest et al. 12
Unlike our participants, Cimino's participants had, on average, 19 months experience with the site, giving them numerous opportunities to explore it. Despite this added experience, even these users had trouble with the graphing function. 4,5
Task completion: Participants in our study desired features that add “reassurance” that they had successfully completed a task. They also wanted to know where information was being sent and who received their messages. For users like ours that are concerned about the transfer of medical information via electronic communication, future usability testing should include tasks to assess if users understand who has access to their messages.
We were surprised by the length of time participants were willing to persevere with difficult tasks, especially looking for and interpreting laboratory results. The mean task time was over seven minutes. We could not locate any other studies which reported task times. We hypothesize that these extended times may be partially due to participants' desire to comply with the requested tasks in a research setting. Further, unlike searching a typical public health information site, the tasks performed were tied directly to their child's care. Therefore, the importance of the information may also have contributed to participants' willingness to continue with a time-consuming task. Future studies of page view duration during actual portal use may help clarify this issue.
Satisfaction and ease of use: Participants' satisfaction and ease of use scores as measured by the CUSQ were lower than scores reported in most other studies, with error messages receiving the lowest ratings. These satisfaction scores did not improve significantly between rounds of testing, even though there were significant improvements in task completion. Despite the scores on satisfaction being highest for interface pleasantness and likeability, the low satisfaction with the error messaging system is of more importance in regards to system usability. In commercial systems, the focus tends to be more on having a likeable and pleasant interface while aspects of the program that would hinder use are often overlooked. Therefore, problematic error messaging should be addressed as an area of significant concern and necessary improvement because its effectiveness can ultimately determine if people will continue to use the system and use it as intended. We are addressing this in our portal.
In addition, less than half of participants said that they were satisfied with the system overall and almost 60% chose more neutral items when answering if they were satisfied with how easy it was to use the system. In previous non-scenario based studies, participants said the electronic health record they accessed was easy to use. 5,10,16,19,32
The difference in the scores on ease of use between our participants and those in other studies may be due to several factors. For example, participants in the studies by Cimino et al. and De Clercq et al. all had experience with computers and may have been more computer savvy than our users. 5,10
Further, previous scenario-based studies provided participants training before they used the electronic system, 16,19
which we did not. We wanted to understand how users would respond with minimal training and experience. This design decision limited our ability to compare our study with others in which participants received prior training. Finally, we used a comprehensive prototype that replicated the complexity of the data parents would actually experience regarding their child with a severe chronic condition. Thus, it may have been more complex than the data presented in other studies, as most of those portals were designed for use in primary care.
Lastly, many problems encountered by participants could be traced back to assuming too much knowledge. Thus, content elements of the portal were not laid out clearly enough and spelled out sufficiently for a novice's understanding. For example, without a clear outline of the process of how to sign up and log in, less computer savvy users were confused by the large amounts of text on introductory pages.
Positive perceptions about the system:
Parents rated convenience and empowerment as key positive attributes of the system, a finding that echoes previous research. Participants in the study by Pyper et al. also cited benefits, including being better informed about their health, having medication lists, and having reminders of future appointments and screenings. 16
In addition to liking the convenience of the system, participants liked the medication pages and laboratory results, and the majority thought they would access both frequently. Knowing information about medications and laboratory results is very important to medical patients so it is expected that such functions that access them would be used often. Our findings are consistent with others who found that participants spent a majority of the time viewing laboratory results. 5,8,21
Participants perceived that understanding the layout of the portal was something they could learn reasonably quickly and they scored the CUSQ highly for learnability. Several participants commented that once they started exploring the site, they could find their way around. Although we did not examine this area, future evaluators might investigate which tasks remain difficult over time for users and thus focus their redesign efforts.
Limitations: There were limitations to our research. First, we studied only parents who had never used the system. Additional study with experienced users would have broadened our knowledge of how people interact with the system over time. We also did not formally test participants' computer skills. We assumed that parents who would participate in a computer-based study would have enough computer skills to navigate a web page effectively. However, because we made this choice, we could not distinguish difficulties due to limited computer skills from specific portal navigation problems. It is possible that many of the problems participants had resulted more from a lack of computer skills than portal issues. This should be addressed in future work.
Participants might have experienced anxiety over not having someone available to help them interpret information. Portals, like other stand alone electronic systems, lack immediate interaction with clinicians, a fact discussed by some participants. Understanding parents' experience with actual use of their children's data would be an important area for understanding risks and benefits of portals.
Additionally, we modified the portals between rounds of testing, making improvements based on findings. We made an a priori decision to conduct formative testing since our main goal was to improve the portals and we could not afford to do sequential usability tests of each portal. This approach balanced available resources, desire for portal development, and scientific certainty.
Performance based usability testing is widely advocated but has rarely been carried out in developing patient portals. While such usability testing can be expensive, the current study demonstrates that it can assist in making healthcare system interfaces for laypersons more user-friendly and potentially more functional for patients and their families.