Our evaluation approach identified a significant number of challenges likely to interfere with patients' abilities to use a particular CHI intervention. While this list is likely not exhaustive, the large number of identified challenges provides intervention designers with a starting place from which to begin redesigning the intervention. Intervention designers can thus resolve challenges before conducting further, higher-cost testing with patients. Actual patients will likely encounter other challenges in addition to those identified through the described methodology, but the additional challenges will become more salient and easier to identify once the initial set of challenges is mediated.
The use case tasks enabled the patient proxies to know their goals, but still allowed them flexibility in completing each task. For instance, we directed each patient to ‘Create a (HealthVault/Google Health) account.’ The research team could have specified subtasks that would enable the patient proxies to complete this task, such as which icons the patient should click on to create an account. However, this fine level of detail takes almost all cognitive load and reasoning away from the patient, thus likely limiting insightful information the researchers would otherwise obtain about what challenges the patient encounters in completing the tasks. The researchers could have, similarly, specified only higher level activities, such as ‘Use (HealthVault/Google Health) to send your blood pressure readings to your nurse.’ This activity-level guidance would likely not provide the patient proxies with enough direction to elicit detailed information on what challenges the patient proxies encountered using various aspects of the system, and the elicited challenges may be so diverse that comparing and synthesizing challenges across the patient proxies would be difficult. Therefore, the approach used (ie, use cases with a set of tasks) is more structured than heuristic usability evaluation (ie, providing little guidance and minimal information regarding the patient proxies' tasks), but more flexible than cognitive walkthrough approaches (ie, using step-by-step task descriptions).33–35
The categorization of the challenges as physical, cognitive, or macroergonomic is useful, in that design changes to mediate these challenges should align with the type of problem patients are experiencing. For instance, one could extract all cognitive challenges from the data set to understand how the system is or is not supporting patients' cognition and how the design could be changed or supplemented to enhance this support.
Physical challenges were most often linked to patients' use of CHI intervention-related devices: both diabetes use cases involved a blood pressure cuff, both bariatric surgery use cases involved a weight scale, and the HealthVault bariatric surgery use case also involved a pedometer. The number of cognitive challenges was quite consistent across use cases, with most cognitive challenges associated with one-time activities such as learning to use the devices and applications. The number of macroergonomic challenges varied more across use cases, with most macroergonomic challenges associated with repeated activities such as uploading device data, and regularly manually entering blood pressure, blood sugar and food intake data.
While some patient challenges are specific to particular use cases, others span devices and applications independent of use cases (eg, HealthVault for both diabetes and bariatric surgery). Device and application designers are best suited to address these challenges. For instance, if a device vendor resolves all challenges associated with its blood pressure cuff, it is resolving a subset of challenges that will arise for all use cases involving that blood pressure cuff. In addition, intervention designers can use supplementary intervention artifacts and processes and training material to guide patients as the patients interact with the devices and applications.
Other challenges are inherent to the task, so the challenges span use cases regardless of which devices and applications patients use (eg, both HealthVault and Google Health for bariatric surgery). These challenges provide rich opportunities for innovators to bring new technologies to the CHI marketplace, which may drive down the costs of existing technologies. For instance, the development of innovative and standardized means to collect and log food intake can mediate the confusion and complexity patients experience when manually logging individual foods and estimating calorie counts for specific foods.
Our study has several limitations. First, our study only included two patient proxies, which limits the extent to which we can generalize the findings from this study to the challenges that a larger number of proxies might encounter. However, the two proxies identified 122 unique patient challenges, suggesting that this method can be used to successfully identify challenges that patients will encounter when using CHI interventions. Second, while our goal was to identify a set of use challenges using patient proxies before evaluating the system with actual patients, we did not compare proxy findings with challenges faced by actual patients. We realize that the set of challenges identified by actual patients will likely differ from those identified in this study and plan to conduct this analysis in future work. Our study focused on two specific patient populations, which have unique characteristics and limitations. However, these two patient populations are quite diverse: one is focused on a prevalent set of chronic diseases (diabetes, high blood pressure), and the other is focused on recovery from an acute intervention (bariatric surgery). It is also unlikely that any challenge is actually particular to only one use case. For instance, we identified challenges associated with uploading blood pressure data through HealthVault's connection center during the HealthVault diabetes use case, but this challenge would be present for any use case involving the blood pressure cuff and HealthVault. Finally, because the study focused specifically on CHI interventions based on low-cost devices and applications that are not jointly designed, the challenges associated with these commercial applications may not well represent the type of challenges associated with customized systems. However, the approaches we used could be generalized to evaluate and redesign any type of CHI intervention.