PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of jamiaAlertsAuthor InstructionsSubmitAboutJAMIA - The Journal of the American Medical Informatics Association
 
J Am Med Inform Assoc. 2010 Mar-Apr; 17(2): 203–211.
PMCID: PMC3000775

Veterans' voices: use of the American Customer Satisfaction Index (ACSI) Survey to identify My HealtheVet personal health record users' characteristics, needs, and preferences

Abstract

Background

Consumer research reveals considerable interest in the use of Personal Health Records (PHRs), yet adoption remains relatively low. Both adopters and nonadopters represent important perspectives from which to understand this paradox.

Objective

This study focuses on direct feedback from adopters obtained using the American Customer Satisfaction Index (ACSI) survey on the My HealtheVet PHR portal (http://www.myhealth.va.gov) of the Veterans Health Administration (VHA). The results represent a source of direct feedback with which to better understand veterans' needs and preferences.

Methods

The ACSI Survey was implemented in October 2007 to measure satisfaction and elicit information about characteristics and preferences of My HealtheVet PHR adopters. The data represent a continuous random sample of site visitors who have navigated at least four pages on the site. A total of 100 617 surveys were completed (17.2%).

Results

Satisfaction with My HealtheVet is high (8.3/10.0), and users are highly likely to return to the site (8.6/10.0) and recommend the site to other veterans (9.1/10.0). The majority of system adopters are male (91%), between the ages of 51 and 70 (68%), and served in the Vietnam War (60%). Most veterans currently visit the site to utilize pharmacy-related features.

Conclusion

VHA has used the ACSI to monitor satisfaction, and to better understand the characteristics, needs, and preferences of early adopters. The data provide an important source of direct feedback to inform program development. Future research will include monitoring the impact of enhancements and new features on satisfaction, and conducting additional research with nonadopters to identify barriers to adoption and use.

Keywords: PHR, personal health record, patient satisfaction, Veterans, eHealth, healthcare, electronic medical record, My HealtheVet, VA, Veterans Health Administration

Introduction

The use of Personal Health Records (PHRs) to enhance the delivery of healthcare services to patients and to promote active engagement of patients in their own care represents a promising use of health information technology. Indeed, the Institute of Medicine (IOM) Committee on Quality Health Care in America noted in its landmark 2001 report, Crossing the Quality Chasm, that, “the advent of the internet and the world wide web has placed us on the threshold of a change that is reshaping virtually all aspects of society, including healthcare delivery.”1 The report recommended that “access to care should be provided over the internet, by telephone, and by other means in addition to in person visits.” This vision cast internet technology and the new tools it has made possible with powerful and transformative potential. As one such tool, the PHR provides a new virtual setting for these activities, yet further research is needed to identify the optimal role of the PHR to support patients and their interactions with providers and complex healthcare systems.

PHRs are relatively new innovations in a rapidly evolving industry. Ultimately, addressing some of the most pressing research questions about the impact of PHR use on health outcomes will be more feasible once PHRs mature, and a critical mass of patient use has accrued.2 At this early stage of development, formative evaluation coupling satisfaction with an assessment of user characteristics and preferences provides important insights as organizations explore the complexities of implementing the technology in ways that are most meaningful for patients, and strategically situated in the context of patient-centered care. Understanding consumer needs and preferences is a crucial starting place to begin to unravel this complexity. While both adopters and nonadopters represent important perspectives from which to understand a paradox of high consumer interest coupled with low rates of adoption, this article focuses on direct feedback from adopters obtained using the American Customer Satisfaction Index (ACSI) survey tool on the My HealtheVet PHR portal (http://www.myhealth.va.gov). The results represent veterans' voices; rich sources of direct feedback with which to better understand veterans' needs and preferences.

Background

State of personal health records

Healthcare organizations are striving to implement health information technology in response to an expanding national agenda, industry drivers to improve efficiency and maintain a competitive advantage, and most importantly the desire to leverage technology to improve patient care, patient safety, satisfaction, and health outcomes. Tang and Lansky3 emphasize the importance of the role of information technology as a pathway to improving healthcare, highlighting the need to build systems in which patients share information and control with healthcare professionals. The Institute of Medicine vision is simply unattainable in a world of paper-based records, characterizing the current state of much of the present healthcare system. Even with the digitization of medical records, transformation can only be realized if the patient is a full partner with the healthcare provider and with the healthcare organization providing care. In this context, personal health information tools are needed to support the free flow of information and provide patients with shared knowledge. The Markle Foundation's Connecting for Health Initiative defines the PHR as “an internet-based set of tools that allow people to access and coordinate their lifelong health information and make appropriate parts available to those who need it. PHRs offer an integrated and comprehensive view of health information, including information people generate themselves.”4 While an electronic health record (EHR) represents a collection of health information that has been gathered by and is managed by an enterprise, a PHR is meant to address the health information needs of the individual patient or consumer. Tang and Lanksy aptly describe this bridging of the patient–provider health information gap by the PHR as “the missing link.”3

Today a PHR provider may be a hospital, physician, vendor, employer, or insurer. More than 200 PHRs are currently available, and many are web-based, enabling secure universal access to important medical information by both patients and providers.5 With the catastrophic hurricane-induced health record access problems of 2005 for a segment of the American population, there is also significant interest in the potential use of electronic PHRs as a means of collecting and safeguarding important health information such as medications, diagnoses, and allergies. Multiple authors have described examples of current PHR models offered by institutions, healthcare organizations, employers, and insurers.6–13 PHRs come in a “multitude of forms, and are developed by different entities with different philosophies to meet different needs.”6 Ball and Gold have described a health record banking model that organizational, regional, or national system interoperability could enable. The entry of additional corporate (Microsoft, Google) and employer-based (Dossia) PHRs was referred to as a “tectonic shift” for the health informatics industry.14 In Piecing Together the PHR, Heubusch noted: “the personal health record, or PHR, might lack a common definition, a common data set, a common format, and a short list of sponsors. But it has one thing in abundance, and that's potential.”10 While it is unclear what impact all of these changes will have on the evolution of PHRs, organizations are being influenced by industry and marketplace drivers, and these tools are becoming more commonly available.

Consumer interest in personal health records

Consumer research consistently reveals significant and growing interest in using PHRs, although actual utilization remains relatively low. In November 2006, a public opinion survey commissioned by the Markle Foundation revealed that consumers view PHRs as an important element in reducing medical errors and increasing quality of care.15 This survey revealed that 89% of the public would want to look over their medical records if they could, and 65% are interested in accessing their own PHR online. Eighty-eight percent (88%) of the respondents agreed that online records would be important in reducing the number of unnecessary or repeated tests and procedures they undergo, and 84% said that they would be interested in accessing their electronic records to check for mistakes. Ninety percent (90%) personally thought that it would be important to track their symptoms or changes in healthcare online. Overall, 96% of respondents indicated that they think it is important for individuals to be able to access all of their own medical records to manage their own health. This growing enthusiasm for electronically available medical information is also tempered with public concerns about privacy and security; 75% of the respondents agreed that the government has a role in establishing rules to protect the privacy and confidentiality of online health information.

A May 2008 consumer attitude survey of a national sample of 1580 US adults matching the demographic proportions of the US adult population revealed that 79% of the public believe that using an online PHR would provide major benefits to individuals in managing their health and healthcare services.16 Nearly half (46.5%) said that they would be personally interested in using an online PHR service, yet utilization of electronic PHRs remains low: only 2.7% of adults have an electronic PHR, even though four out of five of those who do use one consider it to be valuable.

Internet access trends and typologies

Leveraging technology requires access as well as interest. The Pew Internet & American Life Project has been studying the impact of the internet since 2000, publishing a variety of reports that reflect evolving issues, attitudes, and trends. In February 2008, 73% of American adults had used the internet, and 54% had a high-speed internet connection at home.17 Horrigan created a typology of users, sorting American adult users into three broad categories: high end users, medium users, and low-level adopters.18 Earlier Pew reports have revealed important insights about American's online activities related to health information seeking. In Health Information Online, Fox reported that 79% of American adult internet users had used the internet to get medical information.19 In Finding Answers Online in Sickness and in Health, Madden and Fox described the expansion of the internet as a healthcare resource, noting that as more and more Americans come online in general, a growing number rely on the internet as a source of important health information.20 This report also noted that of the 20% of American adults who report a disability, handicap, or chronic disease, fully 86% have looked online for information about at least one of 17 health topics. The majority of these e-patients, both with and without chronic conditions, reported that the information changed their overall approach to maintaining their health, or the health of someone they take care of.21 22 In terms of both access and attitude, these data reveal important implications for eHealth innovations such as web-based PHRs.

Paradox of interest and adoption

Even with the increasing availability of PHRs and evidence of a clear increase in the use of the internet to access health information, there is limited broad-scale “consumer uptake” of PHRs. Given the consumer interest revealed by consumer research, a paradox exists between what consumers report that they want, and the degree to which organizations have been able to mobilize significant consumer engagement with PHRs. Focusing on the drivers of satisfaction and value for PHRs may surface these issues. Implementing techniques which can elicit direct feedback from the constituency of patients served by a healthcare organization is of paramount importance.

My HealtheVet personal health record

In 2003, the Veterans Health Administration (VHA) of the United States Department of Veterans Affairs (VA) introduced a web-based PHR to complement traditional services, improve co-managed care, and empower patients and their families to play a more active role in veterans' healthcare.23–28 The My HealtheVet portal (http://www.myhealth.va.gov) enables veterans to create and maintain a robust PHR that includes access to trusted patient health education information, a comprehensive personal health journal, electronic services such as online VA prescription refill requests, and additional features (see table 1, available as an online data supplement at http://www.jamia.org). The My HealtheVet Program is based on the core belief that knowledgeable patients are better able to make informed healthcare choices, stay healthy, and seek services when needed. The primary goal of the Program is to support veterans as empowered healthcare consumers with improved quality, access, and value of healthcare services, as well as increased satisfaction.

Veterans can visit the My HealtheVet website and self-register to create an account, although registration is not required to view the professionally sponsored health education resources, including topics of special interest to the veteran population. Once registered, veterans can create a customized PHR that is accessible from any computer with internet access. For veterans who are VA patients, a one-time process of In Person Authentication (IPA) at a local VA facility provides access to additional features, such as the ability to import extracted VA electronic medical record data to create a complete summary of both VA medications and the patient's self-entered prescriptions, over-the-counter medications, herbals, and supplements. The site also includes a customizable health calendar, online learning modules, and self-assessment tools. The ability to delegate access to all or selected parts of the PHR has been piloted successfully and is being developed as an enhancement.

As of October 2008, the My HealtheVet program had served more than 670 000 registered users, which represented 12% of VA patients receiving healthcare services. More than 90 000 of these registrants had upgraded their account through IPA, enabling access to the initial extracts of VA medical record data from VistA, VA's electronic health record system. By October 2009, the number of registered users had increased to 850 000 (16% of VA patients receiving services), with 150 000 having upgraded their accounts through IPA. The total number of visits to My HealtheVet since it was launched on November 11, 2003 had exceeded 32 million. Since the availability of online interactive ordering of VA prescription refills in August 2005, veterans had refilled more than 11 million prescriptions using My HealtheVet.

Program expansion will include the incremental release of additional features, enabling patients to view their VA appointments, laboratory test results, and additional portions of their VA electronic medical records. By releasing new features incrementally, system impact can be monitored and adjustments made to optimize system performance and the quality of the user experience, both of which are critical for adoption. Veteran feedback obtained via usability testing is incorporated into iterative releases, promoting adaptability and flexibility in meeting user needs. The My HealtheVet Program Office also continues to develop services to improve patient–provider communication. My HealtheVet Secure Messaging provides veterans and their healthcare providers with the ability to communicate electronically as a supplement to traditional healthcare interactions. Secure Messaging enables veterans to interact with healthcare providers and VA staff electronically to exchange non-urgent health-related information, to attend to administrative needs such as scheduling appointments, to request VA prescription renewals, or to ask health-related questions. This capability is currently available for a limited number of participants to support alignment with clinical practice workflows and enable participant feedback. Incremental expansion of Secure Messaging will make the service available at VA clinics nationwide.

Measuring satisfaction and soliciting feedback is an important part of the My HealtheVet Program Performance Evaluation Plan, as well as a requirement for management of federal information resources.29 One survey technique that has been deployed is the American Customer Satisfaction Index (ACSI) Survey. The My HealtheVet ACSI Survey was implemented in October 2007 in order to measure and monitor veteran satisfaction, and to identify and prioritize areas for improvement. Ongoing analysis has informed custom question development and data segmentation to generate additional insights about veterans' needs and preferences. This article reports data collected using the ACSI methodology from October 2007 through October 2008. The data provide a rich and informative view of My HealtheVet users' characteristics, preferences, and patterns of adoption, laying an important foundation for additional studies.

Methods

The American Customer Satisfaction Index (ACSI) survey is an industry standard tool for assessing consumer drivers of satisfaction and prioritizing areas of improvement.30 Established in 1994, the ACSI tracks trends in customer satisfaction and is produced by a partnership of the University of Michigan Business School, the American Society for Quality (ASQ), and the CFI Group. In addition to broad economic customer evaluations, the ACSI also is used to compile the E-Gov Satisfaction Index, a rating of more than 100 federal government websites in terms of how well they are satisfying citizens via the internet.31 ACSI scores are based on data from randomized voluntary web-based surveys and are reported on a scale of 0–100. The ACSI model computes an index score each time an adequate quantity of data has been collected through completed surveys.30

The ACSI methodology uses the “voice of the customer” technique to produce actionable results (scores, priority maps, and benchmarks) which represent customer wants and needs in a hierarchical structure that reflects both relative importance and satisfaction. The ACSI is a psychometric model which incorporates multiple measures of experiential factors and satisfaction levels in order to reduce measurement error. Multiple item measures focused on the drivers of satisfaction (content, functionality, look and feel, navigation, search, and site performance) are combined algorithmically to compile a satisfaction index which also predicts the impact of change as well as capturing future behaviors (likelihood to recommend the site to others, and to return to the site). Impacts are calculated using latent variable partial least-squares regression and are used in the model to prioritize website enhancements that will improve overall satisfaction. The derivation of optimal measure weights is based on cause and effect relationships between customer experiences, evaluations, and intentions; and has been shown to be valid and reliable.32 Deployment of the ACSI across multiple federal websites of the National Institutes of Health from September 2004–2006 demonstrated the usefulness of the ACSI as a standardized web evaluation tool for government websites with public audiences.33

An important advantage of using the ACSI as a tool for program evaluation is that it provides a direct measurement of customer satisfaction which incorporates preapproved Office of Management and Budget (OMB) clearance. OMB approval for public surveys is a requirement of all federal agencies under the Paperwork Reduction Act. The ability to include custom questions in addition to ACSI model questions enables the collection of rich data about users' characteristics, needs, and preferences. Collecting direct veteran and stakeholder feedback in an efficient and adaptable way has been an important strategy to support incremental program improvements. Another advantage of using the ACSI is that it enables benchmarking against other government and industry sites. A growing number of federal agencies are using this tool as a standard way of evaluating website satisfaction. The ACSI also supports the prioritization of website enhancements based on potential impact on satisfaction. VHA has used these data to identify needed enhancements, and also as a foundation for developing more targeted evaluation strategies. Since the My HealtheVet Program represents not only a federal website, but also a PHR portal, this evaluation can inform both areas.

Implementation

In October 2007, the VHA contracted with ForeSee Results as part of a Federal Consulting Group Interagency Agreement to deploy the ACSI Survey Tool for the My HealtheVet PHR portal (http://www.myhealth.va.gov). The ACSI Survey is implemented on the portal as a web-based pop-up browser window. A random sample of site visitors are invited to participate in the survey, based on a customized loyalty factor and sampling percentage. The loyalty factor, currently set at four pages, ensures that respondents have visited multiple pages on the site before being prompted to participate in the survey. The sampling percentage, currently 4%, ensures that an optimal number of site visitors are surveyed in order to enable the collection of new data on a regular basis with minimal respondent burden. Participation in the survey is voluntary and anonymous. The survey is deployed as a continuous survey, and weighted averages are combined to construct a single index that is tracked over time. The response rate for the My HealtheVet ACSI Survey averages 17.2%, which is significantly higher than the ForeSee client average of 6.5%. In addition to the ACSI model questions, custom questions are included in the pool of questions which a particular respondent may be asked. Although custom question responses are not used to derive ACSI results, they do represent an opportunity to collect important data from site visitors, while also enabling analytic data segmentation to focus on specific user populations.

Limitations

One limitation of this study technique is the potential that exists for response bias. Response rates for web-based ACSI surveys are typically low, with an average rate of 4–8%. The ACSI methodology recognizes this constraint but emphasizes the random presentation of the survey invitation and large sample size, noting that the anonymous nature of site visitors in a web environment precludes further investigation of non-response bias, since there is no identifying information available for non-respondents that would allow subsequent contact. Although the initial sampling rate was established at 13%, a relatively high response rate led us to decrease the sampling rate to 4% in order to optimize data collection in alignment with the ACSI methodology.30 32 While the response rate for the My HealtheVet ACSI survey (17.2%) was significantly higher than the average response rate for all ForeSee clients during this time period (6.5%), it is important to emphasize that the results reflect the characteristics and perspectives of survey responders, and may not be fully generalizable to the larger population.

Thus, it is important to characterize the respondent pool as a random sample of visitors to the My HealtheVet PHR website who opt to participate in the survey. Comparisons of respondents to the population of My HealtheVet registrants and to the veteran population overall have been made (age, gender, location) in order to contextualize the respondent pool in comparison to both populations of interest. To assess the rate of repeat survey takers, we also later added a custom question which asked if the respondent had previously taken the survey (91% had not). To date, we have no evidence of systemic response bias, but to further address these study limitations we intend to deploy additional techniques for triangulation. Keeping these nuances and potential biases in mind, the study provides an unprecedented look at the characteristics and preferences for this large cohort of PHR users. As previously noted, additional evaluation strategies are also needed to elicit input and feedback from nonadopters, and to identify barriers that may exist to system access and use.

Results

ACSI My HealtheVet satisfaction summary: model questions

Visitors to the My HealtheVet website were offered an online survey using the ACSI methodology beginning October 2007. A 4% random sample of users visiting four or more pages was prompted to take the survey. From October 2007 through October 2008, surveys were presented to 585 039 site visitors, and 100 617 surveys were completed (17.2%). A summary of ACSI scores and satisfaction elements for the time period is shown in figure 1, available as an online data supplement at http://www.jamia.org. Element and satisfaction scores are weighted averages of the responses to two to four individual survey questions. The impact on satisfaction represents the amount of improvement to overall satisfaction that we would anticipate if an element score was increased by five points. The impact on future behavior represents the amount of improvement that we would expect to see if overall satisfaction is improved by five points. The aggregate My HealtheVet ACSI score remained steady throughout the time period with a mean score of 75. This score exceeded the 3rd quarter 2008 E-Gov aggregate score of 73.9, and citizen satisfaction with government overall of 67.8 as reported in the ACSI E-Government Satisfaction Index in October 2008.31 By October 2009, the aggregate My HealtheVet ACSI score had increased to 78, while the third quarter 2009 E-Gov Index increased slightly to 75.2.

Quarterly reviews during the time period demonstrated that My HealtheVet site visitors score satisfaction notably higher than key benchmarks including averages for ForeSee Results Company-wide, E-Gov, and E-Gov Information sites. Each of the My HealtheVet element scores (content, functionality, look and feel, navigation, and search) was higher than these benchmarks, while site performance was equivalent. With this established baseline, the ACSI data enable an assessment of the impact of system changes on user satisfaction, including the release of additional services rated most highly by ACSI respondents. Summary satisfaction ratings indicate that veterans are highly satisfied with the My HealtheVet Program (8.3 on a scale of 1 to 10), highly likely to recommend the site to other veterans (8.6 on a scale of 1 to 10), and highly likely to return to the site (9.1 on a scale of 1 to 10).

The ACSI Priority Map is used to identify drivers of satisfaction which represent the top priorities for improvement. For My HealtheVet, satisfaction is highest with content, functionality, look and feel, and site performance, as shown in figure 2. The top priority areas for improvement are navigation and search, based on level of satisfaction and potential impact of improvements. Site redesign and search-engine refinement projects are planned for this fiscal year, and additional custom questions are currently being used to collect more specific data related to these two elements to inform action plans for improvement. Monitoring subsequent ACSI scores will enable assessment of the impact of these improvements.

Figure 2
American Customer Satisfaction Index My HealtheVet priority map.

ACSI My HealtheVet satisfaction summary: custom questions

A pool of more than 50 custom questions has been rotated through the survey since its inception, resulting in data that have been used to better understand system adopters. Custom questions are reviewed on a monthly basis, and questions are added and suspended as needed, based on data analysis performed by the multidisciplinary My HealtheVet Clinical Advisory Board Performance Evaluation Workgroup. Because of the dynamic nature of the custom question pool, the total number of completed surveys for a particular custom question may vary. For this reason, sample sizes are provided for all data shown in tables 2–5. These sample sizes represent the number of respondents who completed each survey question.

Table 2
Demographics and characteristics of American Customer Satisfaction Index My HealtheVet respondents
Table 3
Access patterns of American Customer Satisfaction Index My HealtheVet respondents
Table 4
Usage of American Customer Satisfaction Index My HealtheVet respondents
Table 5
Preferences of American Customer Satisfaction Index My HealtheVet Respondents (N=100 617)

Demographics and characteristics for respondents are shown in table 2. Survey respondents are predominantly veterans (93%), although a small percentage are family members of veterans (5%). Multiple roles may be selected, and although other roles are indicated, these each represent 1% or less of the respondent pool. Likewise, for military period of service, respondents are able to select multiple categories. The majority of respondents served in the Vietnam War (60%). The second most common period of service indicated is Peacetime Service (19%), followed by Desert Shield/Desert Storm (13%). Other periods of service are represented, enabling further segmentation for additional analysis.

The majority of respondents are between the ages of 51 and 70 years (68%), while another 16% are 71 years or older. A smaller number (15%) are between the ages of 31 and 50 years old. In July 2008, additional data were collected using narrower age ranges to enable comparison with the population of My HealtheVet registrants and the veteran Population (see figure 3). Consistent with earlier data, 67% of the respondents are between the ages of 50 and 69. Survey respondents are predominantly male (91%), consistent with both the population of My HealtheVet registrants (88%) and the veteran population overall (93%). For the subset of respondents who indicated that they served in the Global War on Terror (n=5626), a greater percentage are female veterans (19% compared to 9%). When asked to rate their overall general health status, the majority of respondents report good (38%) or fair (29%) health, but 10% rate their health as poor.

Figure 3
Age-range comparisons for My HealtheVet (MHV) registrants, veteran population, and American Customer Satisfaction Index (ACSI) respondents.

The top 10 states of residence for respondents are identical to the top 10 states for My HealtheVet account registrations, with the highest number of respondents from Florida (10%), Texas (9%), and California (8%). In comparison, California, Florida, and Texas also have the greatest number of veterans overall. All other states are represented in the ACSI respondent pool, while a small percentage (1%) live elsewhere. These data correlate well with My HealtheVet registration data, indicating that the respondent population in general mirrors the population of registered users.

Table 3 summarizes access patterns for ACSI respondents. Travel time to the nearest VA facility is an important user characteristic in terms of both access to services and ability to complete the IPA process. Notably, 37% indicate that their travel time to the nearest facility is 1 h or more. Eight percent must travel 2 h or more to reach their nearest VA facility. Nearly all of the respondents (96%) have used VA services in the past 12 months, reflecting the high proportion of users eligible to obtain an upgraded account by visiting their VA facility and becoming authenticated. The majority of respondents have used mostly VA facilities for routine outpatient care in the last 12 months (84%), although a significant percentage (14%) have instead mostly used a non-VA hospital or clinic. When asked about their use of urgent care or emergency-room services in the last 12 months, of those who did receive these services (54%), more than half used a VA hospital or clinic.

Nearly all respondents reported accessing the My HealtheVet website from home (96%), although visitors also access the site from their workplace (11%), VA medical center (3%), and other locations. For 93% of the respondents, home is the most frequent site of access. Fully 91% of respondents report using a high-speed internet connection to access the portal, although the program will continue to focus on meeting the needs of all users by ensuring compatibility with dial-up internet access (7%) as the lowest common denominator. Interestingly, most users rate their ability in using the internet as advanced (68%) or intermediate (29%). For the subset of respondents who indicated that they served in the Global War on Terror (n=5626), 78% rate their ability as advanced, supporting the belief that system adopters from the newest generation of veterans often have higher levels of computer literacy.

Respondents are predominantly registered account holders at the My HealtheVet website (95%), giving them access to a fuller range of tools and features. For VA patients, a one-time process of IPA enables access to the full range of site features, including online VA prescription refill renewal by medication name and access to their VA prescription history. Although the majority of respondents have been authenticated (60%), a significant percentage have not been authenticated (24%) or are not sure (15%). A qualitative review of open-ended responses using a comment cluster tool revealed some of the reasons respondents cite for not being in person authenticated. The data showed that many users are not familiar with “authentication,” or do not understand how to become authenticated. Other respondents noted that they forgot to accomplish this at their most recent visit to a VA facility, or intend to at an upcoming appointment. Improvements in communication, marketing, and processes used for authentication are under way to ensure that eligible users will have access to important features that rely upon authentication.

As shown in table 4, most respondents indicate that they visit the My HealtheVet website about once a month (49%), while another 25% visit about once a week. Twelve percent (12%) reported being first-time visitors. Some users (2%) visit more than once a day.

Understanding what users are trying to accomplish when they visit the site is an important factor in prioritizing portal features to meet user needs. As the most requested transactional service, online VA prescription refills is reported by respondents as the top objective of their visit (75%). The second highest objective reported is access to a user's prescription history from their VA medical record (24%), followed by looking up information about a medication (18%). These pharmacy-related features currently represent users' top objectives. As additional transactional services are rolled out, this question will continue to reveal important insights about users' priorities. To locate information, most respondents report browsing the site to locate information (41%), while another 31% utilize navigational quick links. Only 16% of visitors report using the site's search feature. Although the majority of respondents (67%) indicate that they accomplished the goal of their visit, a significant percentage indicate that they only partially accomplished (17%) or did not accomplish (12%) what they wanted to. In May 2008, an additional response choice (“not finished”) was added to determine if respondents were being prompted to participate in the survey prior to accomplishing their goal (5%). A qualitative review of open-ended responses using a comment cluster tool provided by ForeSee reveals some of the objectives respondents wanted to accomplish but could not. Most of the responses revealed the desire to view or manage VA appointments, or view VA medical record data. Both of these capabilities are currently being field-tested in preparation for release. As additional transactional services are rolled out, it is anticipated that an even greater number number of users will achieve their desired goals using the site.

When asked about additional features desired, respondents indicate that 87% (79 892 veterans) wish to view upcoming VA appointments, 74% want to be able to schedule or change their appointments, 73% want to view information from their VA medical record, and 64% would like online secure communication with their doctor (see table 5). Each of these features is currently being developed for upcoming release. The rank order of desired features represents important direct feedback from veterans and stakeholders which will continue to be used to prioritize site development. When asked about the kinds of medical records users wished to have as part of their My HealtheVet PHR, respondents predominantly want access to their VA medical record (90%), although 64% also desire access to their military service health records, and 33% want access to their medical records from other non-VA providers. Only 3% (634 of 20 244 respondents) indicate that they do not want their records to be available on My HealtheVet. Users do have the capability to utilize the site and PHR functions without necessarily integrating medical record data.

My HealtheVet site visitors are also presented with an opportunity to add free text comments and suggest improvements through the ACSI survey. A qualitative review using the comment cluster tool for open-ended responses reveals the main improvements that users suggest. The most common improvement noted is for the site to enable VA appointment views and management, both of which are currently being developed for release, as previously mentioned. The second most common improvement suggested is the ability to view VA medical record data, a feature that is currently available for VA prescription data with additional extract types in queue for upcoming release. Of the comments related to prescription refill, many ask for the ability to view medication names when requesting prescription refills. Although this feature is currently available to VA patients who have been in-person-authenticated, lack of awareness of this feature reveals the need to develop additional communication strategies. In-depth analysis of this direct veteran and stakeholder feedback will continue in order to be responsive to user-suggested improvements. The majority of ACSI respondents (56%) agree that use of My HealtheVet has improved their ability to manage their health. It is anticipated that the further implementation of new features identified as desirable services by users will have a positive impact on user goal accomplishment and satisfaction.

Discussion

The VHA has used the ACSI to measure and monitor satisfaction; and also to better understand the characteristics, needs and preferences of these early PHR adopters. Although the potential for response bias exists due to the nature of the survey, we also examined the degree of match with the population of registrants for key demographic characteristics such as age, gender, role, and geographic location. The majority of ACSI respondents are between the ages of 51 and 70 (68%), served in the Vietnam War (60%), and are in good or fair health (67%). Although we have assumptions that the newest population of veterans returning from Operation Enduring Freedom/Operation Iraqi Freedom as part of the Global War on Terror may have both higher levels of computer skill and increased expectations for eHealth services, these currently represent only 5% of the respondent pool, in relative proportion with the population of veterans. Multiple communication strategies are needed to enhance awareness of My HealtheVet, especially in this growing segment of the veteran population.

Understanding what users are trying to accomplish when they visit the site is an important factor in prioritizing portal features to meet user needs. For My HealtheVet, pharmacy-related features currently represent users' top objectives, although all portions of the site show significant use. As additional transactional services are rolled out, this question will continue to reveal important insights about users' priorities. Not surprisingly, goal accomplishment on the site leads to higher satisfaction (ACSI score=84). Additional usability testing is under way, using task-based scenarios to continue to optimize the site and improve the user experience. When asked about additional desired features, respondents indicate their preferences, and each of the top four most highly ranked features is planned for upcoming release. The rank order of these and other desired features represent important direct feedback from veterans and stakeholders to prioritize site development. Modification of response choices for this custom question will enable new data collection to inform prioritization of future development. Of note is the high percentage of users who rank their internet ability as advanced (68%) or intermediate (29%). While health literacy remains a complex issue for many consumers, the data also point to the need for strategies to enhance computer literacy so that the PHR is a usable and effective tool for all veterans who wish to use it. Local education and training initiatives have been deployed at many VA medical centers to address this need.

Conclusion

This article provides a summary of ACSI data which reveals veteran satisfaction, identifies priority areas for improvement, and provides important information about users' characteristics, preferences, and needs. Additional data collection and analysis will be used to determine what works well for users and what may need to be improved. The value of direct veteran feedback is critical in this process. With the release of several new highly desired features based on veteran feedback, VHA aims to optimize the program and expand reach by increasing the number of veterans using the portal. The ACSI will continue to be used to capture the “voice of the veteran” and to measure customer satisfaction trends as program improvements are implemented and new features rolled out in response to veterans' needs. Additional research will be conducted with nonadopters to identify barriers to system adoption and use.

Supplementary Material

Web Only Data:

Acknowledgments

The author thanks the My HealtheVet Clinical Advisory Board Performance Evaluation Workgroup for their analytical insights and support of the ACSI Survey Initiative.

Footnotes

Contributed by

Contributors: Preliminary data for a subset of these results were presented as a poster at the Fall American Medical Informatics Association 2008 Symposium.

Competing interests: None.

Provenance and peer review: Not commissioned; externally peer reviewed.

References

1. Institute of Medicine Committee on the quality of healthcare in America. Crossing the quality chasm: a new health system for the 21st century. Washington: National Academy Press, 2001.
2. Rainie L. E-health research. In: Murero M, Rice RE, editors. , eds. The internet and health care. Mahweh, New Jersey: Erlbaum, 2006.
3. Tang PC, Lansky D. The missing link: bridging the patient–provider health information gap. Health Aff (Millwood) 2005;24:1290–5. [PubMed]
4. Connecting for Health The personal health working group: final report. New York: Markle Foundation, 2003.
5. Gearon C. Perspectives on the future of personal health records. California: California HealthCare Foundation, 2007. http://www.chcf.org/documents/chronicdisease/PHRPerspectives.pdf(accessed 15 Oct 2008).
6. Ball MJ, Gold J. Banking on health: personal records and information exchange. J Healthc Inf Manag 2006;20:71–83. [PubMed]
7. Cronin C. Personal health records: an overview of what is available to the public. Washington DC: AARP Public Policy Institute, 2006. http://www.aarp.org/research/health/carequality/2006_11_phr.html(accessed 15 Oct 2008).
8. Endsley S, Kibbe DC, Linares A, et al. An introduction to personal health records. Fam Pract Manag 2006;13:57–62. [PubMed]
9. Halamka JD, Mandl KD, Tang PC. Early experiences with personal health records. J Am Med Inform Assoc 2008;15:1–7. [PMC free article] [PubMed]
10. Heubusch K. Piecing together the PHR. J AHIMA 2007;78:28–32. [PubMed]
11. Sprague L. Personal health records: the people's choice? NHPF Issue Brief 2006;820:1–13. [PubMed]
12. Tang PC, Ash JS, Bates DW, et al. Personal health records: definitions, benefits, and strategies for overcoming barriers to adoption. J Am Med Inform Assoc 2006;13:121–6. [PMC free article] [PubMed]
13. Reinke T. Rise of the PHR. Manag Care 2007;16:40–2. [PubMed]
14. Mandl KD, Kohane IS. Tectonic shifts in the health information economy. N Engl J Med 2008;358:1732–7. [PubMed]
15. Connecting for Health Survey finds Americans want electronic personal health information to improve own health care. New York: Markle Foundation, 2006. http://www.markle.org/downloadable_assets/research_doc_120706.pdf(accessed 15 Oct 2008).
16. Connecting for Health Americans overwhelmingly believe electronic personal health records could improve their health: Markle Foundation, June 2008. Available at: http://www.connectingforhealth.org/resources/ResearchBrief-200806.pdf(accessed 15 Oct 2008).
17. Fox S. Degrees of access: Internet, cell phone, and home broadband: Pew Internet & American Life Project. Presented to the Usability Professionals' Association in Washington February 20, 2008.
18. Horrigan JB. A typology of information and communication technology users: pew internet & American life project, May 7, 2007. http://www.pewinternet.org/pdfs/PIP_ICT_Typology.pdf(accessed 15 Oct 2008).
19. Fox S. Health information online: pew internet & American life project, May 17, 2005. http://www.pewinternet.org/pdfs/PIP_Healthtopics_May05.pdf(accessed 15 Oct 2008).
20. Madden M, Fox S. Finding answers online in sickness and in health: pew internet & American health project, May 2, 2007. http://www.pewinternet.org/pdfs/PIP_Health_Decisions_2006.pdf(accessed 15 Oct 2008).
21. Fox S. E-patients with a disability of chronic disease: pew internet & American life project, October 8, 2007. http://www.pewinternet.org/pdfs/EPatients_Chronic_Conditions_2007.pdf(accessed 15 Oct 2008).
22. Fox S. Recruit doctors. Let e-patients lead. Go mobile: pew internet & American life project. Presented to the Health 2.0 conference in San Diego, CA March 4, 2008. http://www.pewinternet.org/pdfs/Fox_Health_March_2008.pdf(accessed 15 Oct 2008).
23. Schneider J. My Health eVet: fighting for health with information. Journal of Consumer Health on the Internet 2008;12:13–21.
24. Naditz A. Telemedicine at the VA: VistA, MyHealth eVet, and other VA programs. Telemed J E Health 2008;14:330–2. [PubMed]
25. Nazi K, Cowell F. My Health eVet: VA's new web-based tool helps veterans stay healthy—and involved in their care. PN 2007;61:30–4.
26. Buxbaum P. Envisioning troop-centric health care . Government Health IT, 2008. http://www.govhealthit.com/print/4_18/features/350395-1.html(accessed 15 Oct 2008).
27. Nazi K. “My HealtheVet personal health record,” testimony to AHIC CE workgroup. September 12, 2007. http://www.dhhs.gov/healthit/ahic/materials/09_07/ce/nazi.html(accessed 15 Oct 2008).
28. Nazi K. “My HealtheVet personal health record, part II,” testimony to AHIC CE workgroup, October 16, 2007. http://www.dhhs.gov/healthit/ahic/materials/10_07/ce/nazi.html(accessed 15 Oct 2008).
29. Office of Management and Budget. Circular No A-130 Revised. Management of Federal Information Resources. http://www.whitehouse.gov/omb/circulars/a130/a130trans4.html(accessed 10 Oct 2008).
30. ACSI ACSI methodology. http://theacsi.org(accessed 15 Oct 2008).
31. ACSI Government satisfaction scores: October 2008: http://www.theacsi.org(accessed 1 Nov 2008).
32. Fornell C, Johnson MD, Anderson EW, et al. The American Customer Satisfaction Index: nature, purpose, and findings. J Mark 1996;60:7–18.
33. Wood FB, Siegel ER, Feldman S, et al. Web evaluation at the US National Institutes of Health: use of the American Customer Satisfaction Index online customer survey. J Med Internet Res 2008;10:e4. [PMC free article] [PubMed]

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of American Medical Informatics Association