|Home | About | Journals | Submit | Contact Us | Français|
We examine key study design challenges of using stated-preference methods to estimate the value of whole genome sequencing (WGS) as a specific example of genomic testing. Assessing the value of WGS is complex because WGS provides multiple findings, some of which can be incidental in nature and unrelated to the specific health concerns that motivated the test. Additionally, WGS results can include actionable findings (variants considered to be clinically useful and can be acted on), findings for which evidence for best clinical action is not available (variants considered clinically valid but do not meet as high of a standard for clinical usefulness), and findings of unknown significance.
We consider three key challenges encountered in designing our national study on the value of WGS – layers of uncertainty, potential downstream consequences with endogenous aspects, and both positive and negative utility associated with testing information – and potential solutions as strategies to address these challenges. We conceptualized the decision to acquire WGS information as a series of sequential choices that are resolved separately: a) To determine the value of WGS information at the initial decision to undergo WGS, we used a contingent valuation questions, and b) to elicit respondent preferences for reducing risks of health problems and the consequences of taking the steps to reduce these risks, we used a discrete choice experiment. We conclude by considering the implications for evaluating the value of other complex health technologies that involve multiple forms of uncertainty.
Genomic testing, such as whole genome sequencing (WGS), can be used to predict future disease risk or inform treatment for current disease and for which there is growing demand from patients. Genomic testing provides an excellent example of the challenges of measuring the value of personalized health technologies in general, and WGS as an example of a test that provides more than one result or results for multiple diseases. As the costs of genomic testing decrease, it is possible that it will become more routine and eventually used for general population screening. However, it is unclear whether the benefits of the information received from genomic testing outweigh the potential harm from anxiety, unnecessary follow-up testing and overtreatment.
Whether genomic testing can achieve its potential to improve patient outcomes ultimately will depend on what information patients receive and how patients and providers value and respond to test information. As noted in the Institute of Medicine report on genomic diagnostic testing, there is a need to evaluate how these technologies and the information they generate can best be integrated into the clinical setting to maximize patient benefit and minimize harm. Phillips et al. suggests there is a need for evidence on the value of personalized medicine technologies to inform decision making. Further, the value of genomic testing needs to consider health and non-health benefits, and the impact on downstream health services.
Neumann et al. found that people value information from predictive tests for both medical and non-medical decision making. People also value test information if it can alter their behaviour. Research in the area of diagnostic tests has found that there are multiple types of value to consider when evaluating diagnostic testing, such as ‘value of knowing’, medical value, and psychological value. Lee, Neumann & Rizzo suggest that methods such as discrete choice experiments (DCEs) would be useful in isolating the value of knowing versus medical value. However, the value of knowing may encompass other values, like the value for the option to do something in the future when new health technologies are available. Furthermore, the value of knowing is likely disease specific, which highlights the challenges faced with tests that provide multiple results or results for multiple diseases, such as WGS.
The objective of this paper is to examine key study design challenges of using stated-preference methods to estimate the value of WGS. We use WGS as a specific example of genomic testing and other personalized medicine technologies more broadly, where the study design challenges we explore are inherent in the complexity of the decision problem. We start with a brief overview of stated preferences and methods for measuring preferences in health. Then we describe how we conceptualized the problem of measuring the value of WGS as a decision, how we designed a DCE considering three challenges posed by the problem complexity, and our proposed solutions. We conclude with the limitations of our solutions, the implications, and the consequence of these limitations for evaluating the value of other complex health technologies.
Patient preferences are increasingly relevant in the era of personalized medicine and patient-centered care. Patients who know and understand their preferences may experience less decisional regret, increased satisfaction, and improved communication with their health care provider. Stated-preference studies quantify tradeoffs that respondents are willing to accept among multiple characteristics of alternative health care interventions or technologies and can be used to estimate the value or utility of the characteristics of these interventions or technologies. Quantifying stated-preferences can identify differences amongst individuals, or groups of individuals, and can be applied in samples of patients, care providers or the general population. Aggregate or mean preference estimates are used to inform resource allocation or health-policy decisions. Individual-level or group-level preferences are relevant to clinical decision making.
While methods for measuring preferences in health have developed considerably,[11-13] there remain significant study design challenges, particularly in estimating preferences for complex health technologies that could simultaneously affect multiple disease conditions, include multiple sequential or conditional risks, and occur over different time periods with varying levels of uncertainty. WGS results can include actionable findings (variants considered clinically useful and can be acted on; i.e., variants for which there are medical treatment guidelines or that are associated with preventable diseases), findings for which evidence on effective clinical action is not available (variants that are considered clinically valid but do not meet as high a standard for clinical usefulness; i.e., variants for which there is unclear medical treatment), and findings of unknown clinical significance (variants considered to have unknown or no clinical significance).[14, 15] Furthermore, elective interventions to reduce or avert the risk of health problems have their own risks (e.g., potentially serious side effects). Assessing the value of WGS information must jointly consider the potential benefits, harms, and costs associated with the findings and their short- and long-term downstream sequelae.
We consider three key challenges encountered in designing our study  on the value of WGS – positive and negative utility associated with testing information, layers of uncertainty, and potentially downstream consequences with endogenous aspects– and potential solutions to address these challenges. We conclude by considering the limitations of our solutions, and the implications and consequence of these limitations for evaluating the value of other complex health technologies.
We conceptualized the decision to acquire WGS information as a series of sequential choices that are assumed to be resolved separately. Untangling the decisions that link the acquisition of WGS information to outcomes is facilitated by the use of various preference-elicitation formats; each selected to deal with study-design challenges related to specific decision points.
In genomics, personal utility is defined as the meaning and worth an individual gives to a genomic or genetic test from their personal perspective.[17, 18] Previous research suggests that individuals, regardless of health status, value having choices about the WGS information they receive.[19-22] The initial decision to acquire WGS information involves assessing whether a broad set of uncertain outcomes, including both the WGS findings and willingness to act on the information received, is likely to offer enough benefit to justify the cost of sequencing. We used contingent valuation (CV) questions to determine the value of WGS information at the initial decision to undergo WGS. CV is a method used to value commodities or services for which there are no market data or for which market data are uninformative about values to consumers, such as health and health care. CV surveys elicit the money-equivalent value (willingness to pay) of a specified commodity or service [23, 24] and thus can be used to estimate the perceived benefit of WGS information. Figure 1 illustrates the bid structure of the CV questions in our survey to elicit willingness to pay for WGS information (experimental design and analysis details in Marshall et al.). The findings from our CV analysis (see results) highlight that a substantial proportion of respondents did not value obtaining WGS testing information even if it were free, and for those who were interested, they were willing to pay more for actionable findings from WGS than findings for which treatment is currently unavailable. In contrast, our DCE findings reported here focus on whether a person is willing to act on the information received.
We considered the final decision in the sequence of choices to be whether a person is willing to act on the information received. Specifically, to what degree is a person willing to accept medical interventions with risks and costs, given the likelihood and severity of the health problems exposed by WGS results. There are many actions people could take in response to WGS findings. For example, they could share the information with family members, move to a location with better access to specialized health services, or alter education, employment, saving, reproductive or other plans. We focused on the medical actionability of WGS information to evaluate the tradeoffs people are willing to accept among factors related to their genetic predisposition for health problems, and the potential downstream consequences.
We used a discrete-choice experiment (DCE) to elicit respondent preferences for reducing risks of health problems and the consequences of taking the steps to reduce risks. DCEs are a systematic approach for eliciting stated preferences to quantify the relative importance that respondents assign to various characteristics of a health care service or treatment. The options are described by a set of attributes, each of which has different levels. Attribute levels are combined into intervention profiles and the profiles are assembled into sets of two or more alternatives using an experimental design with known statistical properties. Respondents choose their preferred alternative from among each set of constructed profiles in a series of choice questions. The pattern of choices reveals the implicit relative importance of each attribute level and can be used to quantify the rates at which respondents are willing to accept tradeoffs among attributes in the study design.
Interpreting the resulting preference-weight estimates requires several underlying assumptions :
Although these assumptions are necessary in the context of this problem posed as a DCE, people rarely decompose such decisions as values that are separate and additive constructs, despite routinely making decisions that involve similar complexities.
Quantifying the value and determinants of value of WGS information required dealing with three challenges: valuing positive and negative effects of incidental findings on utility, multiple cascading uncertainties related to multiple gene variants identified in a single test, and downstream consequences conditional on elective interventions
Assessing the value of WGS is complex because it provides multiple results that have varying levels of clinical usefulness. Additionally, results include primary findings (variants in a gene(s) relevant to the diagnostic indication for which sequencing was ordered) and secondary findings (also termed incidental findings, variants in genes not apparently relevant to a diagnostic indication for which sequencing was ordered) for multiple different diseases. Reporting secondary findings that are currently determined to have unclear medical treatment could generate anxiety and unnecessary medical tests, but patients could miss valuable information if not reported. Further, acting on findings prematurely could result in harm from potential overtreatment and unnecessary healthcare resource use.[26, 27] However, individuals may prefer to learn about these findings to reassess personal priorities and/or get affairs in order if their chance of death is increased (changes in bequest values), or they may hope information on long-term risk becomes actionable in the future (option values).
We addressed this challenge of valuing multiple test results by embedding two CV questions into our survey, varying the initial WGS report cost across respondents to elicit willingness-to-pay. One questions was about a basic genomic report that includes only actionable genomic status information and one about additional genomic information for which the medical treatment is currently unclear and would not be included in a basic genomic report (Figure 1).
In addition to the usual challenges associated with communicating risk to survey respondents, WGS is a complex technology with multiple layers of probabilities and associated uncertainties. For example, for each of the multiple genes examined in WGS, each gene variant is associated with a probability of being identified, and a probability that having the variant will result in a health problem. Stated-preference surveys are designed to provide experimental control over the decision problem. Evidence from previous DCE studies suggests that, although the choice of one uncertain outcome over another should not be affected by the timing of uncertainty or the sequence in which multiple uncertainties are revealed, people often make decisions that are inconsistent with this assumption.[29, 30] As a consequence of the nature of the WGS decision problem, individual reactions to information influence cascading probabilities and sequelae that are outside experimental control. This endogeneity then makes it challenging to obtain enough information through choices to determine how these uncertainties affect the validity of preference measures for features of WGS information.
In response to a similar methodological challenge, Gonzalez et al., demonstrated the feasibility of using a certainty equivalent function to obtain estimates for respondents’ interpretation of treatment-related risk using choice experiment data. However, an important assumption is that the only impact associated with the risk is the risk of the event itself. Thus, certainty-equivalent functions do not consider subsequent changes in decision-relevant factors conditional on the event occurring. For example, the certainty equivalent for survival assumes that survival corresponds only to premature death without any effects on quality of life. This assumption likely does not hold true in the case of WGS testing.
Thus, given current constraints on preference-elicitation capabilities, we had to simplify the problem considerably to ensure experimental control over multiple factors, including the risk of a specific variant finding, the risk of an associated health problem and the risk of subsequent comorbidities and mortality. We asked respondents to imagine that they have a specific gene variant identified through WGS that will result in a health problem. The health problem could be mild or severe, defined by limitations on activities of daily living, pain and discomfort, anxiety and a chance of dying in 10 years. We randomly assigned 1 of 2 gene variants to each respondent to consider with different associated health problems:
Gene variant 1:
Gene variant 2:
Respondents were randomly assigned 1 of 2 chances that they develop the health problems associated with the gene variant: 20% or 60% chance of developing the health problem. The combination of the severity of the health problems (mild vs severe) and chance of developing the health problems associated with the gene variant (20% vs 60%) resulted in four possible scenarios to which a respondent could be randomly assigned (Table 1). This study design reduced the otherwise intractable dimensionality of the problem, but at the cost of limiting what we are able to infer about more realistic decision making.
Unlike other health interventions and technologies, WGS produces no outcomes per se. It produces information that could lead to outcomes depending on how respondents react to the information. Evaluations of preferences for WGS must consider the downstream impacts on people’s health conditional on such reactions. If the findings are actionable, it is likely there could be positive outcomes from the associated treatments or prevention activities. For findings for which evidence for the best clinical action is not available, and findings of unknown significance, individuals could be better off without such knowledge. Also, for each scenario, there could be benefits, harms, and costs associated with any possible elective interventions to confirm a diagnosis or reduce the risk. There also is the potential for both positive and negative externalities to individuals other than the patient, such as awareness of potential genetic predisposition to health problems by family members or concerns about the quality of life of family members who support the person with potential health problems. That is, there could be risks and benefits to family members who did not choose to incur that risk or benefit.
In the design of our preferences survey for WGS, we had to decide how to capture interventions and downstream consequences and present these to respondents in the context of tradeoffs to prevent health problems associated with gene variants found by WGS. The challenge was to characterise the intervention and downstream consequences in a way that was specific enough for respondents to interpret and conceive of these aspects of WGS testing results, but broad enough to capture the range of interventions and consequences that might be associated with subsequent actions.
Furthermore, we chose not to specify specific gene variants or health problems (e.g. cancer) to reduce bias associated with ‘branding’ the disease. The association with disease is an important aspect of the value associated with testing information. Since the value of knowing is likely disease specific, by not including disease labels, we are not able to capture preferences that are driven by the utility of knowing about a specific disease that is of particular relevance or importance to an individual. Thus, our findings are even less likely to reflect revealed preferences (what respondents would actually do when faced with this information) than might be typically expected in DCEs because we are not including this detailed information.
We sought to impose experimental control over possible reactions to WGS information by defining 3 hypothetical interventions for preventing particular health problems: medicine, surgery and watchful waiting. After considering the scenario to which they were randomized, respondents evaluated 6 DCE questions, each with 2 hypothetical interventions and 3 scenarios (Figure 2). Each intervention was described by 3 attributes – chance that a side effect makes you unable to do everyday activities or take care of yourself, out-of-pocket cost, and follow-up requirement. In each DCE question, respondents indicated which intervention they would choose if these were the only interventions available. Half of the respondents had a choice from two different medical interventions (with different levels of associated side effects, cost and follow-up requirements) and watchful waiting that required a check-up every 6 months. The other half of the respondents had a choice from two different surgical interventions (with different levels of associated side effects, cost and follow-up requirements) and watchful waiting that required a check-up every 6 months.
We designed and administered an online survey to elicit preferences for WGS information to a general population sample of adults (21 years and older) in the United States. Of the 873 individuals invited to participate in the survey, 410 consented and fully completed the survey. This allowed for robust statistical analyses with a minimum acceptable level of statistical precision (standard error less than 0.05). Data were weighted to represent the US general population. We described WGS and a WGS report, and respondents were asked CV questions to elicit their willingness to pay for genomic status information using a double-bounded dichotomous choice elicitation format two-question bidding game (Figure 1).
For the DCE, attributes were identified based on a literature review, pre-test interviews (n=13) and consultation with genetics experts. We framed the survey to ask respondents to imagine that they have a specific gene variant identified through WGS that could result in a mild or severe health problem. The health problem was described by aspects of activities of daily living, pain/discomfort, anxiety and a chance of dying in 10 years. The DCE profiles described interventions that could be used to prevent health problems associated with gene variants identified through WGS. The attributes included: chance that a side effect makes you unable do everyday activities or take care of yourself, ongoing out-of-pocket cost, and requirements for follow-up monitoring of their condition.
We created the experimental design using a common SAS algorithm to maximize statistical and respondent efficiency and ensure all attribute levels are identifiable. The final design included 6 DCE questions and 14 profiles. A main-effects random-parameters logit (RPL) regression model was used to estimate relative preferences for each attribute level to avoid potential estimation bias from unobserved preference heterogeneity and account for within-sample correlation.[32, 33]
Our findings suggest that some people would be better off, as indicated by their willingness to pay, if they were allowed to make an informed decision to opt in to receive genomic information for which no medical treatment currently is available. This may be associated with getting their affairs in order if a gene variant were to affect their life expectancy, or hope that information on long-term risks may be actionable in the future. A substantial proportion of respondents did not value obtaining genomic information included in the basic report (38%, 95% CI: 33- 43%) or genomic information for which no treatment currently is available (55%, 95%CI: 50- 60%), even if this information were free. If they were interested in obtaining genomic information, respondents were willing to pay more for the basic WGS report ($299; actionable findings) than for the additional genomic findings for which treatment currently is unavailable ($180), suggesting that respondents valued the prophylactic or therapeutic benefits of the information in the WGS report.
Figures 3 and and44 show the estimated weights for the relative preference for outcomes associated with medicines and surgeries to prevent health problems from gene variants. The weights for watchful waiting indicate the relative preference for opting out of medicine or surgery when gene variants result in a health problem with a specific chance and severity. Results showed that preferences largely were consistent across respondents who evaluated interventions that involved surgery (Figure 3) and those who evaluated only medications (Figure 4).
For respondents who evaluated the surgery intervention, the risk of side effects was the most important attribute, followed by the cost of the intervention and follow-up requirement. There was no statistically significant difference between no cost and $50 per month. For respondents who evaluated the medicines intervention, the risk of side effects and the cost of the intervention were the most important attributes, followed by follow up requirement. One difference between respondents who evaluated surgery and medicines was their preference for watchful waiting over either medicine or surgery. Relative to surgery, respondents were more willing to take medicines to prevent health problems associated with gene variants. However, given the severity of the health problem, watchful waiting was consistently preferred over the average surgery when the likelihood of having health problems from the gene variants was 20% and symptoms from the potential health problems were described as mild. There also were statistically significant differences between relative preferences for watchful waiting when the chance of health problems was 20% with mild symptoms, and watchful waiting when the chance of health problems was 60% with severe symptoms.
Stated preferences surveys using DCEs are methodologically robust in part because the survey designer has control over the stimuli to create an experimental design to generate unbiased parameter estimates. DCEs use advanced quantitative methods in a utility theoretic framework where people’s underlying preferences are identified by experimentally controlled choice data.[11, 34] We used a DCE to quantify respondents’ preferences for attributes of alternative interventions that would be available to respondents if they developed health problems from gene variants identified through WGS.
Our findings appear logical and have face validity. They are consistent with our a priori expectations about the relative magnitude of the values encompassed in the value of WGS information. We found that respondents valued WGS information most if they could take preventive action to reduce their risk of a health problem through non-invasive interventions (medicines or watchful waiting were preferred to surgical intervention). This is consistent with the idea that people value the ability to plan and do other things such as medical interventions or monitoring rather than surgical interventions which have a risk of serious adverse events.
Secondly, preference weights and ordering for each of the attributes (prefer lower chance of side effects, lower out-of-pocket costs, minimal requirements for follow up care) were similar for both medicine and surgical interventions, except for the severity/likelihood of health problem resulting from gene variant. This difference between medicine and surgical interventions has face validity and highlights the complexity of estimating the value of WGS and the heterogeneity of preferences when the benefit-risk tradeoffs include both potential current and future downstream consequences for both the individual getting tested and their family. This preference heterogeneity is consistent with the literature on the extent to which people will change behaviour or accept an intervention in response to knowledge that they are at an increased risk of experiencing a negative health outcome. An extreme example is the ‘Angelina Jolie’ effect, where some women go to extreme measures (prophylactic breast removal) to avoid a risk of a negative outcome (breast cancer). This heterogeneity in preferences present challenges for evaluating health policies, particularly when considering complex technologies such as WGS and other personalized medicine interventions. Lee, Neumann & Rizzo have suggested that the value of knowing may encompass other values and the value of knowing is likely disease specific, which further highlights the challenges faced with tests that provide more than one result or results for multiple diseases, such as WGS.
Under our stated assumptions, we conclude that our DCE was successful in generating meaningful preference weights. In response to the first challenge of valuing multiple test results, we embedded CV questions into our survey to elicit willingness-to-pay for a basic genomic report that includes only actionable genomic status information and one for which the medical treatment is currently unclear and would not be included in a basic genomic report. For the second challenge of cascading uncertainty, we added an experimental design to control multiple factors – the risk of a specific gene variant, the risk of a health problem and the risk of comorbidities and mortality – on top of the actual DCE question. To address the third challenge of measuring the value of information and capturing the associated downstream consequences of testing, we introduced 3 hypothetical interventions to prevent the potential health problems in the scenario they were presented. However, solving the challenges posed by this valuation problem required simplifying a number of realistic complications of WGS testing and subsequent decision making.
Our experience has implications for designing stated-preference studies to answer questions about complex technologies such as WGS in the future. Although we believe that we have at least partly addressed the 3 challenges summarised in this paper, it is unclear how much simplification is acceptable for a DCE to still be useful and relevant. WGS testing requires people to make decisions that involve multiple choices that we were not able to fully describe in a stated-preference survey, or capture in sufficient detail to reflect all of the combinations and permutations of the sequelae or the timing of these events, all of which have an effect on the associated value of the intervention.
We were forced to simplify and adapt the design of the survey to try to control for cascading uncertainties and possible behavioral responses. Within the current limits of stated-preference methods, we realized that we either could ask questions about the information we were seeking that were too difficult for most respondents to answer, or could ask questions that most respondents could answer, but which misrepresented in critical ways the real-world complexity of the research problem. One alternative to consider in future studies would be to administer the DCE survey in a limited set of respondents who had previously decided to acquire WGS information. It may be feasible for these respondents to answer a more complex set of questions about their willingness to act on this information.
Although our findings are logical in general, a limitation is that they may not be generalizable to inform policy since our findings could be an artifact of the necessary problem simplifications to conform with the theoretical requirements and good research practices. If we were to assess our survey development, design and analysis against commonly accepted standards using the 10 points on the ISPOR checklist for good research practices, our survey would fare well. However, we are left with the possibility that a DCE survey may simply not be the right method to elicit stated preferences for these kinds of complex technologies that include endogeneity associated with multiple decisions and uncertainties. As researchers, while we are often required to simplify problems, at some point, it may be that the available methods for stated preferences are simply not adequate. While there is an important role for stated-preferences research in the context of patient centered care, we need to be willing to re-examine the alignment of the methods to the problem and explore methodological advances to stated-preferences design and analytic techniques that can capture meaningful preferences for these complex tests and outcomes. Future research should evaluate whether the assumptions and simplifications we applied to model respondents’ decisions are consistent with how people make decisions to acquire WGS information and disease-specific effects on these decisions.
We thank Dr. Kathryn A Phillips and Michael Douglas for their contributions to the study conceptualization and interpretation. We thank Denise Lautenbach, Jill Oliver Robinson and the MedSeq study team for their advice and guidance with developing our survey and recruiting participants for pre-test interviews.
This study was partially funded by a NHGRI grant to Kathryn A Phillips (R01HG007063), a NCI grant to the UCSF Helen Diller Family Comprehensive Cancer Center (5P30CA082013-15), and the UCSF Mount Zion Health Fund. Deborah A Marshall is supported by a Canada Research Chair, Health Services and Systems Research and the Arthur J.E. Child Chair in Rheumatology Outcomes Research.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.