|Home | About | Journals | Submit | Contact Us | Français|
Qualitative research can inform the development of asthma patient-reported outcome (PRO) measures and user-friendly technologies through defining measurement constructs, identifying potential limitations in measurement and sources of response error, and evaluating usability.
The goal of the current study was to inform the development of a comprehensive asthma PRO assessment with input from patients and clinical experts.
Self-reported adult asthma sufferers recruited from a 3,000 member New England-area research panel participated in either one of three focus groups (N=21) or individual cognitive item debriefing interviews (N=20) to discuss how asthma impacts their health-related quality of life (HRQOL), and provide feedback on a preliminary set of asthma impact survey items and prototype patient report. Focus groups and cognitive interviews were conducted using traditional research principles (e.g., semi-structured interview guide, probing, and think aloud techniques). An Expert Advisory Panel (N=12) including asthma clinical specialists and measurement professionals was convened to review results from the focus group and cognitive interview studies and make recommendations for final survey and report development.
Domains of health impacted by asthma included physical (recreation, play, competitive sports, and exercise), social (activities, family relationships), emotional (anger, upset, frustration, anxiety, worry), sleep, role (recreational/leisure activities; work), and sexual functioning. Most items in the impact survey were easily understood, covered important content, and included relevant response options. Items with contradictory examples and multiple concepts were difficult to comprehend. Suggestions were made to expand survey content by including additional items on physical and sexual functioning, sleep, self-consciousness, stigma, and finances. Reports were considered useful and participants saw value in sharing the results with their doctor. Graphic presentation of scores was not always understood; participants preferred tabular presentation of score levels with associated interpretative text. Display of inverse scores for different measures (higher scores equaling better health on one scale and worse health on another) shown on a single page was confusing. The score history section of the report was seen as helpful for monitoring progress over time, particularly for those recently diagnosed with asthma.
Expert panelists agreed that displaying inverse scores in a single summary report may be confusing to patients and providers. They also stressed the importance of comprehensive interpretation guidelines for patients, with an emphasis on what they should do next based on scores. Panelists made recommendations for provider and aggregate-level reports (e.g., “red flags” to indicate significant score changes or cut-points of significance; identification of subgroups that have scored poorly or recently gotten worse).
Incorporating input from patients, clinicians, and measurement experts in the early stages of product development should improve the construct validity of this PRO measure and enhance its practical application in healthcare.
Increasingly, the success of healthcare is being defined in terms of its ability to improve patients’ functional status and well-being, constructs that are often measured using patient self-report.[1–3] Use of self-report measures may enhance the clinical encounter, since patient perceptions of health and illness, treatment satisfaction, and self-management skills have an influence on treatment adherence and preventative health behaviors.[4–6]
In recent years, there have been innovations in technology designed to support patients and their health care providers in achieving positive health outcomes, including the development of electronic systems for data capture, information management, health record integration, and reporting.[7–11] Additionally, advancements in psychometric methods, such as the use of Item Response Theory (IRT) in test development, offer substantive practical advantages to classical methods of test construction. PRO measures can be improved using item response theory (IRT), and competing objectives of more practical and more precise tools can be achieved over a wide range of severity levels using computerized adaptive testing (CAT).[13–19] During the last decade, CAT applications have been increasingly used in the assessment of health outcomes.[15–30] These applications require a large set of items (banks) in any one functional area; items that consistently scale along a dimension of low to high functional proficiency; and rules guiding starting, stopping and scoring procedures.
Qualitative research can inform the development of PRO measures and user-friendly technologies.[31–36] In 2006, the Federal Drug Administration published draft guidelines for the development of PRO measures, recognizing that survey development “is incomplete without patient input.” For example, focus groups can help researchers identify and define constructs that are relevant from the patient point-of-view, detect possible limitations in a survey, and generate ideas for hypothesis testing. Cognitive item testing can evaluate sources of response error in the survey questionnaire that may affect results (e.g., problems that will reduce response reliability or change the meaning of responses). Usability studies can evaluate the survey and the technology platform upon which it is delivered (interface design, navigational elements, and user preferences). Such research enables the developers of PRO measures to evaluate the application of basic design principles.[38–40]
Numerous studies have used qualitative research to understand risk factors associated with asthma, the impact of asthma on quality of life, beliefs about asthma and perceptions of treatment, and how health technologies and treatment interventions may improve health outcomes for asthma patients.[41–53]
Qualitative research was used to inform the development a comprehensive PRO measure, collectively referred to as the ASTHMA-CAT™. The ASTHMA-CAT assessment combines asthma impact, asthma control, and generic health-related quality of life (HRQOL) measures in a single administration, yielding patient, provider, and aggregate feedback reports.
The DYNHA Asthma Impact Survey (DYNHA® AIS™) is the CAT component of the ASTHMA-CAT assessment.[54,55] Its development was modeled after the DYNHA Headache Impact Test (DYNHA HIT). DYNHA AIS draws on a 37-item impact bank constructed to develop disease-specific CATs.
A prototype DYNHA AIS was piloted in a disease management population and was found to reduce response burden while providing equally precise scores as compared to the full AIS item bank. Static (full bank) and dynamic AIS versions discriminated between respondents with differing levels of asthma severity, and mean static and dynamic AIS scores were equivalent within severity categories. However, findings also indicated the need for wider coverage of the functional impact of asthma, including items with greater specificity in content, in order to precisely assess asthma patients at varying levels of severity, and to capture changes in impact due to treatment.[56–58]
The goal of the current study was to construct and evaluate an expanded AIS item bank by: (1) conducting focus groups with asthma patients to confirm the importance of hypothesized domains and gaps in content coverage, and to collect feedback on the important components necessary for inclusion in a patient feedback report; (2) conducting cognitive interviews with asthma patients to evaluate sources of response error in the survey that may affect results (e.g., problems that will reduce response reliability or change the meaning of responses); and (3) convening an Expert Advisory Panel of measurement professionals and clinical experts in asthma to advise survey and report development efforts.
Participants (N=21) were sampled from a research panel database including approximately 3,000 members from the southern New England area. The database was developed by American Institutes for Research (AIR) over a period of several years and continues to be updated through active recruitment on websites, listservs, and discussion boards; local newspapers; postings at businesses, community centers, libraries, hospitals, and other public venues; and referrals from research partners and past study participants. Sampling criteria included self-reported adult (ages 18 years and older) asthma sufferers, who were able to read and write in English. Current smokers and those reporting current depression, congestive heart failure, chronic bronchitis, chronic obstructive pulmonary disease, emphysema, pneumonia, and respiratory conditions other than asthma were excluded from the study. AIR emailed potential panelists who met at least one of the study criteria and informed them of the study. Individuals interested in study participation contacted the study coordinator for screening. To ensure adequate representation across control levels, the Asthma Control Test (ACT™) was fielded in the screening survey. Seventy individuals were screened before meeting target enrollment.
The following measures and materials were used in the focus groups:
SF-8 Health Survey (SF-8™) is an 8-item generic survey measuring functional health and well-being, administered solely as a means to explain origination of SF-8 scores in the patient feedback report.[61–63]
Background Information Survey is a 13-item module assessing participant characteristics.
Chronic Conditions Checklist is a 2-item module assessing presence of co-existing conditions: “Has a doctor ever told you that you had any of the following conditions?”, and “Do you now have any of the following conditions?”
Three focus groups, held at a Boston-area research facility, were led by the same moderator, a PhD-level researcher with nearly 20 years of experience in measurement and evaluation. Participants signed informed consent documents prior to the start of their focus group. The moderator used a semi-structured interview guide and each session included structured discussion to define health-related quality of life (HRQOL); identify ways asthma impacts HRQOL; rank order HRQOL domains most impacted by asthma; review AIS items for content, meaning, clarity, layout; identify content areas not covered in the AIS; suggest improvements to the survey and report content; and recommend possible uses for the survey and report. Through discussion, the group identified ways asthma impacts quality of life, and then independently rank ordered their top five impacts. A standard list of probes was developed for the discussion, but used only when needed. Each 2-hour session was videotaped and transcribed. Participants received $100 incentive for taking part in the study.
Recordings from each focus group session were transcribed, and two trained researchers independently reviewed the transcripts in their entirety. Data from the activity to identify the top ways asthma impacts HRQOL were tallied. Content analyses were performed manually. Unique and repetitive responses to all questions were noted. Following a grounded theory approach, themes emerged from the data (emic approach). After reading and reviewing the data multiple times to look for similarities and differences, the two researchers categorized themes using open coding. Any conflicting opinions were resolved through debate until consensus was achieved.
A sample of participants (N=20, non-overlapping with the focus group sample) was drawn from a research panel database including approximately 3,000 members from the southern New England area. Recruitment and sampling criteria were identical to those in the focus group study. Sixty individuals were screened before meeting target enrollment.
Measures included the AIS, Chronic Conditions Checklist, and 36 experimental asthma impact bank items constructed based on focus group results (covering general health, sleep, finances, stigma, and physical, mental, role and social functioning).
Remaining ASTHMA-CAT measures were administered to participants, solely for the purpose of explaining the origin of scores in the patient feedback report.
Score reports were edited based on focus group results, and a revised version was shared with cognitive test participants for their input.
Individual semi-structured cognitive interviews were conducted in a Boston-area research facility by one of two AIR (PhD and MS-candidate) researchers. Participants signed informed consent documents prior to the start of each session which was recorded in its entirety on audiotape. Also, the moderator recorded verbatim notes and responses to all questions in an electronic database.
Participants were asked to carefully review each item and to think aloud as they formulated their responses. This allowed data on the cognitive processes respondents used to comprehend information and formulate their responses to be recorded.[65,66]
Participants were then asked to review the report options, and provide feedback on the clarity, layout, organization, and perceived usefulness of the report.
Participants received $100 incentive for taking part in the study.
Recorded feedback from each cognitive interview was used to evaluate cognitive processes, item interpretation and comprehension, memory recall of relevant information, decision processes, and response processes.[65,66] Each item was evaluated for potential sources of response error (e.g., problems that may reduce response reliability or change the meaning of responses). The analytic focus was on identifying possible problems and frequency with which they occurred.
An Expert Advisory Panel of asthma clinical specialists (n=7, including 4 MD, 1 MD-PhD, 2 PhD-level) and measurement professionals (n=5, including 4 PhD and 1 MA-level) was convened to review and advise survey and report development efforts. The panelists were selected from health management organizations, academic medical centers, and private practice to represent diverse interests and expertise in asthma clinical practice and research, disease management, patient advocacy, and HRQOL measurement (including three authors on this paper). Panelists were paid for their time and travel, and the meeting was facilitated by an expert moderator from the American Institutes for Research.
Following the focus group and cognitive debriefing studies, a full-day meeting of the panel was held to solicit feedback on content areas most impacted by asthma, content additions to the AIS bank, and feedback reports.
Twenty-one adults self-reporting asthma from Massachusetts, varied in terms of age (range 24–59 years), gender (62% female), race/ethnicity (76% White, 19% African American, 5% Asian), and asthma control (38% controlled, 29% somewhat controlled, 33% uncontrolled) participated in one of three focus group sessions. Table I summarizes sample characteristics. Five individuals participated in group 1, and eight participated in groups 2 and 3, respectively.
Figure 2 summarizes how participants defined and described quality of life. In particular, they associated good quality of life with the ability to engage in and enjoy exercise, physical activity, and recreational, travel and leisure activities. Good quality of life was also associated with happiness, independence, and health. Others emphasized the importance of personal time, and time spent with family and loved ones.
Participants identified several areas impacted by asthma, and primarily focused on physical, social, and emotional aspects of health (see Table II). Participants described limitations in various physical activities, including recreation, play, competitive sports, and exercise. Several noted physiological effects, associating asthma with increased susceptibility to infection and as a factor in escalating illness. Most indicated that exposure to environmental triggers resulted in negative physical, emotional, financial, and, most notably, social consequences. Respondents described negative effects on social activities and family relationships. Interestingly, one participant noted a positive effect on family life when other family members benefited from a household no-smoking rule.
When describing emotional impact from asthma, participants cited feelings of anger, upset, frustration, anxiety, and worry. These feelings were often mediated by other functional limitations (e.g., social). The anticipation of a future asthma attack and how this anticipation affects subsequent behavior was a cause for concern for many participants. Those with poorly controlled or uncontrolled asthma were particularly vocal about these issues. Additionally, several respondents discussed the impact of asthma on sleep and resulting fatigue; the effects of asthma treatment and associated burden; limitations on travel and other recreational/leisure activities; financial burden; impact on work and productivity; and other personal consequences.
In rank order, HRQOL domains most impacted by asthma were physical, social, emotional, financial, sleep, and work.
Participants identified favorable characteristics of the AISsurvey, instructions, items and response options, reporting that most items were clear and easy to understand, underlining emphasized key text, content covered the important areas, and response options were appropriate. Participants also noted some problem areas. Due to the seasonal effects of asthma, some preferred a 3-month recall period to the 4-week recall used in the existing bank. Some participants noted difficulty attributing health problems solely to asthma. Items with contradictory examples (e.g., “reading a book or exercising”) were difficult to answer, and participants noted that some items contained repetitive content. Content coverage could be expanded to include more items on anxiety, worry, and fear; and hierarchical probing items are needed for more depth of coverage. Participants favored response options presented in the same direction.
Participants were interested in feedback reports that showed their progress over time, with sufficient score interpretation, and saw value in sharing results with their doctor. Specifically, participants considered asthma-specific scores (ACT and AIS) to be very useful. Tabular presentation of score ranges and interpretive text was familiar and recognizable, and easier to understand than bar graphs. Participants said the report could be used to track asthma outcomes over time, and this was seen as a particularly useful feature for those newly diagnosed with the condition. Participants indicated that they would share this report with their doctor as a way to communicate about their health, and thought the information provided would add value to the overall clinical encounter.
Participants also identified problems with the report and suggested possible improvements. Bar graphs displaying norm-based scores (t-scores) were difficult for most to interpret. Scores, surveys, and normative comparisons could be improved with better labeling and more thorough explanation. Participants preferred to view their own progress over time, rather than detailed information on normative comparisons. Bold, red font was interpreted by some as bad or severe; while others thought it helped bring attention to important information. It was difficult for participants to match score output with the corresponding survey that had been administered. Participants preferred a focus on asthma-specific rather than generic outcomes, and some were uncomfortable with the emotional health score output. A few participants indicated that the mental health scores and associated interpretation text may make them feel depressed or may set them back.
Twenty adults self-reporting asthma from the Boston area who varied in terms of age (range 18–64 years), gender (65% female), race/ethnicity (80% White, 10% African American, 5% Asian, 5% Middle Eastern or Indian), and asthma control (30% controlled, 35% somewhat controlled, 35% uncontrolled) participated in the one-on-one interview sessions (see Table I).
Participants indicated that most items and instructions were worded clearly, easily understood, and included a relevant recall period. Instructions and items were “easy” or “very easy” to understand. However, comprehension was reduced when items contained multiple concepts (e.g., participants reported that they would respond differently depending on whether asked about work, school, or daily activities), vague terms, words with dual meaning, or perceived irrelevant content (e.g., asthmatics might be asked “How often did you sit down and rest”, but not “…lie down and rest” since lying down often increases distress). Interpretation improved with simplified language and when specific examples were provided in the item stem. Participants suggested it may be important to include a “not applicable” response option, particularly for items with activities not common to all adults (e.g., “…participate in competitive sports?”, “…run a short distance?”, “…playing with children?”). Some participants expected the survey to cover the impact of asthma medications and the impact of specific triggers and/or environments on asthma.
Generally, participants reported no difficulty recalling events over the past 4 weeks, and as they answered items, considered “average asthma impact” over this timeframe. However, for some items, a 4-week recall period seemed a mismatch (e.g., in the case of assessing impact of “cost of asthma treatment”, participants reported that past 4 weeks seemed too short a recall timeframe).
Similar to the focus group sample, cognitive testing participants were interested in feedback reports that showed their progress over time, with sufficient score interpretation, and saw value in sharing results with their doctor. Most preferred to see a summary report showing scores for all survey modules, with options to learn more (e.g., hyperlink to additional interpretation). And participants who viewed the feedback summary before the individual survey report modules seemed better able to interpret scores and interpretative information than those who viewed the single reports first.
Replicating prior results, participants had difficulty correctly interpreting scores shown in a graph, and preferred a table of possible score levels with associated interpretative text. Participants said that the survey results could help them communicate better with their doctors, and those newly diagnosed with asthma may find the information particularly useful for self-management. Further, participants reported that the survey would be more useful if they could access it from home on a periodic basis. The same respondents liked the “Your Progress” section of the report (see Figure 1).
Panelists provided feedback on content areas most impacted by asthma (see Figure 3), and then reviewed focus group and cognitive testing results. Although several content areas were discussed, the panelists seemed to focus on physical, role, and sleep domains. Panelists also reviewed each item in the developmental AIS item bank, recommending specific revisions to item stem and response options and additional content coverage (e.g., sexual functioning domain).
Panelists reviewed and provided input on patient, provider, and aggregate feedback reports. One of the main discussion points centered around score/scale direction; as the generic survey is scored so that higher scores equal better health, while higher scores indicate worse health on the asthma-specific impact survey. Panelists recommended that all scales shown in a combined report should be scored in the same direction. For patient reports, the focus was on ensuring efficient, yet comprehensive interpretation guidelines with more emphasis on the “What You Should Do” section (including actionable next steps for self-management). Panelists agreed that progress over time should be shown for as many time points as possible (e.g., hyperlink to full score history). Also, they agreed that the report should inform patients when they should take the survey next. They requested that the report take patient reading level into account.
Panelists suggested that the provider report display item-level responses, so clinicians can understand which items are driving the resulting score. Also, panelists suggested that the reports display a “red flag” when scores change substantially or exceed a cut-point of significance. For instance, this system could be used to inform a clinician that his/her patient screens positive for depression, or it could be used to instruct a patient to contact her/his doctor immediately for treatment follow-up.
They also suggested that the aggregate report should display distribution of scores, and flag sub-groups (e.g., clinic groups, regional groups, etc.) that have scored poorly, as well as those that have recently gotten worse. Finally, they recommended that the aggregate report highlight which groups are doing better or worse than normative averages.
The goal of this study was to incorporate patient and provider perspectives into the development of an asthma outcomes assessment and feedback reports.
Similar to previous work exploring how asthma impacts HRQOL[67–68], several areas were identified, with a focus on physical (recreation, play, competitive sports, and exercise), social (activities, family relationships), and emotional (anger, upset, frustration, anxiety, worry) functioning; sleep and resulting fatigue; the effects of asthma treatment and associated burden; limitations on travel and other recreational/leisure activities; financial burden; impact on work and productivity; and sexual functioning.
Most items in the AISsurvey were clear and easy to understand, covered many of the important content areas, and included relevant response options. Some participants suggested that a 3-month recall period may change how they report asthma impact, due to the seasonal effects of the condition. Since the overall ASTHMA-CAT assessment is intended to be administered at regular intervals to monitor changes in asthma impact and control over time, and since respondents had no difficulties recalling events over the past 4 weeks, the shorter recall period will be retained for future work.
Additionally, participants favored response options presented in the same direction, and some requested a “N/A”, or not applicable, option for content areas not likely to be experienced by them. Future iterations of item bank development will test the inclusion of a N/A response option and other ways to tailor the assessment (i.e., content selection procedures) will be considered.
Items with contradictory examples and those containing multiple concepts were difficult to comprehend. For example, a single question asked respondents how much asthma limits their performance in housework, work, school, or social activities. Revised AIS bank items have been developed based on detailed findings from the current study. The new bank includes the original items, plus a set of experimental items designed to be more concise and focused in their content coverage (limiting each item to one idea).
Also, suggestions were made to expand AIS content and depth of coverage. In addition to cognition, fatigue, general social support, mental health, and role and social functioning, the revised item bank provides additional coverage for finance, physical functioning, self-consciousness, sleep, stigma, and sexual functioning. This bank, covering new content areas and providing shorter and more specific items, will be evaluated in a forthcoming large-scale item calibration study.
In response to patient feedback reports, participants considered this information to be useful, saw value in sharing the results with their doctor, and thought the information would add value to the clinical encounter. Score interpretation was facilitated by tabular presentation of score levels with associated interpretative text. Some participants reported that the ability to track progress over time would be particularly useful for those recently diagnosed with asthma. In addition, participants reported that the assessment would be especially useful if they could access it from home on a periodic basis, with the goal of monitoring their progress over time.
Expert panelists agreed that displaying scales with varied score direction (i.e., high scores indicating better health on one scale and worse health on another) in a single summary report may be confusing to patients and providers. They also stressed the importance of comprehensive interpretation guidelines for patients with more emphasis on the “What You Should Do” section of the report. Future work will evaluate reporting options to enhance patient self-management.
Panelists requested item-level responses for provider reports and “red flags” to indicate significant score changes or cut-points of significance (i.e., likelihood for depression), information that may be useful for clinical decision-making. Panelists also recommended red flags in aggregate reports to identify subgroups that have scored poorly or recently gotten worse, and indicators to highlight those groups doing better or worse than normative comparisons.
Participants in this study generally were well educated, a potential limitation, which should be considered prior to finalizing the tool. Possible ways to address this issue might include application of readability statistics, heuristic evaluation, and/or validation studies with asthma sub-groups.
Although qualitative research often relies on small participant samples, it enables researchers to gather detailed information useful for product conceptualization, design, and implementation.[69–71] In this study, patient-centered research helped to identify gaps in measurement, possible sources of survey response error, areas for improvement in survey and report development, and new hypotheses for future testing that may not have been discovered vis-à-vis traditional quantitative methods.
Incorporating input from patients, clinicians, and measurement experts in the early stages of product development should improve the construct validity of this PRO measure and enhance its practical application in healthcare.
This research was supported in part by an NIH-sponsored grant (National Heart, Lung, and Blood Institute, #2R44HL078252-02) and QualityMetric Incorporated from its own research funds. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Heart, Lung, and Blood Institute or the National Institutes of Health. Dr. Turner-Bowker, Dr. Saris-Baglama, and Mr. DeRosa are employed by QualityMetric Incorporated. The focus groups and cognitive interviews were conducted by Dr. Paulsen and Mr. Bransfield while they were employed at American Institutes for Research under a subcontract from QualityMetric Incorporated.