|Home | About | Journals | Submit | Contact Us | Français|
Our aim was to assess the content validity of the Patient-Reported Outcomes Measurement Information System (PROMIS) social health item banks by comparing a prespecified conceptual model with concepts that focus-group participants identified as important social-health-related outcomes. These data will inform the process of improving health-related quality-of-life measures.
Twenty-five patients with a range of social limitations due to chronic health conditions were recruited at two sites; four focus groups were conducted. Raters independently classified participants' statements using a hierarchical, nested schema that included health-related outcomes, role performance, role satisfaction, family/friends, work, and leisure.
Key themes that emerged were fulfilling both family and work responsibilities and the distinction between activities done out of responsibility versus enjoyment. Although focus-group participants identified volunteerism and pet ownership as important social-health-related concepts, these were not in our original conceptual model. The concept of satisfaction was often found to overlap with the concept of performance.
Our conceptual model appeared mostly comprehensive but is being further refined to more appropriately (a) distinguish between responsibilities versus discretionary activities, and (b) situate the outcome of satisfaction as it relates to impairment in social and other domains of health.
Patient-reported outcomes (PROs) encompass aspects of health-related quality of life (HRQoL) including symptoms, limitations, well-being, and patient preferences regarding health states, treatments, and costs. PROs measure patients' health status using information directly from the patient, without interpretation of the patient's responses by a physician or anyone else . The Patient-Reported Outcomes Measurement Information System (PROMIS) project was undertaken to help advance PRO measurement for clinical, social science, and epidemiological research, as well as for clinical practice. The overall aim of PROMIS is to develop reliable and valid approaches to measuring PROs using item response theory and computer-adaptive testing. A multidisciplinary panel of researchers from institutions nationwide collaborated to construct a framework for the measurement of general health concepts. As a starting point, PROMIS adapted the original structure of the World Health Organization's (WHO) physical, mental and social framework. The WHO framework is broad and inclusive, and flexible enough to allow integration of other theoretical perspectives [2-5]. More recently, the WHO (2001) has replaced its three-domain framework of health with an International Classification of Functioning, Disability and Health (ICF) . Multiple domains were identified for each of the three original WHO health concepts (physical, mental and social); for more in-depth discussion of this process, see Cella et al. . Five domains were ultimately selected for the initial work on PROMIS: pain, fatigue, emotional distress, physical functioning, and social role participation. These five constructs were chosen because each met the following criteria: 1) the construct cuts across many chronic health conditions; 2) the construct is frequently used as an outcome variable in clinical trials; 3) the latent trait level could be modeled on a continuum; and 4) candidate items could be measured on an ordinal scale and eventually tested with classic and modern psychometric methods [2, 3, 6]. The selected domains represent the initial foray of PROMIS toward developing a measurement system and are not intended to cover all areas of the ICF Framework.
This study focuses on the social domain, which is defined in PROMIS as perceived well-being regarding social activities and relationships, including the ability to relate to individuals, groups, communities and society as a whole. Primary components of social health and functioning measured in current research include social role participation, social network quality, interpersonal communication, and social support [7-12]. We were most concerned for PROMIS with social role participation and satisfaction since these concepts most closely align with outcomes rather than processes. The concept of social role participation is distinct from the others mentioned (social network quality, interpersonal communication, and social support), and reflects one's involvement in, and satisfaction with, usual social roles including those within marriage/partnership, parenting, work, and leisure activities [8, 13]. Social role participation has also been referred to as “social adjustment”. In contrast, measures of social support generally seek information about a person's perception of the availability or adequacy of resources provided by other persons . Social support was excluded from the social health domain because to date it has been generally deemed a “process,” not “outcome” variable.
For the social health domain, a multi-institutional panel of PRO experts convened regularly with two main initial objectives: to define the social health domain through a conceptual model (also termed “domain map”), and to construct banks of items to measure the most salient aspects of social health-related outcomes . These two tasks were carried out concurrently and interdependently, and are described below. These objectives were in the larger goal of using both qualitative and quantitative data to inform the development of improved measures of HRQOL. The present study describes the use of qualitative focus group data.
The strategy for defining the domain was to review past literature, draw on the expertise of the panel members and the larger PROMIS network, and come to consensus on the domain's subcategories. To reach consensus, we used the Delphi panel technique. This technique was originally designed in the 1950s to help forecast future events; it is now used in a number of fields to obtain expert input on planning activities from individuals who are widely dispersed geographically . The process began with an open-ended query given to a panel of experts (“what are the outcome-related components of social health and well-being?”), who submitted itemized feedback in response. In subsequent iterations, the panel members rated the relative importance of the feedback received, and made changes to phrasing and content. Consensus was reached after three such iterations. Two subcategories of social role participation were proposed as a result of this consensus process - performance of and satisfaction with social roles in three contexts: family/friends, work/school, and leisure activities (Figure 1) .
The strategy for item bank development was to identify, evaluate, and modify items from existing surveys, and lastly, to develop new items where necessary. To identify existing items, we conducted comprehensive literature searches for social health questionnaires resulting in 1781 social health items. Construction of the item banks and domain map, as well as the processes used to narrow the item pool are described in detail elsewhere . To summarize, items were re-written to improve clarity, reduce the number of response options, and utilize a 7-day time frame. The final item pool contained 112 items, the majority of which resulted from new item development. When necessary, intellectual property agreements were obtained from instrument authors to provide PROMIS permission for use.
We were concerned that aspects of the domain that are important to patients may be over- or underemphasized in the academic literature. To address this problem, we conducted focus groups to elicit feedback directly from participants who had reported social health limitations. We aimed to obtain input that would help fill in potential conceptual gaps in the domain map, and inform writing of new items if necessary. Our goal was to qualitatively assess the content validity of the social health item banks by evaluating the fit between those concepts that make up the social health domain map and those concepts that focus group participants identified as important aspects of social health-related outcomes. A key assumption of this approach was that the focus group discussions would provide qualitative information that would either support or refute the concepts of the social health domain as it was defined.
We evaluated the extent to which the information gained from the focus groups confirmed, contradicted, or added to the social health domain map (Figure 1), via the following research questions:
Both in developing the item banks and in defining the domain, we aimed to capture aspects of social health and well-being that would be most likely associated with changes in health status.
We analyzed data from 25 participants who took part in four focus groups at two sites. Raters worked independently to classify participants' statements based on a schema derived from the domain map. The resultant coded transcripts were assessed to determine the comprehensiveness and appropriateness of the domain map in covering important social health PROs.
Because a goal of PROMIS is to design instruments to measure domains that cross many chronic health conditions, we did not believe it was feasible to conduct condition-specific focus groups. Instead, we adopted the strategy of selecting a sample of patients who stated they had experienced a range of social limitations [3, 17]. Recommendations for both the number of focus groups and sample size vary. The number of recommended sessions depends on the complexity of the study design and the target sample's level of distinctiveness [18-21]. Stewart et al. (2007) observed that rarely are more than 3-4 focus groups conducted in the social sciences. We felt that two groups at each site would limit bias that might be seen in a single group or site and allow us to examine themes common across groups. We chose a sample size of at least five participants per group because recommendations in the literature range from 4-8 participants , 4-12 participants , 6-12 participants [21, 23], or 8-12 participants per group . Smaller groups allowed facilitators to do more in-depth follow-up queries about social limitations in order to examine the conceptual coverage for the PROMIS social health domain map. Finally, different facilitators conducted the groups in order to minimize the potential for investigator bias.
At both sites, we recruited outpatients who reported impairment in social functioning. A range of health conditions was represented (e.g., arthritis, heart disease, diabetes, and depression), and participants were heterogeneous with regard to mental versus physical impairment. Participants were recruited from outpatient clinics at two of the PROMIS study sites: the University of North Carolina (UNC) General Internal Medicine Clinics and the University of Pittsburgh adult outpatient psychiatric clinics at Western Psychiatric Institute and Clinic (WPIC). A multimodal recruitment strategy (including print media, flyers, mailed letters, and health care provider referrals) was employed in order to get as many patients together as possible, and to aim for demographic diversity. A non-response rate could be calculated only for the portion of the sample that was recruited by mail (n=115). Of those, 96 (83%) did not respond to the letters. The majority of non-respondents were Caucasian and female, as were the majority of the respondents. Respondents and non-respondents did not differ significantly with respect to ethnicity, gender or education. To be included in the study, all participants had to be at least 18 years old; had seen a physician for a chronic or mental health condition within the past 5 years; had to speak and read English; and had responded affirmatively to recruitment materials asking for participants with limitations in their activities due to health problems. No participants who met the eligibility criteria were excluded.
All participants provided written informed consent in person after hearing the facilitator describe the study and consent processes. Institutional Review Boards at each site approved the study (UNC IRB#05-2751; WPIC IRB#0511028), and all procedures were conducted in compliance with the Declaration of Helsinki. Appendix A is a sample facilitator's script that was based on the domain map. Participants were not shown individual items from the item banks; rather, they were led through a series of open-ended discussion questions relating to social activities, well-being, and limitations. Each session lasted a maximum of 1.5 hours, was digitally recorded, and transcribed.
Three raters, each of whom was previously unfamiliar with the domain map, were trained to use qualitative analysis software (Atlas.ti 5.2)  to independently code statements within the social health focus group transcripts. Coding consisted of selecting and classifying statements that related to the following hierarchical, nested classification schema (see Table 1 for definitions and examples):
Each rater designated blocks of text independently, with the instruction to include more, rather than less, context for each given concept (i.e., raters were advised against selecting just one or two words at a time). Raters were instructed to assign multiple codes in cases where more than one concept was mentioned. The coding process was done iteratively in three passes, corresponding to the three schema levels above. In the first pass, raters read all text and assigned the “HRO” code to applicable statements. In the second pass, raters examined HRO statements, and assigned additional codes of “RPP” (role participation performance), “RPS” (role participation satisfaction), or “both RPP and RPS”. In the final pass, raters further refined the coding into one or more of the context areas (family/friends, work, leisure, or other).
If a statement was too general to be categorized beyond the HRO level, it was left as is. For example, the statement “I'm just not able to do things I used to do,” in the absence of further context or specificity, would be coded as “HRO.” However, if the participant gave multiple contexts (e.g., “…to do things I used to do, like playing tennis or spending time with my grandchildren”), this would also have been coded as both “leisure” and “family”. Statements in the last pass that were specific but did not fit into the categories of family/friends, work, or leisure were coded as “Other.” To improve consistency and address questions and ambiguities in the coding process, the raters reconvened for discussion after the completion of one transcript out of the four. The coding process was geared toward the content analysis and the assessment of the domain map. Thus, it involved non-mutual exclusivity of codes; it also lacked a finite set of instances for potential agreement between raters. These characteristics ruled out assessment of inter-rater agreement using tools that rely on these discrete counts, such as the kappa measure of agreement [25-27].
In assessing fit between the domain map and focus group content, the first step was to identify recurrent themes among those statements coded as social health-related outcomes. The next step was to identify potential qualitative indicators of lack of fit. Prior to conducting the analysis, we envisaged two types of such potential indicators. Recurrent relevant themes among statements that were not assigned any code, or coded as “Other” would indicate that our domain map was not comprehensive. In addition, consistent difficulty by the raters in assigning the codes to particular content would indicate that our domain map was not appropriately specified.
A total of four focus groups were conducted, with 10 participants at the UNC site and 15 at the WPIC site, totaling 25 participants. Table 2 shows descriptive characteristics of the focus group participants, who were heterogeneous with regard to gender, mental/physical condition, and severity of impairment. Multiple chronic comorbidities were common.
Tables 3 and and44 show examples of quotes that were coded as role participation performance and/or role participation satisfaction. Participants expressed varying degrees of engagement in social roles involving family/friends, work, and leisure. A theme that emerged in statements about performance was the distinction between doing things out of responsibility (e.g., childcare) versus enjoyment (e.g., reading poetry); see Table 3. The balance between family, work, and leisure was also often cited as important to participants. In statements about satisfaction (Table 4), participants often spoke about frustration or other emotions associated with their ability to get along with others, carry out responsibilities, or do things for enjoyment.
Among the four transcripts analyzed, there were few instances of text left uncoded by the raters. Raters reported difficulties in coding and/or did not code text that was vaguely stated or inaudible, lacked precise context, expressed uncertainty, told stories about the past or about other people, contained philosophizing/observations about life, or described hypothetical and/or future scenarios that did not reflect reality in the present (see Table 5). It appeared that the uncoded text did not cover social health-related outcomes and therefore carried negligible implications for our assessment of the domain's comprehensiveness and appropriateness.
The concept of volunteerism was the only discernable theme that emerged and recurred in the statements coded as “Other.” Table 6 gives examples of such statements. The ability to do volunteer work was reported as a source of satisfaction and fulfillment; conversely, limitations in ability to engage in volunteerism were reported as a potential source of dissatisfaction. Volunteerism and work undertaken for pay may share certain aspects, such as contractual arrangement, defined tasks or outputs, expectation of a standard of quality, and time constraints. However, while in general the primary goal of work undertaken for pay is to support oneself financially, participants' statements implied that the primary goal of volunteerism was to gain a sense of well-being from the act of helping others. This well-being seemed to derive from the fulfillment of religious, spiritual, or moral beliefs about doing things for others, motivated by altruism.
Three areas of content-related coding difficulties emerged in the coding process: overlapping concepts, hypothetical speech, and references to pets.
As mutual exclusivity among the domain subcategories was not one of our a priori goals, raters were instructed to assign more than one code where appropriate. Our examination of frequently overlapping themes informed our assessment of the domain map's appropriateness. The most common incidents of overlapping concepts occurred when participants spoke of responsibilities toward other family members, especially children, in the same way that they spoke of work responsibilities. Many participants had children, and their social well-being was often related to the degree to which they fulfilled expectations, desires, needs, and demands originating from the simultaneous requirements for both childcare and financial security. Another area of overlap was between the concepts of performance and satisfaction. The raters often simultaneously assigned both of these two codes to the same statements.
Another difficulty that the raters encountered was in coding participants' hypothetical statements or statements about the future. Such statements most likely reflected health and social concerns or worries on the part of participants. One stated: “You start thinking ‘what if I get cancer - and that's painful - how will I know I have cancer because I hurt all the time?’” All three raters labeled this statement as a health-related outcome, but none made a further designation.
There were numerous references to pets as significant members of participants' households. In the attachment literature, pets are considered secondary attachment figures; pet ownership has been found to have positive health correlates . Focus group participants often expressed the sentiment that pets were a vital source of support and companionship. However, because we had chosen not to evaluate social support at this time, and because we chose to define the term “social” as having to do with other humans, our coding efforts focused mainly on those statements about taking care of pets as part of household responsibilities. It is nonetheless noteworthy that pet ownership was mentioned frequently when participants spoke of their satisfaction and well-being.
When asked whether they preferred a 30- versus 7-day timeframe for questions about social well-being, four UNC participants expressed a preference for 30 days. No WPIC participants expressed a preference and no UNC participants said they preferred a 7-day recall period. However, this feedback should be considered in light of the long-term quantitative measurement goal of PROMIS to retain accuracy. Recent FDA guidance on reliability in quantitative PRO measurement states that longer periods of recall may threaten the accuracy of PRO data . Depending on the disease area and the frequency of those symptoms being assessed, shorter recall periods may be preferable to longer ones in quantitative assessment of PROs .
Social health is a complex construct in terms of conceptualization and measurement. Given also the variety of possible subdomains and their interactions with each other in a possible framework for social health, it was important to assess the adequacy of our domain map in defining social concepts of value to real people. Our aims in the present study were to assess whether our domain map and its subcategories were comprehensive and appropriate in describing social health-related outcomes.
Overall, the findings supported comprehensiveness of the domain map. However, as a result of our findings the subcategory of work was expanded to include volunteerism (see Recommendation 1, Table 7). In the future, if social support is eventually included in the domain, the concept of pets should be re-examined (Recommendation 2, Table 7).
In assessing appropriateness, we gained valuable information that will help us to improve the domain map. First, the distinction between activities undertaken out of obligation versus enjoyment was important to participants (Recommendation 3, Table 7). Second, our findings suggest that we should revise how the concept of satisfaction is integrated into the social health domain map (Recommendation 4, Table 7). Statements about satisfaction reflected the degree to which participants' social role expectations, desires, needs, or demands (imposed either internally or externally) were fulfilled. The theme of satisfaction was not distinct from performance, as many times statements coded as satisfaction reflected “satisfaction with [one's] ability to perform” a given activity. Pervasive overlap of these concepts in the content analysis prompted the question of how best to measure “performance” versus “satisfaction” (see Figure 1). Although the concepts were frequently mentioned together when raised in the focus group discussions, this does not necessarily imply that they cannot be distinguished conceptually. Further examination of conceptual distinctions between performance and satisfaction is warranted. In addition to coding difficulty surrounding the concept of satisfaction, strong expressions of emotion were also present in statements about satisfaction (Table 4). Because there is a separate emotional distress domain in PROMIS, further efforts should explore how the concept of satisfaction with social health overlaps with emotional distress.
One limitation was that our findings on responsibility versus enjoyment may have been an artifact of the design of the interview guide (see Appendix), which asked participants to think specifically about things they are expected to do and things that they like to do. This question of potential circularity will be in part addressed in the quantitative analyses of the item banks underway; these analyses are expected to give us greater insight into the underlying latent traits for these measures.
Selection bias may have had an effect on our findings. First, volunteerism may have appeared as a salient theme among these participants because they had self-selected to be in the focus group. Second, the range of limitation represented by our sample excluded patients whose health problems would have prevented them from leaving their house (i.e., persons who were extremely socially disengaged). Since one overall goal of the PROMIS quantitative analysis is to reduce “floor” and “ceiling” effects of functionality and limitation, more diverse sampling procedures will be needed to cover a fuller range.
Lastly, the semi-public nature of any focus group study means that sensitive issues may be discussed less comprehensively than they would in a more private setting. Stronger personalities and “group think” threads of conversation may also have dominated the discussion and precluded some individuals from sharing as much as others.
The main strength of the present study was that the information gained from this qualitative analysis helped us make substantive improvements in refining the domain map, making it more comprehensive and appropriate in covering social health concepts. These qualitative efforts are expected to contribute to an item bank for social health that improves upon past measures of social health: specifically, the PROMIS items are broadly applicable across different illnesses and health states, and in keeping with the definition of PROs , they do not impose external numeric thresholds on respondents' assessment of their own social well-being. For examples, several extant measures ask questions like “how many people do you feel you can count as close to you?”, creating a need for further interpretation of the marginal qualitative value of a response of “5” versus a “2”. In addition, the PROMIS effort has used a combination of methods - panel consensus, qualitative methods, and quantitative methods - to construct the item bank, integrating past literature, extant measures, and most importantly, the patient perspective. Thus these qualitative findings are an important step in a process that provides new and unique contributions to the existing methods and literature for assessing social health.
A wide variety of medical and psychiatric conditions was represented in the sample (e.g., arthritis, heart disease, diabetes, and depression). We were therefore able to capture a spectrum of social health limitations and themes in social health-related outcomes that would be pertinent in a variety of chronic health conditions. This fits with the overall goal of PROMIS to develop measures of HRQoL that can be used in a variety of contexts for medical and psychosocial evaluation.
Another strength of the present study was that data were collected across two different sites – one a predominantly rural area in the Southeast and the other a predominantly urban area in the Northeast – and using different facilitators; this design feature minimizes the potential for effects of investigator bias to bring about systematic bias in the findings.
Quantitative analyses of data collected in the social health domain should examine how the concepts of performance, satisfaction, responsibility, enjoyment, and limitation interact with each other to further inform our domain map. These analyses should also be able to identify areas for further potential refinements of the domain map (e.g., perhaps a distinction between relationships within versus outside the home). The decision to use focus groups in the present study was made on the basis of our need for qualitative information that would confirm, contradict, or add to the domain map. Thus, pre-coded methods such as pile sorting and concept mapping would not have allowed us to identify and assimilate new information. However, such methods may prove useful in further assessments of the revised domain, as well as in future cross-cultural applicability research .
Because our long-term goals are to create item banks that (a) are amenable to item response theory and computer adaptive testing for measuring constructs within social health, and (b) reflect and track with health outcomes and impairments in other areas of HRQoL, including other PROMIS domains (emotional distress, fatigue, pain, and physical functioning), further validation of the social health domain map and item bank through both qualitative and quantitative analysis efforts is necessary.
The Patient-Reported Outcomes Measurement Information System (PROMIS) is a National Institutes of Health (NIH) Roadmap initiative to develop a computerized system measuring patient-reported outcomes in respondents with a wide range of chronic diseases and demographic characteristics. PROMIS was funded by cooperative agreements to a Statistical Coordinating Center (Evanston Northwestern Healthcare, PI: David Cella, PhD, U01AR52177) and six Primary Research Sites (Duke University, PI: Kevin Weinfurt, PhD, U01AR52186; University of North Carolina, PI: Darren DeWalt, MD, MPH, U01AR52181; University of Pittsburgh, PI: Paul A. Pilkonis, PhD, U01AR52155; Stanford University, PI: James Fries, MD, U01AR52158; Stony Brook University, PI: Arthur Stone, PhD, U01AR52170; and University of Washington, PI: Dagmar Amtmann, PhD, U01AR52171). NIH Science Officers on this project are William Riley, Ph.D., Susan Czajkowski, PhD, Lawrence Fine, MD, DrPH, Louis Quatrano, PhD, Bryce Reeve, PhD, James Witter, MD, and Susana Serrate-Sztein, PhD. This manuscript was reviewed by the PROMIS Publications Subcommittee prior to external peer review See the web site at www.nihpromis.org for additional information on the PROMIS cooperative group.
This work was also supported by AHRQ National Research Service Award Research Training Grant T32 HS000032-17.
We gratefully acknowledge the editorial assistance provided by David Cella of Evanston Northwestern Healthcare, as well as the data coding efforts of the three raters at the UNC site: Katherine Buysse, Jessica Dilday, and Ashley Hink. Significant acknowledgements at the Pittsburgh site include Emily Huisman, Catherine Maihoefer, and Nathan Dodds.
Competing interests: The authors declare that they have no competing interests.
Authors' contributions: LC, RD, DD, DI, and KW conceived of and designed the study. LC, DI, AS, and KW conducted the data collection. HB, LC, DD, EH, SE, MK, DI, AS, JM, and KW carried out analysis and interpretation. LC created the initial draft of the article. HB, RD, DD, EH, SE, DI, MK, AS, JM, and KW aided in subsequent drafting and critical revision. DD and RD obtained funding. LC held overall responsibility for this work. All authors read and approved the final manuscript.