|Home | About | Journals | Submit | Contact Us | Français|
Daniel Walker, PhD, MPH, Postdoctoral Researcher, Department of Family Medicine, Ohio State University, 273 Northwood and High Building, 2231 North High Street, Columbus, OH 43201, Phone: 614-293-8007; Fax: 614-293-2715, ude.cmuso@reklaW.leinaD
Alexandra DeNardis Moss, BA, Department of Family Medicine, Ohio State University, 273 Northwood and High Building, 2231 North High Street, Columbus, OH 43201, Phone: 614-293-8007; Fax: 614-293-2715, ude.rehcuog.liam@100somla
Nina A. Bickell, MD, MPH, Professor, Department of Population Health Sciences & Policy, Department of Medicine, Co-Director, Center for Health Equity & Community Engaged Research, Mount Sinai School of Medicine, 1 Gustave L. Levy Place, Box 1077, New York, NY 10029, tel (212) 659-9567, fax (212) 423-2998, email@example.com
Qualitative Comparative Analysis (QCA) is a methodology created to address causal complexity in social sciences research by preserving the objectivity of quantitative data analysis without losing detail inherent in qualitative research. However, its use in health services research (HSR) is limited, and questions remain about its application in this context.
To explore the strengths and weaknesses of using QCA for HSR.
Using data from semi-structured interviews conducted as part of a multiple case study about adjuvant treatment underuse among underserved breast cancer patients, findings were compared using qualitative approaches with and without QCA to identify strengths, challenges, and opportunities presented by QCA.
Ninety administrative and clinical key informants interviewed across ten NYC area safety net hospitals.
Transcribed interviews were coded by three investigators using an iterative and interactive approach. Codes were calibrated for QCA, as well as examined using qualitative analysis without QCA.
Relative to traditional qualitative analysis, QCA strengths include: (1) addressing causal complexity, (2) results presentation as pathways as opposed to a list, (3) identification of necessary conditions, (4) the option of fuzzy-set calibrations, and (5) QCA-specific parameters of fit that allow researchers to compare outcome pathways. Weaknesses include: (1) few guidelines and examples exist for calibrating interview data, (2) not designed to create predictive models, and (3) unidirectionality.
Through its presentation of results as pathways, QCA can highlight factors most important for production of an outcome. This strength can yield unique benefits for HSR not available through other methods.
Innovative research designs, often borrowed from other disciplines, can help surmount challenges to conducting health services research (HSR) studies (1). Healthcare providers exhibit great variation in organizational structures, care processes, and outcomes (2), necessitating a methodological approach flexible enough to capture high levels of detail, yet rigid enough for validity, reliability, and generalizability. Furthermore, HSR approaches may need to be adapted to accommodate smaller sample sizes, as gathering site-specific data can require significant time and limit the number of sites that can feasibly be studied. While quantitative approaches offer many benefits, they may be inappropriate to deal with small sample sizes, and may not take into account the different causal pathways that can produce an outcome. Alternatively, traditional qualitative approaches may not provide the level of objectivity and reproducibility desired to understand the relationship between site characteristics and low-quality care in complex delivery systems.
Qualitative Comparative Analysis (QCA) provides an approach that captures the objectivity of quantitative designs, yet can take advantage of the detail from qualitative interviews (3-5). QCA is considered ideal for small- to medium-N case-oriented research studies, thus making it a good fit for many investigations. Originally developed as a research tool for social science, QCA has been used sparingly in HSR (6,7), typically to uncover patterns to achieve a given outcome in large data sets (8). Examples of QCA based on key informant interviews, particularly in HSR, are even more rare.
This paper evaluates the use of QCA in HSR using key informant interview data, and compares this approach to more traditional qualitative methods. This analysis focuses on the strengths and weaknesses of QCA methodology, using a case to help investigators evaluate the applicability of QCA within the HSR context.
Interview data were collected as part of a larger intervention study designed to reduce underuse of adjuvant therapies for breast cancer patients by closing referral loops between surgeons and oncologists in safety net hospitals (9). The aim of the ongoing study is to uncover factors associated with high-quality (i.e., low underuse) cancer care in resource-limited safety net hospitals. Ten hospitals serving high proportions of minority breast cancer patients in the New York metropolitan area were recruited to participate in the intervention project.
Prior to intervention implementation, principal investigators at each hospital site identified key informants who could discuss how cancer care and quality improvement occur at their institutions. Key informants included hospital leaders in both cancer care and quality, oncologists, nurses and nurse managers, clerks, and patient navigators. In all, 90 in-depth, semi-structured interviews were conducted with: 59 clinicians; 16 administrators; 12 clerical staff; and 3 others.
Interviewees were asked about the structure and processes of breast cancer care delivery and perceived facilitators of and barriers to patients obtaining their prescribed adjuvant treatments. Interviews, lasting 30-60 minutes, were recorded and transcribed verbatim.
An integrated approach was used to code interview data that included both deductive and inductive methods (10). The three-member coding team, consisting of senior investigators (authors) and an experienced research assistant (author), held regular meetings to discuss decisions about codes and themes as they emerged.
A preliminary coding dictionary was developed based on broad topics from the key informant interview guide. This coding dictionary defined categories a priori based on the question domains, such as “Current Process for Coordinating Breast Cancer Care,” or “Barriers to Handoffs and Tracking,” or “Readiness for Innovation Implementation.” Each coder applied this preliminary structure to three common interview transcripts, then met to reach consensus about codes and definitions.
Each coder subsequently coded additional interviews independently, using the constant comparative method of qualitative data analysis to develop and refine codes in vivo (11,12). Frequent meetings helped to ensure consistency and accuracy of coding, as well as clarification about emergent codes. This approach followed the standards of code development for rigorous qualitative analysis (10, 13). ATLAS.ti 7.0 software was used to facilitate the coding process (14).
QCA involves listing and counting different combinations of variables in a given data set, and then applies logical inference rules (i.e., Boolean algebra) to evaluate whether alternative inferences are supported by the data (15). This approach identifies one or more solution pathways that produce an outcome of interest (15), thereby providing information about alternative approaches to achieve a given outcome. The QCA methodology contains five steps: 1) identify an outcome of interest and a list of conditions that may be associated with that outcome; 2) develop calibration metrics using a crisp, dichotomous or a fuzzy, ordinal set; 3) calibrate the data; 4) develop a truth table showing all possible combinations of conditions, identify necessary conditions, and use logic to minimize the truth table and arrive at pathways to the outcome; and, 5) assess these pathways using two parameters of fit called consistency and coverage (16). A summary of these QCA-specific terms and accompanying definitions are provided in Table 1.
In this case example, underuse of adjuvant treatments is the outcome of interest. Similar codes and themes were grouped into conditions that could be associated with the outcome. For example, the theme of “no shows” was combined with the emergent theme of “tracking across specialties” to specify “follow-up” as a condition.
Fuzzy set calibration structure was then created by assigning a value between 0 and 1, based on degree of membership in the condition. To continue the example above, the “follow-up” condition was calibrated through a multi-step process that involved defining measures for the condition, creating anchor points (1, .5, 0) for each measure, and then calibrating the fuzzy set (1, .8, .6, .4, .2, 0). The first measure defined for this condition focused on whether “no shows” received phone calls and letters. An anchor point value of 1 was assigned to those sites that always made phone calls and/or sent letters; .5 was assigned if the site “sometimes” did this; and 0 was assigned to sites where this was never done. Once these anchors were established, more specific scores were assigned.
Finally, relative weights were assigned to each measure within a condition according to their importance to the outcome. For example, because follow-up is important for adjuvant treatment rates, the basic approach of phone calls and letters was not as important as additional steps some hospitals took to reaching patients (e.g., telegrams, urgent patient home visits, missed appointment notification). As a result, the “basic” no show follow-up was weighted lower (e.g., .3) than the “extra steps” measure (e.g., .7). The output from this calibration process was represented as the weighted levels shown in a data matrix, as described above.
Next, we developed a truth table showing all possible combinations of conditions, and assigned cases (hospitals) to truth table rows based on the presence/absence of identified conditions (e.g., follow-up). Using Boolean logic, the truth table was minimized to arrive at pathways to the outcome (e.g., (8)). Three solution pathways were found to achieve the outcome of low underuse of adjuvant therapies. Finally, the fifth step required assessing these pathways using the consistency and coverage parameters of fit (16, 19).
Five strengths of QCA, relative to a traditional qualitative approach, were identified: 1) the methodology addresses equifinality and causal complexity, 2) findings can be presented as a “pathway” as opposed to a list, 3) necessary conditions are helpful to characterize actionable results, 4) QCA offers the option of fuzzy-set calibrations, and 5) QCA-specific parameters of fit allow researchers to compare pathways to an outcome. These strengths are each discussed in more detail next.
Equifinality is defined as two or more pathways that lead to the same outcome. QCA allows for multiple pathways to be identified as part of a solution, which is especially useful when the cases under study have different strategies to achieve a desired outcome. As an example, the three solution pathways showed alternative approaches to achieve low rates of underuse for breast cancer adjuvant therapies, rather than only one best way. Additionally, QCA hinges on the principle of causal complexity--the idea that more than one condition can be combined to produce an outcome. This case study included as many as six different factors, reflecting the complexity inherent in the care coordination process. The value of a methodology that accommodates causal complexity is particularly clear in healthcare organizations where outcomes are often the result of specific combinations of environmental, procedural, and social conditions.
In a traditional thematic analysis of qualitative interview data, such as grounded theory, multiple factors that are associated with an outcome are typically identified. QCA allows researchers to take this analysis a step further because it enables them to a) identify relevant factors as well as b) determine the ways in which those factors combine to produce the outcome. The ability to identify pathways to an outcome as opposed to a list of factors that might impact an outcome helps to present more detailed recommendations for practice.
A necessary condition is a condition that must be present for the outcome to occur. For a favorable outcome, making sure all necessary conditions are in place is imperative to achieve that outcome, yet often necessity varies. For example, while follow-up was consistently necessary, flexibility was necessary in the absence of system support, but unnecessary when system support was present. When necessary conditions are identified, implementing or removing these conditions is an important first practical step towards achieving or preventing an outcome.
A fuzzy set analytic approach enables researchers to classify degrees of a condition's presence for a given case and allows them to retain much of the detail present in in-depth qualitative interviews. In this study, degrees of follow-up were measured as “always,” “usually,” “occasionally,” “rarely,” and “never.” Fuzzy set calibrations thus enable researchers to assess the degree to which a condition is present at each site, and allow them to see how closely each case follows a given pathway to the outcome.
The two QCA-specific parameters of fit are consistency and coverage. Consistency is a measure of the degree to which cases that share a combination of conditions (pathway) also share the same outcome, and coverage measures the prevalence of a given pathway. For this study, the level of consistency for the three solution pathways ranged from .82 to 1.0, consistent with acceptable levels in social science research, and providing confidence that the presence of these conditions in these hospital sites was consistent with low underuse of adjuvant therapies. Coverage, on the other hand, ranged from .19 to .49, reflecting variability in these different pathways, as well as the complexity of impacting the outcome of low underuse. When all identified pathways are equally grounded in theory, consistency and coverage can help researchers assess the accuracy and empirical relevance of each individual pathway. These parameters of fit can help researchers decide whether to recommend one pathway over another.
Three challenges of using QCA in a HSR study with interviews as the primary data source were identified: 1) very few guidelines and examples exist for calibrating interview data, and calibration content is likely to vary based on the outcome of interest, 2) QCA was not designed to create predictive models, and 3) QCA is unidirectional. These drawbacks are each discussed further below.
Ragin and other QCA scholars have developed multiple methods for converting numerical data into crisp and fuzzy sets (16, 20), but they have not proposed any guidelines for calibrating interview data. Basurto and Speer (21) have published suggested guidelines, but they note that most QCA studies which might otherwise be used as examples do not disclose their calibration methods, and also that their guidelines are just one of many possible ways to calibrate interview data. Furthermore, these guidelines center around more general social science research, and are not specific to HSR projects. Thus, in keeping with the general principles of qualitative analytic rigor, designing a project-specific calibration structure takes careful attention and several revisions, and often requires an iterative process to ensure that final calibrations are justified. In this case example, the calibration of the “follow-up” condition involved multiple iterations, with members of the research team holding extensive discussions about how to characterize conditions (e.g., what themes were appropriate to include), how to define measures (e.g., “basic” follow-up vs. “extra”), and how to weight measures (e.g., what measures would have the most impact on the outcome). In the absence of information about the different HSR-related conditions in the literature, an iterative process is necessary to ensure that the final calibrations are not arbitrary.
QCA is very useful in identifying pathways to an outcome. However, the pathways developed by a qualitative comparative analysis are not designed to determine how outcomes would change if specific sites moved to a different calibration bracket. Although this is possible to some extent on a case-by-case basis, it would be very time consuming to make predictions on a large scale. This challenge is only a limitation in the methodology if predictions are necessary to develop convincing results.
Finally, the pathways produced by QCA are only associated with the outcome in question; the opposite of a pathway is not necessarily associated with the opposite of the outcome. If researchers are interested in pathways both to an outcome and to its opposite, two separate analyses must be conducted, and different conditions may need to be used for each analysis (5). This second analysis follows the same procedure as the original analysis (identifying conditions, summarizing the data, creating calibration structures, reducing the truth table, etc.), and takes a comparable amount of time to conduct as the first analysis. As a result, investigating an outcome as well as its opposite is a very time-consuming analytical procedure.
Using QCA to further analyze interview data that had been coded thematically provided new insights that were not achievable with traditional qualitative methods. Some of the opportunities provided by QCA are highlighted in Table 2, and described in greater detail below.
First, with respect to the level of detail that could be presented in the results, QCA permitted comparison across sites by focusing on the level that a condition was present at each site, represented numerically; the corresponding level of detail might have been cumbersome to express in prose comparisons of multiple sites. For instance, the theme of “no shows” involved how the hospitals tracked patients who did not show up for their appointments. The sites varied in their ability to address “no shows,” and describing this variability entails different examples from different sites. Using traditional qualitative approaches, this description includes addressing both similarities and differences across sites, and involves a considerable amount of textual description. In contrast, QCA enables the assignment of a level for the condition at each site, providing the ability to parsimoniously compare each hospital's level in a data matrix. For the condition that included “no shows,” levels ranged from .10 to .96, clearly reflecting considerable variability across sites. QCA also requires calibration, and therefore specifications of the key elements that contributed to a theme, including forced ranking of the included elements. This enabled the collapse of themes into overarching domains, allowing the emergence of different patterns of associations that had remained more diffuse in the absence of QCA.
Another important opportunity presented by QCA is the type of results that can be produced. QCA permitted the identification of three distinct solution pathways associated with low underuse and forced classification of necessary conditions for these solution pathways. The different solution pathways included different conditions, suggesting that there may be more than one way to reduce underuse of adjuvant therapies. For example, two hospitals had good follow-up, a patient-centered culture, shared information, and good system support. However, another solution pathway was shared by two different hospitals and involved good follow-up, a patient-centered culture, and shared information, but did not require good system support. This level of specificity enabled us to move beyond merely developing a list of factors associated with low underuse (e.g., follow-up, culture) but with no sense of strength of association among the proposed causal factors. Further, QCA enabled us to clarify how trade-offs might exist among the different factors (e.g., flexibility vs. system support).
A third opportunity presented by QCA is the generation of testable hypotheses. The development of the solution pathways provides an opportunity to test the veracity of these assumptions in different settings. In the context of quality improvement, determining conditions that can lead to high quality provides managers with a testable basis for practice change. For research, testable hypotheses generated from QCA provide the basis for larger scale and more generalizable studies. For example, shared information was a necessary part of both solution pathways described above. This finding could be used to support a hypothesis about the role of information systems in providing high-quality care. Finally, while bias is inherent in any coding process, the process of creating specific and objective measures and conditions in QCA forces a greater level of proof and objectivity of findings in the analysis process.
QCA has both value and limitations as a methodology for HSR that uses interviews as the primary data source. Its potential to present pathways to outcomes of interest in a causally complex, equifinal manner and its ability to identify actionable “necessary conditions” enable researchers to synthesize their results in a more detailed and pragmatic manner than is typically possible. However, limitations such as its inability to evaluate multiple outcomes or to create predictive models are clearly considerations.
An important limitation of QCA highlighted in this study is the paucity of examples available to help advise the process of calibration of qualitative data in HSR. Most recent studies in HSR that have used QCA (6-8) have used surveys (occasionally combined with interviews) to collect their data. When surveys are used as the QCA data source, it is much easier to match respondents’ answers with conditions that can be both identified and calibrated a priori. However, for studies involving interviews, the process typically allows for reflection on the part of the interviewee, and provides the interviewer with an opportunity to adjust or amend the interview questions. The tradeoff between structure and flexibility is especially evident when considering the amount of time required to use QCA in a study where interviews are the primary data source. Nonetheless, using surveys in a study such as this one would have made it extremely difficult to uncover emergent issues (22), and would have limited the ability to characterize some conditions in the solution pathways. Researchers therefore must be prepared to create calibration structures suited specifically to their studies, including spending sufficient time calibrating interview data.
While prior studies have compared QCA to conventional statistical methods in the context of quantitative data analysis (e.g., (18)), the use of QCA reported here suggests that there is an important opportunity to use QCA in HSR when interviews are the primary data source. While decision modeling can yield similarly actionable results (23), QCA is especially relevant to HSR because its presentation of results as a pathway to an outcome can allow healthcare organizations to make informed decisions about how to allocate their resources. The benefits that accompany presenting results in this manner are unique to QCA and should be considered by researchers as they select among methodological approaches.
Similar to other qualitative methods, the time and energy required for in-depth analyses of key informant interview data in QCA is considerable and can create challenges for large studies. Although QCA was designed for studies with as few as 10 cases, studies with more cases typically make it easier to create and analyze the truth table – particularly in highly complex contexts, such as HSR. This study of 10 hospitals including 90 key informant interviews would be considered quite large; thus a lower limit of 10 cases implies a considerable amount of work. Further, as described above, QCA-specific limitations include the inability to be used for predictive modeling, as well as the potential problem of bias inherent in any coding process. Finally, while the specificity of the conditions enables greater objectivity, it is also possible that this specificity threatens generalizability to other conditions or scenarios.
QCA can afford researchers unique insights and actionable results, making QCA an appropriate and valuable methodology for certain HSR endeavors. It is particularly applicable to studies with clear, specific outcomes in question, and especially for those studies in which there is likely more than one way to reach the outcome and each way of reaching that outcome is likely to depend on multiple factors. In these instances, QCA can bring increased clarity and detail to the analysis of interview data, and provide specific guidance given evidence about how different factors impact an outcome.
The authors are extremely grateful to the organizations and informants who participated in this study, and especially to the site principal investigators who facilitated this research. We also thank our research team member, Rebeca Franco, affiliated with the Mount Sinai School of Medicine during the study. This research was funded by a grant from the National Cancer Institute, NCI R01-CA 149025, but the study sponsors had no involvement in the study design, in the collection, analysis and interpretation of data; in the writing of the manuscript; or in the decision to submit the manuscript for publication. There are no conflicts of interest associated with this manuscript.
STATEMENT ON CONFLICTS OF INTEREST: The authors have no conflicts of interest with this manuscript.