No less an object and instance of comparability work in research synthesis studies than methodological diversity is topical diversity. Systematic reviews typically begin with efforts to define the topic of review with a view toward achieving a topically identical set of studies. Yet, a continuing concern in the systematic review literature is the often-cited ‘‘apples and oranges’’ problem (Deeks et al., 2005
, Section 8.1.2; Sharpe, 1997
) requiring researchers to decide whether they will treat two entities (e.g., apples and oranges) as one entity (fruit) or preserve the distinctions between them.
A case in point from our antiretroviral adherence studies is the diversity in the entities studied as aspects of ‘‘antiretroviral regimen’’ (e.g., number of pills per dose, number of doses per day, side effects, difficulty swallowing). Comparability work entails deciding whether to treat them as one variable (i.e., antiretroviral regimen) influencing adherence, or to preserve the individuality of each regimen aspect. The first option would efface what some will regard as regimen features too different to be meaningfully combined, while the second option would leave few findings available for synthesis as relatively few of the adherence studies reviewed (and often only one study) addressed the same regimen aspects. In either option, difference is managed; in the first option, by erasing it as the act of comparing is moved up the continuum of abstraction and in the second, by preserving it as the act of comparing remains at a more empirical level.
According to Glass (2000
, p. 6), whose name is virtually synonymous with meta-analysis, to compare apples to apples would not only be ‘‘trivial’’, but also redundant as nothing but the study of fruit makes sense or is ‘‘worthy of true scientists’’. Drawing from Nozick’s (1981
, p. 29) ‘‘closest continuer theory’’, Glass concluded that the question ‘‘of how two things could be the same ultimately resolves itself into an empirical question’’ (p. 8) of what researchers conceive of as the important differences. In other words, comparability work is here directed toward deciding what can be seen as similar enough or too different to be combined. In instructional texts on the systematic review of quantitative research, researchers are advised to determine what comparisons should be made and which findings should be used in each comparison. They are also cautioned that such decisions are ‘‘inevitably subjective and not amenable to statistical solutions’’ (Deeks et al., 2005
, Section 8.1.2). In his widely used guide to synthesizing quantitative research, Cooper (1998
, p. 116) observed that any one ‘‘cumulative analysis’’ should ‘‘test the same comparison or estimate the same relationship’’. Yet, he also noted that researchers should not combine findings at a level that would elide ‘‘distinctions meaningful to the users of the synthesis’’ (p. 109). As these texts indicate, achieving comparability becomes a matter of technique (e.g., of converting different statistical expressions of data into effect size indexes) only after judgments are made about what comparisons are useful and will appeal to the audiences to which they are directed.
Topical diversity in qualitative versus quantitative studies
In mixed research synthesis studies, comparability work around topical diversity is influenced by divergent views concerning whether qualitative and quantitative studies can address the same topics. One view is that qualitative and quantitative studies can address the same topic (e.g., participants’ ‘‘views’’ of a target event; Harden et al., 2004
), while a contrasting view is that qualitative and quantitative studies are defined, in part, by their addressing different topics: as Barbour and Barbour (2003
, p. 180) described it, in comparison to quantitative research, qualitative research ‘‘taps into … a different sort of curiosity’’.
In the antiretroviral adherence studies in our project, the quantitative findings emphasize numerically measured variables (e.g., CD4 count, viral load, number of pills in the drug regimen) and demographic characteristics (e.g., age, education, injection drug use) as dichotomous or continuous correlates of adherence. In contrast, the qualitative findings emphasize people’s experiences with, attitudes toward, and beliefs about antiretroviral therapy. While the quantitative findings in this body of research focus on predictors of extent of adherence or non-adherence, the qualitative findings focus on the reasons for adherence and non-adherence.
Here difference can be managed by treating ‘‘predictors’’ as topically different from ‘‘reasons’’, or conceiving reasons as explanations for predictors. Alternatively, difference can be managed by treating predictors and reasons as equivalent, or conceiving the qualitative findings as more thematically precise versions of the quantitative findings and the quantitative findings as more numerically precise versions of the qualitative findings. In the first instance, difference is imposed whereas in the second, similarity is imposed.
Achieving topical similarity by creating the body of research
Whether the topical differences between qualitative and quantitative studies in a domain of research are preserved or effaced, the management of topical diversity is always complicated by the fact that no two studies of any kind in any body of research deemed to address the same topic ever actually address the same topic, let alone address ‘‘it’’ in the same way. For example, in the body of research addressing HIV-positive women’s adherence to antiretroviral therapy are studies directed toward ascertaining what predicts adherence. Yet, these apparently topically similar studies vary widely in the attributes, conditions, events, and other ‘‘factors’’ studied and the ways in which these factors are conceived, measured, and linked to each other, the way antiretroviral therapy is conceived and measured, the way adherence is conceived and measured, and the persons and sites chosen to study the operation of these factors. Adherence in these studies varies widely even in the aspects of number examined, such as percent of time prescription orders are followed, percent of pills taken, and percent of doses consumed over the last day, 2 days, week, or month. In the end, only a few of dozens of studies that researchers will have reviewed in a target domain will have actually addressed the influence of the same set of factors on the same set of other factors in the same way. For example, of the 199 bivariate relationships featured in the quantitative studies we reviewed, 57 were assessed in only 1 study. Moreover, 40 of ostensibly the same relationships (e.g., between education and adherence) were operationalized or analyzed so differently as to resist comparison (e.g., use of different assessment instruments, mean versus dichotomous scoring).
Despite the absolute lack of topical identity between studies in ostensibly the same topical area, the research synthesis enterprise requires reviewers to act as if a designated set of studies is similar enough to treat it as one body of research. Even to talk about a body of research is to create an identity between disparate entities. Indeed, reviewers create a body of research for each synthesis project. The very act of deciding what studies to include and exclude is not simply a sampling issue but a form of comparability work whereby reviewers engineer a certain kind of sample. Such engineering efforts are typically directed toward reducing topical diversity in order to have a topically comparable data set for analysis.
Many of the studies we considered for inclusion never actually contained the words adherence
anywhere in the reports of them and addressed such topics as patterns of access, use, and prescription, and how these patterns correlated with such factors as women’s race, class, drug use, psychiatric condition, or CD4 count and viral load. We began our project with the prevailing definition of adherence as something patients do or do not do, namely, follow providers’ prescriptions. Yet adherence (i.e., taking medicines as prescribed by providers) does not come into play until a provider actually prescribes medicine for someone, which is, in turn, dependent on the provider seeing that person as suitable to receive that drug. In addition, adherence does not come into play until individuals are able to fill prescriptions, which is, in turn, dependent on whether they have access to a pharmacy to drop off and pick up the drugs and the means to pay for them. The ‘‘arena’’ (Clarke, 2005
) of adherence includes so many more topics than are generally conceived of as constituting the body of adherence research, yet this arena is too topically diverse for any one research synthesis study.
Accordingly, the body of research constituting studies of antiretroviral adherence in HIV-positive women—at any one time for any one research purpose—might include or exclude studies of such topics as: (a) provider practices related to prescribing antiretroviral drugs for HIV-positive women (because these practices determine whether and which women are in a position to adhere, even if these practices were not linked to adherence in the study); (b) factors that facilitate or impede getting prescriptions filled, such as transportation to pharmacies, means to pay (because these factors also determine whether and which women are in a position to adhere, even if these factors were not linked to adherence in the study); (c) side effects of antiretroviral medications (because they may preclude adherence, even if these effects were not linked to adherence in the study); (d) progression of HIV disease (because such disease indexes as CD4 count & viral load also indicate adherence to antiretroviral therapy, even if these indexes were not linked to adherence in the study); (e) provider practices related to selecting and altering specific antiretroviral drugs and drug combinations prescribed (because different drugs and drug regimens will have different effects and, through these effects, influence adherence, even if specific drugs were not linked to adherence in the study); or (f) attitudes toward, beliefs about, or intentions to use antiretroviral drugs (because they can influence adherence practice, even if these factors were not linked to adherence in the study).
As this by no means comprehensive list of contenders for inclusion into the body of antire-troviral-adherence-in-HIV-positive-women research shows, any number or configuration of reports of studies may constitute this body. Indeed, adherence
research can include studies that never addressed adherence per se, but rather entered the systematic review process because these studies suggested a link between what they did cover and adherence. This variability in the work object (Law & Singleton, 2005
) of systematic review explains why the results of different reviews of ostensibly the same body of research will yield different and even conflicting results (Linde & Willich, 2003
) and why it is difficult even to talk about a single body of research or of two systematic reviews of the same body of research.
No one project is likely to contain all of the studies that address the broad terrain of ‘‘health work’’ (Mykhalovskiy et al., 2004
, p. 323) in which HIV-positive persons’ medicine-getting and medicine-taking practices are located, that is, the larger ‘‘social, discursive, and institutional context(s)’’ (p. 317) in which HIV-positive persons ‘‘do medications’’ (p. 324). This health work is, in turn, situated in the larger arena of the health work required in other chronic and stigmatizing diseases, which is, in turn, situated in the larger arena of other work that competes with health work, which, is, in turn, situated within still other arenas.
The feasibility of the systematic review enterprise requires cutting a review down to size because, without boundaries, no systematic review is possible. The inclination in systematic review is, therefore, toward exclusion of studies (or findings in studies) to achieve a comparable data set. The systematic review enterprise is less about inclusion, or taking stock (Hunt 1997
) of a field of research, and more about exclusion, or finding defensible ways not to take stock of all of it (MacLure, 2005
; Torrance, 2004
). A systematic review is judged to be credible, in part, to the extent that it makes this ‘‘boundary work’’ (Gieryn, 1983
; Lamont & Molnár, 2002
) transparent. Reviewers are obliged numerically and narratively to account for every exclusion occurring at each successive stage of the review process, namely, at the search, retrieval, and initial and subsequent data extraction, analysis, and evaluation stages. A report of a study in
at the search and retrieval phase may be out
at the data extraction phase as its findings are determined to resist comparison with other findings.
Indeed, the boundary work that defines systematic review is often so exclusionary as to eliminate most of what constitutes the larger arena in which that phenomenon is situated. Reviewers are typically in the strange position of actually synthesizing the findings of only a handful of the reports meeting their initial search criteria. The bias toward exclusion has generated criticisms of the systematic review enterprise as a disciplinary technology aimed at amassing reasons not to include studies (MacLure, 2005
). Yet boundary work is necessary to achieve a manageable (i.e., comparable) data set. In the end, what constitutes a body of research for review is the result of topical differences that reviewers have found ways to ‘‘bridge’’ or ‘‘pacify’’ (Harbers, 2005
, p. 578).