PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of intjepidLink to Publisher's site
 
Int J Epidemiol. 2009 April; 38(2): 370–373.
Published online 2009 January 28. doi:  10.1093/ije/dyn354
PMCID: PMC2663720

Response: The value of a historically informed multilevel analysis of Robinson's data

Our story begins where Robinson's classic study ended.1 Could a study of relationship between two variables measured only at the individual-level—emblematic of most epidemiologic and social science research when individual data are available—lead to an impoverished description of the relationship? Using the same data on illiteracy and race that Robinson employed and supplementing it with relevant ecologic data that would have been available at that time, we showed that, in this particular case, studying solely the ‘behavior of individuals’, while ignoring their historical and ecological state context, was both limiting and misleading.2 The comments of the two discussants,3,4 whom we thank for their efforts, underscore the importance of a multilevel approach to scientific research. Neither of their commentaries, importantly, alters the central tenets and conclusions of our study and instead serves only to bolster them.

Firebaugh reiterates the importance of a multilevel conceptual and analytical approach and concurs with our empirical observations and interpretations. His insightful and illustrative example of race and voting behaviour makes the crucial distinction between, and shows the simultaneous importance of, the influence of individual race and racial context on individual voting behaviour.

Oakes too seems to broadly agree with our overall approach and conclusions. Notably, his reanalysis of our research questions with additional individual-level socio-economic variables led to the same finding demonstrating the robust nature of the empirical association we report. However, Oakes’ lengthy, and occasionally tangential, comment includes problematic statements, some of which misrepresent our study, while others are factually inaccurate. Statements necessitating correction include (i) attribution of motive; (ii) bibliometric searches; (iii) methodological individualism as the basis of multilevel analysis; (iv) the historical realities and impact of Jim Crow; (v) technical and conceptual aspects of multilevel modelling; and (vi) relationships between theory, study design, data analysis and causal inference.

On attributing personal motive to Robinson's study

Contrary to what Oakes asserts, it is extremely important to note that our study did not seek to discern Robinson's motivation and personal beliefs. We categorically stated that, ‘we do not wish to imply that Robinson was deliberate in ignoring the strong synergies between race, illiteracy and states that were present at that time’.2 (p. xx) Instead, our aim was to situate Robinson's conclusion, urging an exclusive focus on individual-level relationship, in its historical and political context. Specifically, we argued that in the context of the example that Robinson used on illiteracy and race, to not consider the salience of the states (even while examining an individual relationship) was a conceptual shortcoming. Our results demonstrated that state-context, as characterized by the presence or absence of Jim Crow laws, mattered for the ‘behavior of individuals’ vis-à-vis race and illiteracy. We argued, using historical evidence, that a focus on ‘behavior of individuals’ was perhaps reflective of the socio-political milieu of the time. Oakes does not offer any evidence to alter this claim. Although it certainly is Oakes’ prerogative to speculate about Robinson's personal motivations and beliefs, we reiterate that we did not.

On the reach and influence of Robinson's study

Oakes claims that we exaggerate the reach of Robinson's work in epidemiology. He ignores our documentation that shows virtually every major epidemiologic textbook published since the 1950s has warned against ecologic fallacy and, more importantly, only very recently have a few begun to note the possibility of individualistic fallacy. Instead, Oakes presents a citation analysis of Robinson's work in a remarkably limited set of two public health/epidemiology journals for solely the years from 1981 to 2002, the reasoning behind which remains unstated. Arguably, a search in peer-reviewed journals is not an appropriate way of assessing a particular scientific contribution that occurred in one field (i.e. sociology, and appearing in a premier sociological journal American Sociological Review) on another (i.e. epidemiology/public health). For this reason, we believe, our strategy of reviewing major epidemiologic text books is perhaps a more appropriate one with regard to assessing the reach of Robinson's work. It is highly likely that most epidemiologists are unaware of Robinson and his classic study, but are aware of the concept of ‘ecological fallacy’; in lectures on advanced quantitative methods that one of us SV Subramanian often gives to graduate and post-graduate scholars in epidemiology/public health, when participants have been asked ‘who has heard of Robinson’, a rare few raise their hand; however, when asked ‘who has heard of ecological fallacy’, almost everyone raise their hand. As Neil Pearce recently observed, the ‘modern’ epidemiologic methodological paradigm is fundamentally based on the ‘theory of randomised controlled trials of individuals’ with ecologic studies being consigned to a ‘relic of the “pre-modern” phase of epidemiology’.5 (p. 326) In short, although we may wish Oakes’ belief that Robinson's work had only a limited reach in epidemiology was true, the evidence, we are afraid, suggests the contrary.

On methodological individualism and multilevel thinking

Oakes states that ‘abundant data’—which are neither provided nor cited—show that our interpretation of multilevel thinking as overcoming methodological individualism is incorrect. In fact, Oakes goes so far as to make a surprisingly incoherent claim that ‘methodological individualism is the foundation of multilevel thinking’.4 (p. xx) Individuals, not individualism, ‘in conjunction with’ groups are indeed one obvious foundation for a multilevel thinking; to reiterate a point we make in our article, ‘recognition of “individuality” does not require embracing the philosophical stance of “individualism”‘. 2 (p. xx) Besides, multilevel thinking also requires understanding how societal features not reducible to individuals, anathema to methodological individualism, can influence individual-level phenomena. Neither in our article nor in this rejoinder do we expect to resolve centuries of debate over the premises and interpretations of methodological individualism. We did, however, based on historical evidence, develop a case that Robinson's conclusion was reflective of the rise of post World War II quantitative approaches premised on methodological individualism.

Oakes also misrepresents the ‘ecosocial’ framework, which motivated the multilevel thinking reflected in our study, by incorrectly stating that it lacks principles of agency necessary for a dynamic relationship between individuals and their context. An examination of any article discussing this theory would readily find that it explicitly emphasizes agency and accountability at multiple levels, from individual to societal, and does not consider society, or for that matter individuals, as a given.6–8

On framing and estimating the impact of Jim Crow laws and historical realities

Oakes’ discussion of Jim Crow is historically inaccurate. Oakes asks ‘Why would Jim Crow inhibit the literacy of whites?’,4 (p. xx) and answers, ‘Simply put, it did not’.4 (p. xx) We are unaware of the historical analyses that Oakes read to support this claim, since he cites none. In fact, an extensive scholarship regarding Jim Crow, a small fraction of which we reviewed to inform our article (see references 93–102), suggests that Jim Crow was part of a deliberate strategy by the white political and economic elite in the US south to bolster their power by pitting poor Whites against poor Blacks, thereby ensuring differentially reduced resources for both groups, including for public education. Thus, Oakes’ carving of the data to identify a ‘causal’ effect of Jim Crow on illiteracy, presuming that it should have no effect on Whites, disregards historical evidence. Nonetheless, it is interesting that in two of three ways in which Oakes slices the data, he still shows that Jim Crow was associated with ‘both’ Black and White illiteracy, with the association being stronger for Blacks; a finding that is in agreement with what we report.

Oakes also asserts that ‘the enactment of Jim Crow laws emerged in the first place through some function of the illiterate and otherwise intellectually retarded views of whites in the US south’;4 (p. xx) a statement for which Oakes, yet again, provides no support. Notwithstanding Oakes’ speculative and historically inaccurate (as documented in the evidence we cite above) assertion that illiteracy among Whites caused the emergence of Jim Crow laws, and not the other way around, we conducted a sensitivity analysis by conditioning the effect of Jim Crow on an effect associated with state-level illiteracy rates for White males over 21 years of age in 1870. The 1870 census data were chosen because it was the earliest post-Civil War census year preceding the period when de jure segregation in the form of Jim Crow laws began to emerge. Specifically, the 1870 census was the first census conducted: (i) after the end of the US Civil War in 1865 and during the period of Reconstruction (1866–73), which saw passage of the 13th, 14th and 15th Amendments to the US Constitutions (which, respectively, outlawed slavery, made prior slaves citizens, and gave Black men the right to vote) as well as the 1875 federal Civil Rights Act9 and (ii) before the rising political backlash, during the period known as the ‘Redemption’, which gained strength starting in the mid-1870s and during which the dominant White political elite secured passage of legislation at the federal and state level which overturned the advances made during the Reconstruction, culminating in President Rutherford Haye's 1877 withdrawal of federal troops from the US South and the collapse of the remaining Reconstructionist southern state governments.9

As shown in Table 1, specifying illiteracy rates among White adult males in 1870 as a prior common cause of both Jim Crow laws and subsequent illiteracy levels, per Oakes’ speculative hypothesis, did not alter the associations we reported in our study in any significant manner. Notably, Native Whites living in Jim Crow states continued to have a substantially higher risk of being illiterate compared with Native Whites living in non-Jim Crow states.

Table 1
Odds ratios and 95% credible intervals of being illiterate for the different combinations of race/nativity and presence or absence of Jim Crow laws, before (Model A) and after (Model B) adjusting for White illiteracy levels in different states in 1870 ...

On multilevel models and analysis

Oakes’ comments on multilevel analysis is scattered in scope, and most of it is not specific to our study. We do, however, feel impelled to make two corrections (one technical, and one conceptual) to avoid misrepresentation of our study.

First, Oakes’ discussion of statistical inferences in multilevel models cautions readers to be mindful of the statistical precision of our results. Contrary to what Oakes states, we did not use ‘frequentist’ estimation and testing strategy for our analysis, and neither did we present ‘confidence intervals’ or ‘P values’ based on ‘asymptotic Z-distribution’. We used a Bayesian estimation strategy,2 (p. xx) and reported the mean value (not point estimate) and the 95% credible interval (not confidence interval) of the distribution related to each of the parameter. Further, the credible intervals for the effects associated with Jim Crow laws are from a model that includes a state-level random part, which in turn correctly recognized, as would a ‘frequentist’ multilevel model estimation approach,10 that Jim Crow laws was a state-level variable with only 49 data points. Oakes is misinformed in stating that the degree of freedom for testing the effect of presence or absence of Jim Crow laws is 1. The degree of freedom depends not on the number of ‘treatment’ arms (2 in this instance) but on the number of groups (i.e. the number of states) in each arm.

Second, Oakes wrongly states that we believe that multilevel modelling ‘reveals the pitfalls of Robinson's argument against ecological correlations’.4 (p. xx) We can find no statement in our study that makes this claim. On the contrary, we make it clear we are not even analysing the same question that Robinson analysed. Indeed, if that were the case, we would not require a multilevel framework. Our whole motivation was that there is no need to pose the question as individual or ecological; instead, by analysing data simultaneously at different relevant levels we overcome this dilemma of having to choose a level of analysis. Doing so, we showed, led to a richer description of the relationship under scrutiny.

It is also worth stressing that the multilevel model estimates a parameter that summarizes between-state differences, and does so for different population groups, as opposed to including dummy indicator variables for every state (a fixed effects approach), making it impossible to assess the effect of Jim Crow laws as we would have used all the degrees of freedom at the state level.

On causal inference

We conclude by drawing attention to avoidable confusions that Oakes introduces with regards to the role of theory, study design, data analysis and causal inference in scientific research. Oakes’ criticism of multilevel regression analysis from a causal inference perspective is illustrative of such confusion, whereby he conflates issues of study design with statistical analysis; a conflation that Oakes has made before,11 and has been discussed elsewhere.12,13 On matters related to causal inference, the combination of theory (which informs the precise framing of the causal relationships and quantities of interest as well as the choice of variables) and study design (which concerns how the study should be designed in order to facilitate causal interpretation) trumps any form of statistical data analysis.14 In the absence of theory, and without appropriate study design and measurement of the quantities of interest, all statistical analyses produce GIGO (‘garbage in, garbage out’). At best, Oakes’ comment can be interpreted as a criticism of all statistical analysis (and it is highly likely that far more ‘single-level’ regressions are being estimated than ‘multilevel’ regressions in epidemiology, including social epidemiology). Moreover, Oakes is creating a false antithesis; it does not have to be multilevel or causal analysis—it can be both.15,16 We entirely support a deeper engagement on issues pertaining to causal inference in observational studies, including those involving contextual exposures,17 but such engagement also warrants theoretical precision and historical context.

In conclusion, we hope that by revisiting Robinson we encourage greater reflexivity about the contextual nature of our science, including the questions we ask and the methods we employ, thereby improving the likelihood of generating richer knowledge.

Funding

National Institutes of Health Career Development Award (NHLBI K25 HL081275 to S.V.S.).

References

1. Robinson WS. Ecological correlations and the behaviour of individuals. Am Sociol Rev. 1950;15:351–57.
2. Subramanian SV, Jones K, Kaddour A, Krieger N. Revisiting Robinson: the perils of individualistic and ecologic fallacy. Int J Epidemiol. 2009;38:342–60. [PMC free article] [PubMed]
3. Firebaugh G. Is the social world flat? W.S. Robinson and the ecologic fallacy. Int J Epidemiol. 2009;38:368–70. [PubMed]
4. Oakes JM. Commentary: individual, ecological and multilevel fallacies. Int J Epidemiol. 2009;38:361–68. [PubMed]
5. Pearce N. The ecological fallacy strikes back. J Epidemiol Community Health. 2000;54:326–27. [PMC free article] [PubMed]
6. Krieger N. Theories for social epidemiology in the 21st century: an ecosocial perspective. Int J Epidemiol. 2001;30:668–77. [PubMed]
7. Krieger N. Proximal, distal, and the politics of causation: what's level got to do with it? Am J Public Health. 2008;98:221–30. [PubMed]
8. Krieger N. Epidemiology and the web of causation: has anyone seen the spider? Soc Sci Med. 1994;39:887–903. [PubMed]
9. Foner E. Reconstruction: America's Unfinished Revolution, 1863–1877. New York: HarperCollins; 2002.
10. Goldstein H. Multilevel Statistical Models. London: Arnold; 2003.
11. Oakes M. The (mis)estimation of neighborhood effects: causal inference for a practicable social epidemiology. Soc Sci Med. 2004;58:1929–52. [PubMed]
12. Subramanian SV. The relevance of multilevel statistical methods for identifying causal neighborhood effects. Soc Sci Med. 2004;58:1961–67. [PubMed]
13. Diez Roux AV. Estimating neighborhood health effects: the challenges of causal inference in a complex world. Soc Sci Med. 2004;58:1953–60. [PubMed]
14. Rubin DB. For objective causal inference design trumps analysis. Annals Appl Stat. 2008;2:808–40.
15. Hong G, Raudenbush SW. Evaluating kindergarten retention policy: a case study of causal inference for multi-level observational data. J Am Stat Assoc. 2006;101:901–10.
16. Hong G, Raudenbush SW. Causal inference for time-varying instructional treatments. J Edu Behav Stat. 2008;33:333–62.
17. Subramanian SV, Glymour MM, Kawachi I. Identifying causal ecologic effects on health: a methodologic assessment. In: Galea S, editor. Macrosocial Determinants of Population Health. New York: Springer Media; 2007. pp. 301–31.

Articles from International Journal of Epidemiology are provided here courtesy of Oxford University Press