PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Child Dev. Author manuscript; available in PMC 2010 November 1.
Published in final edited form as:
PMCID: PMC2836492
NIHMSID: NIHMS181304

Estimating the “Impact” of Out-of-Home Placement on Child Well-Being: Approaching the Problem of Selection Bias

Abstract

This study used data on 2,453 children age 4 to 17 from the National Survey of Child and Adolescent Well-Being and 5 analytic methods that adjust for selection factors to estimate the impact of out-of-home placement on children's cognitive skills and behavior problems. Methods included ordinary least squares (OLS) regressions and residualized change, simple change, difference-in-difference, and fixed effects models. Models were estimated using the full sample and a matched sample generated by propensity scoring. Although results from the unmatched OLS and residualized change models suggested that out-of-home placement is associated with increased child behavior problems, estimates from models that more rigorously adjust for selection bias indicated that placement has little effect on children's cognitive skills or behavior problems.

Developmental researchers have long been interested in understanding how various caregiving experiences, both within and outside of children's homes, impact child development and well-being. An important extension of this line of inquiry focuses on changes in caregiving contexts due to child maltreatment and the subsequent out-of-home placement of children by Child Protective Services (CPS). Knowing whether out-of-home placement by CPS mitigates or heightens developmental risk for maltreated children is crucial for assessing the efficacy of child welfare policies and interventions, as well as for understanding the relative risks and benefits of major discontinuities in children's care. Despite the importance of this issue for both child welfare policy and developmental theory, however, studies of the effects of out-of-home placement on child well-being have yet to overcome important challenges related to selection bias in who enters state custody; that is, children who remain in the care of their parents and those who are placed out-of-home are likely to differ on a host of observable and unobservable factors, including socioeconomic characteristics and the types and severity of maltreatment they have experienced. Such differences pose a considerable barrier to producing unbiased estimates of the effects of out-of-home placement on child well-being. As such, it remains unclear whether placement is generally beneficial, harmful, or inconsequential for the development and well-being of maltreated children (Courtney, 2000; McDonald, Allen, Westerfelt, & Piliavin, 1996), especially those at the margin of placement (Doyle, 2007).

This study used data on 2,453 children age 4 to 17 from the National Survey of Child and Adolescent Well-Being (NSCAW) and five analytic methods that adjust for selection factors—including ordinary least squares (OLS) regressions with extensive controls and residualized change, simple change, difference-in-difference (a repeated measures analysis of group differences in change over time), and fixed effects models—to estimate the impact of out-of-home placement on children's cognitive skills and behavior. All models were estimated using both the full sample and a matched sample generated by propensity scoring. Comparing estimates produced by each of the methods may provide insight into whether associations between out-of-home placement and child well-being are likely to be causal.

The study extends prior research in its use of multiple methods to adjust for selection factors and in two additional ways. First, the NSCAW data enabled us to account for a wide range of confounding factors—including child and family background characteristics, CPS case characteristics, and children's pre-placement levels of cognitive skills and behavior problems—many of which have been omitted from previous studies. The NSCAW sample is also larger, more geographically diverse, and includes a wider age-range of children than other existing child welfare datasets. Coupled with the longitudinal design of the study, this allowed us to follow a larger sample of children over time on multiple indicators of well-being than has been possible in most prior work. Second, our approach improved upon recent research using NSCAW in that we defined our sample to include only children for whom we had in-home (i.e., pre-placement) data. Prior research with NSCAW has utilized samples that included children already placed out-of-home at the baseline interview (Barth, Guo, Green, & McCrae, 2007; Stahmer et al., 2007). As such, “baseline” measures of family characteristics and child well-being were actually assessed during an out-of-home placement for at least some children, thereby reflecting the characteristics of these children's placement families and their levels of well-being while in placement. By excluding children who were never observed in their home of origin, our sample enabled us to better isolate changes in child outcomes that may result from out-of-home placement.

Background

An estimated 3.6 million children were investigated or assessed by CPS agencies in the United States in 2006; more than a quarter were substantiated or indicated for child abuse or neglect (U.S. Department of Health and Human Services [USDHHS], 2008a). In all, more than 300,000 U.S. children entered and approximately 510,000 were residing in some form of out-of-home placement as a result of CPS involvement (USDHHS, 2008b). The traditional objectives of the U.S. child welfare system have been to ensure safety and promote permanency for children. Although improving child well-being has often been viewed as an implicit goal of the system, it has only more recently, with the passage of the Adoption and Safe Families Act of 1997 (ASFA), been made explicit (Wulczyn, Barth, Yuan, Harden & Landsverk, 2005). Indeed, since 2001 state CPS agencies have been required to undergo federal Child and Family Service Reviews which assess and monitor their progress towards promoting child safety and permanency, as well as meeting children's educational, physical, and mental health needs. Whether safety, permanency, and well-being are better served by removing children from their homes or keeping family units intact is likely to depend, at least in part, on the relative risks and benefits of continuity versus discontinuity of care for children. Out-of-home placement may provide children with physical safety and opportunities to develop nurturing relationships with adults, but its abrupt and indefinite nature may also place additional burdens on already vulnerable children. It may therefore present both opportunities for resilience and new stressors (c.f., Rutter, 2000).

Early work by Maas and Engler (1959) indicated that children in substitute care fared less well than community samples of youth on a range of developmental outcomes, a finding that has been repeatedly replicated (Blome, 1997; Clausen, Landsverk, Ganger, Chadwick, & Litrownik, 1998; Shin, 2004). Research also indicates, however, that children who come to the attention of CPS have experienced a variety of other adversities that alone might jeopardize functioning. These adversities often include parental substance use and mental health problems, poverty, and abuse or neglect (Barth, Wildfire, & Green, 2006; Child Welfare League of America, 2001; Reams, 1999), each of which may challenge development (Barnard & McKeganey, 2004; Brooks-Gunn & Duncan, 1997; Cicchetti & Toth, 2005; Glaser, 2000; Stone, 2007) and thereby partially or fully account for differences in outcomes between children experiencing out-of-home placement and those in community samples. Indeed, an early large-scale longitudinal study of children in out-of-home placement found that many showed improvement with regard to physical, cognitive, and school-related outcomes during their first 6 months in care (Fanshel & Shinn, 1978), suggesting that some of the adverse outcomes associated with placement may reflect pre-placement experiences rather than the effect of placement itself (Waldfogel, 2000).

Further complicating efforts to understand the unique impact of out-of-home placement on child well-being, children who are placed out-of-home are also likely to differ from those who are substantiated for maltreatment but are not removed from home: They are likely to have experienced a greater severity of maltreatment and higher levels of prior contact with CPS (USDHSS, 2005). The two groups also tend to differ on factors such as parental cooperation with CPS, parental stress, parenting skills, social support, substance abuse, domestic violence, and criminal justice involvement (Shlonsky, 2007). As such, isolating the impact of out-of-home placement on well-being poses a number of analytic challenges.

Analytic Challenges and Identification Strategies

Because it is not possible to simultaneously observe a child both in his or her home and in an out-of-home placement, or to randomly assign children to out-of-home placement, researchers must rely on statistical methods to adjust for selection bias in who enters state care. Four strategies have been used in existing studies. These include controlling for correlates of both out-of-home placement and child outcomes, employing matching techniques, estimating change models, and utilizing instrumental variables methods. Below, we discuss the benefits and disadvantages of each of these approaches and highlight the value of applying additional methods to the study of relations between out-of-home placement and child well-being.

The most commonly used strategy for attempting to adjust for pre-placement ecological adversity and maltreatment experiences is to control for confounding covariates while comparing outcomes for children who have experienced out-of-home placement and those who have not. Most existing studies of this type have compared children experiencing placement to those who were not maltreated (or were not reported to CPS) but had similar levels of socioeconomic disadvantage (Bilaver, Jaudes, Koepke, & Goerge, 1999; Blome, 1997; Buehler, Orme, Post, & Patterson, 2000; Fantuzzo & Perlman, 2007; Pears and Fisher, 2005a; 2005b; Viner & Taylor, 2005) or to children who were maltreated (or reported to CPS) but remained in their homes (Johnson-Reid & Barth, 2000; Leslie, Gordon, Ganger, & Gist, 2002). Studies using this approach have found adverse associations of placement with child and adult outcomes in domains such as health, emotional and behavioral adjustment, cognitive development, criminal justice involvement, educational attainment, and economic status. Although this strategy is useful for adjusting for the confounding effects of observed variables that are associated with both out-of-home placement and child outcomes, estimates are subject to bias due to unobserved factors. Studies of this type are further limited in that they do not account for baseline differences in children's scores on the outcome(s), despite that children placed out-of-home and those remaining in-home are likely to differ in this regard. The detailed background information available in NSCAW allowed us to control for an extensive set of potentially confounding variables; we also employed a series of additional methods that accounted for children's baseline scores on the outcome measures and adjusted for some forms of unobserved heterogeneity.

A second strategy utilizes propensity score matching methods (Rosenbaum & Rubin, 1983) to identify treatment and comparison groups that are statistically equivalent on observable background characteristics and differ only in terms of whether children have experienced out-of-home placement. Associations between out-of-home placement and child outcomes are then estimated for the matched groups. Matching methods offer a distinct advantage over simply controlling for confounding covariates in that they can ensure appropriate overlap in the covariate distributions of the treatment and comparison groups, such that the models are not extrapolating over portions of the covariate distributions in which there is no support. Like comparison group studies that control for background characteristics, however, they adjust only for measured selection factors; estimates continue to be subject to bias due to unobserved factors.

Although several studies (Barth et al., 2007; Rubin, O'Reilly, Luan, & Localio, 2007) have used matching methods to estimate group differences in well-being among children in out-of-home care, we are aware of only one that directly compared outcomes for children experiencing placement and those remaining in-home. Berzin (2008) compared young adult outcomes for youth who had experienced foster care placement at some point during childhood with those of matched and unmatched samples of youth who had not. Results using the unmatched data indicated that youth who experienced placement had lower levels of educational attainment and higher rates of public assistance use, teen parenting, and criminal justice involvement. In contrast, results using one-to-one nearest neighbor propensity score matching, the matching scheme that produced the most similar treatment and comparison groups, revealed no associations between placement and any of these outcomes. This suggests that the associations identified in the unmatched data likely reflected differences in the characteristics and experiences of youth who did and did not experience placement, rather than the experience of placement itself. As noted above, however, matching methods are subject to bias due to omitted variables. Indeed, Berzin (2008) notes that several important background factors, such as parental substance abuse and criminal justice involvement, as well as the nature of maltreatment experienced by sample children, were omitted from her models. In addition, her analyses utilize a relatively small treatment group sample of 136 youth who experienced foster care placement at some point during childhood and were observed between the ages of 17 and 24. The study is unable to account for the timing of out-of-home placement during childhood or to assess the influence of placement on proximal outcomes. In contrast, we assessed the influence of out-of-home placement on children's cognitive and behavioral development over approximately the 2 1/2 years encompassing placement. Furthermore, our sample included considerably more children who experienced placement (N = 342), and our matching models accounted for a wider range of background factors, including pre-placement assessments of the outcome measures.

A third approach to reducing selection bias involves using longitudinal data to estimate changes in well-being over time as a function of out-of-home placement. Models for doing so can take different forms, including the residualized change, simple change, difference-in-difference, and fixed effects models that we utilized in the current study. In general, however, they reduce bias by accounting for children's baseline (or mean) levels of well-being when predicting later levels of well-being or changes in well-being over time. They thereby adjust—to varying degrees—for pre-existing (unmeasured) differences between children who are subsequently removed from home and those who are not. For example, change models may be useful for adjusting for prior maltreatment severity, which likely influences children's baseline and final levels of cognitive skills and behavior problems, as well as whether they are placed out-of-home, but is difficult to measure (Litrownik et al., 2005) and often omitted from empirical analyses.

Existing studies of this type have generally estimated either what we refer to as “residualized” change models (National Institute of Child Health and Human Development Early Child Care Research Network [NICHD] & Duncan, 2003), in which a given outcome is modeled as a function of an earlier score on that outcome and a set of background characteristics and experiences (Stahmer et al., 2007), or repeated measures analyses using MANOVA (Davidson-Arad, 2005; Davidson-Arad, Englechin-Segal, & Wozner, 2003) or ANCOVA (Lawrence, Carlson, & Egeland, 2006). By directly modeling the baseline score, such methods adjust for factors that are not directly observed, but that determine initial functioning on the outcome of interest and thereby influence the follow-up score. Several such studies have found placement to be associated with improvements over time on physical (physical activity, orientation, health, and safety) and psychological (self-actualization, relaxation, mental health, and identity) dimensions of quality of life (Davidson-Arad et al., 2003; Davidson-Arad, 2005). In contrast, others have reported associations of out-of-home placement with increases in behavior problems (Lawrence et al., 2006) and decreases in cognitive and language development (Stahmer et al., 2007).

Existing studies using MANOVA or ANCOVA are limited in that they have adjusted for few observable covariates and have utilized very small samples. A noteworthy shortcoming of the residualized change approach is that it does not adjust for factors that differentially impact an outcome at baseline and follow-up. As such, NICHD and Duncan (2003) emphasize the value of comparing results from multiple types of change models when attempting to estimate causal relations. Thus, in addition to OLS and residualized change models, we also employed simple change, difference-in-difference, and fixed effects models to estimate associations of out-of-home placement with child well-being. Our simple change and difference-in-difference models both used between-child variation in placement status to assess whether changes in well-being over time differed for children who did and did not experience placement. The simple change model adjusted for initial differences in well-being at the individual level, whereas the difference-in-difference model did so at the group level. Our fixed effects model used within-child variation in placement status to identify effects of out-of-home placement on intra-child change in well-being for children observed both in-home and during or after out-of-home placement. As discussed in the Method section, each of these strategies is likely to reduce bias to a greater extent than OLS and residualized change models, albeit under different assumptions. To further adjust for selection factors, we also estimated all of our models utilizing both our full and matched samples.

A fourth strategy for estimating causal effects involves the use of instrumental variables. We are aware of only one existing study of out-of-home placement that utilized this approach. Doyle (2007) used unique data on Illinois CPS caseworkers' propensities to remove children from home as an instrument to isolate the exogenous component of out-of-home placement and, thereby, to estimate causal effects. Results suggested that children assigned to caseworkers with higher propensities for removal were more likely to be placed out-of-home and, subsequently, to exhibit higher levels of delinquency and teen childbearing, and lower earnings. Although instrumental variables techniques can be used to estimate causal relations by purging an independent variable of the bias that results from unobserved heterogeneity (e.g., omitted variables) or endogeneity (Gennetian, Magnuson, & Morris, 2008), identifying a valid instrument is difficult (Currie, 2005). In the present case, we would have required an instrument that produced random variation in the likelihood that a child was placed out-of-home, but that was otherwise uncorrelated with child outcomes. We could not utilize state level policies regarding child removal as instruments, for example, because such policies are likely to be correlated with other state level factors that may independently influence child outcomes. Thus, we did not utilize instrumental variables methods in our analyses.

Placement Characteristics

Associations between out-of-home placement and child well-being are likely to vary by the length, stability, and type(s) of placements children experience. Placement length and stability may moderate these associations by influencing the consistency of care to which children are exposed and whether children experience disruptions in care (James, Landsverk, & Slymen, 2004). Children who spend a short period of time out-of-home may be affected differently than those who spend months or years in care. Likewise, children experiencing multiple placements may be impacted differently than those with stable placement experiences. Results from prior research on these factors have been inconsistent. Several studies demonstrate links between placement instability and adverse child outcomes (Newton, Litrownik, & Landsverk, 2000; Rubin et al., 2004; Rubin et al., 2007); other work finds no variation in outcomes by placement length or number of placements (Pears & Fisher, 2005a, 2005b).

Research on placement type has consistently indicated that children placed in residential facilities experience greater developmental adversity than those placed in family foster care, even after adjusting for initial levels of functioning (Davidson-Arad, 2005; Davidson-Arad, et al., 2003; McDonald et al., 1996). Among children placed in family foster homes, developmental outcomes may be influenced by the kinship status of foster parents. Residing with kin is thought to minimize the trauma of out-of-home placement and thereby promote child well-being (Chamberlain et al., 2006; Chapman, Wall, Barth, & the NSCAW Research Group, 2004; Ehrle & Geen, 2002; Holten, Rønning, Handegård, & Sourander, 2005; James, 2004). At the same time, kin foster parents tend to be less advantaged (Berrick, Barth & Needell, 1994; Ehrle & Geen, 2002) and to exhibit lower quality caregiving behaviors than non-kin foster parents (Gaudin & Stuphen, 1993; Harden, Clyman, Kriebel, & Lyons, 2004). As such, the extent to which developmental outcomes may be differentially affected by kin and non-kin placements is theoretically ambiguous. Empirical results have been similarly mixed. Findings from several studies suggest that children placed in non-kin foster homes exhibit poorer outcomes than those placed with kin (Holten et al., 2005; Keller, Wetherbee, Le Prohn, Payne & Lamont, 2001; Landsverk, Davis, Ganger, Newton, & Johnson, 1996; Lawrence et al., 2006; Leslie et al., 2002), whereas others provide little evidence of differences (Barth et al., 2007; Benedict, Zuravin & Stallings, 1996; Shore, Sim, Le Prohn, & Keller, 2002; Zimmer & Panko, 2006).

Although multiple existing studies have examined associations of placement characteristics with child outcomes, it is important to recognize that placement length, stability, and type are likely to be endogenous (i.e., determined by similar factors or processes) with children's cognitive skills and behaviors. As such, it is unlikely that estimates of associations between placement characteristics and child outcomes will reflect causal relations even when rigorous methods of adjusting for selection factors are employed. Nonetheless, to be consistent with prior work, and also in the hope of shedding further light on the potential moderating roles of placement length, stability, and type, we estimated supplemental models that allowed associations of out-of-home placement with child outcomes to vary by each of these factors. We note, however, that these estimates should not be interpreted as reflecting causal relations.

Method

Participants

Our analyses necessitated the use of data with highly detailed information on children's out-of-home placement experiences, background characteristics, and pre- and post-placement levels of well-being. NSCAW, the first national probability study to collect data directly from children and families coming into contact with CPS in the U.S., is the only available data source that provides such information on a national level. The full NSCAW sample includes 5,501 children from birth to age 14 (at baseline), who were investigated by CPS between the fall of 1999 and the winter of 2000. The study attempted to follow all sample children for 36 months, regardless of case disposition. Data were collected via interviews with children, caregivers, teachers, and CPS investigators at baseline and 12-, 18-, and 36-months after enrollment.

We used data provided by children, parents, current caregivers, and CPS caseworkers via the baseline, 18- and 36-month interviews (child assessments were not conducted at the 12-month interview). Our sample included only those children who entered the child welfare system due to a new CPS investigation during the initial NSCAW sampling period and were observed in-home at either the baseline interview (conducted on average about 14 weeks after the initial investigation) or, for children placed out-of-home at the baseline interview, the 18-month interview. We excluded children who were placed out-of-home at the time of both their baseline and 18-month interviews because we had no in-home (baseline) data on them or their families of origin. As such, we focused on estimating associations between subsequent out-of-home placement and child well-being over a two and a half year period (on average) for children who were between the ages of 4 and 14 at the time of NSCAW sampling and were observed living in their home of origin at the time of the baseline assessment used in this study.

From the full NSCAW sample of 5,501, we excluded 2,288 children because they were younger than age 4 at baseline and, therefore, were not assessed on the outcome measures of focus. We excluded an additional 694 children who entered the NSCAW sample as a result of a new CPS investigation but were not observed in-home at either the baseline or 18-month interview. Finally, we excluded 66 children for whom contradictory living arrangement data provided by caseworkers and caregivers precluded us from determining whether a child was in-home or out-of-home during the NSCAW assessments. Our final analysis sample consisted of 2,453 children. All of these children were retained in our sample throughout the observation period, regardless of their subsequent placement experiences. Given our exclusion criteria, our analysis sample was no longer representative of all children entering the child welfare system, nor of the original NSCAW sample; however, about 66% of children entering and 71% of children residing in out-of-home care in 2006 were between the ages of 4 and 17 (USDHHS, 2008b). Furthermore, descriptive statistics (not shown) indicated that our analysis sample was similar to the full NSCAW sample of children age 4 to 14 at baseline on most background characteristics, with the exception that the mean family risk score (described below) was higher in the full sample. Given that children excluded from our sample were out-of-home at both baseline and 18-months, it makes sense that their homes of origin were considerably less safe than those of children included in our sample. As such, our results may not be generalizable to children who are most likely to be immediately removed from home and kept in care for a long period of time. Although we selected the most appropriate sample with which to estimate causal effects of out-of-home placement on child well-being, our strategy increased the internal validity of our study at the expense of external validity.

Measures

Out-of-home placement

Our primary predictor variable was an indicator (1 = yes) that a child was removed from home by CPS between his or her baseline and follow-up assessments. This measure included placements in kin and non-kin family foster homes as well as other types of care (i.e., group homes, emergency shelters, psychiatric hospitals, residential treatment facilities, detention centers, transitional living arrangements, or other settings). Because children experiencing these ‘other’ types of placement may substantially differ from those experiencing only family foster homes, we conducted supplemental analyses that excluded the former group.

We designated the initial NSCAW interview as the baseline assessment point for all children observed in-home at that time (94%) and the 18-month interview as the baseline assessment point for the 6% of children who were not observed in-home at baseline, but were observed in-home at 18-months. We designated the 36-month interview as the follow-up assessment point for all children observed (either in-home or out-of-home) at 36 months (91%). We designated the 18-month interview as the follow-up assessment point for the 9% of children who were observed in-home at baseline and were assessed (either in- or out-of-home) at 18-months, but not at 36-months. In all models, we controlled for the number of months between a child's baseline and follow-up observations. We also controlled for whether a child experienced placement prior to their (in-home) baseline assessment. Because children who experienced placement prior to their baseline assessment are likely to differ from those who did not, we also estimated supplemental models which excluded the former.

Utilizing a dichotomous indicator of whether a child experienced placement to predict cognitive skills and behavior problems (and changes therein) provides evidence of the average association between placement and these outcomes, regardless of the length of time spent in care and the number and types of placements experienced. However, each of these factors—though likely endogenous to children's well-being—may moderate the associations of interest. Thus, in supplemental analyses, we also estimated models that used as the key predictors: (1) the total proportion of time a child spent out-of-home between the baseline and follow-up interview (logarithm), as well as the number of placements the child experienced during the observation period; and (2) the proportion of time (logarithm) a child spent in each placement type, as well as the number of placements the child experienced during the observation period.

Cognitive skills and behavior problems

Cognitive skills and behavior problems were measured at both baseline and follow-up. Children's cognitive skills were assessed using the Kaufman Brief Intelligence Test (K-BIT; Kaufman & Kaufman, 1990). The K-BIT measures verbal and non-verbal intelligence for individuals ranging in age from 4 to 90. The test is comprised of vocabulary (word knowledge and concept formation) and matrices (ability to perceive relationships and complete analogies) subtests and has demonstrated adequate internal consistency and test-retest reliability among children and adolescents (Kaufman & Kaufman, 1990). Among the NSCAW sample, internal consistency was .76 for the vocabulary subtest and .79 for the matrices subtest. Behavioral problems were assessed by children's raw scores on the internalizing and externalizing behavior problems subscales of the Child Behavior Checklist (CBCL) for children ages 4 to 18 (Achenbach, 1991). The CBCL was completed by the child's primary caregiver (usually the child's biological or foster mother) at each time point. The internalizing subscale (α = .91) measures withdrawn behaviors, somatic complaints, and anxious/depressed behaviors; the externalizing subscale (α = .92) measures delinquent and aggressive behaviors. Each of these instruments has favorable psychometric properties and has been widely used in prior research.

Control variables

Our models controlled for a rich set of covariates. Baseline demographic characteristics included continuous measures of child age, caregiver age, and the family's income-to-poverty ratio, as well as indicators (1 = yes) for whether the child was female; the child was Black, Hispanic, or of another race/ethnicity (with White children serving as the reference category); the primary caregiver was single (versus married); the primary caregiver was born outside of the U.S.; the primary caregiver's education was less than high school, high school (reference category), or more than high school; and a grandparent was present in the household. Baseline child maltreatment and CPS involvement measures included a 24-item family of origin risk score that was completed by the caseworker in the initial CPS investigation (α = .71); indicators (1 = yes) for whether the initial CPS investigation included physical abuse, sexual abuse, neglect-failure to provide, neglect-failure to supervise, and other maltreatment; an indicator for whether the initial investigation was substantiated; and an indicator for whether the focal child experienced out-of-home placement prior to the baseline assessment. The family of origin risk score was comprised of a series of indicators at the child (prior maltreatment reports, poor ability to self-protect, special needs), primary or secondary caregiver (substance abuse or mental health problems, criminal justice involvement, cognitive impairments, poor physical health, poor parenting, unrealistic expectations for the child, history of maltreatment, excessive discipline, recognition of problems in the home and motivation to change them, reasonable level of cooperation with CPS), and family (high stress, low social support, trouble accessing basic necessities, domestic violence) levels. We standardized this measure to have a mean of 0 and standard deviation of 1. Finally, we controlled for the number of months between a child's baseline and follow-up assessments in all models.

Missing Data

Like most longitudinal studies, NSCAW contains a considerable amount of missing data. Although nonresponse analyses suggested that missing data in NSCAW “is unlikely to be consequential for most types of analyses” (USDHHS, 2005, p. 2-12), complete case analysis can nonetheless lead to biased estimates (Little & Rubin, 1987). Furthermore, had we performed complete case analyses based on all of the variables included in our study, our sample size would have been limited to 1,610 children (249 of whom experienced out-of-home placement). Were we to conduct complete case analyses for each outcome, our sample sizes would have ranged from 1,626 to 1,794 children. Doing so, however, would have impeded comparisons across analyses as each would be based on a different sample. Thus, we imputed missing data for our full analysis sample using multiple imputation techniques (Allison, 2002; Rubin, 1987; Schafer, 1997). Multiple imputation is based on more plausible assumptions than complete case analysis. Complete case analysis assumes that the subsample of complete cases essentially comprises a random draw from the original sample such that each case has an equal probability of missing data; multiple imputation assumes only that data are missing at random such that the probability of missing data on a variable is unrelated to the value of that variable after taking into account the other variables in the analysis. Analyses of missing data in our analysis sample (not shown) revealed that grandparent presence in the household of origin, having a non-U.S. born caregiver, and experiencing an out-of-home placement prior to the baseline assessment were associated with an increased probability that a child had missing values on one or more variables; having an initial report for neglect-failure to provide and being observed for a slightly shorter duration between baseline and follow-up were associated with a decreased probability of missing data. Neither out-of-home placement experiences during the period of observation, nor any of the baseline or follow-up measures of cognitive skills or behavior problems was associated with the likelihood that a child had missing data, increasing our confidence that multiple imputation was an appropriate strategy. We used Stata's ice program to impute 5 datasets using all variables included in our analyses, and its mim program to conduct our analyses.

Empirical Strategy

We estimated the effects of out-of-home placement on cognitive skills and behavior problems for children over (on average) a two and a half year observation period utilizing five analytic methods: (1) OLS regressions with a rich set of controls, (2) residualized change models, (3) simple change models, (4) difference-in-difference models, and (5) fixed effects models. Each model was estimated using our full analysis sample and a sample generated by propensity score matching. As discussed above, adjusting for unobserved differences between children experiencing placement and those remaining in-home, and thus ensuring comparable treatment and comparison groups, poses a major challenge to identifying causal effects of out-of-home placement. Because OLS regression adjusts only for observable factors, results from the (particularly unmatched) OLS models will reflect more bias than those from the change models. As such, we present unmatched OLS results primarily for comparison with those of the matched OLS models and the change models. Although each of the additional methods should operate to reduce omitted variable bias in a similar way, each relies on different assumptions with associated shortcomings, such that there is likely to be variation in results across methods. Furthermore, none of the methods can be assumed to independently produce causal estimates; however, a comparison of estimates across models may provide insight as to whether associations between out-of-home placement and child well-being are likely to be causal.

We first estimated OLS regressions that related the level of well-being at follow-up to whether a child experienced out-of-home placement between baseline and follow-up, net of a host of background factors. The general form of these models was:

CWFi=α+β1OOHi+β2Xi+εi
(1)

where CW-Fi is a cognitive skills or behavior problems measure for child i at follow-up; OOHi is the key independent variable, whether child i experienced an out-of-home placement between baseline and follow-up; Xi is a vector of covariates; and ε is a disturbance term. β1 is the “treatment effect” of out-of-home placement on well-being at follow-up. We first estimated this model controlling only for the number of months between the baseline and follow-up assessments. We then added baseline demographic characteristics, followed by child maltreatment and CPS involvement factors (in two steps) in order to assess the extent to which the associations of interest varied with the inclusion of more extensive controls.

Despite the inclusion of a host of controls, estimates produced by the OLS models are subject to bias due to unmeasured factors that are correlated with both placement and child well-being. These factors may partially be reflected in children's baseline scores on the outcomes. To attempt to account for such bias, the second step of our analysis consisted of what we refer to as residualized change models (NICHD & Duncan, 2003), but which have also been referred to as lagged dependent variable and regressor variable methods (Allison, 1990), of the form:

CWFi=α+β1OOHi+β2Xi+β3CWBi+εi
(2)

where CW-Bi is the baseline measure of the outcome (CW-Fi). In equation (2), well-being at follow-up is modeled as a function of out-of-home placement, baseline well-being, and the full set of controls. The initial well-being score functions as a proxy for unobserved pre-existing differences between children. Its inclusion in the model serves to adjust for the average influence of baseline well-being on later well-being, assuming that the baseline measure (and any associated unobserved factors) has an identical effect on the follow-up measure for children who did and did not experience placement. That is, the model adjusts for persistent child characteristics (e.g., genetic factors) that have consistent effects on CW-Bi and CW-Fi for children in both groups. Resulting estimates are less subject to bias than those produced by the standard OLS model. The treatment effect is interpreted as the effect of out-of-home placement on child well-being at follow-up, net of initial well-being; the model does not directly estimate the effect of placement on changes in well-being between baseline and follow-up.

The third step in our analysis was to estimate simple change models (Allison, 1990; NICHD & Duncan, 2003; Rogosa, Brandt, & Zimowski, 1982; Rogosa & Willett, 1983) that assessed the influence of out-of-home placement on changes in well-being, net of a host of covariates. We did so using the following equation:

ΔCWi=α+β1OOHi+β2Xi+εi
(3)

where ΔCWi is the difference between a child's follow-up and baseline well-being scores. This model uses variation between children to identify differences in the average change in well-being over time for those children who did and did not experience placement. Coefficients are interpreted as the effects of the predictor variables on changes in the outcome. The model provides a significant advantage over the OLS and residualized change models by reducing bias associated with unobserved factors that have the same effect on well-being at both time points, while also allowing observed fixed characteristics to have time varying effects (i.e., to directly affect the change in well-being). Although the simple change model should reduce bias to a greater extent than the OLS and residualized change models, it has less precision to detect associations and does not adjust for unobserved variables that differentially affect well-being at different points in time.

Our fourth and fifth steps consisted of estimating difference-in-difference and fixed effect models. Both required that our data be organized in child-wave observations. As such, for these analyses we utilized 4,906 observations in which each child was represented twice (once at baseline and once at follow-up). The difference-in-difference model allowed us to compare the average effect of placement between non-identical treatment and comparison groups while accounting for both within group differences over time and between group differences at each time point. The model operates on the assumption that the relative difference in well-being between groups at each time point will be statistically equivalent in the absence of out-of-home placement. The approach removes bias that could be due to stable differences between the two groups, as well as that which could be due to time trends. We used these models to assess whether the average change in well-being between baseline and follow-up was equivalent for children who did and did not experience placement. The treatment effect (difference-in-difference estimator) is computed by subtracting the average change in the outcome between the two time points for the control group from the average change in the outcome for the treatment group (Murray, 2006). The model is similar to the simple change model except that it adjusts for average initial differences at the group, rather than individual, level. We estimated models of the form:

CWit=α+β1OOHi+γ1FUit+δ1OOHiFUit+β2Xi+εit
(4)

where CWit is a well-being measure for child i at time t (baseline or follow-up); OOHi is an indicator that a child experienced placement, and is equal to 1 at both time points for children who did so; FUit is an indicator (1 = yes) of whether a given observation occurred at follow-up; and OOHi*FUit is the interaction between out-of-home placement and the observation occurring at follow-up. Here, β1 is a group difference in well-being that is assumed to be constant over time, such that it reflects the initial difference in well-being between children who were subsequently placed out-of-home and those who remained in-home (it does not reflect the effect of out-of-home placement on well-being); γ1 is the average change in well-being between baseline and follow-up for all children (regardless of out-of-home placement status); and δ1 is the treatment effect or difference-in-difference estimate, which reflects the extent to which the average change in well-being between baseline and follow-up differs for children who did and did not experience out-of-home placement. The difference-in-difference estimate is computed as (CW¯(OOH=1,FU=1)CW¯(OOH=1,FU=0))(CW¯(OOH=0,FU=1)CW¯(OOH=0,FU=0)), where CW¯ is the adjusted group mean on the outcome. We corrected the standard errors produced by the difference-in-difference models for intra-cluster correlation due to multiple observations of each child. In the tables that follow, we report estimates for only δ1, which reflect the (treatment) effects of out-of-home placement on changes in well-being over time; we do not present estimates for β1 or γ1.

Although the difference-in-difference model reduces bias by accounting for pre-existing group differences in well-being and assessing whether the treatment and control groups experience similar changes in well-being over time, estimates will be biased to the extent that unobserved factors (e.g., a CPS policy change during the observation period that resulted in additional services for only one group of children) differentially affect changes over time in well-being for the two groups. Like the simple change model, the difference-in-difference model allows permanent characteristics to have time varying effects on the outcome (i.e., such characteristics are not differenced out of the model).

All of the change models described thus far use between-child variation in placement status to identify differences in the average change in well-being over time for children who did and did not experience placement. In contrast, fixed effects models identify effects via intra-child change for children observed both in-home and during or after out-of-home placement (Duncan, Magnuson, and Ludwig 2004). The method expresses each variable as a deviation from a child's mean value (across time) on that measure and differences the regression equation across time periods. This eliminates all time invariant observed and unobserved variables from the model. However, fixed effects estimates are subject to bias if unobserved variables (or their effects on the outcome) are time-varying, or if permanent characteristics have time-varying effects. Our fixed effects models took the following form:

ΔCWi=Δβ1OOHi+Δβ2Xi+Δεi
(5)

where β1, the treatment effect, is identified only for children who experienced placement.

Because 4 of the 5 models presented above identify effects through between-child variation in placement status, ensuring that our models are estimated for similar treatment and comparison groups with adequate overlap on their covariate distributions is a pressing concern. For this reason, the final step in our analysis consisted of re-estimating all of the models described above after using propensity score matching methods (Rosenbaum and Rubin, 1983) to construct treatment and comparison groups that were statistically equivalent with regard to all background characteristics. The primary advantage of this strategy is that it restricts inference to treatment and comparison samples with adequate overlap in the covariate distributions, thereby avoiding unwarranted model extrapolation. In addition, inferences drawn from the matched sample are specifically applicable to children on the margin of removal—the most relevant group among which to assess the impact of out-of-home placement on well-being. For models that compare well-being across children (all but the fixed effects models), this strategy allowed us to more rigorously adjust for selection bias by estimating differences between comparable groups of children. For the fixed effects models, it allowed us to estimate within-child change in well-being for those children who experienced placement and were also most like those who did not.

To construct the matched treatment and comparison groups, we first used a probit regression to estimate each child's conditional probability (i.e., “propensity score”) of being placed out-of-home based on all of our covariates (child age, race, and gender; caregiver age, marital status, education, and nativity; grandparent presence; family income-to-poverty ratio and risk score; types of maltreatment alleged; whether the initial report was substantiated; and whether the child experienced an out-of-home placement prior to the initial assessment), as well as children's baseline vocabulary, matrices, internalizing behavior problems, and externalizing behavior problems scores. We then used one-to-one matching without replacement to match each treatment (out-of-home placement) group child to the comparison group child with the closest propensity score, thereby limiting the sample to treatment and control children for whom there was sufficient overlap in propensity scores (common support). Unmatched children were discarded from the sample. Additionally, we trimmed the 10% of treatment observations at which the propensity score density of the comparison group was the lowest. After matching the treatment and comparison groups, we conducted additional analyses (not shown) to ensure adequate balance across their covariate distributions and found no significant differences on any of the background characteristics between the groups. The resulting matched sample (N = 616) was comprised of 308 children who experienced placement and 308 children who did not. These analyses were performed using Stata's psmatch2 program.

Because our analysis sample was not representative of the full NSCAW sample, the sample weights were not applicable and, therefore, not used; this should not influence our results aside from limiting their generalizability (Barth, Gibbons, & Guo, 2006), as discussed above.

Results

Descriptive Statistics

Table 1 presents descriptive statistics for out-of-home placement characteristics for children who experienced placement, as well as baseline demographic characteristics and child maltreatment and CPS involvement measures for children who did and did not experience placement. Approximately 14% of the children in our analysis sample experienced out-of-home placement between their baseline and follow-up assessment. On average, children who were removed from home spent 48% of the observation period out-of-home and experienced slightly more than 2 placements; they spent 17% of the observation period in non-kin foster homes, 17% in kin foster homes, and 13% in other types of placement.

Table 1
Sample Characteristics

The raw data also indicated that children who experienced placement differed from those who did not on several important background characteristics. At baseline, children who were removed from home were more likely to have had a U.S. born caregiver. They also had older caregivers with lower levels of educational attainment, as well as lower family income-to poverty ratios and higher family risk scores. Finally, they were less likely both to have had an initial investigation that was for sexual abuse and to have been removed from home prior to their baseline assessment, but more likely to have had their initial CPS investigation substantiated.

Table 2 presents mean baseline and follow-up scores on the outcome variables for the full sample of children and for those who did and did not experience placement. The raw data indicate that, on average, children in our sample experienced increases in vocabulary and matrices skills, and decreases in internalizing and externalizing behavior problems between baseline and follow-up. Mean vocabulary and matrices scores did not significantly differ at either baseline or at follow-up for children who were removed from home and those who were not; children in both groups experienced significant (and similar) gains in cognitive skills between the two time points. Raw data for behavior problems, however, indicated that children who were not removed from home had significantly fewer internalizing and externalizing difficulties at both baseline and follow-up than children who experienced placement. In addition, children remaining in-home exhibited decreases in both internalizing and externalizing behavior problems between baseline and follow-up, whereas children who were removed from home displayed similar levels of internalizing behavior problems at both time points, but fewer externalizing behavior problems at follow-up than at baseline.

Table 2
Descriptive Statistics for Child Cognitive Skills and Behavior Problems Measures

To further examine associations of baseline cognitive skills and behavior problems with out-of-home placement, we estimated a series of models in which an indicator (1 = yes) for whether a child experienced out-of-home placement was regressed on a baseline cognitive skills or behavior problems measure and the full set of baseline covariates. Results (not shown) suggested that, after accounting for these selection factors, lower baseline cognitive skills and higher baseline behavior problems were associated with an increased likelihood of placement. This finding underscores the importance of adjusting for pre-existing differences when estimating effects of placement on child outcomes. We next present results from a series of regression models in which we employed multiple strategies of adjusting for selection factors.

Regression Results

Full sample

Table 3 presents regression results when the full (unmatched) sample was employed. Models 1 through 3 are the OLS regressions controlling for an increasingly detailed set of covariates. Models 4 through 6 are the residualized change, simple change, difference-in-difference, and fixed effects models. With regard to cognitive skills, results from Model 1, in which we controlled only for the number of months between the baseline and follow-up assessments, were consistent with the patterns indicated by the raw data: We found no significant differences in cognitive skills at follow-up between children who experienced placement and those who did not. This pattern held across all 7 models. Results for behavior problems were also consistent with the pattern found in the raw data: Model 1 revealed associations of placement with higher levels of both internalizing and externalizing behavior problems at follow-up. Although these associations were modestly attenuated with the addition of baseline demographic and child welfare case characteristics to the models, they retained statistical significance even after the full set of controls was included (Model 3). Results from the residualized change model (Model 4) suggested that, after controlling for baseline scores on the outcome measures, placement continued to be associated with higher internalizing behavior problems at follow-up, but was not associated with externalizing behavior problems. In contrast, results from the simple change, fixed effects, and difference-in-difference models indicated that placement was not associated with internalizing behavior problems, whereas results from the simple change and fixed effects models indicated that children experiencing placement exhibited a decrease in externalizing behavior problems between baseline and follow-up (the negative coefficient from the difference-in-difference model for externalizing behavior problems was also marginally significant at p = .066). Effect sizes, computed by dividing the relevant coefficient by the standard deviation (SD) of externalizing behavior problems for the full sample at follow-up (10.47), suggested that placement was associated with a 0.14 to 0.15 SD decrease in externalizing behavior problems between baseline and follow-up. On the whole, these results imply that, once unobserved time invariant selection factors have been taken into account, placement has a modest protective effect with regard to externalizing behavior problems and a neutral effect with regard to both cognitive skills and internalizing behavior problems.

Table 3
OLS and Change Models of the Effects of Out-of-Home Placement on Subsequent Cognitive Skills and Behavior Problems, Unmatched Sample

Matched sample

Table 4 presents results from models 3 through 7 when estimated using the matched sample. These results are quite different from those shown in Table 3. Indeed, regardless of the method used to estimate associations of out-of-home placement with children's cognitive skills and behavior problems, we found no significant estimates when the matched sample was employed. A potential concern here is that we lacked the statistical power to detect effects in the matched sample given its considerably reduced sample size. However, an examination of tables 3 and and44 revealed that the differences in estimates were not solely due to imprecise estimation (larger standard errors) when the matched sample was employed; in many cases the coefficients themselves differed substantially across the tables suggesting that adjusting for differences in background characteristics via propensity score matching accounted for all remaining associations of out-of-home placement with cognitive skills and behavior problems.

Table 4
OLS and Change Models of the Effects of Out-of-Home Placement on Subsequent Cognitive Skills and Behavior Problems, Matched Sample

Extensions

We conducted a series of extensions (results not shown) to our primary analyses. First, we re-estimated the internalizing and externalizing behavior problems models using the clinical cutoffs for these measures as outcome variables. Results from both the OLS model with the full set of controls and the residualized change model revealed associations of out-of-home placement with clinical levels of both internalizing and externalizing behavior problems when the unmatched sample was employed. However, we found no associations between out-of-home placement and clinical levels of behavior problems when estimated via any of the other methods in the unmatched sample. Likewise, we found no significant associations when the matched sample was employed, regardless of the estimation method utilized.

Second, because children experiencing what we have categorized as ‘other’ types of placements (i.e., other than family foster homes) are likely to differ from those placed in family foster homes on a range of background characteristics (Davidson-Arad et al., 2003; McDonald et al., 1996), we estimated supplemental models that excluded this group of children. Likewise, we estimated models that excluded the 156 children (6%) who experienced out-of-home placement between the time of their initial CPS investigation and the time of their baseline assessment. In both cases, results were qualitatively consistent with our primary findings.

Finally, we exploited our matched sampled to examine whether associations between out-of-home placement and child well-being may vary by a child's propensity for removal. Here we divided the matched sample into subsamples of children whose propensities for removal were above and below the median. We then re-estimated all of our models (separately) for each subsample. We found no significant associations in any of the models when estimated using either subsample. We note, however, that this may reflect a lack of statistical power through which to detect effects given the limited size (N = 308) of each subsample.

Placement characteristics

Finally, to examine whether relations of out-of-home placement and child outcomes differed by placement length, stability, and type, we estimated models (results not shown) that (1) allowed these associations to vary by the proportion of the observation period that a particular child spent out-of-home (logarithm), as well as the number of out-of-home placements a child experienced, and (2) allowed them to vary by the proportion of time a child spent in each type of placement (non-kin foster home, kin foster home, or other), as well as the number of out-of-home placements a child experienced. We found no associations of placement length or stability with any of the outcomes when the matched sample was employed.

Turning to placement type, when the matched sample was employed, we found that time spent in ‘other’ (non-foster home) placements was associated with increases in both internalizing and externalizing behavior problems when estimated by the OLS model with the full set of controls. We also found associations of time in ‘other’ placements with increases in internalizing behavior problems when estimated by both the residualized and simple change models, and with decreases in vocabulary skills when estimated by the difference-in-difference model. Finally, results from the fixed effects models revealed that time spent in both kin and non-kin foster placements was associated with increased vocabulary and math skills, and that time spent in non-kin foster homes was associated with decreased externalizing behavior problems.

Discussion

This study used NSCAW data and five analytic methods of adjusting for selection factors to estimate the impact of out-of-home placement on child well-being among 4 to 17 year old children using both matched and unmatched treatment and comparison samples. We first demonstrated that children who were removed from home differed from those who were not on a host of background factors. This confirmed that there was differential selection into placement and highlighted the importance of adjusting for selection factors, including baseline well-being scores, when attempting to estimate effects of out-of-home placement on child well-being.

We then estimated a series of models that increased in methodological rigor and ability to adjust for selection factors. We found no significant effects of out-of-home placement on cognitive skills in any of the models. With regard to behavior problems, when the unmatched sample was employed, our basic OLS results indicated that children who experienced placement exhibited significantly more internalizing and externalizing behavior problems at follow-up than children who did not. However, estimates produced by the change models did not support these findings. Results from the residualized change models suggested an association between placement and increased internalizing behavior problems, but no relationship between placement and externalizing behavior problems. Results from the simple change and fixed effects models, both of which are likely to reduce bias to a greater extent than the OLS and residualized change models, revealed associations between placement and decreases in externalizing behavior problems, thus raising considerable doubt that estimates produced by OLS and residualized change models lend themselves to causal interpretation.

Next, we used propensity score matching methods to construct treatment and comparison samples of children who were similar on all measured background characteristics and differed only in terms of whether they experienced out-of-home placement. We then re-estimated all of our models using the matched sample and found no significant associations between placement and either children's cognitive skills or their behavior problems. This did not appear to primarily reflect reduced statistical precision (increased standard errors) when the matched sample was employed; rather, many of the point estimates differed substantially when estimated using the matched and unmatched samples. This pattern is consistent with Berzin's (2008) findings which, though based on a sample of young adults, also revealed associations between out-of-home placement and adverse outcomes using unmatched data, but none using matched data.

Finally, to examine the ways in which experiences during out-of-home placement may have influenced child outcomes, we estimated supplemental models that accounted for variability in placement length, stability, and type. We found no associations of placement length or stability with children's cognitive skills or behavior problems once the matched sample was employed. Notably, however, our measure of placement stability, defined only by number of out-of-home placements, is quite crude; future research should more fully explore the potential moderating role of placement stability using more refined measures of stability (see, e.g., James et al., 2004; Rubin et al., 2007).

Our supplemental analyses did reveal several associations between placement type and child outcomes, the most consistent of which suggested links between non-foster home placements and increased behavior problems. However, as noted above, placement length, stability, and type are likely to be endogenously determined with children's cognitive skills and, in particular, their behavior problems. For example, if out-of-home placement was associated with reduced externalizing behavior problems (as suggested by our simple change and fixed effects results when estimated using the unmatched sample), then longer placements would likely be associated with larger reductions in this outcome. At the same time, if placement were effective at reducing externalizing behavior problems, and such reductions caused children to return home more quickly, then shorter placements would likely be associated with greater reductions in externalizing behavior problems. Counteracting effects of this sort may help to explain our lack of findings with regard to placement length. Similarly, children's levels of cognitive ability and behavior—both prior to and during placement—are likely to influence the types of placements to which they are assigned as well as the quality of those placements. As such, including placement characteristics in models that attempt to estimate causal relations between out-of-home placement and child outcomes may not be appropriate; we have little confidence that these estimates reflect causal relations.

On the whole, our results suggest that, on average, out-of-home placement appears to neither place additional burden on the already vulnerable children who enter State custody, nor contribute to improved well-being for these children, at least in terms of short-term changes in cognitive skills and behavior problems. Lack of findings for cognitive skills may, perhaps, reflect a growing emphasis on neighborhood-based foster care (Berrick, 2006) which may prevent children who enter placement from also experiencing changes in schooling that might engender significant improvements or decrements in cognitive skills. More generally, that our overall pattern of results suggests few impacts of placement on child well-being may not be particularly surprising given that previous research comparing children placed out-of-home to similar children who remained in-home has reported both positive (Davidson-Arad, 2005; Davidson-Arad, et al., 2003) and negative (Doyle, 2007; Lawrence, et al., 2006; Stahmer et al., 2007) effects. It is possible that our analytic methods have reduced bias that may be operating in different directions across prior studies depending on the characteristics of the treatment and comparison groups, as well as the particular child outcomes and placement measures utilized.

The current study differs from prior research in several important respects, most notably in its use of multiple methods of adjusting for selection factors that may influence placement decisions and child outcomes, but also in its use of national survey data that includes children experiencing CPS involvement in a variety of child welfare policy and practice contexts. To the extent that differences in our results and those of prior research can be explained by differences in analytic methods, the results of this study may suggest that findings of earlier work likely reflected bias due to unobserved heterogeneity across groups of children who did and did not experience placement. To the extent that differences in results reflect differences in samples utilized (i.e., local versus national), findings from prior studies may have been influenced by idiosyncratic aspects of the (local) macro-level environments in which children experienced out-of-home placement.

This study also has several limitations. First, our sample excluded infants and toddlers, as well as children who were never observed in their homes of origin. This ensured that baseline child and family assessments for the entire sample were conducted while children were in-home and also enabled us to assess changes in well-being using identical measures at baseline and follow-up. However, children who were excluded from our analysis sample because they were never observed in-home (i.e., were out-of-home at both the baseline and 18-month assessments) likely came from homes that were considerably less safe than those of the children included in our sample. As such, our findings may not be generalizable to children with the highest likelihood of being removed from home and spending a long period of time in care. Moreover, associations between out-of-home placement and child well-being may be very different for infants and young toddlers than for older children, an issue that warrants attention in future research.

Second, our outcome measures reflect only limited aspects of well-being. We focused on children's cognitive skills and behavioral problems, but did not consider the extent to which out-of-home placement affects other domains of functioning. Additionally, both the K-BIT and CBCL have been constructed such that they are appropriate for use with a broad age range of children. They may therefore lack the sensitivity or specificity to precisely detect some effects of placement on children's cognition or behavior during various stages of development. Furthermore, our behavior problems measures were drawn from the caregiver-reported version of the CBCL. As such, a large proportion of reports on children who experienced placement were provided by different reporters at baseline and follow-up (about 84%, as compared to 35% for children who did not experience placement). Notwithstanding the favorable psychometric properties and widespread use of the CBCL, utilizing reports from multiple informants at different time points may be problematic (Achenbach, McConaughy, & Howell, 1987). However, this is a common problem in studies of the effects of out-of-home placement on child behavior (Newton et al., 2000) and we have no reason to expect that it would bias our results more so than those of prior work. Nonetheless, future research would benefit from observational measures of child behavior provided by the same reporter(s) at multiple points in time. Future research that considers a wider range of outcomes with regard to major developmental tasks will also help to provide a more complete picture of how placement may influence children.

Third, our results are based on only a two and a half year observation period (on average). This may be too short a window to observe the full range of effects of placement on child outcomes, as such effects may manifest over time. Existing studies using rigorous methods to adjust for selection factors have produced mixed evidence regarding the long-term effects of placement. For example, Doyle (2007) used instrumental variables models to adjust for selection bias and found strong links between placement during childhood and a range of adverse outcomes in adulthood, whereas Berzin (2008) used propensity score matching methods and found none. Thus, additional research on the long-term effects of placement is warranted.

Fourth, there is likely to be considerable heterogeneity in associations of placement with children's cognitive skills and behavior problems that is obscured in our analyses. In particular, there may be differences in these associations by child age and gender (Horowitz, Balestracci, and Simms, 2001), as well as race and ethnicity (Jonson-Reid and Barth, 2000; Keller et al., 2001) and the types and severity of maltreatment children have experienced (Myers et al., 2002). In supplemental analyses, we attempted to examine whether any associations of placement with child cognitive skills and behavior problems varied by each of these factors using both the full and matched samples. To do so, we estimated a series of separate models in which child age at the time of removal, gender, race and ethnicity, maltreatment type, and (caseworker reported) severity of maltreatment risk were individually interacted with out-of-home placement. Although the models revealed several significant interaction terms, we found no consistent or theoretically relevant pattern by any of these factors. However, our confidence in these estimates was limited both because the cell sizes for these analyses were quite small and because we conducted a large number of statistical tests, potentially increasing the likelihood that we would identify significant effects in error. Furthermore, given the inconsistent pattern of these results and the lack of prior theory or evidence to support them, we could not rule out that they were due to chance. Future work would benefit from further exploring potential heterogeneity in associations of out-of-home placement and child well-being by demographic and maltreatment-related factors.

Finally, this study focused on estimating the “full” associations of out-of-home placement with children's cognitive skills and behavior problems, but did not explore the mechanisms through which such associations may operate. For example, we did not investigate whether placement influenced the cognitive stimulation, warmth, or support children received. Yet, these factors are important pathways through which any associations of placement with child outcomes are likely to occur. Future research on potential mechanisms, such as the quality of various substitute care arrangements and the degree of continuity in other aspects of children's lives, including peers, schools, and neighborhoods, is vital for understanding the circumstances under which placement may mitigate or heighten children's well-being.

Conclusion

The results of this study suggest that, once selection factors are taken into account, out-of-home placement has little influence on child outcomes, at least in the short-term. That placement does not appear to help or harm children in the domains of cognitive functioning or behavior problems may reflect the continued influence of other ongoing risks or of early maltreatment experiences and other ecological adversities on child well-being. It may also reflect that the child welfare system has historically focused on promoting safety and permanency for children, rather than child well-being. Our findings both that estimates produced by OLS models frequently differ from those produced by change models, and that estimates produced using unmatched observations frequently differed from those produced using matched observations, underscore the importance of rigorously adjusting for selection factors when estimating associations of out-of-home placement with child well-being.

Though based on a relatively limited set of well-being indicators, our findings also have implications for child welfare policy and practice. Although the primary focus of the child welfare system continues to be promoting safety and permanency, CPS is now also explicitly charged with improving child well-being (Wulczyn et al., 2005). To the extent that out-of-home placement does little to positively influence children's development, CPS agencies should ensure that decisions to remove children from home continue to be driven solely by concerns for child safety. Out-of-home placement is an integral and necessary means through which CPS aims to protect children from harm in their own homes. Our results should not be interpreted to suggest that CPS should move away from this form of intervention, nor that placement decisions should consider aspects of child well-being beyond assuring that children are protected from abuse and neglect. Furthermore, it is important to acknowledge that, although out-of-home placement may, on average, have little influence on children's well-being, there is likely to be considerable variation in how individual children respond to placement, which should be reflected in CPS guidelines and practices with regard to children who have been removed from home due to safety concerns. As such, if the child welfare system intends to actively engage in promoting child well-being, CPS agencies should carefully assess and balance the needs of individual children when designating supportive services for children in out-of-home care. Developmental research can play an important role in this effort by specifying mechanisms of risk and resilience for children in out-of-home placement.

Acknowledgments

This research was supported by Administration for Children and Families Grant #90PH0010. The authors are grateful to Xianqun Luan for assistance in identifying and coding children's out-of-home placements in the NSCAW data. We thank Marcy Carlson, Heather Hill, Katherine Magnuson, Matt Stagner, Jane Waldfogel, and seminar participants at the Taubman Center for Public Policy and American Institutions at Brown University and the Merrill Palmer Skillman Institute at Wayne State University, as well as three anonymous reviewers, for excellent comments and advice.

Note

This study used data from the National Survey on Child and Adolescent Well-Being, which was developed under contract with the Administration on Children, Youth, and Families, U.S. Department of Health and Human Services (ACYF/DHHS). The data were provided by the National Data Archive on Child Abuse and Neglect. The information and opinions expressed herein reflect solely the position of the authors. Nothing herein should be construed to indicate the support or endorsement of its content by ACYF/DHHS.

Contributor Information

Lawrence M. Berger, University of Wisconsin—Madison.

Sarah K. Bruch, University of Wisconsin—Madison.

Elizabeth I. Johnson, University of Tennessee—Knoxville.

Sigrid James, Loma Linda University.

David Rubin, University of Pennsylvania.

References

  • Achenbach T. Manual for the Child Behavior Checklist 4 – 18 and 1991 profile. Burlington: University of Vermont, Department of Psychiatry; 1991.
  • Achenbach TM, McConaughy SH, Howell CT. Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin. 1987;101:213–232. [PubMed]
  • Allison PD. Change scores as dependent variables in regression analyses. Sociological Methodology. 1990;20:93–114.
  • Allison PD. Missing data. Thousand Oaks, CA: Sage Publications; 2002.
  • Barnard M, McKeganey N. The impact of parental problem drug use on children: What is the problem and what can be done to help? Addiction. 2004;99:552–559. [PubMed]
  • Barth RP, Gibbons C, Guo S. Substance abuse treatment and the recurrence of maltreatment among caregivers with children living at home: A propensity score analysis. Journal of Substance Abuse Treatment. 2006;30:93–104. [PubMed]
  • Barth RP, Guo S, Green RL, McCrae JS. Kinship care and nonkinship foster care: Informing the new debate. In: Haskins R, Wulczyn F, Webb MB, editors. Child protection: Using research to improve policy and pratice. Washington, DC: The Brookings Institution; 2007. pp. 187–206.
  • Barth RP, Wildfire J, Green RL. Placement into foster care and the interplay of urbanicity, child behavior problems, and poverty. American Journal of Orthopsychiatry. 2006;76:358–366. [PubMed]
  • Benedict MI, Zuravin S, Stallings RY. Adult functioning of children who lived in kin vs. non-kinship foster care. Children and Youth Services Review. 1996;22:37–54.
  • Berrick JD. Neighborhood-based foster care: A critical examination of location-based placement criteria. Social Service Review. 2006;80:569–583.
  • Berrick JD, Barth RP, Needell B. A comparison of kinship foster homes and family foster homes: Implications for kinship foster care as family preservation. Children and Youth Services Review. 1994;16:33–63.
  • Berzin SC. Difficulties in the transition to adulthood: Using propensity scoring to understand what makes foster youth vulnerable. Social Service Review. 2008;82:171–195.
  • Bilaver LA, Jaudes PK, Koepke D, Goerge RM. The health of children in foster care. Social Service Review. 1999;73:401–417.
  • Blome WW. What happens to foster kids: Educational experiences of a random sample of foster care youth and a matched group of non-foster care youth. Child and Adolescent Social Work Journal. 1997;14:41–53.
  • Brooks-Gunn J, Duncan GJ. The effects of poverty on children. Future Children. 1997;7:55–71. [PubMed]
  • Buehler C, Orme JG, Post J, Patterson DA. The long-term correlates of family foster care. Children and Youth Services Review. 2000;22:595–625.
  • Chamberlain P, Price JM, Reid JB, Landsverk J, Fisher PA, Stoolmiller M. Who disrupts from placement in foster and kinship care? Child Abuse and Neglect. 2006;30:409–424. [PubMed]
  • Chapman MV, Wall A, Barth RP, National Survey of Child and Adolescent Well-Being Research Group Children's voices: The perceptions of children in foster care. American Journal of Orthopsychiatry. 2004;74:293–304. [PubMed]
  • Child Welfare League of America. Alcohol, other drugs, and child welfare. Washington, DC: Author; 2001.
  • Cicchetti D, Toth SL. Child maltreatment. Annual Review of Clinical Psychology. 2005;1:409–438. [PubMed]
  • Clausen JM, Landsverk J, Ganger W, Chadwick D, Litrownik A. Mental health problems of children in foster care. Journal of Child and Family Studies. 1998;7:283–296.
  • Courtney ME. Research needed to improve the prospects for children in out-of-home placement. Children and Youth Services Review. 2000;22:743–761.
  • Currie J. When do we know what we think we know? Determining causality. In: Bianchi SM, Casper LM, King RB, editors. Work, family, health, and well-being. Mahwah, NJ: Erlbaum; 2005. pp. 279–296.
  • Davidson-Arad B. Fifteen-month follow-up of children at risk: Comparison of the quality of life of children removed from home and children remaining at home. Children and Youth Service Review. 2005;27:1–20. [PubMed]
  • Davidson-Arad B, Englechin-Segal D, Wozner Y. Short-term follow-up of children at risk: Comparison of the quality of life of children removed from home and children remaining at home. Child Abuse and Neglect. 2003;27:733–750. [PubMed]
  • Doyle JJ. Child protection and child outcomes: Measuring the effects of foster care. American Economic Review. 2007;97:1583–1610.
  • Duncan GJ, Magnuson K, Ludwig J. The endogeneity problem in developmental studies. Research in Human Development. 2004;1:59–80.
  • Ehrle J, Geen R. Kin and non-kin foster care: Findings from a national survey. Children and Youth Services Review. 2002;24:15–35.
  • Fanshel D, Shinn E. Children in Foster Care. New York: Columbia University Press; 1978.
  • Fantuzzo J, Perlman S. The unique impact of out-of-home placement and the mediating effects of child maltreatment and homelessness on early school success. Children and Youth Services Review. 2007;29:941–960.
  • Gaudin JM, Sutphen R. Foster care vs. extended family care for children of incarcerated mothers. Journal of Offender Rehabilitation. 1993;19:129–147.
  • Gennetian LA, Magnuson K, Morris PA. From statistical associations to causation: What developmentalists can learn from instrumental variables techniques coupled with experimental data. Developmental Psychology. 2008;44:381–394. [PMC free article] [PubMed]
  • Glaser D. Child abuse and neglect and the brain—a review. Journal of Child Psychology and Psychiatry. 2000;41:97–116. [PubMed]
  • Harden BJ, Clyman RB, Kriebel DK, Lyons ME. Kith and kin care: Parental attitudes and resources of foster and relative caregivers. Children and Youth Services Review. 2004;26:657–671.
  • Holten A, Rønning JA, Handegård BH, Sourander A. A comparison of mental health problems in kinship and nonkinship foster care. European Child and Adolescent Psychiatry. 2005;14:200–207. [PubMed]
  • Horowitz SM, Balestracci KMB, Simms MD. Foster care placement improves children's functioning. Archives of Pediatric and Adolescent Medicine. 2001;155:1255–1260. [PubMed]
  • James S. Why do foster care placements disrupt? An investigation of reasons for placement change in foster care. Social Service Review. 2004;78:601–627.
  • James S, Landsverk J, Slymen DJ. Placement movement in out-of-home care: patterns and predictors. Children and Youth Services Review. 2004;26:185–206.
  • Jonson-Reid M, Barth RP. From maltreatment report to juvenile incarceration: The role of child welfare services. Child Abuse & Neglect. 2000;24:505–520. [PubMed]
  • Kaufman AS, Kaufman NL. Kaufman brief intelligence test (K-BIT) Circle Pines, MN: American Guidance Service, Inc.; 1990.
  • Keller TE, Wetherbee K, Le Prohn NS, Payne V, Lamont ER. Competencies and problem behaviors of children in family foster care: Variations by kinship placement status and race. Children and Youth Services Review. 2001;12:915–940.
  • Landsverk J, Davis I, Ganger W, Newton R, Johnson I. Impact of child psychosocial functioning on reunification from out-of-home placement. Children and Youth Services Review. 1996;18:447–462.
  • Lawrence CR, Carlson EA, Egeland B. The impact of foster care on development. Development and Psychopathology. 2006;18:57–76. [PubMed]
  • Leslie LK, Gordon JN, Ganger W, Gist K. Developmental delay in young children in child welfare by initial placement type. Infant Mental Health Journal. 2002;23:496–516.
  • Litrownik AJ, Lau A, English DJ, Briggs E, Newton RR, Romney ST, Dubowitz H. Measuring the severity of maltreatment. Child Abuse and Neglect. 2005;29:553–573. [PubMed]
  • Little RJA, Rubin DB. Statistical analysis with missing data. New York: Wiley; 1987.
  • Maas HS, Engler RE. Children in need of parents. New York: Columbia University Press; 1959.
  • McDonald TP, Allen RI, Westerfelt A, Piliavin I. Assessing the long-term effects of foster care: A research synthesis. Washington, DC: Child Welfare League of America; 1996.
  • Murray MP. Econometrics: A modern introduction. Boston: Pearson Addison Wesley; 2006.
  • Myers JEB, Berliner L, Briere J, Hendrix CT, Jenny C, Reid TA. The APSAC Handbook on Child Maltreatment. 2nd. Thousand Oaks, CA: Sage Publications; 2002.
  • National Institute of Child Health and Human Development Early Child Care Research Network. Duncan GJ. Modeling the impacts of child care quality on children's preschool cognitive development. Child Development. 2003;74:1454–1475. [PubMed]
  • Newton RR, Litrownik AJ, Landsverk JA. Children and youth in foster care: Disentangling the relationship between problem behaviors and number of placements. Child Abuse and Neglect. 2000;24:1363–1374. [PubMed]
  • Pears K, Fisher PA. Emotion understanding and theory of mind among maltreated children in foster care: Evidence of deficits. Development and Psychopathology. 2005a;17:47–65. [PubMed]
  • Pears K, Fisher PA. Developmental, cognitive, and neuropsychological functioning in preschool-aged foster children: Associations with prior maltreatment and placement history. Developmental and Behavioral Pediatrics. 2005b;26(2):112–122. [PubMed]
  • Reams R. Children birth to three entering the state's custody. Infant Mental Health Journal. 1999;20:166–174.
  • Rogosa DR, Brandt D, Zimowski M. A growth curve approach to the measurement of change. Psychological Bulletin. 1982;92:726–748.
  • Rogosa DR, Willett JB. Demonstrating the reliability of the difference score in the measurement of change. Journal of Educational Measurement. 1983;20:335–343.
  • Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70:41–55.
  • Rubin DB. Multiple imputation for nonresponse in surveys. New York: Wiley; 1987.
  • Rubin DM, Alessandrini EA, Feudtner C, Mandell DS, Localio AR, Hadley T. Placement stability and mental health costs for children in foster care. Pediatrics. 2004;113:1336–1341. [PubMed]
  • Rubin DM, O'Reilly ALR, Luan X, Localio AR. The impact of placement stability on behavioral well-being for children in foster care. Pediatrics. 2007;119:336–344. [PMC free article] [PubMed]
  • Rutter M. Children in substitute care: Some conceptual considerations and research implications. Children and Youth Services Review. 2000;22:685–703.
  • Schafer JL. Analysis of incomplete multivariate data. London: Chapman & Hall; 1997.
  • Shin SH. Developmental outcomes of vulnerable youth in the child welfare system. Journal of Human Behavior in the Social Environment. 2004;9:39–56.
  • Shlonsky A. Initial construction of an actuarial risk assessment measure using the National Survey of Child and Adolescent Well-Being. In: Haskins R, Wulczyn F, Web MB, editors. Child protection: Using research to improve policy and practice. Washington, DC: Brookings Institution Press; 2007. pp. 62–80.
  • Shore N, Sim KE, Le Prohn NS, Keller TE. Foster parent and teacher assessments of youth in kinship and non-kinship foster care placements: Are behaviors perceived differently across settings? Children and Youth Services Review. 2002;24:109–134.
  • Stahmer AC, Hurlburt M, Horowitz SM, Landsverk J, Zhang J, Leslie LK. Associations between service level and child development among young children in child welfare. Manuscript, Child and Adolescent Services Research Center, University of California; San Diego: 2007.
  • Stone S. Child maltreatment, out-of-home placement and academic vulnerability: A fifteen-year review of evidence and future directions. Children and Youth Services Review. 2007;29:139–161.
  • U.S. Department of Health and Human Services, Administration on Children, Youth and Families. Child Maltreatment 2006. Washington, DC: U.S. Government Printing Office; 2008a.
  • U.S. Department of Health and Human Services, Administration on Children, Youth and Families. The AFCARS Report: Preliminary FY 2006 estimates as of January 2008. Washington, DC: U.S. Government Printing Office; 2008b.
  • U.S. Department of Health and Human Services. National Survey of Child and Adolescent Well-Being (NSCAW): CPS sample component wave 1 data analysis report. Washington, DC: USDHHS; 2005. Available at: http://www2.acf.dhhs.gov/programs/opre/abuse_neglect/nscaw/reports/cps_sample/cps_report_revised_090105.pdf.
  • Viner RM, Taylor B. Adult health and social outcomes of children who have been in public care: Population-based study. Pediatrics. 2005;115:894–899. [PubMed]
  • Waldfogel J. Child welfare research: How adequate are the data? Children and Youth Services Review. 2000;22:705–741.
  • Wulczyn F, Barth RP, Yuan YT, Harden BJ, Landsverk J. Beyond Common Sense: Child Welfare, Child Well-Being, and the Evidence for Policy Reform. New Brunswick, NJ: Aldine Transaction; 2005.
  • Zimmer MH, Panko LM. Developmental status and service use among children in the child welfare system: A national survey. Archives of Pediatric and Adolescent Medicine. 2006;160:183–188. [PubMed]