PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Psychometrika. Author manuscript; available in PMC 2010 July 16.
Published in final edited form as:
Psychometrika. 2010 June; 75(2): 243–248.
doi:  10.1007/s11336-009-9135-y
PMCID: PMC2905175
NIHMSID: NIHMS127970

Ensuring Positiveness of the Scaled Difference Chi-square Test Statistic *

Abstract

A scaled difference test statistic T~d that can be computed from standard software of structural equation models (SEM) by hand calculations was proposed in Satorra and Bentler (2001). The statistic T~d is asymptotically equivalent to the scaled difference test statistic Td introduced in Satorra (2000), which requires more involved computations beyond standard output of SEM software. The test statistic T~d has been widely used in practice, but in some applications it is negative due to negativity of its associated scaling correction. Using the implicit function theorem, this note develops an improved scaling correction leading to a new scaled difference statistic Td that avoids negative chi-square values.

Keywords: Moment-structures, goodness-of-fit test, chi-square difference test statistic, chi-square distribution, non-normality

Introduction

Moment structure analysis is widely used in behavioural, social and economic studies to analyze structural relations between variables, some of which may be latent see, e.g., Bollen and Curran (2006), Grace (2006), Yuan and Bentler (2007), and references therein. In such analyses it frequently happens that two nested models M0 and M1 are compared using estimation methods that are non-optimal (asymptotically) given the distribution of the data; e.g. ML estimation is used when the data are not multivariate normal. In those circumstances, the usual chi-square difference test Td = T0 - T1, based on the separate models’ goodness of fit test statistics, is not χ2 distributed. A correction to Td by a scaling factor was proposed by Satorra (2000) and Satorra and Bentler (2001) to improve the chi-square approximation. The latter is the focus of this paper.

Satorra and Bentler’s (2001) correction provided a simple procedure to obtain an approximate scaled chi-square statistic using hand calculations on regular output of SEM analysis; it has the drawback, however, that a positive value for the scaling correction is not assured. The present paper develops a simple procedure by which a researcher can compute the exact SB difference test statistic based only on output from standard SEM programs.

Setup and Notation

Throughout we adhere to the notation and results of Satorra and Bentler (2001). Let σ and s be p-dimensional vectors of population and sample moments respectively, where s tends in probability to σ as sample size n → +∞. Let n(sσ) be asymptotically normally distributed with a finite asymptotic variance matrix Γ (p × p). Consider the model M0 : σ = σ(θ) for the moment vector σ, where σ(.) is a twice-continuously differentiable vector-valued function of θ, a q-dimensional parameter vector. Consider a discrepancy function F = F (s, σ) in the sense of Browne (1984), and the estimator [theta w/ hat] based on F or on an (asymptotically equivalent) weighted least squares (WLS) analysis with weight matrix V=122F(s,σ)σσ evaluated at σ = s.

Let M0 : σ = σ[open star](δ), a(δ) = 0, and M1 : σ = σ[open star](δ) be two nested models for σ. Here δ is a (q+m)-dimensional vector of parameters, and σ[open star](.) and a(.) (an m-valued function) are twice-continuously differentiable vector-valued functions of δ [set membership] ϴ1, a compact subset of Rq+m. Our interest is in the test of a null hypothesis H0 : a(δ) = 0 against the alternative H1 : a(δ) ≠ 0.

For the developments that follow, we require the Jacobian matrices Π(p× (q + m)) := [partial differential]σ[open star](δ)/[partial differential]δ’ and A(m × (q + m)) := [partial differential]a(δ)/[partial differential]δ’, which we assume to be regular at the true value of δ, say δ0. We also assume that A is of full row rank. By using the implicit function theorem, associated to M0 (more specifically, to the restrictions a(δ) = 0), there exists (locally in a neighborhood of δ0) a one-to-one function δ = δ(θ) defined in an open and compact subset S of Rq, and a θ0 in the interior of S such that δ(θ0) = δ0 and σ(δ(θ)) satisfies the model M0. Let H = [partial differential]δ(θ)/[partial differential]δ’ ((q + m) × q) be the corresponding Jacobian matrix evaluated at θ0. Hence, by the chain rule of differentiation, Δ= [partial differential]σ/[partial differential]θ’ = ([partial differential]σ(δ)/[partial differential]δ’)([partial differential] δ(θ)/[partial differential]θ’)= ΠH. Since a(δ(θ)) = 0, it holds that with A evaluated at δ0, AH = 0 with r(A)+r(H) = q+m, and r(.) denoting the rank of a matrix. Thus, H’ is an orthogonal complement of A. Let P ((q + m) × (q + m)) := Π’VΠ. Associated to M1, the less restricted model σ = σ[open star](δ), the goodness-of-fit test statistic is T1=nF(s,σ~), where σ~ is the fitted moment vector in model M1 with associated degrees of freedom r1 = r0 - m and scaling factor c1 given by

c11r1tr{U1Γ}=1r1tr{VΓ}1r1tr{P1ΠVΓVΠ},
(1)

where

U1VVΠP1ΠV.
(2)

We refer to Satorra and Bentler (2001) for further details.

Scaling Correction for the Difference Test

When both models M0 and M1 are fitted, for example by ML, then we can test the restriction a(δ) = 0, assuming M1 holds, using the chi-square difference test statistic Td := T0 - T1. Under the null hypothesis, we would like Td to have a χ2 distribution with degrees of freedom m = r0 - r1. This is the restricted test of M0 within M1. For general distributions of the data, the asymptotic chi-square approximation may not hold. To improve on the chi-square approximation, Satorra (2000) gave explicit formulae that extends the scaling corrections proposed by Satorra and Bentler (1994) to the case of difference, Wald, and score type of test statistics. General expressions for those corrections were also put forward in Satorra (1989, p.146). Specifically, for the test statistic Td we are considering, Satorra (2000, p. 241) proposed the following scaled test statistic:

TdTdc^d,wherecd1mtr{UdΓ}
(3)

with

Ud=VΠP1A(AP1A)1AP1ΠV.
(4)

Here, ĉd denotes cd after substituting consistent estimates of V and Γ, and evaluating the Jacobians A and at the estimate [delta with circumflex]0 when fitting M0 (or M1). Since tr {UdΓ} can be expressed as the trace of the product of two positive definite matrices, tr {UdΓ} > 0, and thus cd > 0; the same for ĉd > 0. Consequently, Td is ensured to be a non-negative number.

A practical problem with the statistic Td is that it requires computations that are outside the standard output of current structural equation modeling programs. Furthermore, difference tests are usually hand computed from different modeling runs. Satorra and Bentler (2001) proposed a procedure to combine the estimates of the scaling corrections c0 and c1 associated to the chi-square goodness-of-fit test for the two fitted models M0 and M1 in order to compute a consistent estimate of the scaling correction cd for the difference test statistic. A modified (easy to compute) scaled test statistic T~d with the same asymptotic distribution as Td was proposed. Both statistics were shown to be asymptotically equivalent under a sequence of local alternatives (so they have the same asymptotic local power). Their procedure to compute T~d is as follows (see Satorra and Bentler, 2001, p. 511).

  1. Obtain the unscaled and scaled goodness-of-fit tests when fitting M0 and M1 respectively; that is, T0 and T0 when fitting M0, and T1 and T1 when fitting M1;
  2. Compute the scaling corrections ĉ0 = T0/T0, ĉ1 = T1/T1, and the unscaled chi-square difference Td = T0 - T1 and its degrees of freedom m = r0 - r1;
  3. Compute the scaled difference test statistic as
    T~dTdc~dwithc~d=(r0c^0r1c^1)m.

Here r0 and r1 are the respective degrees of freedom of the models M0 and M1.

The basis for computing the scaling correction for the difference test statistic is the following alternative expression for Ud of (4) (see Satorra and Bentler, 2001, p. 510)

Ud=U0U1,
(5)

where U1 is given in (2) and U0 := V - VΠH(H’Π’VΠH)-1H’Π’V. Since (5) implies

mcd=tr{UdΓ}=tr{(U0U1)Γ}=r0c0r1c1,
(6)

it follows that cd = (r0c0 - r1c1)/m. This is the theoretical basis for Satorra and Bentler’s (2001) proposal as given in steps 1-3 above.

Problem with the Current Scaled Difference Test

For an arbitrary matrix V > 0, (5) and (6) are exact equalities when the matrices Ud, U0 and U1 are evaluated at a common value δ (as, e.g., the fitted value under model M0 or under M1); however, they are just asymptotic equalities when Ud, U0 and U1 are evaluated at different estimates that converge to the same true value under the null hypothesis. Under Satorra and Bentler’s (2001) proposal, c~d evaluates U0 and U1 at the estimates [delta with circumflex]0 and [delta with circumflex]1 respectively. Since [delta with circumflex]1 will in general not satisfy the null model M0 (i.e., it will not be of the form δ = δ(θ) for the function implied by the implicit function theorem), when it deviates highly from M0, the estimated difference c~d=(r0c^0r1c^1)m may turn out to be negative. This may happen in small samples, or when M0 is highly incorrect; a result can be an improper value for T~d. Satorra and Bentler (2001, p. 511) warned on the possibility that an improper value of T~d could arise.

In order to be sure to avoid a negative value for c~d and hence T~d, currently one would need to resort to computing Td using the (3). Unfortunately this is impractical or impossible for most applied researchers who only have access to standard SEM software.

Fortunately, as we show next, the exact value of Td can also be obtained from the standard output of SEM software, using a new hand computation.

A New Scaled Test Statistic Td

Denote by M10 the fit of model M1 to a model setup with starting values taken as the final estimates obtained from model M0, and with number of iterations set to 0. Consider ĉ1(10) := T1(10)/T1(10), where T1(10) and T1(10) are the standard unscaled and scaled test statistic of this additional run. Note that the estimate ĉ1(10) uses model M1 but the matrices and A are now evaluated at [delta with circumflex]0 := δ([theta w/ hat]), where [theta w/ hat] is the estimate under M0. Since now all the matrices involved in (5) are evaluated at δ^0, the equality (6) holds exactly, and not only asymptotically, as when U0 is evaluated at δ^0 and U1 at δ^1. The scaling correction that is now computed is

c^d(10)(r0c^0r1c^1(10))m,
(7)

which, in view of (6), is the scaling correction of (3) when Ud is evaluated at the estimate δ^0 when fitting M0, and thus it is a positive number. The new scaled difference statistic is thus defined as

Td(10)(T0T1)c^d(10),
(8)

Clearly, Td(10)=Td that is, Td(10) coincides numerically with the scaled statistic (3) proposed in Satorra (2000).

An Illustration

We use this data just for illustrative purposes, and because it provides an example where the standard scaling correction fails to be positive. We use a latent variable model discussed for this data by Bentler, Satorra and Yuan (2009). The Bonett-Woodward-Randall (2002) test shows that these data have significant excess kurtosis indicative of non-normality at a one-tail .05 level, so test statistics derived from ML estimation may not be appropriate and we do the scaling corrections.

The model considered is a structured means model, with the mean cigarette sales indirectly affecting the mean rates of the various cancers. The specified model is

Vj=λjF+Ej,j=2,,5,F=βV1+D1,V1=μ+E1,

where the Vj’s denote observed variables; F, D1, and Ej are the common, residual common, and unique factors respectively; λj denotes a factor loading parameter, β is the effect of cigarette smoking on the cancer factor, and μ is the mean parameter for rates of smoking. The units of measurement for the factor were tied to V2, with λ2 = 1. The following values for the ML and Satorra-Bentler (1994) (SB) scaled chi-square statistics are obtained

T1=107.398,T1=65.3524,r1=9,c^1=1.6434,

along with the degrees of freedom r1 and the scaling correction ĉ1. The model does not fit, though for the sake of the illustration we are aiming for, this is not of concern to us.

Restricted Model, M0

The same model is now fitted with the added restriction that the error variances of the kidney and leukemia cancers, E4 and E5, are equal. This model gives the following statistics

T0=139.495,T0=97.4034,r0=10,c^0=1.4322.

Difference Test

Our main interest lies in testing the difference between M0 and M1, which we do with the chi-square difference test. The ML difference statistic is

Td=139.495107.398=32.097,

which, with 1 degree of freedom (m), rejects the null hypothesis that the error variances for E4 and E5 are equal. Since the data is not normal, we compute the SB (2001) scaled difference statistic. This requires computing the scaling factor c~d=(r0c^0r1c^1)m given by

[10(1.4322)9(1.6434)]1=14.32214.7906=.4686.

The scaling factor c~d is negative, so the SB difference test cannot be carried out; or, if carried out, it results in an improper negative chi-square value.

New Scaled Difference Test

As described above, to compute the scaled statistic Td we implement (7) and (8). The output that is missing in the prior runs is the value of the SB statistic obtained at the final parameter estimates for model M0 when model M1 is evaluated. This can be obtained by creating a model setup M10 that contains the parameterization of M1 with start values taken from the output of model M0. Model M10 is run with zero iterations, so that the parameter values do not change before output including test statistics is produced.1 The new result gives

T(10)=139.495,T(10)=94.9551,r1=9,c^(10)=1.4691,

where as expected, T10 = T0 as reported above (i.e., the ML statistics are identical), and the value ĉ10 is hand-computed. As a result, we can compute

c^d(10)=(r0c^0r1c^1(10))m=[(10)(1.4322)(9)(1.4691)]=1.10,

which, in contrast to the SB (2001) computations, is positive. Finally, we can compute the proposed new SB corrected chi-square statistic as

Td=Td(10)=(T0T1)c^d(10)=(139.495107.398)1.10=29.179,

which can be referred to a χ12 variate for evaluation.

Discussion

The implicit function theorem was used to provide a theoretical basis for the development of a practical version of the computationally more difficult scaled difference statistic proposed by Satorra (2000).2 The proposed method is only marginally more difficult to compute than that of Satorra and Bentler (2001) and solves the problem of an uninterpretable negative χ2 difference test that applied researchers have complained about for some time.

Like the method it is replacing, the proposed procedure applies to a general modeling setting. The vector of parameters σ to be modeled may contain various types of moments: means, product-moments, frequencies (proportions), and so forth. Thus, this scaled difference test applies to methods such as factor analysis, simultaneous equations for continuous variables, log-linear multinomial parametric models, etc.. It can easily be seen that the procedure applies also in the case where the matrix Γ is singular, when the data is composed of various samples, as in multi-sample analysis, and to other estimation methods. It applies also to the case where the estimate of Γ reflects the fact that we have intraclass correlation among observations, as in complex samples. Hence this new statistic should be useful in a variety of applied modeling contexts. Simulation work will be needed to understand its virtues and limitations, relative to other alternatives, in such contexts.

Footnotes

*Research supported by grants SEJ2006-13537 and PR2007-0221 from the Spanish Ministry of Science and Technology and by USPHS grants DA00017 and DA01070. This paper is in press in Psychometrika and presently available on line at www.springerlink.com.

1For this particular example, the Appendix of Satorra and Bentler (2008) illustrates this procedure with EQS (Bentler, 2008). In the same reference, there is a second illustration with a larger degrees of freedom.

2Satorra (2000) provides also Monte Carlo evidence—on a specific model context and various sample sizes - of the superiority of the scaling correction over other alternatives such as the adjusted (mean and variance corrected) statistic.

Contributor Information

Albert Satorra, Universitat Pompeu Fabra, Barcelona.

Peter M. Bentler, University of California, Los Angeles.

References

  • Bentler PM. EQS 6 structural equations program manual. Multivariate Software; Encino, CA: 2008.
  • Bentler PM, Satorra A, Yuan K-H. Smoking and cancers: Case-robust analysis of a classic data set. Structural Equation Models. 2009;16:382–390. [PMC free article] [PubMed]
  • Bollen KA, Curran PJ. Latent curve models : A structural equation perspective. Wiley; New York: 2006.
  • Bonett DG, Woodward JA, Randall RL. Estimating p-values for Mardia’s coefficients of multivariate skewness and kurtosis. Computational Statistics. 2002;17:117–122.
  • Browne MW. Asymptotically distribution-free methods for the analysis of covariance structures. British Journal of Mathematical and Statistical Psychology. 1984;37:62–83. [PubMed]
  • Grace JB. Structural equation modeling and natural systems. Cambridge; New York: 2006.
  • Satorra A. Alternative test criteria in covariance structure analysis: A unified approach. Psychometrika. 1989;54:131–151.
  • Satorra A. Scaled and adjusted restricted tests in multi-sample analysis of moment structures. In: Heijmans DDH, Pollock DSG, Satorra A, editors. Innovations in multivariate statistical analysis: A Festschrift for Heinz Neudecker. Kluwer Academic; Dordrecht: 2000. pp. 233–247.
  • Satorra A, Bentler PM. Corrections to test statistics and standard errors in covariance structure analysis. In: von Eye A, Clogg CC, editors. Latent Variables Analysis: Applications for developmental research. Sage; Thousand Oaks, CA: 1994. pp. 399–419.
  • Satorra A, Bentler PM. A scaled difference chi-square test statistic for moment structure analysis. Psychometrika. 2001;66:507–514.
  • Satorra A, Bentler PM. Ensuring positiveness of the scaled dfference chi-square test statistic. Department of Statistics, UCLA Department of Statistics Preprint; 2008. http://repositories.cdlib.org/uclastat/papers/2008010905. [PMC free article] [PubMed]
  • Yuan K-H, Bentler PM. Structural equation modeling. In: Rao CR, Sinharay S, editors. Handbook of statistics 26: Psychometrics. North-Holland; Amsterdam: 2007. pp. 297–358.