Dynamic Regime Marginal Structural Mean Models for Estimation of Optimal Dynamic Treatment Regimes, Part II: Proofs of Results

doi:10.2202/1557-4679.1242

Int J Biostat. 2010 January 6; 6(2): Article 9.

Published online 2010 March 3. doi: 10.2202/1557-4679.1242

PMCID: PMC2854089

Dynamic Regime Marginal Structural Mean Models for Estimation of Optimal Dynamic Treatment Regimes, Part II: Proofs of Results^*

Liliana Orellana,^* Andrea Rotnitzky,^† and James M. Robins^‡

^*Instituto de Cálculo, Universidad de Buenos Aires, Email: orellana/at/dm.uba.ar

^†Universidad Torcuato Di Tella and Harvard School of Public Health, Email: arotnitzky/at/utdt.edu

^‡Harvard School of Public Health, Email: robins/at/hsph.harvard.edu

This article has been cited by other articles in PMC.

Abstract

In this companion article to “Dynamic Regime Marginal Structural Mean Models for Estimation of Optimal Dynamic Treatment Regimes, Part I: Main Content” [Orellana, Rotnitzky and Robins (2010), IJB, Vol. 6, Iss. 2, Art. 7] we present (i) proofs of the claims in that paper, (ii) a proposal for the computation of a confidence set for the optimal index when this lies in a finite set, and (iii) an example to aid the interpretation of the positivity assumption.

Keywords: dynamic treatment regime, double-robust, inverse probability weighted, marginal structural model, optimal treatment regime, causality

1. Introduction

In this companion article to “Dynamic regime marginal structural mean models for estimation of optimal dynamic treatment regimes. Part I: Main Content” (Orellana, Rotnitzky and Robins, 2010) we present (i) proofs of the claims in that paper, (ii) a proposal for the computation of a confidence set for the optimal index when this lies in a finite set, and (iii) an example to aid the interpretation of the positivity assumption.

The notation, definitions and acronyms are the same as in the companion paper. Througout, we refer to the companion article as ORR-I.

2. Proof of Claims in ORR-I

2.1. Proof of Lemma 1

First note that the consistency assumption C implies that the event

is the same as the event

So, with the definitions

we obtain

Next, note that the fact that equation M5

unless

implies that

Then, it follows from the second to last displayed equality that

So, part 1 of the Lemma is proved if we show that

(1)

Define for any k = 0, ..., K,

To prove equality (1) first note that,

where the second to last equality follows because given equation M15

and

is a fixed, i.e. non-random function of equation M18

and consequently by the sequential randomization assumption, equation M19

is conditionally independent of A_K given equation M20

and

. The last equality follows by the definition of λ_K (·|·, ·).

We thus arrive at

This proves the result for the case k = K. If k < K – 1, we analyze the conditional expectation of the last equality in a similar fashion. Specifically, following the same steps as in the long sequence of equalities in the second to last display we arrive at

the last equality follows once again from the sequential randomization assumption. This is so because given equation M24

and

and

are fixed, i.e. deterministic, functions of equation M28

and the SR assumption ensures then that equation M29

and

are conditionally independent of A_K_–1 given equation M31

and

Equality (1) is thus shown by continuing in this fashion recursively for K – 2, K – 3, ..., K – l until l such that K – l = k – 1.

To show Part 2 of the Lemma, note that specializing part 1 to the case k = 0, we obtain

Thus, taking expectations on both sides of the equality in the last display we obtain

This shows part 2 because B is an arbitrary Borel set.

2.2. Proof of the Assertions in Section 3.2, ORR-I

2.2.1. Proof of Item (a)

Lemma 1, part 2 implies that the densities equation M35

factors as

In particular, the event equation M37

has probability 1. Consequently,

Therefore,

(2)

2.2.2. Proof of Item (b)

Lemma 1, part 1 implies that

The left hand side of this equality is equal to

and this coincides with the right hand side of (2) which, as we have just argued, is equal to [var phi]

_k+₁ (ō_k).

2.3. Proof of Lemma 2 in ORR-I

Let X be the identity random element on equation M42

and let E_{P^marg × P_X} (·) stand for the expectation operation computed under the product law P^marg × P_X for the random vector (O, A, X). Then the restriction stated in 2) is equivalent to

(3)

and the restriction stated in 3) is equivalent to

(4)

To show 2) let d (O, A, X) [equivalent]

ω_K (Ō_K, Ā_K) {u (O, A) – h_par (X, Z, β*)}.

(ORR-I, (14)) [implies]

(3).

where the last equality follows because E_{P^marg × P_X} [d (O, A, X) |X = x, Z] = E_P^marg [d (O, A, x) |Z] by independence of (O, A) with X under the law P^marg × P_X and, by assumption, E_P^marg [d (O, A, x) |Z] = 0 μ-a.e.(x) and hence E_P^marg [d (O, A, x) |Z] because P_X and μ are mutually absolute continuous.

(3)

(ORR-I, (14)). Define b^* (X; Z) = E_{P^marg × P_X} [d(O, A, X)|X, Z]. Then,

consequently, E_{P^marg × P_X} [d (O, A, X) |X, Z] = 0 with P^marg × P_X prob. 1 which is equivalent to (ORR-I, (14)) because P_X is mutually absolutely continuous with μ.

To show 3) redefine d (O, A, X) as ω_K (Ō_K, Ā_K) {u (O, A) − h_sem (X, Z, β*)}.

(ORR-1, (15)) [implies]

(4)

where the third equality follows because E_{P^marg × P_X} {d (O, A, X) |X = x, Z} = E_P^marg {d (O, A, x) |Z} and E_P^marg {d (O, A, x) |Z}= q (Z) μ-a.e.(x) and hence P_X-a.e.(x) by absolute continuity.

(4)

(ORR-I, (15)). Define b^* (X; Z) = E_{P × P_X} [ d(O, A, X)|X, Z]. Then,

Consequently, b^* (X, Z) = E_{P^marg × P_X} [b^* (X, Z) |Z] [equivalent]

q (Z) P_X – a.e. (X) and hence μ_X –a.e. (X) by absolute continuity. The result follows because b^* (x, Z) = E_{P^marg × P_X} [d (O, A, X) |X = x, Z] = E_P^marg [d (O, A, X) |Z].

2.4. Derivation of Some Formulas in Section 5.3, ORR-I

2.4.1. Derivation of Formula (26) in ORR-I

Any element

of the set Λ is the sum of K + 1 uncorrelated terms because for any l, j such that 0 ≤ l < l + j ≤ K + 1,

Thus, Λ is equal to Λ₀ [plus sign in circle]

Λ₁

. . .

Λ_K where

and

stands for the direct sum operator. Then,

and it can be easily checked that Π [Q|Λ_k] = E (Q|Ō_k, Ā_k) – E [Q|Ō_k, Ā_k–₁].

2.4.2. Derivation of Formula (27) in ORR-I

Applying formula (26, in ORR-I) we obtain

So, for k = 0, ..., K,

But,

So formula ((27), ORR-I) is proved if we show that

(5)

This follows immediately from the preceding proof of Result (b) of Section 3.2. Specifically, it was shown there that

Consequently, the left hand side of (5) is equal to

where the last equality follows by the definition of equation M59

and the fact that equation M60

(as this is just the function equation M61

resulting from applying the integration to the utility u (O, A) = 1).

2.4.3. Derivation of Formula (31) in ORR-I

It suffices to show that equation M62

where

But by definition

where the last equality follows because

2.5. Proof that b_{·, opt} is Optimal

Write for short, _· (b) [equivalent]

_· (b, _{·, opt}),

We will show that J_· (b) = E {Q_· (b) Q_· (b_{·, opt})^′} for · = par and · = sem. When either model (16, ORR-I) or (29, ORR-I) are correct, β^* = β^†. Consequently, for · = par we have that J_par (b) is equal to

For · = sem and with the definitions (x, Z) [equivalent]

b (x, Z) – (Z) and _sem (; β^†, γ^†, τ^†) [equivalent]

Q_sem (; β^†, γ^†, τ^†) – _sem (; β^†, γ^†, τ^†), the same argument yields J_sem (b) equal to

Now, with var_A (_· (b)) denoting the asymptotic variance of _· (b), we have that from expansion ((32) in ORR-I)

and consequently

Thus, 0 ≤ var_A (_· (b) – _· (b_{·, opt})) = var_A (_· (b)) + var_A (_· (b_{·, opt}) – 2cov_A (_· (b), _· (b_{·, opt})) = var_A (_· (b)) – var_A (_· (b_{·, opt})) which concludes the proof.

3. Confidence Set for x_opt (z) when equation M71

is Finite and h_· (z, x; β) is Linear in β

We first prove the assertion that the computation of the confidence set B_b entails an algorithm for determining if the intersection of equation M72

half spaces in

^p and a ball in

^p centered at the origin is non-empty. To do so, first note that linearity implies that equation M73

for some fixed functions s_j, j = 1, ..., p. Let equation M74

and write

. The point x_l is in B_b iff

(6)

Define the p × 1 vector equation M77

whose j^th entry is equal to s_j (x_l, z) – s_j (x_k, z), j = 1, ..., p. Define also the vectors equation M78

and the constants equation M79

. Then

iff

. Noting that β in C_b iff equation M82

is in the ball

we conclude that the condition in the display (6) is equivalent to

The set

is a hyper-plane in

^p which divides the Euclidean space

^p into two half-spaces, one of which is equation M86

. Thus, the condition in the last display imposes that the intersection of N – 1 half-spaces (each one defined by the condition equation M87

for each k) and the ball equation M88

is non-empty.

Turn now to the construction of a confidence set equation M89

that includes B_b. Our construction relies on the following Lemma.

Lemma. Let

where u₀ is a fixed p × 1 real valued vector and Σ is a fixed non-singular p × p matrix.

Let α be a fixed, non-null, p×1 real valued vector. Let τ₀ [equivalent]

α′ u₀ and α^* = Σ^1/2α. Assume that α₁ ≠ 0. Let, equation M91

be the p×1 vector equation M92

. Let

be the linear space generated by the p×1 vectors equation M93

and define

where

Then there exists equation M97

satisfying

if and only if

Proof

Then, with τ₀ [equivalent]

−α′ u₀ and α* = Σ^1/2α, we conclude that there exists equation M101

satisfying α′ u = 0 if and only if there exists u* [set membership]

R^p such that

Now, by the assumption equation M103

we have −α*^′ u* = τ₀ iff equation M104

. Thus, the collection of all vectors u* satisfying −α*^′ u* = τ₀ is the linear variety

where

and

are defined in the statement of the lemma. The vector equation M107

is the residual from the (Euclidean) projection of equation M108

into the space [Upsilon]

Thus, −α*^′ u* τ₀ iff equation M109

for some

. Consequently, by the orthogonality of equation M111

with

we have that for u* satisfying −α*^′ u* = τ₀ it holds that

Therefore, since equation M113

is unrestricted,

if and only if

(7)

This concludes the proof of the Lemma.

To construct the set equation M116

we note that the condition in the display (6) implies the negation, for every subset equation M117

, of the statement

(8)

Thus, suppose that for a given x_l we find that (8) holds for some subset equation M120

, then we know that x_l cannot be in B_b. The proposed confidence set equation M122

is comprised by the points in equation M123

for which condition (8) cannot be negated for all subsets equation M124

. The set

is conservative (i.e. it includes B_b but is not necessarily equal to B_b) because the simultaneous negation of the statement (8) for all equation M126

does not imply the statement (6). To check if condition (8) holds for any given subset equation M127

and x_l, we apply the result of Lemma as follows. We define the vector α [set membership]

^p whose j^th component is equal to equation M128

, j = 1,..., p and the vector equation M129

. We also define the constant equation M130

, and the matrix Σ = [Gamma]

_· (b). We compute the vectors equation M131

and the matrix V* as defined in Lemma. We then check if the condition (7) holds. If it holds then this implies that the hyperplane comprised by the set of β’s that satisfy the condition in display (8) with the < sign replaced by the = sign, intersects the confidence ellipsoid C_b, in which case we know that (8) is false. If it does not hold, then we check if condition

(9)

holds. If (9) does not hold, then we conclude that (8) is false for this choice of equation M134

. If (9) holds, then we conclude that (8) is true and we then exclude x_l from the set equation M135

4. Positivity Assumption: Example

Suppose that K = 1 and that equation M136

with probability 1 for k = 0, 1, so that no subject dies in neither the actual world nor in the hypothetical world in which g is enforced in the population. Thus, for k = 0, 1, O_k = L_k since both T_k and R_k are deterministic and hence can be ignored. Suppose that L_k and A_k are binary variables (and so are therefore equation M137

and

) and that the treatment regime g specifies that

Assume that

(10)

Assumption PO imposes two requirements,

(11)

(12)

Because by definition of regime g, equation M143

, then requirement (11) can be re-expressed as

Since indicators can only take the values 0 or 1 and equation M145

, l₀ = 0, 1 (by assumption (10)), the preceding equality is equivalent to

that is to say,

By the definition of λ₀ (·|·) (see (3) in ORR-I), the last display is equivalent to

(13)

Likewise, because equation M149

, and because equation M150

by the fact that equation M151

, requirement (12) can be re-expressed as

or equivalently, (again because the events equation M153

and

have the same probability by equation M155

Under the assumption (10), the last display is equivalent to

which, by the definition of λ₀ (·|·, ·, ·) in ((3), ORR-I), is, in turn, the same as

(14)

We conclude that in this example, the assumption PO is equivalent to the conditions (13) and (14). We will now analyze what these conditions encode.

Condition (13) encodes two requirements:

i) the requirement that in the actual world there exist subjects with L₀ = 1 and L₀ = 0 (i.e. that the conditioning events L₀ = 1 and L₀ = 0 have positive probabilities), for otherwise at least one of the conditional probabilities in (13) would not be defined, and
ii) the requirement that in the actual world there be subjects with L₀ = 0 that take treatment A₀ = 1 and subjects with L₀ = 1 that take treatment A₀ = 0, for otherwise at least one of the conditional probabilities in (13) would be 0.

Condition i) is automatically satisfied, i.e. it does not impose a restriction on the law of L₀, by the fact that equation M159

(since baseline covariates cannot be affected by interventions taking place after baseline) and the fact that we have assumed that equation M160

, l₀ = 0, 1.

Condition ii) is indeed a non-trivial requirement and coincides with the interpretation of the PO assumption given in section 3.1 for the case k = 0. Specifically, in the world in which g were to be implemented there would exist subjects with L₀ = 0. In such world the subjects with L₀ = 0 would take treatment equation M161

, then the PO assumption for k = 0 requires that in the actual world there also be subjects with L₀ = 0 that at time 0 take treatment A₀ = 1. Likewise the PO condition also requires that for k = 0 the same be true with 0 and 1 reversed in the right hand side of each of the equalities of the preceding sentence. A key point is that (11) does not require that in the observational world there be subjects with L₀ = 0 that take A₀ = 0, nor subjects with L₀ = 1 that take A₁ = 1. The intuition is clear. If we want to learn from data collected in the actual (observational) world what would happen in the hypothetical world in which everybody obeyed regime g, we must observe people in the study that obeyed the treatment at every level of L₀ for otherwise if, say, nobody in the actual world with L₀ = 0 obeyed regime g there would be no way to learn what the distribution of the outcomes for subjects in that stratum would be if g were enforced. However, we don t care that there be subjects with L₀ = 0 that do not obey g, i.e. that take A₀ = 0, because data from those subjects are not informative about the distribution of outcomes when g is enforced.

Condition (14) encodes two requirements:

iii) the requirement that in the actual world there be subjects in the four strata (L₀ = 0, L₁ = 0, A₀ = 1), (L₀ = 0, L₁ = 1, A₀ = 1), (L₀ = 1, L₁ = 0, A₀ = 0) and (L₀ = 1, L₁ = 1, A₀ = 0) (i.e. that the conditioning events in the display (14) have positive probabilities), for otherwise at least one of the conditional probabilities would not be defined, and
iv) the requirement that in the actual world there be subjects in every one of the strata (L₀ = 0, L₁ = 0, A₀ = 1), (L₀ = 0, L₁ = 1, A₀ = 1), (L₀ = 1, L₁ = 1, A₀ = 0) that have A₁ = 0 at time 1 and the requirement that there be subjects in stratum (L₀ = 1, L₁ = 0, A₀ = 0) that have A₁ = 1 at time 1, for otherwise at least one of the conditional probabilities in (14) would be 0.

Given condition ii) and the sequential randomization (SR) and consistency (C) assumptions, condition iii) is automatically satisfied, i.e. it does not impose a further restriction on the joint distribution of (L₀, L₁, A₀). To see this, first note that by condition (ii) the strata (L₀ = 0, A₀ = 1) and (L₀ = 1, A₀ = 0) are non-empty. So condition (iii) is satisfied provided

But

and

by (10). An analogous argument shows that equation M165

. Finally, condition (iv) is a formalization our interpretation of assumption PO in section 3.1 for k = 1. In the world in which g was implemented there would exist subjects that would have all four combination of values for equation M166

. However, subjects with equation M167

will only have equation M168

, so in this hypothetical world we will see at time 1 only four possible recorded histories, equation M169

and

. In this hypothetical world subjects with any of the first three possible recorded histories will take equation M173

and subjects with the last one will take equation M174

. Thus, in the actual world we must require that there be subjects in each of the first three strata (L₀ = 0, L₁ = 0, A₀ = 1), (L₀ = 0, L₁ = 1, A₀ = 1), (L₀ = 1, L₁ = 0, A₀ = 0) that take A₁ = 0 and subjects in the stratum (L₀ = 1, L₁ = 1, A₀ = 0) that take A₁ = 1. A point of note is that we don t make any requirement about the existence of subjects in strata other than the four mentioned in (iii) or about the treatment that subjects in these remaining strata take. The reason is that subjects that are not in the four strata of condition (iii) have already violated regime g at time 0 so they are uninformative about the outcome distribution under regime g.

Footnotes

^*This work was supported by NIH grant R01 GM48704.

References

Orellana L, Rotnitzky A, Robins JM. 2010. Dynamic regime marginal structural mean models for estimation of optimal dynamic treatment regimes, Part I: Main content The International Journal of Biostatistics 62Article 7. [PubMed]

Articles from The International Journal of Biostatistics are provided here courtesy of
Berkeley Electronic Press