We use the info-gap robustness function to evaluate alternative interventions aimed at controlling the relative TB prevalence, *C*(*t*), at a specified target time, *t*_{m}, in the future. An intervention is specified by the values of the control variables. The evaluation leads to realistic assessment of outcomes and preferences among the interventions.

Interpreting robustness curves: trade off and zeroing

All info-gap robustness curves have two properties, mentioned earlier: trade off between performance and robustness, and zeroing of the robustness curve. These properties are central in using robustness curves to evaluate public health policy.

The coefficients of the epidemiological models are specified in Tables and . Thoughout our examples, the initial conditions correspond to low TB and low HIV prevalence (the first data-column of Table ) unless specified otherwise. The control variables specified in Appendix “Control variables” section are themselves model parameters. The robustness curve in Figure is evaluated for the nominal values of the control variables specified in Tables and . This set of control variables is the “baseline intervention”. The uncertain variables specified in Appendix “Uncertainty” section are also model parameters. Their nominal values and uncertainty estimates are specified in Table . These nominal values are the same as appear in Tables and for these variables. The total case load is evaluated at time *t*_{m}=10 years after initiation unless indicated otherwise.

| **Table 1**Model parameters in the Murray-Salomon basic model |

| **Table 2**Model parameters in the Murray-Salomon basic model |

| **Table 4**Nominal values and error weights of uncertain variables |

Figures and show the temporal evolution of the relative prevalence of TB cases, *C*(*t*), and relative relapses, *R*(*t*), based on the nominal estimates of the model parameters, with moderately low initial TB and HIV prevalence. *C*(*t*) and *R*(*t*) are fractions of the initial total population size. Figure shows that the total number of TB cases starts at about 4.2% of the initial population and decays to about 3% in the first 1.5 years, thereafter decaying more slowly, reaching 2.1% of the initial population size after 10 years. The relapse population starts very small, rises rapidly in the first year and thereafter decays gradually. The reduction in the rate of decrease of the TB cases after 1.5 years, Figure , results from the influx of relapses which have built up since initiation of the intervention.

Trade off

Key to understanding the trade off expressed by the robustness curve is the concept of satisficing. In contrast to optimizing, satisficing asks for an outcome that meets minimal needs but may not be the best imaginable. The satisficing strategy is not merely “accepting second best.” Satisficing is aspirational, setting a goal just like optimization, but also requiring robustness to uncertainty. The satisficing strategy induces a trade off between the aspiration for good outcome and the robustness against uncertainty in attaining that outcome.

The robustness curve in Figure is based on satisficing the relative TB prevalence: requiring that the prevalence not exceed the critical value, *C*_{m}. Figure shows the robustness vs. the critical prevalence. The positive slope of the robustness curve in Figure expresses the trade off between robustness and performance: large robustness entails large prevalence at the specified target time (10 years). Equivalently, requiring low relative prevalence entails low robustness to uncertainty in the epidemiological model. The robustness curve quantifies the intuition that more demanding outcomes (small prevalence) are more vulnerable to model uncertainty (small robustness).

We can interpret the numerical values along the robustness curve as follows. The prevalence, *C*(*t*), and its critical value, *C*_{m}, are normalized to the initial population size. For instance, *C*_{m}=0.025 means that the prevalence at time *t*_{m} must not exceed 2.5% of the initial population size. The robustness corresponding to this value of *C*_{m}, is 0.1 as seen in Figure . This means that the performance requirement is guaranteed if the uncertain model parameters vary from their nominal values by no more than 10% of their error estimates. (The model parameters are constrained to be positive since they are first-order rate constants.)

The public health practitioner may feel that robustness to 10% uncertainty in the model parameters is rather small, given the substantial uncertainty in the epidemiological dynamics of TB with HIV prevalence. If we want robustness to, say, 25% uncertainty in the model parameters we must accept a larger final case load, namely, *C*_{m}=0.033 as seen in Figure . Greater robustness is obtained only by accepting poorer outcome; this is an irrevocable trade off that is quantified by the robustness curve.

Zeroing

We note that the robustness curve in Figure reaches the horizontal axis at the value *C*_{m}=0.021. This means that requiring the prevalence not to exceed 2.1% of the initial population has no robustness against model uncertainty. The value of *C*_{m} at which the robustness becomes zero is precisely the nominal prediction of the prevalence at time *t*_{m} as seen by the right end-point in Figure . That is, the value of *C*(*t*_{m}), evaluated with the best estimates of the model parameters, equals 0.021. The horizontal intercept in Figure is an example of the property of zeroing that holds for all info-gap robustness curves: The outcome predicted by the model, when adopted as the performance requirement, has no robustness against uncertainty in the model.

It is not surprising that the predicted outcome is extremely vulnerable to error in the model upon which the prediction is based. However, the zero-robustness of predicted outcomes has an important implication for policy selection.

The robustness curve in Figure is for a particular choice of values of the control variables: the baseline intervention. The zeroing property—no robustness of the predicted outcome of these control values—implies that we should not assess these control values in terms of their predicted outcome. The predicted prevalence of 0.021 at time *t*_{m}=10 years does not reliably reflect the performance of these control variables. Due to the trade off property, only larger prevalence can reliably be expected to result from this choice of the control variables. Predicted outcomes are not reliable for prioritizing the interventions.

Equivalent interventions

Different combinations of interventions can yield essentially equivalent results, as in Figure . The baseline intervention (solid), is characterized by low diagnosis rate and high relapse rate. The other intervention (dash) has higher diagnosis rate and lower relapse rate as specified in Table . (Interventions are specified by the values of control variables presented in Table ). The robustness curves for these two control strategies, at 10 years, are nearly the same, suggesting that the public health practitioner may choose freely between them, perhaps employing additional criteria such as cost or ease of implementation. Equivalence may be lost if parameters are changed. For instance, we will see later (Figure ) that these interventions evaluated at 10, 20 or 30 years have very different robustness curves.

| **Table 5**Control variables for robustness curves |

Figure shows a different aspect of the equivalence of interventions. The figure shows robustness curves for two strategies specified in Table . Both strategies aim to control the relative prevalence of TB, but one (solid) is geared for a 10-year target time, while the other (dash) considers a 30-year target. The estimated outcomes—prevalence—are very nearly the same for these two strategies, each at its respective target time, as shown by their shared horizontal intercept at *C*_{m}=0.018. These predictions result from estimated model parameters, so one might be inclined to conclude that TB prevalence of 0.018 can be achieved at either 10 or 30 years by using the corresponding intervention.

However, the epidemiological model is highly uncertain, and the robustness curves in Figure of these two strategies are quite different. Not surprisingly, the 30-year target is much less robust to uncertainty. It would be erroneous to treat these two strategies as outcome-equivalent since their performances at positive robustness are quite different. Nominal equivalence (equivalence of the predicted outcome) does not imply robustness equivalence.

Impact of initial TB and HIV prevalence

We now consider higher initial prevalences. The overall shape of the dynamic response is very similar in each case, except that the prevalence increases significantly as the initial prevalence increases. As in Figures and , in each scenario the initial TB prevalence decreases rapidly during the first 2 years, and thereafter decreases more slowly as the new relapse population—which peaks around the end of the first year—flows back into active cases.

Figure shows robustness curves for a target time 10 years after initiation, for low (solid), medium (dash) and high (dot-dash) initial prevalence of TB and HIV. The low-prevalence curve (solid) is the same as Figure . The robustness curves shift dramatically to the right as the baseline prevalence of TB and HIV increases, indicating poorer estimated outcome and lower robustness to uncertainty.

Intervention aggressiveness

Figure shows robustness curves for low initial TB and HIV prevalence with interventions specified in Table . The solid curve is the baseline intervention, against which the other curves entail more aggressive intervention in either or both the active cases and the relapse population.

The progression from solid to dash to dot-dash in Figure represents increasingly aggressive intervention in the active TB case population. We see that increasing aggressiveness, in this specific parameter configuration, results in increasing prevalence and decreasing robustness to model error at the target time. The explanation is that aggressive treatment of active cases enlarges the relapse population which flows back into the active case population.

The top curve in Figure modifies the most aggressive case (dot-dash) by also including more aggressive intervention in the TB relapse population. This reduction in relapse reduces the predicted prevalence after 10 years, and increases the robustness to uncertainty.

Different target times

Most of the results discussed so far evaluated the robustness for a target time 10 years after initiation. We now consider the implications of different target times.

Figure shows robustness curves at target times, *t*_{m}, of 10, 20 and 30 years (solid, dash, dot-dash respectively). The initial prevalences of TB and HIV are low. The interventions are all at the baseline.

The predicted prevalence decreases as the target time increases, as shown by the horizontal intercepts in Figure . The baseline intervention is predicted to reduce the prevalence, (in units of initial population size), as the time horizon increases. However, the zeroing property means that these predictions have no robustness to uncertainty in the model used for prediction. Only higher prevalence has positive robustness.

From Figure we see that, for critical TB prevalence *C*_{m} less than 3%, the 30-year TB prevalence is more robust than the 20-year prevalence which is more robust than the 10 year prevalence. For instance, at critical TB prevalence of *C*_{m}=0.02, the robustnesses for 10, 20 and 30 year horizons are 0, 0.08 and 0.12, respectively. This intervention has no robustness to uncertainty when requiring a 2% prevalence after 10 years; in fact, the estimated prevalence at 10 years is greater than 2%. The prevalence at 20 years will be no worse than 2% provided that the model coefficients err by no more than 8%, and at 30 years the robustness to error is 12%.

The practitioner may feel that even 12% robustness against model-coefficient error is rather small, given the severe uncertainty of TB epidemiology in the context of epidemic HIV. This means that, even at a 30-year horizon, this intervention cannot reliably achieve a relative prevalence as low as 2%.

Suppose we are willing to aim at a final TB prevalence of 3.7%. We see from Figure that now the 10-year horizon is more robust than 20 years which is more robust than 30 years. The robustnesses are now 30%, 24% and 22% for 10, 20 and 30 years. The robustness curves have intersected one another and the robustness rankings are reversed. As the target time decreases, the predicted outcome becomes worse (horizontal intercept moves right) but the cost of robustness improves. This causes the robustness curves to cross one another. More intuitively, we can say that prediction of TB prevalence is more reliable for short time horizon than for long times. But since a long time is required to overcome the relapse effect, we observe the intersection of the robustness curves and the consequent reversal of their robust dominance.

Results like Figure have important policy implications for TB control over long time periods. The policy maker may be tempted to choose one option that is predicted to yield better short term results. However, that choice might be wrong when one opts to satisfice the outcome with robustness to uncertainty. Predictions of mathematical models (horizontal intercepts) are not sufficiently reliable for comparing and prioritizing interventions; the cost of robustness (slope) must also be considered. In the example in Figure one might conclude that prevalence less than 3% is not achievable at any target time, that 3.7% is feasible at 10-years but not beyond, and that other interventions are needed for longer-term outcomes.

Impact of HIV mortality

Figure shows 10-year robustness curves for various HIV infection rates, with low initial TB and HIV prevalence, as specified in Table . The HIV infection rate decreases in the progression from solid, dash, dot-dash to dot-dot. As the HIV infection rate decreases, the estimated 10-year TB prevalence increases and the robustness decreases. The explanation lies in the high mortality rate of the HIV population. As the HIV infection rate decreases, the size of the relapse population decays more slowly, allowing greater flow back into the active TB case population. Interventions that decrease HIV infection rates or restore immunity to HIV patients, will counter-intuitively tend to increase TB prevalence unless compensating measures are taken. Significantly, the cost of robustness (slope of the robustness curve) does not change as a result of decreased HIV infection rate. Reducing HIV infection rate shifts the robustness curve to the right, with almost no change in slope.