Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Eur J Oper Res. Author manuscript; available in PMC 2017 March 16.
Published in final edited form as:
Eur J Oper Res. 2016 March 16; 249(3): 1005–1013.
doi:  10.1016/j.ejor.2015.07.038
PMCID: PMC4669975

Modelling Adherence Behaviour for the Treatment of Obstructive Sleep Apnoea

Yuncheol Kang, Ph.D., Amy M. Sawyer, Ph.D., RN, Paul M. Griffin, Ph.D., and Vittaldas V. Prabhu, Ph.D.


Continuous positive airway pressure therapy (CPAP) is known to be the most efficacious treatment for obstructive sleep apnoea (OSA). Unfortunately, poor adherence behaviour in using CPAP reduces its effectiveness and thereby also limits beneficial outcomes. In this paper, we model the dynamics and patterns of patient adherence behaviour as a basis for designing effective and economical interventions. Specifically, we define patient CPAP usage behaviour as a state and develop Markov models for diverse patient cohorts in order to examine the stochastic dynamics of CPAP usage behaviours. We also examine the impact of behavioural intervention scenarios using a Markov decision process (MDP), and suggest a guideline for designing interventions to improve CPAP adherence behaviour. Behavioural intervention policy that addresses economic aspects of treatment is imperative for translation to clinical practice, particularly in resource-constrained environments that are clinically engaged in the chronic care of OSA.

Keywords: Behavioural OR, obstructive sleep apnoea, treatment adherence behaviour, Markov models, cost-effective interventions

1. Introduction

1.1. Application Background

Obstructive Sleep Apnoea (OSA) is a sleep disorder that afflicts an estimated 5% of the U.S. adult population; it is the second-most prevalent sleep disorder in the U.S. (NIH 2011, Young, et al. 2002). OSA is characterized by repetitive nocturnal cessation of breathing, resulting in intermittent hypoxia and cortical arousals from sleep (Malhotra and White 2002, Patil, et al. 2007). It is well known that untreated OSA contributes to serious health-related issues, including hypertension, stroke, cardiovascular disease, metabolic disorders, and mood and cognitive impairments, and OSA increases the probability of occupational injuries and motor vehicle accidents (Billings and Kapur 2013, Hernández-Lerma and Lasserre 1996, Iber, et al. 2007, Young, et al. 2002). Continuous Positive Airway Pressure therapy (CPAP) is known to be a highly efficacious treatment when it is used consistently for the duration of each sleep bout (Giles, et al. 2006, Guest, et al. 2008, Sadatsafavi, et al. 2009). CPAP therapy is conducted at the patient’s home and requires the patient to wear a mask connected to an air pressure pump during sleep to provide positive pneumatic pressure to the upper airway, effectively reducing airway closures and the consequences of intermittent nocturnal breathing cessation. This therapy presents numerous challenges, however, for patients. Nonadherence to CPAP therapy is reported as a common behavioural issue among an estimated 50% of patients with CPAP-treated OSA (Parthasarathy, et al. 2013, Sawyer, et al. 2011, Sawyer, et al. 2014).

A typical CPAP device automatically records the duration of usage on a daily basis, measured in hours of use per 24 hours for all intervals of use at an effective pressure for more than 20 minutes. This technology permits an objective measure of CPAP use, or adherence, and permits early recognition of nonadherence during the initial home treatment period. In order to improve CPAP adherence, several types of interventions have been proposed and preliminary tests performed. Intervention strategies reported to date include simplistic, stand-alone educational interventions, CPAP device adaptations such as heated humidification, flexible pressure and auto-adjusting positive airway pressure, cognitive behavioural interventions in both small groups and with individual patients, social support interventions such as peer-to-peer support (Parthasarathy, et al. 2013), multi-dimensional approaches, habit-formation interventions, and self-management strategies (Sawyer, et al. 2011). OSA is not a disorder that can be treated on a one-time basis; OSA requires a patient to use the device as a long-term treatment, and such chronic care necessitates cost-effective interventions. To address this challenge and implement highly effective chronic care for CPAP-treated OSA in a way that substantially reduces CPAP nonadherence, empirically-derived decision policies are needed.

1.2. Behavioural Issues and Markov Models

Nonadherence can be regarded as a behavioural issue that must be addressed in order to treat OSA optimally. Interestingly, such treatment adherence behaviours vary across particular sub-populations (e.g., low socioeconomic position, race) and change dynamically (Kang, et al. 2013, Sawyer, et al. 2014). For example, a patient may show a low level of CPAP usage during the first few days but increased usage after some time has passed, while others show a constant low level of usage. Other patients demonstrate an acceptable level of usage at the outset of treatment, but lower usage levels over time. Moreover, some patients demonstrate usage behaviours that fluctuate randomly over the entire observed treatment period. We hypothesize that such changes are due to stochastic dynamics that result from social, personal, contextual and environmental factors. Modelling such stochastic adherence behaviours can be demanding, however, given its complexity and uncertainty. The use of system dynamics has been widely suggested and utilized in understanding such complex behavioural problems (Gino and Pisano 2008, Hämäläinen, et al. 2013); The application of systems science approaches to complex behavioural health problems is further substantiated by agency and scientific leaders in the U.S (Grady and Daley 2013, OBSSR 2014). Recent studies in chronic care management have approached complex behavioural problems with a system dynamics approach (Luke and Stamatakis 2012). One solution for dealing with these challenges is to use Markov models, which can help to predict an individual’s behaviour in uncertain environments.

Most behavioural intervention studies assume that patients, regardless of their behavioural patterns, will react similarly to interventions. In reality, however, some patients exposed to a behavioural intervention may demonstrate a ceiling effect for the primary outcome; this results in varying degrees of behavioural intervention effectiveness among different patients. Recent research suggests that interventions for improving CPAP usage can be less effective for patients showing moderate adherence compared to the patient group with low adherence (Wozniak, et al. 2014). Such bias between assumed and actual effectiveness should be considered when evaluating cost-effectiveness of the corresponding interventions. To decrease this bias, intervention designers can consider adaptive strategies that are applicable to different patient groups showing different behaviours. From this perspective, we consider multiple Markov models for each available intervention and formulate as a stochastic decision problem—namely, the Markov decision process (MDP)—that can optimize the intervention strategy for behavioural changes depending on the individual’s current behaviour.

In this paper, we suggest using Markov models to capture the stochastic dynamics of adherence behaviours and MDP to explore the optimal strategy for improving a patient’s adherence, a behavioural issue, during OSA treatment. From the estimated models, we reveal some notable behavioural trends in CPAP usage and address cost-effectiveness issues relative to intervening in all patients. By examining the impact of intervention scenarios using the MDP, we further suggest a guideline for designing interventions, dependent on an individual patient’s current behaviour (i.e., adherence), to effectively improve the target behaviour.

The structure of the paper is as follows. In the next section we define Markov models for CPAP adherence behaviour and estimate transition probabilities using data obtained from a recent prospective cohort study (Sawyer, et al. 2014). In Section 3, we develop MDP models to determine cost-effective intervention policies, along with some structural properties. Then, in Section 4, we illustrate the predicted optimal policy patterns using MDP models for probable intervention cases in a chronic care scenario. In particular, we use the result of the Markov model estimated in Section 2 and derive from our own recent research as the basis for design of the MDP model discussed in Section 3, and the baseline transition probabilities (as a ‘Do nothing’ action) for our MDP model in Section 4. Additionally, we validate the cost-effectiveness of two clinical interventions using Markov models in Section 4. Conclusions, limitations, and future work are described in Section 5.

2. Markov Model for CPAP adherence behaviours

In this Section, we discuss the development of a Markov model for CPAP adherence and explain how we characterize the dynamics underlying CPAP usage behavioural trends.

2.1. Data Description

We examined the first month of CPAP usage data from a prospective single cohort study that included 97 patients with moderate to severe OSA (Sawyer, et al. 2014). The goal of the primary study was to examine the predictive utility of a risk screening questionnaire to prospectively identify patients with OSA who were at high risk for CPAP nonadherence at the outset of the treatment period. Participants were recruited at an academic sleep centre after OSA diagnosis was established during an overnight sleep study, i.e., polysomnogram. OSA severity was determined by the apnoea-hypopnoea index (AHI), or apnoeas plus hypopnoeas divided by hours of sleep during polysomnogram (AASM 2005). OSA severity is categorized as mild, ≥5 and <15 events/hour, moderate, ≥15 and <30 events/hour, or severe, ≥30 events (Epstein, et al. 2009).

For the primary study, the inclusion criteria for newly-diagnosed OSA patients were: (1) an apnoea-hypopnoea index (AHI) greater than or equal to 5 events/hour on an in-laboratory polysomnogram, conducted and scored in accordance with standard criteria (Iber, et al. 2007); (2) referral to CPAP titration polysomnogram, i.e., an in-laboratory, single night sleep study during which CPAP pressure is titrated to reduce/eliminate apnoeas and hypopnoeas and normalize oxygenation during sleep; and (3) ability to speak and read English. Participants had no previous experience with CPAP prior to study enrolment. After informed consent, the participants’ demographic and OSA features were collected, and an in-laboratory CPAP titration polysomnogram was performed. Thereafter, participants received home CPAP treatment for 30 days and objective CPAP usage data were collected at study termination. Such usage data were recorded sequentially at a daily basis for 30 days.

We pre-processed the data by excluding patient outcomes with erroneous or missing values; ninety-two patients remained for the final analysis. Table 1 shows summary statistics for the included patients.

Table 1
Summary Statistics (n=92)

2.2. State Definition

A patient's adherence behaviour can be quantified by observing his or her usage behaviour. From this perspective, we define the amount of a patient’s CPAP usage as a state of the Markov model, which we define in discrete units as shown in Table 2. Generally, standard, commercially available CPAP machines record usage in time, so a patient’s CPAP usage value can be regarded as continuous. In order to avoid computational issues, however, we define this continuous state as discrete in this paper by grouping the continuous values into bins. To determine the threshold for discretizing state, we note the empirical findings in the related literature, that is, more than 4 hours of usage is classified as an acceptable level of usage (Sawyer, et al. 2011). This 4 hour threshold for determining adherence is clinically stable across the empiric literature (Sawyer, et al. 2011), and it is the level of usage that is used by third-party payers (i.e., health insurance) for reimbursing or paying for CPAP treatment. Following this clinical standard, we use the 4 hour threshold as a cut point for separating adherers from those who do not adhere to the treatment. We further subdivide the patients who do not adhere to the treatment (i.e., less than 4 hours usage), since several published studies use a 2 hour threshold, which is stricter, to determine nonadherence (Chasens, et al. 2005, Zimmerman, et al. 2006). We validate this 2 hour threshold assumption using an asymptotic distribution theory (Anderson and Goodman 1957) with 95% confidence interval to compare thresholds. We find that our results under the 2 hour threshold are still valid even if we change the threshold by ±25 minutes from 2 hour (i.e., between 1 hour 35 minutes and 2 hours 25 minutes). Thus, we use the 2 hour threshold to differentiate complete non-adherers from the patients showing better but not full adherence to the treatment. Throughout this paper, we designate the former as non-adherers (less than 2 hours usage) and the latter as intermediate adherers (greater than or equal to 2 hours and less than 4 hours usage).

Table 2
State definition

Consequently, we discretize into three bins and label them as State 1, 2 and 3 as shown in Table 2. All Markov models that follow will use the same state definition.

2.3. Estimation of Transition Probabilities

Transition probabilities for the Markov models are estimated using maximum likelihood estimators (MLE) (Lee 1977). Specifically, the MLE for each transition probability is defined as the ratio of the number of transitions for a specific transition over the total number of transitions that have a common exit state. Note that the estimated transition probabilities may be nonhomogeneous over time. Therefore, we use a statistical test of homogeneity (Anderson and Goodman 1957), which uses a χ2 test of the null hypothesis, such that the transition probabilities are constant over time.

The transition probabilities and steady state probability for each patient group have been estimated as shown in Table 3. These results are obtained from the data grouped using demographic information. For the χ2 test (significance level = 5%), the null hypothesis was never rejected for any cases, which implies that the estimated probabilities for all cases are constant over time.

Table 3
Estimated Transition Probabilities

2.4. Interpretation of estimated transition probabilities

For insights into estimated transition probabilities, we examined the average trend of transition probabilities for the entire group of patients. Specifically, we notice that a patient who shows the lowest usage level (i.e., the current state is s1) is less likely to improve his or her usage, as compared to all other patients. Let us denote p(sj|si) as the probability of transitioning from state i to state j at the next time. First, p(s1|s1), the probability of staying at the current lowest usage level, is 3.49 times greater on average than p(s3|s1), which is the greatest improvement in usage. Even though we consider any improvement trend from the lowest usage (i.e., p(s2|s1) + p(s3|s1)), we notice that this improvement ratio is 2.56 times less on average than p(s1|s1). This result can be considered as a motivation for investing in individuals with the lowest usage levels by developing effective adherence promotion interventions.

Compared to the lowest usage group, we notice that a patient is less likely to decrease his or her usage level, once at a higher adherent state. When we examine the entire group of patients, p(s3|s3), the transition probability for staying at the highest usage level, is 5.14 times greater on average than the other possibilities (i.e., p(s1|s3) + p(s2|s3)). Even for the intermediate usage group (s2), transition to the better adherent state (i.e., p(s3|s2)) doubled on average as opposed to the transition probability to the less adherent state (i.e., p(s1|s2)). This finding implies that intervention to promote CPAP usage may not be cost-effective for all groups, especially for groups whose usage level is already high. Furthermore, this finding supports the appropriate allocation of scarce and expensive resources, such as those resources needed for adherence promotion interventions, to those patients most likely to benefit and demonstrate improvement for adherence outcomes. In the U.S. cost-constrained healthcare market, such evidence-based decision-making approaches are imperative. In the next section, we investigate the characteristics of cost-effective policies for promoting CPAP adherence, and suggest guidelines for designing interventions that are economically advantageous for the improvement of CPAP adherence.

3. MDP Model for CPAP adherence behaviours

It has been established that adherence to CPAP therapy is crucial in order to treat OSA optimally (Campos-Rodriguez, et al. 2012, Young, et al. 2013); therefore, a chronic care management approach is necessary. From this perspective, any intervention used in chronic care must be cost-effective for the long term, in order to reduce the overall cost burden over the period of treatment, which is often life time. In this section, we model a Markov decision process (MDP) to examine some notable structural properties of the suggested model in terms of cost-effectiveness.

3.1. MDP model definition

CPAP adherence level is defined as a state (si,i={1,2,3} [set membership] S), as previously described. The action set S includes available interventions (ah [set membership] A) for improving adherence, and also 'Do nothing'. For convenience, ‘Do nothing’ is set to a1 and all possible interventions for promoting CPAP adherence are indexed from 2 to n (e.g. a2, a3 …) in order of efficacy. When the state is si and the action is ah, then the associated reward r(si, ah) is defined. In our case, such a reward term consists of a benefit component and a cost (that is, a cost for each intervention). Considering a chronic care paradigm, we set an infinite planning horizon for making intervention decisions. The decision period is set as a day, and p(sj|si, ah) is the probability of transitioning to state j at the next time, when the current state is i and action is ah. We assume that p(sj|si, ah) is available for each action ah and homogeneous over time. With these definitions, the optimal intervention policy, d(si), will identify the action that results in the maximum value, consisting of both the current reward and the cumulative expected value of reward onward, u(sj) with a discount factor λ(< 1), and can be expressed as follows:


3.2. Probable Interventions

Before we examine the underlying structural properties of our problem, we define probable interventions that promote a patient’s CPAP adherence behaviour. Since we do not have any intervention data that can be used to estimate transition probabilities for interventions (that is, p(sj|si, ah), h ≥ 2), we assume the following conditions for probable interventions.

Assumption 3.2

Probable interventions either improves patient’s adherence states or increases the chances of patient’s staying at the highest adherence states

In our model, p(s2|s1, ah), p(s3|s1, ah) and p(s3|s2, ah) represent transitions toward better adherence states and p(s3|s3, ah) indicates staying at the highest adherence state when an intervention ah is taken. Thus, these four transition probabilities are assumed to be greater than or equal to the corresponding transition probabilities of the baseline model. In our case, ‘Do nothing’ is the baseline action, and its transition probabilities (that is, (sj|si, a1)) were estimated in Section 2. For the sake of conveniently expressing the amount of change in transition probabilities, let us denote ratios of improvement of intervention ah for non-adherers, which are αh = p(s2|s1, ah)/p(s2|s1, a1) and βh = p(s3|s1, ah)/p(s3|s1, a1). Also, let γh be p(s3|s2, ah)/p(s3|s2, a1), representing the rate of improvement by intervention ah for an intermediate adherer and δh be p(s3|s3, ah)/p(s3|s3, a1), representing the rate of staying at the highest adherence when intervention ah is taken. By assumption 3.2, αh, βh, γh and δh are assumed to be greater than or equal to 1. When a baseline model is available, these factors can be calculated only after the transition probabilities corresponding to each intervention are estimated.

3.3. Structural Properties

With Assumption 3.2, we further supplement the model with the detailed characteristics of each action by assuming the following conditions:

Assumption 3.3.1

For the reward function, costs depend only on the intervention and not the adherence state.

Assumption 3.3.2

For each action, the patient benefit does not decrease as the adherence level increases.

Assumption 3.3.3

For each action, the cumulative probability of maintaining or improving the current adherence level up to the highest adherence level increases as adherence level increases. In other words, a patient at a higher adherence level is highly likely to improve his or her adherence level.

Assumption 3.3.4

The incremental effect of using a less-intensive intervention on the probability that the adherence level will exceed a fixed level is smaller if current adherence levels are greater.

Assumption 3.3.1 and 3.3.2 are straightforward. Assumption 3.3.3 is made based on one of the findings in Section 2.4, that is, a higher self-improving trend is observed for adherers over non-adherers in case of ‘Do nothing.’ We further assume that this trend holds in the case of behavioural intervention. Technically, this assumption can be interpreted as IFR (Increasing Failure Rate (Proschan and Barlow 1967)), and a similar approach can be found in papers dealing with clinical decision-making problems (Alagoz, et al. 2004, Shechter, et al. 2008). In our model, Assumption 3.3.3 can be expressed such that Σj=k p(sj|si, ah) is nondeacreasing in si for all k and ah.

Assumption 3.3.4 can be interpreted as the ceiling effect of intervention (Lewis-Beck, et al. 2003). In other words, a chosen action may result in different outcomes, depending on the current state and the overall efficacy of the intervention, which tends to be lower when applied to patients in a higher state (e.g., s3). This assumption can be expressed such that Σj=k p(sj|si, ah) is a subadditive function on S × A for all k. Supporting evidence for Assumption 3.3.4 can be found in related survey research (Wozniak, et al. 2014), which showed that intervention for improving adherence is less effective for adherers and more beneficial for non-adherers. The less intensive intervention was more beneficial for adherers, as compared to a more intensive intervention (Aloia, et al. 2013). Assumptions 3.3.3 and 3.3.4 are closely related to a behavioural OR model, in that it is based on particular behavioural aspects observed in our baseline results and related studies.

To reveal structural properties, we temporarily truncate the planning horizon to the finite horizon, i.e., the notation, d(s), which is the optimal action of infinite-horizon MDP, is changed to dt*(s), representing the optimal action of finite-horizon MDP. We then return it to the infinite horizon MDP problem by assuming that the optimal value of the discount infinite-horizon problem will inherit the same property of the optimal value of the finite-horizon problem (Hernández-Lerma and Lasserre 1996). Under the assumed features, our problem results in a monotone optimal policy in which dt*(s) is nonincreasing in s for t = 1, …, N − 1 (Puterman 1994).

In our context, this monotone optimal policy can be interpreted as follows: for each decision time, the optimal action at the lowest-adherence state provides an upper bound for the optimal action at the other (i.e., improved) adherence state. For example, if the optimal action at s1 is chosen, ‘Do nothing (a1)’ at time t, then the optimal action for the other states (i.e., s2 or s3) at time t must be chosen as ‘Do nothing (a1),’ which means we do not need to calculate optimal decision rules for those states. As a special case, such a boundary condition of a monotone policy plays the role of a control limit when we have only two actions.

3.4. Critical Value of Intervention

Now we define a critical value of intervention ch,s*, as the maximum possible monetary value, ch,s in which the intervention, ah, is the optimal decision at the current state, s, under the given ratio of improvement. For example, ca2,s1*=$10 means that the intervention (a2) becomes cost-effective for the patient whose current state is s1 over the ‘Do nothing (a1)’ action, when the price of the intervention is less than or equal to $10. If the intervention a2 costs more than $10, then the ‘Do nothing (a1)’ action becomes an optimal action. Under a monotone optimal policy, a critical value of intervention has the following characteristics.

Proposition 1

For a non-adherer, a critical value of an intervention increases as the rates of improvement toward better adherence increases.


We show that a critical value increases as either αh or βh increases for non-adherers.

Let ah be the best action for non-adherers (s1) at the decision epoch, n*. From the optimality equation, the following inequality should be satisfied;


Equation (2) can be expressed as


Also, the critical value of intervention ah for non-adherers, ch,s1* can be defined as a maximum value of ch,s1 Thus,


Since νn* (s1) ≤ νn* (s2) ≤ νn* (s3) by a monotone optimal policy and both αh and βh are greater than or equal to 1 by Assumption 3.2.1, none of the terms in the right hand side of equation (4) can be negative as either αh or βh increases. Therefore, ch,s1* increases as either αh or βh increases

Proposition 2

For an intermediate adherer, if an intervention decreases the chance of a reduction in adherence, then the critical value increases as the rate of improvement increases.


Let us denote ζh = p(s1|s2, ah)/p(s1|s2, a1), representing the changes of reducing adherence. Then, we show that if ζh < 1, the critical value increases as γh increases for intermediate adherers.

Let ah be the best action for intermediate adherers (s2) at the decision epoch, n*. Then we can Induce ch,s2* as follows using an approach similar to that we used in Proposition 1.


Since νn* (s1) − νn* (s2) ≤ 0 and νn* (s3) − νn* (s1) ≥ 0 by a monotone optimal policy and γh is greater than or equal to 1 by Assumption 3.2.1. Therefore, if ζh < 1, then a critical value, ch,s2* increases as γh increases for intermediate adherers

Proposition 3

For an adherer, if an intervention decreases the chances of dropping to the lowest adherence, then the critical value increases as an intervention improves the chances of staying at the highest adherence level


Let us denote ηh = p(s1|s3, ah)/p(s1|s3, a1), representing the changes of reducing adherence to the lowest level. Then, we show that if ηh < 1, then the critical value increases as δh increases for an adherer. The proof of Proposition 3 is similar to that of Proposition 2. Note that for an adherer, p(s1|s3, ah) represents a chance to drop to the lowest adherence, and p(s3|s3, ah) represents a chance of staying at the highest adherence

Proposition 4

The critical value for a given intervention becomes greater as the level of adherence becomes higher.


Let ah,j be the best action for sj at the decision epoch n*, where it is not the best action at the next decision epoch n* + 1. By a monotone optimal policy, ah,i at si should be greater than or equal to ah,j at sj at the epoch, n*. Further, at n* + 1, ah,i should be still greater than or equal to ah,j since ah,j = a1,j. This means that ah,i should be still optimal with the price ch,sj*. Therefore, ch,sj*ch,si. Since ch,sich,si*,ch,si* should be greater than or equal to ch,sj*. Thus, the critical value, ch,si* is greater than or equal to ch,sj* if 1 ≤ i < jn

Proposition 1 implies that if a certain intervention shows greater improvement for non-adherers over other actions, its monetary value also increases. For example, suppose there are two interventions, a2 and a3, and the improvement ratio of a3 is greater than that of a2. In this case we can say that a3 has higher monetary value compared to a2. In light of this, Proposition 1 offers a criterion for choosing cost-effective intervention for non-adherers, based on its monetary value. In the case of an intermediate adherer, we can consider three different scenarios after intervention; i) reducing adherence, ii) staying at the current intermediate adherence, iii) improving adherence. Proposition 2 implies that if a certain intervention decreases the chances of reducing adherence for intermediate adherers, its monetary value should increase as the intervention increases chances of improvement more than other interventions. Similarly, Proposition 3 describes the increase in the critical value of intervention for an adherer if the intervention reduces the chances of decreasing adherence. Proposition 4 implies that the monetary value of an intervention increases as we apply it to patients with poorer adherence. For example, if a critical value turns out to be $5 for a patient showing an intermediate level of adherence, then the critical value of the same intervention should not be less than $5 for the patient showing the lowest adherence level. Namely, the cost-effectiveness can be interpreted and applied depending on the patient's current adherence state.

4. Characteristics of Cost-effective Intervention Policy

With Markov models and MDP discussed earlier, we examine the cost-effectiveness of interventions designed to promote CPAP adherence. We consider two types of intervention in this paper: clinical intervention and frequent intervention. First, clinical interventions focus on creating the best-case scenario for CPAP use in the early stages of treatment, and are usually performed in a limited time frame, possibly even in a single delivery of the intervention, often at the outset of treatment initiation. Examples of clinical interventions include educational interventions, behavioural interventions, and supportive interventions (Wozniak, et al. 2014). For the other type of intervention, referred to as frequent, we consider repeatable interventions that can be implemented repeatedly over an extended period of time, such as self-management strategies delivered/accessed with mobile technologies. More importantly, this type of intervention can be considered a component of chronic care management, in that it can help a patient to maintain acceptable CPAP use throughout his or her lifetime.

4.1. Clinical Interventions

Although some studies have reported efficacy of interventions to promote adherence to CPAP use, the cost-effectiveness of interventions have not yet been studied (Wozniak, et al. 2014). Since most clinical interventions are one-time interventions, or consolidated intervention delivery in a short period of time, MDP may not be the best approach for calculating their cost-effectiveness. Therefore, instead of using MDP, we utilize a Markov model and steady state probabilities for each state as inputs for examining the cost-effectiveness of the corresponding intervention.

For the intervention scenarios, we select two evidence-based intervention studies by considering the rigor of each study; one is the use of a training video, which is a type of educational intervention (Basoglu, et al. 2012), and the other is a motivational interviewing intervention (MINT), which is a behavioural intervention (Olsen, et al. 2012). We perform a Monte Carlo simulation to generate CPAP usage data (10,000 patients’ 30 days of usage samples generated) under each intervention scenario using the reported intervention efficacy from both studies. Then, we build a Markov model using the generated usage data and obtain its steady state probabilities in order to estimate the proportion of each state [Table 4]. Finally, we calculate the maximum possible cost that does not exceed the benefit, which is monetized using Quality Adjusted Life Years (QALY) for each steady state probability. In particular, QALY is often used as a measure to describe the trade-off between the cost and the benefit of an intervention in a clinical OR model (Mason, et al. 2014). For parameterization of QALY, we use 0.8 QALY for the adherers and 0.74 QALY for both non-adherers and intermediate adherers, who are exposed to the risk of OSA (Billings and Kapur 2013, Sadatsafavi, et al. 2009). As in several OSA studies (AlGhanim, et al. 2008, Brown, et al. 2005), we assume $50,000 per QALY as a threshold of willingness-to-pay. Given the results shown in Table 4, we can calculate longrun average benefits for both an intervention group and a control group by using steady state probabilities and QALY for each adherence state. By comparing those benefits, the maximum possible cost for the corresponding intervention can be estimated. For the training video, the maximum possible cost is estimated as $418. This result indicates that the corresponding intervention is cost-effective as long as its cost is less than $418. For the MINT, the maximum possible cost is estimated as $448, which is slightly higher than the educational intervention.

Table 4
Intervention scenario and estimated steady state probabilities for each scenario

Since there is no cost information reported for either study, we estimate a reasonable cost range for each intervention by assuming the following. Practically, we can assume that a training video takes a total of 45 minutes with assistance from either a registered sleep technologist (about $26/hr) or a non-registered sleep technologist ($19/hr). Given this assumption, we can determine that the training video is cost-effective for promoting a patient's CPAP adherence, since its estimated cost (i.e., $19.50 per intervention, at most) is less than the maximum cost, $418. As for MINT, if the intervention is assumed to be delivered by a registered nurse ($42/hr) and intervention time is assumed to be 170 minutes total (i.e., 30 minutes for both MINT sessions, 20 minutes for a booster session, and an additional 30 minutes for preparation per session), then it can also be regarded as a cost-effective intervention, since its estimated cost (i.e., $119 per intervention) is less than $448. Consequently, we can conjecture that both clinical interventions are cost-effective under the assumptions specified above.

4.2. Frequent Interventions

We investigate the characteristics of an optimal policy for implementing frequent interventions to promote CPAP adherence using the proposed MDP model, with all conditions previously mentioned. In particular, we examine a trend of critical values of an intervention, especially for non-adherers, as the improvement ratios of the intervention change. Also, we explore characteristics of optimal policies for implementing two probable interventions (i.e., three-action scenario including ‘Do nothing’ action). Few studies have explored the efficacy of a frequent and repeated intervention as a component of a chronic care management strategy for improving CPAP adherence. Thus, we assume probable interventions described in Section 3.2 and 3.3 and illustrate the cost-effectiveness under the MDP model and assumptions previously discussed. For the baseline transition probabilities for each state in this Section, we use the transition probabilities of the ‘Overall’ case shown in Table 3, which comprises the transition probabilities of the ‘Do nothing (a1)’ action.

4.2.1 A Trend of Critical Values of an Intervention

As an illustrative purpose, we consider a two-action case for non-adherers; available actions are ‘Do nothing (a1)’ and one available intervention (a2) to promote adherence. For the QALY, we used 0.74 QALY for both non-adherers and intermediate adherers and 0.8 QALY for adherers, as stated previously. Note that 1.0 QALY is equivalent to $50,000. The discount factor can be varied between 0.95 and 0.99 (Weinstein, et al. 1996). In our model, as we choose a lower discount factor, we observed that the critical value of the intervention becomes lower, indicating less burden in doing the intervention. For example, the critical values with 0.95 as the discount factor are estimated at $1.1 less on average for all cases than with 0.99. However, the overall trend of the resulting optimal policy (e.g. increasing/decreasing trend of critical value as α2 or β2changes) is consistent for different discount factors. In this paper, we use 0.99 as the annual discount factor. The ranges of both α2 and β2, the ratio of improvement for intervention (a2), are restricted under the assumption of a monotone optimal policy. We set γ2 and δ2 as 1.77 and 1.21, respectively, by assuming that both intermediate adherers and full adherers become or remain full adherers after intervention. We find that this setting does not change the results of the illustration. Value iteration was used to solve the infinite-horizon MDP (Puterman 1994).

The critical values for all possible combinations using α2 and β2 are listed in Appendix A. Interestingly, we found that the a critical value for an intervention can exist with different combinations of α2 and β2. For example, the critical value $11.30 appears both when α2 = 2 and β2 = 3.5 and when α2 = 3.5 and β2 = 3.3. From this perspective, we can plot equal critical values on the filled contour map as shown in Figure 1. The overall trend shows that a critical value increases (i.e., shading becomes dark gray) as either αh or βh increases, as shown in Proposition 1. In particular, given α2, the critical value increases as β2 increases. Similarly, the critical value for the intervention also increases when α2 increases, given β2.

Figure 1
Critical values ($) of an intervention given improvement ratio, αh and βh. The annual discount factor is set as 0.99. Color bar (right hand side) shows scale of critical values.

The highest critical value is observed when β2 is highest, compared to when α2 is highest. Intuitively, this trend is reasonable, since β2 indicates that the amount of increase on p(s3|s1, a2), which represents the greatest improvement in CPAP adherence for non-adherers, thereby the most significant factor to increase the critical value. For all combinations of the ratio of improvement, the range of critical values turned out to be between $9.10 and $14.90. This result can be interpreted as follows; if the cost of the intervention is less than or equal to $9.10, then the intervention is always cost-effective for non-adherers over the ‘Do nothing (a1)’ action. Likewise, if the intervention (a2) costs more than $14.90, then doing nothing is always cost-effective for non-adherers over any interventions.

4.2.2. Optimal Policies of Frequent Interventions

For this illustration, we examine the three-action case (‘Do nothing (a1),’ ‘Intervention A (a2),’ and ‘Intervention B (a3)’) for all adherence groups of patients. We have three experimental sets: 1) small differences, 2) intermediate differences, and 3) large differences in βh between the two probable interventions (Table 5). βh is primarily tested in this illustration since it was previously shown as the more critical factor to determine a critical value. For an experiment, αh is set as 2.0, and γh and δh are set as 1.78 and 1.21, respectively; these values do not violate the conditions for a monotone optimal policy.

Table 5
The experiment sets

The optimal action chosen for each experiment set is plotted in Figure 2. Intuitively, the cost of Intervention B (a3) is more expensive than that of Intervention A (a2) since all experiment sets assume β3 is greater than β2. Thus, a2, with lower efficacy, can never be chosen over a3, with higher efficacy, when the cost of a2 is greater than that of a3 and this is well depicted in Figure 2. When cost of a2 is less than that of a3, it turns out that ‘Do nothing (a1)’ is not an optimal strategy for non-adherers (the three leftmost plots in Figure 2) for any case unless the cost is $11 or greater. Another interesting finding is that, as the two interventions show larger differences in improvement, Intervention B (a3) (the more costly intervention) is more frequently chosen as an optimal action for non-adherers. In the case of intermediate adherers, ‘Do nothing’ (a1) and Intervention A (a2) are always chosen as the optimal action over Intervention B (a3) when a2 costs less than a3. For adherers, ‘Do nothing (a1)’ is chosen as the optimal strategy unless the cost of both interventions (a2 and a3) is less than $1.

Figure 2
Optimal policy for three-action case (two interventions and ‘do nothing’). The leftmost, mid and rightmost columns represent non-adherers, intermediate adherers and adherers, respectively. Top, middle and bottom rows are the results of ...

5. Summary and Future Work Directions

In this paper, we developed a Markov model to describe a patient’s treatment adherence behaviour, especially using CPAP application. Specifically, a patient's adherence behaviours were modelled using the objective CPAP usage level, and defined as a set of states within the Markov model. The transition probabilities among the degree of adherence for each cohort were estimated based on demographic information. We found that a patient who shows the lowest-usage level is less likely to improve their usage compared to other groups. This implies investing in adherence interventions for individuals at the lowest-usage can be cost-effective. Further, we noted that the probability of staying at the highest-usage level is fairly high, which implies that interventions to promote/maintain CPAP usage may not be cost-effective for this type of patient. Such results offer insights to effective policy decisions to reduce the burden of nonadherence to CPAP in the treatment of OSA in a cost-conscious approach.

The numerical examples for the MDP model in this paper illustrated differences in cost-effectiveness for cohorts of CPAP users that demonstrate different adherence levels. Specifically, patients showing different adherence levels may benefit from different intervention policies, and interventions may not be appropriate for all adherers. It is imperative that consideration of the economic aspects of treatment be addressed before actual interventions are translated to healthcare settings, particularly in resource-constrained environments that are clinically engaged in the chronic care management of OSA.

Since this paper represents the initial stage of implementing actual interventions with CPAP users, future research will consider the fully developed MDP model with randomized controlled trial intervention data to improve the basic model. A further consideration for improving the model includes the possibility of non-homogeneity of the Markov model. Although transition probabilities are assumed to be time homogeneous throughout the paper, they can be time non-homogeneous in that 1) time may elapse before intervention takes effect, or 2) intervention might cause a beneficial learning effect. In the case of a non-homogeneous MDP model, we can consider 1) using a finite MDP model instead of an infinite model which has stationary probability, or 2) including time dependent states in the model and formulating a non-homogeneous MDP model (Hopp, et al. 1987).

More importantly, compared to most MDP applications in healthcare areas, the Markov model appearing in this paper does not have any absorbing state (e.g., discharge, death) and thereby steady state probabilities for all states exist between 0 and 1. Many perturbation theories for a Markov model have been studied to investigate the change of steady state probabilities against any changes in transitions in a Markov model (Meyer 1975, Schweitzer 1968). Thus, it would be meaningful to utilize such perturbation theories to identify which intervention most effectively helps the patient transition to a higher adherence state, which is a primary goal of chronic care management of OSA.


  • Treatment adherence behaviours are modelled using Markov models.
  • Poor adherence behaviour is less likely to be improved.
  • High adherence behaviour is highly likely to be maintained.
  • MDP results illustrate different cost-effectiveness across various interventions.


This work was, in part, supported by the National Institutes of Health/National Institute of Nursing Research (K99NR011173 Sawyer, AM PI and R00NR011173 Sawyer, AM PI). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Nursing Research or National Institutes of Health.


A table for critical values (unit $) for an intervention with different combinations of α2 and ;2



Note: Critical values for left lower diagonal are excluded from the table because resulting transition probabilities violate the condition such that the sum of probabilities of each row exceed 1.


Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Contributor Information

Yuncheol Kang, Department of Industrial and Manufacturing Engineering, Pennsylvania State University, ude.usp@gnakcy, phone: 1-814-865-9861, address: 236 Leonhard Building, University Park, PA 16802, USA.

Amy M. Sawyer, College of Nursing, Pennsylvania State University, ude.usp@42sma, phone: 1-814-863-1020, address: 201 Health & Human Development East, University Park, PA 16802, USA.

Paul M. Griffin, School of Industrial and Systems Engineering, Georgia Tech, ude.hcetag@niffirgp, phone: 1-404.894.2300, address: 755 Ferst Drive, NW, Atlanta, GA 30332.

Vittaldas V. Prabhu, Department of Industrial and Manufacturing Engineering, Pennsylvania State University, ude.usp.rgne@uhbarp, phone: 1-814-863-3212, address: 310 Leonhard Building, University Park, PA 16802, USA.


  • AASM. Diagnostic and Coding Manual. Westchester, Illinois: American Academy of Sleep Medicine; 2005. International Classification of Sleep Disorders. 2005. Broughton R. Pathological fragmentary myoclonus, intensified sleep starts and hypnagogic foot tremor: three unusual sleep related disorders.
  • Alagoz O, Maillart LM, Schaefer AJ, Roberts MS. The optimal timing of living-donor liver transplantation. Management Science. 2004;50:1420–1430.
  • AlGhanim N, Comondore VR, Fleetham J, Marra CA, Ayas NT. The economic impact of obstructive sleep apnea. Lung. 2008;186:7–12. [PubMed]
  • Aloia MS, Arnedt JT, Strand M, Millman RP, Borrelli B. Motivational enhancement to improve adherence to positive airway pressure in patients with obstructive sleep apnea: a randomized controlled trial. Sleep. 2013;36:1655. [PubMed]
  • Anderson TW, Goodman LA. Statistical inference about Markov chains. The Annals of Mathematical Statistics. 1957:89, 110.
  • Basoglu OK, Midilli M, Midilli R, Bilgen C. Adherence to continuous positive airway pressure therapy in obstructive sleep apnea syndrome: effect of visual education. Sleep and Breathing. 2012;16:1193–1200. [PubMed]
  • Billings ME, Kapur VK. Medicare Long-Term CPAP Coverage Policy: A Cost-Utility Analysis. Journal of Clinical Sleep Medicine. 2013;9:1023–1029. [PubMed]
  • Brown DL, Chervin RD, Hickenbottom SL, Langa KM, Morgenstern LB. Screening for Obstructive Sleep Apnea in Stroke Patients A Cost-Effectiveness Analysis. Stroke. 2005;36:1291–1293. [PubMed]
  • Campos-Rodriguez F, Martinez-Garcia MA, de la Cruz-Moron I, Almeida-Gonzalez C, Catalan-Serra P, Montserrat JM. Cardiovascular mortality in women with obstructive sleep apnea with or without continuous positive airway pressure treatmenta cohort study. Annals of internal medicine. 2012;156:115–122. [PubMed]
  • Chasens ER, Pack AI, Maislin G, Dinges DF, Weaver TE. Claustrophobia and adherence to CPAP treatment. Western Journal of Nursing Research. 2005;27:307–321. [PubMed]
  • Epstein L, Kristo D, Strollo P, Jr, Friedman N, Malhotra A, Patil S, Ramar K, Rogers R, Schwab R, Weaver E. Adult Obstructive Sleep Apnea Task Force of the American Academy of Sleep Medicine. Clinical guideline for the evaluation, management and long-term care of obstructive sleep apnea in adults. J Clin Sleep Med. 2009;5:263–276. [PubMed]
  • Giles TL, Lasserson TJ, Smith B, White J, Wright J, Cates CJ. Continuous positive airways pressure for obstructive sleep apnoea in adults. Cochrane Database Syst Rev. 2006;3:3. [PubMed]
  • Gino F, Pisano G. Toward a theory of behavioral operations. Manufacturing & Service Operations Management. 2008;10:676–691.
  • Grady PA, Daley K. The 2013 National Nursing Research Roundtable: Advancing the science of chronic illness self-management. Nursing outlook. 2013 [PubMed]
  • Guest JF, Helter MT, Morga A, Stradling JR. Cost-effectiveness of Using Continuous Positive Airway Pressure in the Treatment of Severe Obstructive Sleep Apnoea/Hypopnoea Syndrome in the UK. Thorax. 2008;63:860–865. [PubMed]
  • Hämäläinen RP, Luoma J, Saarinen E. On the importance of behavioral operational research: The case of understanding and communicating about dynamic systems. European Journal of Operational Research. 2013;228:623–634.
  • Hernández-Lerma O, Lasserre JB. Discrete-time Markov control processes. New York, NY: Springer-Verlag; 1996.
  • Hopp WJ, Bean JC, Smith RL. A new optimality criterion for nonhomogeneous Markov decision processes. Operations Research. 1987;35:875–883.
  • Iber C, Ancoli-Israel S, Chesson A, Quan S. The AASM manual for the scoring of sleep and associated events: rules, terminology and technical specifications. Westchester, IL: American Academy of Sleep Medicine; 2007.
  • Kang Y, Prabhu VV, Sawyer AM, Griffin PM. Markov models for treatment adherence in obstructive sleep apnea. Proceedings of Industrial and Systems Engineering Research Conference (ISERC); San Juan, Puerto Rico. 2013.
  • Lee T. Estimating the Parameters of the Markov Probability Model from Aggregate Time Series Data. Amsterdam: North-Holland Publishing Company; 1977.
  • Lewis-Beck M, Bryman AE, Liao TF. The Sage encyclopedia of social science research methods. Sage Publications; 2003.
  • Luke DA, Stamatakis KA. Systems science methods in public health: dynamics, networks, and agents. Annual review of public health. 2012;33:357. [PMC free article] [PubMed]
  • Malhotra A, White DP. Obstructive Sleep Apnoea. The Lancet. 2002;360:237–245. [PubMed]
  • Mason JE, Denton BT, Shah ND, Smith S. Optimizing the simultaneous management of blood pressure and cholesterol for type 2 diabetes patients. European Journal of Operational Research. 2014;233:727–738.
  • Meyer J, Carl D. The role of the group generalized inverse in the theory of finite Markov chains. Siam Review. 1975;17:443–464.
  • NIH Fact sheet: Obstructive sleep apnea. [Accessed at Feb/2014];PubMed Health. 2011
  • OBSSR About OBSSR. Office of Behavioral and Social Sciences Research in National Institutes of Health. [Accessed at July/2014];2014
  • Olsen S, Smith SS, Oei TP, Douglas J. Motivational interviewing (MINT) improves continuous positive airway pressure (CPAP) acceptance and adherence: A randomized controlled trial. Journal of Consulting and Clinical Psychology. 2012;80:151. [PubMed]
  • Parthasarathy S, Wendel C, Haynes PL, Atwood C, Kuna S. A pilot study of CPAP adherence promotion by peer buddies with sleep apnea. Journal of clinical sleep medicine: JCSM: official publication of the American Academy of Sleep Medicine. 2013;9:543. [PubMed]
  • Patil SP, Schneider H, Schwartz AR, Smith PL. Adult Obstructive Sleep Apnea: Pathophysiology and Diagnosis. Chest Journal. 2007;132:325–337. [PMC free article] [PubMed]
  • Proschan F, Barlow RE. Mathematical theory of reliability. SIAM series in applied mathematics. 1967
  • Puterman M. Markov decision processes: Discrete stochastic dynamic programming. New York, NY, USA: John Wiley & Sons, Inc; 1994.
  • Sadatsafavi M, Marra CA, Ayas NT, Stradling J, Fleetham J. Cost-effectiveness of oral appliances in the treatment of obstructive sleep apnoea–hypopnoea. Sleep and Breathing. 2009;13:241–252. [PubMed]
  • Sawyer AM, Gooneratne NS, Marcus CL, Ofer D, Richards KC, Weaver TE. A Systematic Review of CPAP Adherence across Age Groups: Clinical and Empiric Insights for Developing CPAP Adherence Interventions. Sleep Medicine Reviews. 2011;15:343–356. [PMC free article] [PubMed]
  • Sawyer AM, King TS, Hanlon A, Richards KC, Sweer L, Rizzo A, Weaver TE. Risk assessment for CPAP nonadherence in adults with newly diagnosed obstructive sleep apnea: preliminary testing of the Index for Nonadherence to PAP (I-NAP) Sleep and Breathing. 2014:1–9. [PMC free article] [PubMed]
  • Schweitzer PJ. Perturbation theory and finite Markov chains. Journal of Applied Probability. 1968:401–413.
  • Shechter SM, Bailey MD, Schaefer AJ, Roberts MS. The optimal time to initiate HIV therapy under ordered health states. Operations Research. 2008;56:20–33.
  • Weinstein M, Siegel J, Gold M, Kamlet M, Russell L. Cost-effectiveness in health and medicine. Vol. 55. New York: Oxford University; 1996.
  • Wozniak DR, Lasserson TJ, Smith I. Educational, supportive and behavioural interventions to improve usage of continuous positive airway pressure machines in adults with obstructive sleep apnoea. Cochrane Database Syst Rev. 2014;1 [PubMed]
  • Young LR, Taxin ZH, Norman RG, Walsleben JA, Rapoport DM, Ayappa I. Response to CPAP withdrawal in patients with mild versus severe obstructive sleep apnea/hypopnea syndrome. Sleep. 2013;36:405. [PubMed]
  • Young T, Peppard PE, Gottlieb DJ. Epidemiology of obstructive sleep apnea: a population health perspective. American Journal of Respiratory and Critical Care Medicine. 2002;165:1217–1239. [PubMed]
  • Zimmerman ME, Arnedt JT, Stanchina M, Millman RP, Aloia MS. Normalization of memory performance and positive airway pressure adherence in memory-impaired patients with obstructive sleep apnea. Chest Journal. 2006;130:1772–1778. [PubMed]