PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of jmirEditorial BoardMembershipSubmitCurrent IssueArchiveReviewSearchAboutJMIR PublicationsJMIR
 
J Med Internet Res. 2017 February; 19(2): e46.
Published online 2017 February 21. doi:  10.2196/jmir.6417
PMCID: PMC5340923

Factors Associated With Dropout During Recruitment and Follow-Up Periods of a mHealth-Based Randomized Controlled Trial for Mobile.Net to Encourage Treatment Adherence for People With Serious Mental Health Problems

Monitoring Editor: Gunther Eysenbach
Reviewed by Til Wykes and Dan Robotham

Abstract

Background

Clinical trials are the gold standard of evidence-based practice. Still many papers inadequately report methodology in randomized controlled trials (RCTs), particularly for mHealth interventions for people with serious mental health problems. To ensure robust enough evidence, it is important to understand which study phases are the most vulnerable in the field of mental health care.

Objective

We mapped the recruitment and the trial follow-up periods of participants to provide a picture of the dropout predictors from a mHealth-based trial. As an example, we used a mHealth-based multicenter RCT, titled “Mobile.Net,” targeted at people with serious mental health problems.

Methods

Recruitment and follow-up processes of the Mobile.Net trial were monitored and analyzed. Recruitment outcomes were recorded as screened, eligible, consent not asked, refused, and enrolled. Patient engagement was recorded as follow-up outcomes: (1) attrition during short message service (SMS) text message intervention and (2) attrition during the 12-month follow-up period. Multiple regression analysis was used to identify which demographic factors were related to recruitment and retention.

Results

We recruited 1139 patients during a 15-month period. Of 11,530 people screened, 36.31% (n=4186) were eligible. This eligible group tended to be significantly younger (mean 39.2, SD 13.2 years, P<.001) and more often women (2103/4181, 50.30%) than those who were not eligible (age: mean 43.7, SD 14.6 years; women: 3633/6514, 55.78%). At the point when potential participants were asked to give consent, a further 2278 refused. Those who refused were a little older (mean 40.2, SD 13.9 years) than those who agreed to participate (mean 38.3, SD 12.5 years; t1842=3.2, P<.001). We measured the outcomes after 12 months of the SMS text message intervention. Attrition from the SMS text message intervention was 4.8% (27/563). The patient dropout rate after 12 months was 0.36% (4/1123), as discovered from the register data. In all, 3.12% (35/1123) of the participants withdrew from the trial. However, dropout rates from the patient survey (either by paper or telephone interview) were 52.45% (589/1123) and 27.8% (155/558), respectively. Almost all participants (536/563, 95.2%) tolerated the intervention, but those who discontinued were more often women (21/27, 78%; P=.009). Finally, participants’ age (P<.001), gender (P<.001), vocational education (P=.04), and employment status (P<.001) seemed to predict their risk of dropping out from the postal survey.

Conclusions

Patient recruitment and engagement in the 12-month follow-up conducted with a postal survey were the most vulnerable phases in the SMS text message-based trial. People with serious mental health problems may need extra support during the recruitment process and in engaging them in SMS text message-based trials to ensure robust enough evidence for mental health care.

ClinicalTrial

International Standard Randomized Controlled Trial Number (ISRCTN): 27704027; http://www.isrctn.com/ISRCTN27704027 (Archived by WebCite at http://www.webcitation.org/6oHcU2SFp)

Keywords: text messaging, mobile health, antipsychotics, mental health, psychiatric services, methodological study

Introduction

Serious mental health problems are a major problem around the world [1]. They are associated with cognitive deficits, such as distortions in thinking or troubles in paying attention or working memory [2], a lack of treatment adherence [3], rehospitalization [4], and lifelong disability [5]. Mobile technology has become a popular way to deliver interventions to facilitate adherence to chronic disease management [6], including to people with serious mental health problems [7-9].

Recently, interventions with mobile phone technology (mHealth) have been applied to randomized controlled trials (RCTs) [10]. However, many methodological concerns in RCTs have been raised when mHealth interventions have targeted people with serious mental health problems [9]. Most concerns are related to inadequately reported details in the studies [10], such as the patient recruitment process [9], participant engagement in mHealth-based interventions [11], or technological details related to intervention delivery (eg, amount of undelivered text messages or patients changing a phone number) [12]. Because the mental health area continues to adopt new technologies in clinical practice [7,13,14], generating high-quality research is essential.

On the other hand, conducting mHealth-based research among people with mental health problems includes challenges [11]. First of all, reaching the target group may be difficult [11,15]. Patients may refuse to participate due to a fear of or suspicious thoughts about mobile devices [16]. They may also distrust the credibility of a mHealth intervention [11]. Other concerns related to mHealth use are user privacy, confidentiality, and online security [11]. People with serious mental health problems may be “digitally divided,” which makes them vulnerable for not benefiting from mHealth services [17]. For example, not everyone is willing to use mobile technology [13], has access to technology [17,18], or is skilful or familiar with mobile phones [19]. Further, low engagement and discontinuation are fundamental problems in technology-based intervention studies [9,20]. The participants may stop the intervention because they feel that it is too complex, time consuming [11], or repetitive and, therefore, the intervention becomes a mere routine.

To better understand how mobile apps could be developed, evaluated, and implemented into routine care, it is important to truly understand which study phases make the RCT the most vulnerable in the field of mental health care. Still, many important parts of the study methodology are inadequately reported in RCTs, particularly regarding interventions targeting people with serious mental health problems [9]. Therefore, we mapped the recruitment and the 12-month trial follow-up periods in order to provide a picture of the dropout predictors from a mHealth-based multicenter RCT, titled “Mobile.Net” (ISRCTN: 27704027). Recruitment and engaging participants in trials involving psychosis is problematic; there are numerous ways in which it can go wrong (eg, consenting, attrition). This paper describes a case study of recruitment and follow-up processes, and problems in this context, based on the Mobile.Net trial. Multiple regression was used to identify which demographic factors are related to recruitment and retention. The main results of the Mobile.Net trial will be reported elsewhere.

Methods

Mobile.Net Trial

Mobile.Net is a nationwide multicenter randomized controlled two-armed trial. The Mobile.Net trial evaluated the effects of tailored short message service (SMS) text messages constructed to encourage patient medication adherence and outpatient care for adult patients with psychosis [21]. Participants in the intervention group received semiautomatic text messages for 12 months (approximately 10 per month; range 2-25 text messages) based on their preferences [22]. They were able to decide the amount, timing, and frequency of the SMS text messages delivered. They were also able to change the content or timing of the messages during the trial. Treatment as usual was offered to all participants. The study was carried out according to the Declaration of Helsinki and approved by the Ethics Committee of the Hospital District of Southwest Finland. Written informed consent was obtained from the study participants after they were given a complete description of the study.

Population

There were a total of 1139 participants, men and women, ranging in age from 18 to 65 years. Each participant had a continuing prescription for antipsychotic medication, access to a mobile phone, and the ability to use the Finnish language. After participants were recruited, they were then randomized. Forensic patients and those having a planned nonacute treatment period were excluded from the study [21].

Procedure

Recruitment, including activities conducted before and during participant enrollment [23], occurred face-to-face in 45 psychiatric hospital wards in Finland (between September 5, 2011 and November 30, 2012). Research nurses in each study ward performed chart reviews to check the eligibility of each patient admitted to the study ward [24]. After completion of the baseline data, patients were allocated randomly (computer-based randomization with four block randomization) into two groups (SMS text message intervention group or control group) [21].

Attrition, including actions after enrollment in the study [23], was assessed at the 12-month follow-up period (between September 5, 2012 and December 31, 2013). Attrition was defined as the loss of eligible participants from the study groups [25]. To gather follow-up data, a postal survey, including the Quality of Life Enjoyment and Satisfaction Questionnaire (Q-LES-Q) by Endicott et al [26] and the Client Satisfaction Questionnaire (CSQ-8) by Atkisson and Greenfield [27], was carried out. Follow-up data were collected by members of the research group. During the study period and follow-up, each point of contact with participants was completely tracked and recorded. Attrition was assessed during the intervention and during the follow-up.

Measures

Data for this paper were divided into two categories: data concerning patient recruitment and data relating to attrition (Textbox 1).

Measures concerning patient recruitment and attrition from the study.

Recruitment (variable: measurement)

1) Screened: (n)

2) Eligible: n (%)

3) Consent not asked: n (%)

4) Refused: n (%)

5) Enrolled: n (%)

Recruitment speed: n/day

Attrition (variable: measurement)

1) During SMS intervention: dropout rate

2) During follow-up

  • Telephone interview: dropout rate
  • Participant’s notification: dropout rate
  • Postal survey: dropout rate
  • Register data retrieval: dropout rate

Recruitment data were categorized into five groups: (1) patients screened for eligibility, (2) eligible participants, (3) eligible participants whose consent was not requested, (4) participants who refused to participate at the point of contact in the psychiatric ward, and (5) those who consented to participate. Outcomes were recorded as screened, eligible, consent not asked, refused, and enrolled. In addition, to track the pace of recruitment, a record of all identified, screened, eligible, unwilling, and successfully recruited patients was kept using a specific monitoring sheet developed for the trial. Patient flow was monitored and recorded daily on the study wards. Daily progress of patient recruitment was reported as “recruitment pace” (ie, how many new patients were recruited e each day) [28].

Attrition data were categorized into two groups: (1) attrition during SMS text message intervention and (2) attrition during the 12-month follow-up period [29]. Attrition during the follow-up was then divided into four categories, based on the follow-up data collection method: (1) telephone interview, (2) participants’ notification (ie, withdrew from the follow-up survey), (3) postal survey, or (4) register data retrieval from the Finnish National Care Register for Health Care [30]. Outcomes were reported as dropout rates. In our study, the dropout rates were calculated based on telephone or postal survey responses, or by whose data were not available in register data retrieval due to an incorrectly entered ID.

Statistical Analysis

Descriptive statistics (frequency, percentage, mean, standard deviation) were used to describe participants’ demographic characteristics, recruitment, and attrition metrics (study participants lost in the follow-up). The demographic variables examined included age, gender, marital status (lives alone, ie, single, divorced, or widowed; lives with someone, ie, married), vocational education (none, vocational education), employment status (employed/self-employed, retired, student, job seeker), diagnosis (International Statistical Classification of Diseases and Related Health Problems, Tenth Revision [31]), and age at first contact with psychiatric services. To analyze possible differences between patients who participated in the study and those who dropped out, t tests and chi-square tests were used.

Multiple logistic regression analysis was used to determine predictors of dropping out of the 12-month postal survey follow-up. Participants’ demographic characteristics, including age, gender, marital status, vocational education, and employment status, were chosen as predictors and added to the analysis [8,32]. All data were analyzed using SPSS version 21. A P value <.05 was interpreted as a statistically significant difference.

Results

Recruitment

A total of 11,530 patients admitted within psychiatric inpatient hospital wards were screened during the 15-month (453 days) recruitment period. There were 6565 who did not meet the eligibility criteria. A total of 779 patients dropped out before the eligibility assessment because they were transferred to another ward or rapidly discharged from hospital. Of the candidates who were screened, 36.31% (4186/11,530) appeared eligible.

Of the 4186 eligible patients, informed consent was asked from 3417 (81.63%). Informed consent was not asked in 18.37% (769/4186) of the cases because the person was quickly discharged from the ward, absconded from hospital, or the research nurses simply forgot to ask.

When age and gender of the screened noneligible and eligible patients were compared, it was found that the eligible patients were generally younger than the noneligible patients (P<.001). Men were more often noneligible than women (P<.001) (Table 1).

Table 1
Demographic characteristics comparable across all stages.

Out of the 3417 eligible participants whose consent was asked, 2278 patients (66.67%) refused to participate in the study. Although reasons for refusal were not asked due to ethical guideline requirements [33], some patients voluntarily offered explanations, such as they did not know how to use mobile phones or text messages, their mobile phone was broken, or they did not have a mobile subscription at the that time. Patients who refused to participate were older than consenting, randomized patients (P=.001) (Table 1).

The pace of recruitment was analyzed based on the number of new patients recruited each day. At the beginning of the study, recruitment was slow. The recruitment rate reached its peak 15 months after enrollment started. For every 10 screens completed, one person was successfully enrolled, at an average recruitment speed of 76 participants each month (2.5 participants per day).

Of the 1139 patients who were enrolled in the study, the data of 16 participants were excluded due to either the withdrawal of informed consent (n=10), the patient did not meet the inclusion criteria (n=5), or a recruitment error (n=1). This left us with a total of 1123 participants (intervention group: n=563; control group: n=560).

Attrition

Attrition During SMS Text Message Intervention Period

A total of 569 eligible participants were allocated to a group to receive tailored SMS text messages for 12 months. The data of six participants were excluded from the analyses due to either a lack of written informed consent (n=2), the patient did not meet the inclusion criteria (n=3), or an erroneous randomization to study group (n=1). This left us 563 participants.

Of the 563 participants who received text messages, 27 dropped out during the 12-month intervention period (4.8%). In cases where a patient did not want to continue with the text message intervention, the researchers were notified by the participant, a relative, or a research nurse. Three participants dropped out before the intervention even began, and 24 within the 12-month intervention period [22]. Participants who dropped out during the intervention were still included in the study [34]. We observed that intervention dropouts were more often women than men (χ21=7.2, P=.009), but found no other statistically significant differences (Multimedia Appendix 1).

Attrition During the 12-Month Follow-Up Period

Information about participants who dropped out after the 12-month follow-up was divided into four categories based on the data collection method: (1) telephone interview, (2) participants’ notification (ie, withdrew from the follow-up survey), (3) postal survey, or (4) register data retrieval.

First, telephone interviews (for the intervention group only) were conducted after the 12-month text message intervention to explore participants’ feedback on the text message service (n=569). We attempted to reach 558 participants by telephone for an interview; after the telephone calls were made, we had 403 completed questionnaires (response rate 72.2%, 403/558) [35]. The dropout rate from these telephone interviews was 27.8% (155/558). Dropouts were younger, usually men, without a vocational education, and were also younger at the time of first contact with psychiatric services compared with those who completed the questionnaire [35].

Second, 35 participants expressed that they wanted to withdraw from the follow-up surveys (intervention group: 5.5%, 31/563; control group: 0.7%, 4/560; χ21=21.4, P<.001). Follow-up surveys were not conducted with these participants, but their register data were retrieved. The dropout rate regarding follow-up surveys was 3.12% (35/1123). Women requested to withdraw from the follow-up surveys more often than men did (χ21=10.1, P=.002). We found no other statistically significant differences (Multimedia Appendix 1).

Third, a postal survey (n=1123) was conducted after the 12-month study period to measure participants’ quality of life (Q-LES-Q [26]) and satisfaction with the treatment (CSQ-8 [27]). Of 1123 participants, 589 did not return the postal survey (dropout rate: 52.45%, 589/1123). There was a statistically significant difference in demographic characteristics between postal survey completers (ie, returned the completed postal survey) and dropouts (Multimedia Appendix 1).

Fourth, register data retrieval was conducted after the 12-month follow-up period. Out of 1123 participants, the register data of four participants were not available from the Finnish National Care Register for Health Care [30], making the dropout rate 0.36%. Demographic characteristics of participants whose register data were available will be reported elsewhere.

Factors Related to Patient Attrition

Through a logistic regression analysis, Table 2 illustrates the associations between participants’ demographic characteristics (age, gender, marital status, vocational education, and employment status) and risk of dropping out from the postal survey. Odds ratios were not estimated from other variables due to missing data. Participants’ age, gender, vocational education, and employment status seemed to predict their risk of dropping out from the postal survey. The participants in this group were older, more often women, had a vocational education, and were more often retired.

Table 2
Associations between participants’ demographic characteristics and risk of dropping out of the postal survey (N=1123).

Discussion

Principal Results

The results of this study demonstrate that it was challenging to recruit, and engage, participants in an SMS text message-based trial follow-up. One-third of patients (36.31%) appeared eligible, and two-thirds of eligible patients (66.67%) refused to participate. Participants were well engaged with the SMS text message intervention provided, but their engagement with the trial follow-up varied: the highest being with the register data retrieval (99.64%) and lowest with the postal survey (47.55%). Participants’ demographic characteristics (age, gender, vocational education, and employment status) were seen as dropout predictors.

In our study, within the context of psychiatric inpatient care, we were able to recruit 1139 individuals (33.33%) out of 3417 eligible participants, whose consent was requested. Age and gender tended to be factors influencing recruitment and refusal. Our refusal rate was 66.67% (2278/3417), which is in line with previous studies suggesting that high refusal rates are a major problem faced during the recruitment process [11,15]. Patients’ illness-related issues [14,19] and perceived stigma were seen as barriers to participation in previous studies [36,37]. This may also be the case in our study, which focused on people treated with antipsychotic medication recruited from hospital wards providing psychiatric care.

Lack of interest in the trial [11], lack of motivation to use mobile technology-based interventions [13], or a lack of capacities may also have affected the refusal rates in our trial. During the participant screening, some patients expressed their inability to use mobile phones or text messaging, stating that as the reason for their refusal. This was unexpected because mobile phone text messages have been proven to be feasible and acceptable among people with severe mental health problems [38], as well as easy to use [35]. Therefore, our findings may reveal that a digital divide may still exist in the area of mental health, although relevant literature identifies this concept as promising [14]. Previous literature also shows that people with mental health problems are able and willing to use mobile-based interventions when given the opportunity [38,39]. In our case, it might have been useful to train those who said they did not know how to use mobile phones, and to offer mobile phones to those who did not have them. This might have given us important knowledge about the digital divide, especially those who are not so familiar with the mobile technology.

The attrition rate during the SMS text message intervention was low (4.8%). This finding does not support previous findings stating that low engagement and discontinuation are major problems in intervention studies [11,14]. However, it is possible that participants who stopped using the SMS text message intervention did not notify the researchers. What we found here was that women dropped out from the SMS text message intervention more often than men did. This is contradictory to the results of Ben-Zeev and colleagues [8], who found that women were significantly more engaged with mHealth interventions than men were. Further, participants’ demographic characteristics (age, gender, vocational education, and employment status) predicted the risk of leaving the study early and, subsequently, not participating in the follow-up. This is in line with previous studies, which reported that participants who left the studies early differed from those who were retained [8,32]. Therefore, it is important to identify feasible and useful mHealth interventions targeting different patient groups and, further, to identify factors that facilitate or prevent patient engagement with mHealth interventions.

Patient engagement in the trial during the follow-up varied depending on the source of data collected, the highest being in the register data retrieval (99.64%) and the lowest in the postal survey (47.55%). Low engagement and discontinuation have been found as fundamental problems in technology-based intervention studies [9,20]. Participants’ age, gender, vocational education, and employment status seemed to predict their risk of dropping out from the postal survey. This group of older, women, and retired participants may also reveal the digital divide in this patient group. It may be that they just do not value technology as much as younger people do and, therefore, did not participate in the postal survey. Recently, however, technology usage has increased among older adults [40]. This is promising and may give researchers new hints on how to engage participants in trials. Given this, it is important to consider which data collection methods are appropriate to use among people with serious mental health problems, and to identify factors that facilitate patient engagement in mHealth-based trial follow-ups.

Limitations

Our study has some limitations. The recruitment data concerning information about screened patients lacked some information, especially about patients’ ages. Therefore, results related to patient demographics concerning recruitment have to be handled with caution. More importantly, we did not gather knowledge about the participants’ actual SMS text message use. Therefore, we lack knowledge about participants’ true engagement with the SMS text message intervention. However, according to our findings before the study actually started, participants were very satisfied with the intervention [35].

A key strength of this study was in its large nationwide sample of people treated with antipsychotic medication. Another was that, to the best of our knowledge, this was the largest trial evaluating a text message system. Our findings regarding attrition are important for those conducting similar RCTs among people with severe mental health problems, although this group may well have different issues with the technology when compared with others [17,41].

Conclusions

Initial patient recruitment and then engagement in the 12-month postal survey follow-up were the most vulnerable phases in the SMS text message-based trial. This may indicate that people with serious mental health problems may need extra support during the recruitment process, and necessitate further support to engage in completion of these follow-up questionnaires—at least within SMS text message trials.

Researchers should acknowledge the possible digital divide for people with serious mental health problems, and choose convenient and efficient data collection methods for study follow-ups. At follow-up, for Mobile.Net, high-grade routine data were almost complete. Methods of trials should take much more consideration of the nature of the target group of participants; otherwise, evidence is dogged with high attrition with the accompanying speculation of researchers. No statistical technique or learned speculation can make up for loss to follow-up. The solutions are likely to vary for different client groups. We think more research is needed both to investigate the support of the recruitment process and methods of follow-up in technology-based RCTs. Asking people to complete forms that are likely to result in grossly incomplete datasets could be considered an unethical—and potentially dangerous—waste of time and resources.

Acknowledgments

The authors would like to warmly thank all the institutions who awarded grants for Professor Välimäki: the Academy of Finland (132581), Turku University Hospital (EVO 13893), Satakunta Hospital District (EVO 12/2010, 81096), Foundations’ Professor Pool, the Finnish Cultural Foundation, and the University of Turku. Many thanks also go to the patients participating in the Mobile.Net study, research nurses, and the staff of the research organizations and wards, without whom the realization of this study would not have been possible. The authors especially wish to thank Kaisa Kauppi, PhD, Minna Anttila, PhD, and Sanna Suni, MA, for their valuable help with monitoring, data collection, and data entry. They also thank the Mobile.Net Safety Committee Group for their efforts and support throughout this study.

Abbreviations

CSQ-8
Client Satisfaction Questionnaire
Q-LES-Q
Quality of Life Enjoyment and Satisfaction Questionnaire
RCT
randomized controlled trial
SMS
short message service

Multimedia Appendix 1

Additional demographic characteristics comparable across latter stages of study.

Footnotes

Conflicts of Interest: None declared.

References

1. Word Health Organization. [2016-11-01]. Schizophrenia http://www.who.int/mental_health/management/schizophrenia/en/ webcite.
2. Harvey PD, Strassnig M. Predicting the severity of everyday functional disability in people with schizophrenia: cognitive deficits, functional capacity, symptoms, and health status. World Psychiatry. 2012 Jun;11(2):73–79. http://onlinelibrary.wiley.com/resolve/openurl?genre=article&sid=nlm:pubmed&issn=1723-8617&date=2012&volume=11&issue=2&spage=73. [PubMed]
3. Velligan DI, Weiden PJ, Sajatovic M, Scott J, Carpenter D, Ross R, Docherty JP, Expert Consensus Panel on Adherence Problems in Serious and Persistent Mental Illness The expert consensus guideline series: adherence problems in patients with serious and persistent mental illness. J Clin Psychiatry. 2009;70 Suppl 4:1–46; quiz 47. http://www.psychiatrist.com/jcp/article/pages/2009/v70s04/v70s0401.aspx. [PubMed]
4. Haddad PM, Brain C, Scott J. Nonadherence with antipsychotic medication in schizophrenia: challenges and management strategies. Patient Relat Outcome Meas. 2014;5:43–62. doi: 10.2147/PROM.S42735. https://dx.doi.org/10.2147/PROM.S42735. [PMC free article] [PubMed] [Cross Ref]
5. Świtaj P, Anczewska M, Chrostek A, Sabariego C, Cieza A, Bickenbach J, Chatterji S. Disability and schizophrenia: a systematic review of experienced psychosocial difficulties. BMC Psychiatry. 2012 Nov 09;12:193. doi: 10.1186/1471-244X-12-193. https://bmcpsychiatry.biomedcentral.com/articles/10.1186/1471-244X-12-193. [PMC free article] [PubMed] [Cross Ref]
6. Hamine S, Gerth-Guyette E, Faulx D, Green BB, Ginsburg AS. Impact of mHealth chronic disease management on treatment adherence and patient outcomes: a systematic review. J Med Internet Res. 2015;17(2):e52. doi: 10.2196/jmir.3951. http://www.jmir.org/2015/2/e52/ [PMC free article] [PubMed] [Cross Ref]
7. Gaebel W, Großimlinghaus I, Kerst A, Cohen Y, Hinsche-Böckenholt A, Johnson B, Mucic D, Petrea I, Rössler W, Thornicroft G, Zielasek J. European Psychiatric Association (EPA) guidance on the quality of eMental health interventions in the treatment of psychotic disorders. Eur Arch Psychiatry Clin Neurosci. 2016 Mar;266(2):125–137. doi: 10.1007/s00406-016-0677-6. [PubMed] [Cross Ref]
8. Ben-Zeev D, Scherer EA, Gottlieb JD, Rotondi AJ, Brunette MF, Achtyes ED, Mueser KT, Gingerich S, Brenner CJ, Begale M, Mohr DC, Schooler N, Marcy P, Robinson DG, Kane JM. mHealth for schizophrenia: patient engagement with a mobile phone intervention following hospital discharge. JMIR Ment Health. 2016;3(3):e34. doi: 10.2196/mental.6348. http://mental.jmir.org/2016/3/e34/ [PMC free article] [PubMed] [Cross Ref]
9. Berrouiguet S, Baca-García E, Brandt S, Walter M, Courtet P. Fundamentals for future mobile-health (mHealth): a systematic review of mobile phone and web-based text messaging in mental health. J Med Internet Res. 2016 Jun 10;18(6):e135. doi: 10.2196/jmir.5066. http://www.jmir.org/2016/6/e135/ [PMC free article] [PubMed] [Cross Ref]
10. Agarwal S, LeFevre AE, Lee J, L'Engle K, Mehl G, Sinha C, Labrique A, WHO mHealth Technical Evidence Review Group Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ. 2016;352:i1174. [PubMed]
11. Olff M. Mobile mental health: a challenging research agenda. Eur J Psychotraumatol. 2015;6:27882. http://europepmc.org/abstract/MED/25994025. [PMC free article] [PubMed]
12. Kannisto KA, Koivunen MH, Välimäki MA. Use of mobile phone text message reminders in health care services: a narrative literature review. J Med Internet Res. 2014;16(10):e222. doi: 10.2196/jmir.3442. http://www.jmir.org/2014/10/e222/ [PMC free article] [PubMed] [Cross Ref]
13. Price M, Yuen EK, Goetter EM, Herbert JD, Forman EM, Acierno R, Ruggiero KJ. mHealth: a mechanism to deliver more accessible, more effective mental health care. Clin Psychol Psychother. 2014;21(5):427–436. doi: 10.1002/cpp.1855. http://europepmc.org/abstract/MED/23918764. [PMC free article] [PubMed] [Cross Ref]
14. Firth J, Cotter J, Torous J, Bucci S, Firth JA, Yung AR. Mobile phone ownership and endorsement of “mHealth” among people with psychosis: a meta-analysis of cross-sectional studies. Schizophr Bull. 2015 Sep 22;42(2):448–455. doi: 10.1093/schbul/sbv132. [PMC free article] [PubMed] [Cross Ref]
15. Furimsky I, Cheung AH, Dewa CS, Zipursky RB. Strategies to enhance patient recruitment and retention in research involving patients with a first episode of mental illness. Contemp Clin Trials. 2008 Nov;29(6):862–866. doi: 10.1016/j.cct.2008.07.005. [PubMed] [Cross Ref]
16. Depp CA, Harmell AL, Vahia IV, Mausbach BT. Neurocognitive and functional correlates of mobile phone use in middle-aged and older patients with schizophrenia. Aging Ment Health. 2016;20(1):29–35. doi: 10.1080/13607863.2015.1008987. http://europepmc.org/abstract/MED/25768842. [PMC free article] [PubMed] [Cross Ref]
17. Ennis L, Rose D, Denis M, Pandit N, Wykes T. Can't surf, won't surf: the digital divide in mental health. J Ment Health. 2012 Aug;21(4):395–403. doi: 10.3109/09638237.2012.689437. http://europepmc.org/abstract/MED/22712756. [PMC free article] [PubMed] [Cross Ref]
18. Harvey PD, Keefe RSE. Technology, society, and mental illness: challenges and opportunities for assessment and treatment. Innov Clin Neurosci. 2012 Nov;9(11-12):47–50. http://europepmc.org/abstract/MED/23346519. [PubMed]
19. Ben-Zeev D, Schueller SM, Begale M, Duffecy J, Kane JM, Mohr DC. Strategies for mHealth research: lessons from 3 mobile intervention studies. Adm Policy Ment Health. 2015 Mar;42(2):157–167. doi: 10.1007/s10488-014-0556-2. [PMC free article] [PubMed] [Cross Ref]
20. Eysenbach G. The law of attrition. J Med Internet Res. 2005;7(1):e11. doi: 10.2196/jmir.7.1.e11. http://www.jmir.org/2005/1/e11/ [PMC free article] [PubMed] [Cross Ref]
21. Välimäki M, Hätönen H, Adams CE. Mobile.net: mobile telephone text messages to encourage adherence to medication and to follow up with people with psychosis: methods and protocol for a multicenter randomized controlled two-armed trial. JMIR Res Protoc. 2012;1(2):e8. doi: 10.2196/resprot.2136. http://www.researchprotocols.org/2012/2/e8/ [PMC free article] [PubMed] [Cross Ref]
22. Kauppi K, Kannisto KA, Hätönen H, Anttila M, Löyttyniemi E, Adams CE, Välimäki M. Mobile phone text message reminders: Measuring preferences of people with antipsychotic medication. Schizophr Res. 2015 Oct;168(1-2):514–522. doi: 10.1016/j.schres.2015.07.044. [PubMed] [Cross Ref]
23. Blanton S, Morris DM, Prettyman MG, McCulloch K, Redmond S, Light KE, Wolf SL. Lessons learned in participant recruitment and retention: the EXCITE trial. Phys Ther. 2006 Nov;86(11):1520–1533. doi: 10.2522/ptj.20060091. [PubMed] [Cross Ref]
24. Kim R, Hickman N, Gali K, Orozco N, Prochaska JJ. Maximizing retention with high risk participants in a clinical trial. Am J Health Promot. 2014;28(4):268–274. doi: 10.4278/ajhp.120720-QUAN-355. [PMC free article] [PubMed] [Cross Ref]
25. Amico KR. Percent total attrition: a poor metric for study rigor in hosted intervention designs. Am J Public Health. 2009 Sep;99(9):1567–1575. doi: 10.2105/AJPH.2008.134767. [PubMed] [Cross Ref]
26. Endicott J, Nee J, Harrison W, Blumenthal R. Quality of Life Enjoyment and Satisfaction Questionnaire: a new measure. Psychopharmacol Bull. 1993;29(2):321–326. [PubMed]
27. Attkisson CC, Greenfield TK. The UCSF Client Satisfaction Scales: I. The Client Satisfaction Questionnaire-8. In: Maruish ME, editor. The Use Of Psychological Testing For Treatment Planning And Outcomes Assessment: Volume 1: General Considerations (3rd Edition) Mahwah, NJ: Lawrence Erlbaum Associates; 2004.
28. Giguere AM, Labrecque M, Légaré F, Grad R, Cauchon M, Greenway M, Haynes RB, Pluye P, Syed I, Banerjee D, Carmichael P, Martin M. Feasibility of a randomized controlled trial to evaluate the impact of decision boxes on shared decision-making processes. BMC Med Inform Decis Mak. 2015;15:13. doi: 10.1186/s12911-015-0134-x. [PMC free article] [PubMed] [Cross Ref]
29. Karlson CW, Rapoff MA. Attrition in randomized controlled trials for pediatric chronic conditions. J Pediatr Psychol. 2009 Aug;34(7):782–793. doi: 10.1093/jpepsy/jsn122. http://jpepsy.oxfordjournals.org/cgi/pmidlookup?view=long&pmid=19064607. [PubMed] [Cross Ref]
30. National Institute for Health and Welfare. 2014. [2016-07-26]. Care register for health care https://www.thl.fi/en/web/thlfi-en/statistics/information-on-statistics/register-descriptions/care-register-for-health-care webcite.
31. The ICD-10 Classification of Mental and Behavioural Disorders: Diagnostic Criteria for Research. Geneva: World Health Organization; 1993. [2016-07-26]. http://www.who.int/classifications/icd/en/GRNBOOK.pdf webcite.
32. Villeneuve K, Potvin S, Lesage A, Nicole L. Meta-analysis of rates of drop-out from psychosocial treatment among persons with schizophrenia spectrum disorder. Schizophr Res. 2010 Aug;121(1-3):266–270. doi: 10.1016/j.schres.2010.04.003. [PubMed] [Cross Ref]
33. Vargas-Barón E. Planning Policies for Early Childhood Development: Guidelines for Action. Paris: United Nations Educational, Scientific and Cultural Organisation (UNESCO); 2005. [2016-07-26]. http://www.unicef.org/ceecis/0611-ECDPolicyGuidelines_en.pdf webcite.
34. Shadish W, Cook T, Campbell D. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin; 2002.
35. Kannisto KA, Adams CE, Koivunen M, Katajisto J, Välimäki M. Feedback on SMS reminders to encourage adherence among patients taking antipsychotic medication: a cross-sectional survey nested within a randomised trial. BMJ Open. 2015;5(11):e008574. doi: 10.1136/bmjopen-2015-008574. http://bmjopen.bmj.com/cgi/pmidlookup?view=long&pmid=26553830. [PMC free article] [PubMed] [Cross Ref]
36. Jørgensen R, Munk-Jørgensen P, Lysaker PH, Buck KD, Hansson L, Zoffmann V. Overcoming recruitment barriers revealed high readiness to participate and low dropout rate among people with schizophrenia in a randomized controlled trial testing the effect of a Guided Self-Determination intervention. BMC Psychiatry. 2014 Feb 03;14:28. doi: 10.1186/1471-244X-14-28. http://bmcpsychiatry.biomedcentral.com/articles/10.1186/1471-244X-14-28. [PMC free article] [PubMed] [Cross Ref]
37. Woodall A, Morgan C, Sloan C, Howard L. Barriers to participation in mental health research: are there specific gender, ethnicity and age related barriers? BMC Psychiatry. 2010;10:103. doi: 10.1186/1471-244X-10-103. http://www.biomedcentral.com/1471-244X/10/103. [PMC free article] [PubMed] [Cross Ref]
38. Ben-Zeev D, Kaiser SM, Krzos I. Remote “hovering” with individuals with psychotic disorders and substance use: feasibility, engagement, and therapeutic alliance with a text-messaging mobile interventionist. J Dual Diagn. 2014;10(4):197–203. doi: 10.1080/15504263.2014.962336. http://europepmc.org/abstract/MED/25391277. [PMC free article] [PubMed] [Cross Ref]
39. Ben-Zeev D. Mobile technologies in the study, assessment, and treatment of schizophrenia. Schizophr Bull. 2012 May;38(3):384–385. doi: 10.1093/schbul/sbr179. http://schizophreniabulletin.oxfordjournals.org/cgi/pmidlookup?view=long&pmid=22234487. [PMC free article] [PubMed] [Cross Ref]
40. Smith A. Pew Research Center. 2014. Apr 03, [2016-11-01]. Older adults and technology use http://www.pewinternet.org/2014/04/03/older-adults-and-technology-use/ webcite.
41. McCauley-Elsom K, Gurvich C, Lee S, Elsom S, O'Connor M, Kulkarni J. Vulnerable populations and multicentred research. Int J Ment Health Nurs. 2009 Apr;18(2):108–115. doi: 10.1111/j.1447-0349.2008.00590.x. [PubMed] [Cross Ref]

Articles from Journal of Medical Internet Research are provided here courtesy of Gunther Eysenbach